Fetch and extract in one call. Ask a question about any web page or describe the fields you want in plain English — the API loads the page past blocks, renders the JavaScript, and returns clean answers or typed JSON.
2,000 free API credits · No credit card required
/ai/question answers a free-form question about any URL. /ai/fields returns typed JSON for the fields you name. No CSS selectors, no XPath, no parser code — and nothing to fix when the site changes its layout.
$ curl -G "https://api.webscraping.ai/ai/question" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://example-store.com/product/42" \
--data-urlencode "question=Is this product in stock, and at what price?"
"Yes, it's in stock at $49.99 (reduced from $64.99)."
AI web scraping uses large language models to read and extract data from web pages instead of hand-written parsing rules. A traditional scraper locates data with CSS selectors or XPath expressions tied to the page's exact HTML structure — so when the site redesigns, the scraper silently breaks. An AI scraper reads the rendered content the way a person would, so it can find "the price" or "the author" wherever they moved.
In practice, an AI scraping pipeline has two jobs: fetching the page (getting past anti-bot systems, rendering JavaScript) and extracting the data (turning the page into answers or structured JSON). The AI only helps with the second — no language model can read a page that Cloudflare refused to serve. That's why WebScraping.AI bundles both: every AI request runs through rotating datacenter, residential, or stealth proxies and headless Chromium rendering before the model reads a single token.
| Traditional scraping | AI web scraping | |
|---|---|---|
| Extraction logic | CSS/XPath selectors per site | Plain-language questions or field descriptions |
| Site redesigns | Break the scraper; manual fix | Handled — the AI reads the new layout |
| New sites | New parser each time | Same request, different URL |
| Output | Raw HTML to parse | Answers or typed JSON |
| Best for | Huge volumes on stable pages | Many sites, changing layouts, LLM pipelines |
Both approaches are available here — /selected gives you CSS-selector extraction at 1 credit when you want it; the AI endpoints add 5 credits when you don't.
Everything between a URL and clean data, in one API
Ask anything about a page and get the answer as text — perfect for quick checks and agent workflows.
Name your fields, describe them in a sentence, get JSON back — your schema, not a parser's.
The /text endpoint strips boilerplate and returns clean page text for RAG pipelines and prompts.
Headless Chromium rendering plus rotating datacenter, residential, and stealth proxies with 13-country geotargeting.
The part most "AI scrapers" skip: AI extraction is useless if the page won't load. Every request here includes the fetching layer — proxies, fingerprinting, retries, JS rendering — and failed requests are free.
7 official SDKs: Python, JavaScript, PHP, Ruby, Go, Java, and C#
curl -G "https://api.webscraping.ai/ai/fields" \
--data-urlencode "api_key=YOUR_API_KEY" \
--data-urlencode "url=https://news.ycombinator.com" \
--data-urlencode "fields[top_story]=Title of the #1 story" \
--data-urlencode "fields[top_story_points]=Points of the #1 story" \
--data-urlencode "fields[top_story_url]=URL of the #1 story"
# Response:
# {
# "top_story": "Show HN: I built a rocket telemetry kit",
# "top_story_points": "342",
# "top_story_url": "https://example.com/telemetry"
# }
# pip install webscraping_ai
# https://pypi.org/project/webscraping-ai/
from webscraping_ai import Client
client = Client(api_key="YOUR_API_KEY")
result = client.fields(
"https://news.ycombinator.com",
fields={
"top_story": "Title of the #1 story",
"top_story_points": "Points of the #1 story",
"top_story_url": "URL of the #1 story",
},
)
print(result)
# Response:
# {
# "top_story": "Show HN: I built a rocket telemetry kit",
# "top_story_points": "342",
# "top_story_url": "https://example.com/telemetry"
# }
// npm install webscraping-ai
// https://www.npmjs.com/package/webscraping-ai
import { WebScrapingAI } from 'webscraping-ai';
const client = new WebScrapingAI({ apiKey: 'YOUR_API_KEY' });
const result = await client.fields({
url: 'https://news.ycombinator.com',
fields: {
top_story: 'Title of the #1 story',
top_story_points: 'Points of the #1 story',
top_story_url: 'URL of the #1 story',
},
});
console.log(result);
// Response:
// {
// "top_story": "Show HN: I built a rocket telemetry kit",
// "top_story_points": "342",
// "top_story_url": "https://example.com/telemetry"
// }
<?php
// composer require webscraping-ai/webscraping-ai-php
// https://packagist.org/packages/webscraping-ai/webscraping-ai-php
require 'vendor/autoload.php';
use WebScrapingAI\Client;
$client = new Client('YOUR_API_KEY');
$result = $client->fields('https://news.ycombinator.com', [
'top_story' => 'Title of the #1 story',
'top_story_points' => 'Points of the #1 story',
'top_story_url' => 'URL of the #1 story',
]);
print_r($result);
// Response:
// {
// "top_story": "Show HN: I built a rocket telemetry kit",
// "top_story_points": "342",
// "top_story_url": "https://example.com/telemetry"
// }
# gem install webscraping_ai
# https://rubygems.org/gems/webscraping_ai
require 'webscraping_ai'
client = WebScrapingAI::Client.new(api_key: 'YOUR_API_KEY')
result = client.fields(
'https://news.ycombinator.com',
fields: {
top_story: 'Title of the #1 story',
top_story_points: 'Points of the #1 story',
top_story_url: 'URL of the #1 story',
}
)
puts result.inspect
# Response:
# {
# "top_story": "Show HN: I built a rocket telemetry kit",
# "top_story_points": "342",
# "top_story_url": "https://example.com/telemetry"
# }
// go get github.com/webscraping-ai/webscraping-ai-go/v4
// https://pkg.go.dev/github.com/webscraping-ai/webscraping-ai-go/v4
package main
import (
"context"
"fmt"
webscrapingai "github.com/webscraping-ai/webscraping-ai-go/v4"
)
func main() {
client, _ := webscrapingai.NewClient(&webscrapingai.Config{APIKey: "YOUR_API_KEY"})
result, _ := client.Fields(context.Background(), &webscrapingai.FieldsOptions{
URL: "https://news.ycombinator.com",
Fields: map[string]string{
"top_story": "Title of the #1 story",
"top_story_points": "Points of the #1 story",
"top_story_url": "URL of the #1 story",
},
})
fmt.Println(result.Result)
}
// Response:
// {
// "top_story": "Show HN: I built a rocket telemetry kit",
// "top_story_points": "342",
// "top_story_url": "https://example.com/telemetry"
// }
// Maven: ai.webscraping:webscraping-ai:4.0.0
// https://central.sonatype.com/artifact/ai.webscraping/webscraping-ai
import ai.webscraping.Client;
import ai.webscraping.Config;
import ai.webscraping.option.FieldsOptions;
import ai.webscraping.result.FieldsResult;
Client client = new Client(Config.builder().apiKey("YOUR_API_KEY").build());
FieldsResult result = client.fields(FieldsOptions.builder()
.url("https://news.ycombinator.com")
.addField("top_story", "Title of the #1 story")
.addField("top_story_points", "Points of the #1 story")
.addField("top_story_url", "URL of the #1 story")
.build());
System.out.println(result.getResult());
// Response:
// {
// "top_story": "Show HN: I built a rocket telemetry kit",
// "top_story_points": "342",
// "top_story_url": "https://example.com/telemetry"
// }
// dotnet add package WebScrapingAI
// https://www.nuget.org/packages/WebScrapingAI
using WebScrapingAI;
var client = new WebScrapingAIClient(new WebScrapingAIClientOptions { ApiKey = "YOUR_API_KEY" });
var result = await client.FieldsAsync(new FieldsRequest {
Url = "https://news.ycombinator.com",
Fields = new Dictionary<string, string> {
["top_story"] = "Title of the #1 story",
["top_story_points"] = "Points of the #1 story",
["top_story_url"] = "URL of the #1 story",
},
});
Console.WriteLine(result.Result);
// Response:
// {
// "top_story": "Show HN: I built a rocket telemetry kit",
// "top_story_points": "342",
// "top_story_url": "https://example.com/telemetry"
// }
The same AI scraping endpoints are exposed through a first-party hosted MCP server — add the URL to Claude, Cursor, or any MCP client, sign in, and your assistant can browse and extract from the live web. No API key needed; a self-hostable open-source version is also available. An n8n node, Zapier, Make, and Pipedream integrations cover the no-code side.
# Hosted MCP server — add to Claude, Cursor, or any MCP client
https://mcp.webscraping.ai/mcp
# Example: Claude Code
claude mcp add --transport http webscraping-ai \
https://mcp.webscraping.ai/mcp
# Sign in with your WebScraping.AI account — no API key needed
AI extraction adds a flat 5 credits to any request — no token math, no dynamic pricing.
250,000 credits
1,000,000 credits
3,000,000 credits
A plain AI extraction is 6 credits (1 + 5); with JS rendering 10; with residential proxies 15–30. Failed requests are always free. Full credit table.
What is an AI web scraper?
An AI web scraper uses a large language model to read a web page's rendered content and extract data from it, instead of relying on hand-written CSS selectors or XPath rules. You describe what you want in plain language — a question or a list of fields — and get back text or structured JSON.
How is AI web scraping different from traditional web scraping?
Traditional scrapers locate data by the page's HTML structure, so they need per-site parser code and break when a site redesigns. AI scraping reads the content semantically, so the same request works across different sites and survives layout changes. Traditional selector-based extraction is still cheaper per request for huge volumes on stable pages — WebScraping.AI offers both.
Does AI web scraping avoid blocks and CAPTCHAs?
Not by itself — the AI only reads pages that were successfully fetched. Anti-bot systems block the fetch, before any model is involved. That's why WebScraping.AI bundles rotating datacenter, residential, and stealth proxies plus headless browser rendering with every AI request, and doesn't bill requests that fail.
How accurate is AI extraction? Do LLMs hallucinate scraped data?
Asking for specific, typed fields dramatically constrains what the model can get wrong compared to dumping a whole page into a chat prompt — the model extracts values that are present on the rendered page rather than generating free text. For fields with strict formats, keep descriptions precise ("price with currency symbol"), and spot-check a sample before scaling any pipeline, as you would with human-written parsers.
Can it scrape JavaScript-heavy websites?
Yes — pages render in headless Chromium before extraction, with configurable JS timeouts and wait_for selectors for content that loads late. JS rendering costs 5 credits instead of 1.
What does AI web scraping cost?
Plans start at $29/mo for 250,000 credits. AI extraction adds a flat 5 credits to a request: a plain AI scrape is 6 credits, so the entry plan covers roughly 41,000 AI extractions per month — and failed requests are free. A free account includes 2,000 credits monthly.
Can I use it with LLM frameworks and AI agents?
Yes — via 7 official SDKs for your own pipelines, a first-party MCP server for Claude/Cursor/Windsurf, a CLI with an installable agent skill, and n8n/Zapier/Make integrations. The /text endpoint also returns clean, LLM-ready page text for RAG ingestion.
Which AI scraping approach should I choose: no-code tools or an API?
No-code tools (Browse AI, Octoparse) suit non-developers who want data in a spreadsheet. An API suits scraping that feeds a product, database, or agent — extraction logic lives in your code, versioned and testable. If you're comparing options, see our honest comparisons with Firecrawl and other tools.
Get started with 2,000 free API credits. No credit card required.