What is Data Scraping? Types, Tools, and Legal Limits

Data scraping is the automated extraction of data from output that was designed for humans to read rather than for machines to consume. That definition is broader than it first sounds: the source can be a web page, a terminal screen, a PDF, a printed report, or a legacy application with no API. Web scraping is the largest subset of data scraping, not a synonym for it.

The distinction matters because it determines your tooling. Scraping a website is an HTTP problem. Scraping a mainframe terminal is a screen-buffer problem. Scraping a scanned invoice is an OCR problem. This guide covers all four types, when each applies, whether the practice is legal, and what it's actually used for.

Key Takeaways

• Data scraping is the parent category; web scraping is one type of it. The others are screen scraping, database/report mining, and API scraping.
• The defining trait is that the source is human-readable output, not a machine-readable feed. If there's a documented API returning JSON, you're integrating, not scraping.
• Screen scraping is what banks and insurers still run on — it's how account aggregation worked before open banking, and how COBOL mainframes get modern front-ends.
• Data scraping is lawful in most jurisdictions when applied to public data, but three things change the answer: authentication, personal data, and copyright.
• "AI scraping" isn't a distinct technique. It's ordinary scraping where a language model replaces hand-written selectors at the extraction step — the fetching problem is unchanged.
• Scraping is also an attack surface: content theft, price espionage, and credential-stuffing reconnaissance all use the same mechanics, which is why security vendors treat the term as a threat category.

Data scraping vs. web scraping

These get used interchangeably, and it's worth being precise once.

Data scraping is any automated extraction from human-facing output. Web scraping is data scraping applied to websites over HTTP.

The relationship is hierarchical: every web scrape is a data scrape, but plenty of data scraping never touches a browser. Pulling account balances off a green-screen terminal, lifting line items out of a fixed-width printed report, or OCR-ing a batch of scanned invoices are all data scraping with no web involved.

In practice most people saying "data scraping" mean web scraping, because the web is where most human-readable data now lives. If that's your case, our web scraping guide covers the HTTP-specific mechanics in depth, and this article stays on the wider category.

The four types of data scraping

Web scraping

The dominant case: request a URL, parse the returned HTML, extract named values. Two complications drive nearly all the engineering effort — content rendered by JavaScript after page load, and anti-bot systems that fingerprint your client. Both are covered in the web scraping guide and the headless browser guide.

Screen scraping

Screen scraping reads what's displayed rather than what's transmitted. It predates the web by decades and remains load-bearing in finance, insurance, healthcare, and government, where systems written in COBOL and RPG still run and have no API.

Two distinct flavours hide under the name:

• Terminal screen scraping. Reading the character buffer of a 3270/5250 mainframe session at known row and column positions. It's exact, not probabilistic — you're reading structured text, just structured by screen coordinates instead of by tags.
• Visual screen scraping. Capturing pixels and recovering text with OCR. This is the fallback when no text buffer is exposed, and it's the least reliable option available — it fails on font changes, scaling, and anti-aliasing.

The rule: use OCR only when there is no text layer to read. If the application exposes accessibility APIs, a text buffer, or a clipboard, all three beat pixels. This is also the reason "is Excel a data scraping tool" gets asked so often — Power Query's web connector does exactly this job for tabular sources, without code.

Report and database scraping

Legacy systems frequently can't export data but can print it. Report mining takes those printed or spooled outputs — fixed-width columns, page headers repeating every 60 lines — and reconstructs records from them.

Note the terminology trap: "database scraping" almost never means querying a database. If you have credentials and a connection string, that's just SQL. Database scraping refers to extracting from a database's human-facing output when direct access is denied — an admin UI, a report, a paginated table.

API scraping

Sites that render client-side fetch their data from JSON endpoints, and those endpoints are usually callable directly. Open your browser's Network tab, filter to XHR/Fetch, and reload — you'll often find the exact data the page renders, already structured.

This is the highest-leverage move in scraping and the most underused. JSON doesn't break when the CSS changes, it's a fraction of the bandwidth, and it frequently includes fields the visible page omits. It's still scraping, though: an undocumented endpoint carries no stability guarantee and can change without notice.

Is data scraping legal?

Scraping publicly accessible data is broadly lawful in the United States. The anchor case is hiQ Labs v. LinkedIn, in which the Ninth Circuit held that accessing a public profile isn't "unauthorized access" under the Computer Fraud and Abuse Act. Several EU and UK decisions land in a similar place for public data.

That protection is narrower than it's usually reported, because the CFAA question is only one of four:

Three specific rules worth internalising:

• Public does not mean unregulated. GDPR and CCPA apply to scraped personal data exactly as to any other collection. Legal basis, minimisation, and deletion obligations all attach.
• Facts aren't copyrightable, expression is. Extracting prices from an article is a different act from copying the article.
• Terms of service are contract, not crime. Breaching them rarely creates criminal exposure, but it's still an enforceable claim, particularly if you accepted them by registering.

Our detailed piece on scraping legality covers the case law properly. None of this is legal advice — jurisdictions differ and the law here is still moving.

Why security vendors treat scraping as a threat

Search "data scraping" and you'll mostly find security companies, because the same mechanics serve legitimate and hostile uses. From the receiving end, scraping shows up as:

• Content theft — wholesale republication of articles or listings.
• Price espionage at a frequency that distorts a competitor's pricing.
• Inventory reconnaissance feeding scalper bots.
• Profile harvesting to assemble data for phishing or credential stuffing.
• Infrastructure load — aggressive scrapers behaving indistinguishably from a denial-of-service.

Worth being honest about, since it explains the defensive posture you'll meet: anti-bot systems aren't paranoia, they're a response to real abuse. It's also the practical argument for restraint. Rate-limit below what would degrade the site, identify yourself where you reasonably can, and take the API when one exists. Scrapers that behave get blocked far less.

What is data scraping used for?

The legitimate applications concentrate in a few areas:

• Competitive and pricing intelligence — price monitoring and stock tracking across retailers.
• Sales and marketing — B2B lead generation and CRM enrichment from directories and company sites.
• Financial research — alternative data and SEC filing monitoring, where the underlying documents are public but not conveniently structured.
• Labour market analysis — job aggregation and salary benchmarking.
• AI training and retrieval — RAG knowledge bases and fine-tuning datasets.
• Legacy system integration — screen scraping as the bridge when replacing a core system isn't viable.

What is AI scraping?

"AI scraping" describes two different things, and conflating them causes most of the confusion.

Scraping with AI means using a language model for the extraction step. Instead of writing div.product > span.price-current and repairing it after every redesign, you describe the field in words and the model finds it. This is a genuine improvement: it's the brittle part of a scraper, and models degrade gracefully where selectors fail outright.

Scraping for AI means collecting training or retrieval data — the corpora behind foundation models, and the fresher, narrower sets behind RAG systems.

Neither changes the hard part. A model can only read what something else already fetched, and it has no answer for IP bans, TLS fingerprinting, or CAPTCHAs. Treat AI as a replacement for your parsing layer, not your infrastructure.

Selector-free extraction, using our AI scraping endpoint:

import requests

response = requests.get(
    "https://api.webscraping.ai/ai/fields",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/listing/42",
        "fields[title]": "Listing title",
        "fields[price]": "Price in USD, digits only",
        "fields[posted_at]": "Publication date in YYYY-MM-DD format",
    },
    timeout=60,
)
print(response.json())
# {"title": "2-bed apartment", "price": "2400", "posted_at": "2026-07-14"}

You can ask a question about a page instead, which suits classification and yes/no checks:

response = requests.get(
    "https://api.webscraping.ai/ai/question",
    params={
        "api_key": "YOUR_API_KEY",
        "url": "https://example.com/product/9",
        "question": "Is this product currently in stock? Answer yes or no.",
    },
    timeout=60,
)
print(response.text)

Both calls handle proxy rotation, browser rendering, and retries — the parts a model can't do for you. AI extraction adds 5 credits on top of the base request cost (1 for a datacenter fetch without JavaScript, 5 with it), and failed requests are never charged. The same endpoints are available through our MCP server and n8n node.

Choosing a tool

A short decision path:

1. Is there a documented API? Use it. You're not scraping.
2. Is there an undocumented JSON endpoint? Check the Network tab. Call it directly.
3. Is the source a website? Start with HTTP + Beautiful Soup. Add Playwright only if the HTML comes back empty. Use a scraping API if you'd rather not run proxies and browsers.
4. Is the source a desktop or terminal app? Screen scraping or an RPA platform. Prefer a text buffer over OCR wherever one exists.
5. Is the source a document or report? A parsing library for structured text; OCR only for scans.

For code-level starting points, see our Python and JavaScript guides, or the Python library comparison.

Frequently Asked Questions

What is meant by data scraping?

Automated extraction of data from output intended for human consumption — a web page, a screen, a report — into a structured format like CSV or JSON. The defining feature is that the source wasn't designed to be machine-read.

Is data scraping the same as web scraping?

No. Web scraping is one type of data scraping, the type that targets websites over HTTP. Screen scraping, report mining, and API scraping are the others. Most casual uses of "data scraping" do mean web scraping.

Is data scraping illegal?

Not inherently. Scraping public data is broadly lawful in the US following hiQ v. LinkedIn. Risk rises sharply when you bypass authentication, collect personal data, or republish copyrighted content. Jurisdiction matters and this isn't legal advice.

What is the difference between data scraping and data mining?

Scraping acquires data; mining analyses it. Scraping gets prices off a hundred retail sites into a database. Mining finds the pattern in them. Scraping is nearly always upstream of mining.

Is Excel a data scraping tool?

Partly. Power Query's "From Web" connector pulls HTML tables into a sheet and refreshes them, which handles simple static tables without code. It can't render JavaScript, rotate IPs, or survive anti-bot systems, so it fits ad-hoc analysis rather than production pipelines.

What is a data scraper?

Either the software performing the extraction, or the person operating it. As software it's typically three parts: an HTTP client or screen reader that acquires the raw output, a parser that structures it, and storage that persists it.

Need the fetching layer handled? Get a free API key — 2,000 credits a month, no credit card required — and see the API documentation for the full parameter reference.