Web Scraping for Machine Learning: Building LLM Training Data

Scraping the web for machine learning is no longer mostly about building feature tables for scikit-learn. In 2026 the dominant jobs are assembling fine-tuning corpora and filling RAG knowledge bases, and both are bottlenecked on the same four steps: get clean text out of HTML, split it sensibly, throw away the duplicates and the junk, and record where every record came from. This guide walks that pipeline end to end with working Python, then covers the legal ground rules for training data, which are genuinely unsettled and worth getting right.

Key Takeaways

• Decide which of four dataset shapes you need before you scrape: pretraining corpus, instruction/fine-tuning pairs, RAG chunks, or a classical feature table. They need different cleaning and completely different quality bars.
• Raw HTML is the wrong input for every one of them. Extract to text or markdown first — WebScraping.AI's /text endpoint with text_format=json returns {title, description, content, links} where content is markdown, which chunks far more cleanly than stripped HTML.
• Chunk on structure, not on character count. Split at markdown headings first, then subdivide anything over ~512 tokens with ~64 tokens of overlap.
• Deduplicate with MinHash/LSH, not exact hashing. Near-duplicates (same article on three syndication sites, paginated boilerplate) are the failure mode that exact hashes miss entirely.
• "Publicly available" is not a licence to train. Copyright, EU TDM opt-outs, GDPR, and AI-specific robots.txt tokens (GPTBot, ClaudeBot, Google-Extended, CCBot) each apply independently of whether the page loaded without a login.
• Store provenance per record — source URL, fetch timestamp, content hash, extractor version — from day one. Retrofitting it onto a finished corpus means re-crawling.

Which dataset are you actually building?

Most bad scraped datasets come from skipping this question. The four common targets have different requirements:

The practical decision rule: if you are fine-tuning, spend your budget on curation and stop scraping early — a few thousand carefully checked pairs beat a million scraped ones. If you are building a RAG knowledge base, spend it on coverage and freshness instead, because retrieval failures are usually "the answer wasn't in the corpus," not "the chunk was slightly noisy."

For fine-tuning specifically, the highest-value scraped sources are ones where a question and its accepted answer sit on the same page: documentation with worked examples, support forums with marked solutions, FAQ pages, changelogs. That structure gives you the pair for free instead of forcing you to synthesise one. Our LLM fine-tuning data use case covers the source-selection side in more depth.

How do you get training-ready text out of HTML?

Parsing HTML yourself with Beautiful Soup works when you know the page structure and want specific fields. It works badly when you are ingesting thousands of unrelated domains, because every site puts its article body somewhere different and every site surrounds it with navigation, cookie banners, and related-post widgets that will end up in your corpus.

For broad collection, use an endpoint that does the extraction. WebScraping.AI's /text endpoint handles the proxy, the headless browser, and the boilerplate stripping in one call:

import os
import requests

API_KEY = os.environ["WEBSCRAPING_AI_API_KEY"]


def fetch_page(url, render_js=False):
    """Return {'title', 'description', 'content', 'links'} — content is markdown."""
    response = requests.get(
        "https://api.webscraping.ai/text",
        params={
            "api_key": API_KEY,
            "url": url,
            "text_format": "json",
            "return_links": "true",
            "js": str(render_js).lower(),
            "proxy": "datacenter",
        },
        timeout=60,
    )
    response.raise_for_status()
    return response.json()


page = fetch_page("https://en.wikipedia.org/wiki/Retrieval-augmented_generation")
print(page["title"])
print(page["content"][:400])
print(len(page["links"]), "links found")

Two parameters carry most of the weight here:

• text_format=json returns the page title, meta description, markdown body, and (with return_links=true) a flat list of absolute URLs. The links list is what you feed back into the crawl frontier, so you rarely need a separate discovery pass.
• js=false skips the headless browser. For documentation sites, blogs, forums, and news — the bulk of a text corpus — the server-rendered HTML already contains the article. That drops the cost from 5 credits to 1 credit per page, so a 10,000-page crawl costs 10,000 credits rather than 50,000. Turn js back on only for the domains where you can see the body is missing without it; our headless browser guide covers how to tell.

Keeping markdown rather than flattening to plain text is deliberate: the heading markers are the signal you will chunk on in the next step, and code fences survive intact, which matters if you are training or retrieving over technical content.

How should you chunk scraped pages for embeddings?

Fixed-width character splitting is the default in most tutorials and it is the wrong default. It cuts mid-sentence, separates a heading from the paragraph that explains it, and produces chunks that are meaningless in isolation — which is exactly the property retrieval depends on.

Split on structure first, then enforce a token budget:

import re
import tiktoken

encoder = tiktoken.get_encoding("cl100k_base")


def split_sections(markdown):
    """Split at markdown headings, keeping each heading with its body."""
    parts = re.split(r"\n(?=#{1,3} )", markdown)
    return [part.strip() for part in parts if part.strip()]


def chunk(markdown, max_tokens=512, overlap=64):
    chunks = []
    for section in split_sections(markdown):
        tokens = encoder.encode(section)
        if len(tokens) <= max_tokens:
            chunks.append(section)
            continue
        step = max_tokens - overlap
        for start in range(0, len(tokens), step):
            chunks.append(encoder.decode(tokens[start:start + max_tokens]))
    return chunks

Working defaults, and when to change them:

• 512 tokens with 64 overlap suits documentation and articles retrieved by short questions.
• Go smaller (~256) when your corpus is dense reference material and queries are precise — smaller chunks sharpen the embedding.
• Go larger (~1024) when answers need surrounding narrative, such as legal or policy text.
• Prepend the document title and heading path to every chunk before embedding. A chunk reading "It defaults to 30 seconds." is useless; "Timeouts > Request timeout — It defaults to 30 seconds." retrieves correctly. This single change usually beats any amount of chunk-size tuning.

For fine-tuning corpora rather than RAG, skip chunking entirely and keep documents whole — the training framework handles packing.

How do you deduplicate a scraped corpus?

Duplication is the most under-treated problem in scraped datasets. It inflates your token count without adding information, biases the model toward whatever text happens to be syndicated most, and — for RAG — fills the top-k results with three copies of the same passage.

Exact hashing catches almost none of it. The duplicates you actually have are near duplicates: the same press release on four sites with different footers, a paginated listing where only the item list changes, a docs page republished per version. Use MinHash with LSH:

from datasketch import MinHash, MinHashLSH


def shingles(text, size=5):
    words = text.lower().split()
    return {" ".join(words[i:i + size]) for i in range(max(len(words) - size + 1, 1))}


def deduplicate(chunks, threshold=0.8, num_perm=128):
    lsh = MinHashLSH(threshold=threshold, num_perm=num_perm)
    kept = []
    for index, text in enumerate(chunks):
        signature = MinHash(num_perm=num_perm)
        for shingle in shingles(text):
            signature.update(shingle.encode("utf-8"))
        if lsh.query(signature):
            continue  # near-duplicate of something already kept
        lsh.insert(str(index), signature)
        kept.append(text)
    return kept

A Jaccard threshold of 0.8 over 5-word shingles is a reasonable starting point: it removes reposts and templated pages while keeping genuinely distinct documents that share vocabulary. Lower it toward 0.7 for boilerplate-heavy corpora, raise it toward 0.9 if you are losing content you wanted.

Then apply cheap quality heuristics. These are adapted from the filters published with large web corpora such as MassiveWeb and RefinedWeb, whose authors report discarding the large majority of raw crawl documents at this stage — expect to lose a lot, and treat that as the pipeline working:

BOILERPLATE = re.compile(r"(accept cookies|sign in to continue|privacy policy)", re.I)


def keep(text):
    words = text.split()
    if len(words) < 50:
        return False  # too short to teach anything
    if sum(char.isalpha() for char in text) / max(len(text), 1) < 0.6:
        return False  # markup debris, tables, ID strings
    if len(set(words)) / len(words) < 0.3:
        return False  # repeated navigation text
    if BOILERPLATE.search(text) and len(words) < 120:
        return False
    return True

Run language identification too if your corpus is meant to be monolingual — fasttext or lingua-py on each chunk, dropping anything below ~0.7 confidence. Mixed-language contamination is common in scraped data and quietly degrades fine-tuning.

Is it legal to scrape data for AI training?

This is contested and moving, so treat anything that gives you a one-line answer with suspicion. The honest summary is that whether you may access a page and whether you may train on it are two separate questions, and most "web scraping is legal" arguments only address the first.

Access. US case law around the Computer Fraud and Abuse Act, notably hiQ Labs v. LinkedIn, narrowed the idea that scraping public pages is unauthorised access. That does not resolve breach-of-contract claims under a site's terms, and it says nothing about what you do with the data afterwards. Our guide to web scraping legality goes through this in detail.

Copyright. Whether training a generative model on copyrighted text is fair use is being actively litigated in multiple US cases as of mid-2026, with no settled answer. In the EU, the DSM Directive's text-and-data-mining exceptions are the relevant frame: Article 3 covers research organisations, while Article 4 permits commercial TDM unless the rightsholder has reserved the right in a machine-readable way. Since August 2025, providers of general-purpose AI models under the EU AI Act must have a copyright policy that respects those reservations and publish a sufficiently detailed summary of their training content.

robots.txt and AI-specific tokens. robots.txt is not a statute, but it is now the main channel for machine-readable opt-out, and publishers use AI-specific user-agent tokens — GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Applebot-Extended, Bytespider, meta-externalagent. A blanket Disallow for those tokens alongside an Allow for search crawlers is a deliberate signal that the site permits indexing but not training. Ignoring it is both a reputational risk and, in the EU, arguably an overridden Article 4 reservation.

Personal data. GDPR and similar regimes apply to personal data regardless of whether it was public. There is no "publicly available" exemption; you need a lawful basis, and Article 14 notification duties are hard to satisfy at crawl scale. The practical rule is to exclude personal data from training corpora unless you have a specific, documented reason to include it, and to run PII detection over the corpus before training rather than after.

Working policy that keeps most teams out of trouble:

1. Honour robots.txt for both your crawler's user agent and the AI-specific tokens, and record the robots decision per fetch.
2. Prefer sources with an explicit licence — Creative Commons, permissive documentation, government publications — and store the licence string in the record.
3. Exclude login-gated content entirely. Passing an authentication wall converts a weak claim into a strong one.
4. Strip or redact personal data at ingestion, not before training.
5. Keep the crawl polite: modest concurrency, backoff on 429s and 5xxs. Aggressive crawling is what turns a tolerated scraper into a blocked and named one.

Collecting image-text pairs for multimodal models

Multimodal fine-tuning needs image-caption pairs, and the web supplies them through alt attributes and figure captions. The scraping part is straightforward — request the HTML rather than the text extraction, then pair each <img> with its best available caption:

from bs4 import BeautifulSoup
from urllib.parse import urljoin

html = requests.get(
    "https://api.webscraping.ai/html",
    params={"api_key": API_KEY, "url": page_url, "js": "false"},
    timeout=60,
).text

soup = BeautifulSoup(html, "html.parser")
pairs = []
for img in soup.find_all("img"):
    src = img.get("src")
    figure = img.find_parent("figure")
    caption = (figure.figcaption.get_text(strip=True) if figure and figure.figcaption
               else img.get("alt", ""))
    if src and len(caption.split()) >= 3:
        pairs.append({"image_url": urljoin(page_url, src), "caption": caption})

Two cautions specific to images. First, quality: most alt text is either empty, a filename, or SEO keyword stuffing, so the three-word minimum above is the floor rather than a real filter — plan to score captions with a vision model and drop the bottom half. Second, licensing is materially stricter than for text. Photographs carry individual copyright, stock imagery carries explicit licence terms, and images of people raise biometric and data-protection issues that plain text does not. Filter to explicitly licensed sources for anything you intend to ship.

A complete scrape-to-dataset pipeline

Putting the pieces together, with provenance recorded per chunk:

import datetime
import hashlib
import json


def build_dataset(urls, output_path, pipeline_version="2026.07.1"):
    all_chunks, records = [], []

    for url in urls:
        try:
            page = fetch_page(url)
        except requests.HTTPError as error:
            print(f"skipped {url}: {error}")
            continue  # failed requests are not billed

        for index, text in enumerate(chunk(page["content"])):
            if not keep(text):
                continue
            all_chunks.append(text)
            records.append({
                "text": f"{page['title']}\n\n{text}",
                "source_url": url,
                "source_title": page["title"],
                "chunk_index": index,
                "fetched_at": datetime.datetime.now(datetime.UTC).isoformat(),
                "content_sha256": hashlib.sha256(text.encode()).hexdigest(),
                "extractor": "webscraping.ai/text?text_format=json",
                "pipeline_version": pipeline_version,
            })

    keep_texts = set(deduplicate(all_chunks))
    records = [r for r in records if r["text"].split("\n\n", 1)[-1] in keep_texts]

    with open(output_path, "w") as handle:
        for record in records:
            handle.write(json.dumps(record) + "\n")

    return len(records)


print(build_dataset(["https://example.com/docs/intro"], "corpus.jsonl"), "records")

Cost, so you can size a crawl: with js=false on datacenter proxies each page is 1 credit, so a 10,000-page corpus costs 10,000 credits. The $29/month plan includes 250,000 credits. Pages that fail are not billed, which matters when you are crawling long tails of domains where a meaningful share will 404 or time out. JavaScript rendering raises the cost to 5 credits per page, and residential proxies to 10 (25 with JS) — see the full parameter reference before you scale a job up.

Versioning and provenance

The metadata above is not bookkeeping; it is what makes the dataset usable six months later. Three things are worth committing to early:

• Content hashes per record. They let you diff two crawl runs and see what actually changed, rather than re-embedding the entire corpus on every refresh.
• An immutable dataset version. Point your training run at corpus-2026-07-28.jsonl, never at corpus.jsonl. DVC or LakeFS handle this properly with Git-style history; a dated file in object storage handles it adequately.
• A dataset card. Sources, collection dates, filters applied, licences, known gaps. The Hugging Face dataset card format and MLCommons' Croissant metadata schema are both reasonable templates. When someone asks "where did this training example come from" — and with the EU AI Act's training-data summary requirement, someone will — the card is the answer.

If you want the collection layer managed rather than built, our training data collection and AI web scraping pages cover the API side, including /ai/fields for pulling structured records out of pages without writing selectors.

Frequently Asked Questions

How much data do you need to fine-tune an LLM?

Far less than for pretraining, and quality dominates quantity. Instruction fine-tuning typically works with thousands to tens of thousands of high-quality pairs; LoRA adapters on a domain corpus often show gains in the low thousands of examples. Scraping millions of noisy pairs usually performs worse than curating a few thousand, because bad pairs teach the model to produce bad outputs rather than being averaged away.

Should you scrape your own data or use Common Crawl?

Use Common Crawl when you need breadth and general web text — it is already crawled, already deduplicated by many downstream projects, and free. Scrape yourself when you need a specific domain, current data, or pages Common Crawl's snapshots miss, which is most of the long tail and nearly all JavaScript-rendered content. In practice most teams do both: Common Crawl for volume, targeted scraping for the domains that actually matter to their model.

What is the difference between scraping for RAG and scraping for fine-tuning?

Freshness and shape. RAG corpora need to be re-crawled continuously because retrieval serves current answers, and they need self-contained chunks with metadata for filtering. Fine-tuning corpora are built once per training run, keep documents whole, and are judged on how well each example demonstrates the behaviour you want. If your requirement is "the model should know today's prices," that is RAG, not fine-tuning.

Does robots.txt legally require you to stop scraping?

No — robots.txt is a convention, not legislation. But it is the standard machine-readable channel for opt-out, and in the EU it is how rightsholders reserve rights under the DSM Directive's commercial TDM exception, which gives it legal weight there. Treat it as binding in practice: honour it, and log the decision per fetch so you can demonstrate you did.

How do you keep a scraped RAG corpus up to date?

Re-crawl on a schedule keyed to how fast each source changes, and compare content hashes rather than re-embedding blindly. Only chunks whose hash changed need new embeddings, which typically cuts refresh cost by an order of magnitude on stable documentation sites. Store the last-fetched timestamp per record so you can prioritise the stalest sources first.

Ready to build a training corpus? Sign up for a free WebScraping.AI account — 2,000 credits per month, no credit card — and point /text at your first ten source pages before you commit to a pipeline design.