Web Scraping for AI: Building Reliable Live Data Pipelines

EXECUTIVE SUMMARY

Feeding live web data into LLMs is an end-to-end system with clearly separable stages: discovery, crawling, extraction, cleaning, chunking, enrichment, grounding, and continuous monitoring. Treat any one as an afterthought and the whole system quietly degrades.

This guide is written as an audit-first playbook. Instead of chasing the newest library, it focuses on three durable properties:

1. INFORMATION GAIN — does each stage preserve the facts that actually matter?
2. TECHNICAL CORRECTNESS — does the output stay faithful to the source page?
3. OPERATIONAL SAFETY — can you detect, explain, and recover when a site changes?

By the end you will understand how to design a pipeline that produces LLM-ready outputs with traceability, schema alignment, and measurable quality — not just "text that looks fine in a demo."

1: PROBLEM STATEMENT

Most teams approach this with a two-line mental model: "scrape the web, then feed it to an LLM." That model breaks in production for reasons that are boringly predictable — and every one of them is preventable.

SIX PREDICTABLE FAILURE MODES:

1. THE EXTRACTED TEXT IS NOISY
   Raw HTML-to-text dumps drag in navigation menus, cookie banners, sidebars, newsletter popups, and footer link farms. The model can't tell the difference between the actual article and the "Subscribe now" block, so retrieval surfaces junk and answers drift.
2. THE CONTENT IS INCOMPLETE
   Naive extractors silently drop the things that carry the most meaning: data tables, code blocks, nested lists, and heading hierarchy. A pricing table flattened into a run-on sentence is worse than no table at all, because it looks like a fact while being unusable.
3. THE DATA IS INCONSISTENT
   Different templates — or even the same site after a redesign — produce different DOM structures. A pipeline tuned to one layout produces clean output on Monday and garbage on Tuesday, with no error thrown.
4. THE PIPELINE IS NOT TRACEABLE
   When a model gives a wrong or surprising answer, you need to point to the exact chunk, the exact source URL, and the exact fetch time that produced it. If you can't, you can't debug it — you can only guess.
5. THE DATA BECOMES STALE
   Live web data changes. Prices, availability, documentation, policies, and version numbers all move. Without freshness signals, your system confidently returns last quarter's truth.
6. THE SYSTEM BREAKS SILENTLY
   The worst failure mode isn't a crash — it's a 200 OK that returns an empty shell, a soft-blocked page, or a login wall rendered as "content." The pipeline keeps running, quality collapses, and nobody notices until users complain.

THE 2026 LENS

Google's ranking systems increasingly reward information gain and genuine helpfulness. That same principle is the right north star for your ingestion pipeline: preserve the facts that matter, remove the noise that harms retrieval, and keep evidence that can be audited. A pipeline optimized for information gain is also a pipeline optimized to rank — and to be trusted.

2: DEFINITIONS

Before designing anything, pin down the two phrases in the title, because ambiguity here causes most architecture mistakes.

WHAT IS "LIVE WEB DATA"?

Live web data means content that CHANGES OVER TIME and therefore must be re-collected or re-validated on a schedule. It is the opposite of a one-time static dump. If a fact can go out of date — a stock level, an API reference, a news event, a policy — it is live data, and your pipeline needs a REFRESH CONTRACT for it, not just an initial crawl.

WHAT DOES "FEEDING INTO LLMs" MEAN?

This can mean three architecturally different things, and they have different validation needs:

1. RAG (Retrieval-Augmented Generation)
   • What it is: Retrieve relevant chunks at query time and pass them into the prompt as context.
   • Best when: Facts change often; you need citations and freshness.
   • Main risk: Poor chunking or retrieval → irrelevant context.
2. FINE-TUNING / CONTINUED TRAINING
   • What it is: Convert web content into a training dataset that reshapes the model's weights.
   • Best when: You need consistent tone, format, or domain reasoning.
   • Main risk: Baked-in staleness; expensive to correct.
3. TOOL-AUGMENTED AGENTS
   • What it is: The agent fetches or scrapes pages during task execution.
   • Best when: Tasks need the very latest state (prices, live docs).
   • Main risk: Untrusted content can hijack the agent (prompt injection).

EMPHASIS OF THIS GUIDE

This guide covers all three at the system level, but emphasizes RAG and agent pipelines — they are the most common in production and, crucially, the easiest to validate and audit because the evidence (retrieved chunks) is inspectable at answer time. Fine-tuning bakes web content into weights, which makes traceability and freshness far harder to guarantee.

3: ARCHITECTURE

A robust pipeline is a sequence of stages, each with a single responsibility and a clean contract with the next. Think of it as an assembly line where every station adds value and passes forward exactly what the next station needs.

THE PIPELINE FLOW

Discovery → Fetch → Extraction → Cleaning → Chunking → Enrichment → Indexing → Retrieval → Grounding → Monitoring

THE 10 STAGES :

1. DISCOVERY — Find URLs and decide what is worth crawling. Sitemaps, internal links, partner feeds, topic filters.
2. FETCH — Download HTML (and optionally assets) with correct headers, rate limits, and render logic for JS-heavy pages.
3. EXTRACTION — Convert HTML into structured text plus metadata (headings, tables, links, author, dates).
4. CLEANING — Remove boilerplate and normalize formatting without deleting meaning.
5. CHUNKING — Segment content into retrieval units with stable, semantic boundaries.
6. ENRICHMENT — Attach entities, timestamps, topic tags, and provenance to each chunk.
7. INDEXING — Store chunks in a vector database and/or a keyword search index.
8. RETRIEVAL — Fetch the most relevant chunks for a given query.
9. GROUNDING — Ensure the model's answer is supported by (and traceable to) retrieved facts.
10. MONITORING — Detect failures, drift, and quality regressions continuously.

THE KEY PRINCIPLE: ALIGNMENT

Every stage should preserve exactly the meaning the NEXT stage needs — no more, no less. Cleaning that strips a heading breaks chunking. Chunking that splits a table breaks retrieval. Extraction that loses the publish date breaks freshness scoring downstream.

4: PILLAR/CLUSTER STRATEGY

This article is a PILLAR PAGE: broad, authoritative, and internally linked to a set of focused cluster articles that each go deep on one subtopic. This structure signals topical authority to search engines and gives readers a clear path from overview to detail.

RECOMMENDED CLUSTER ARTICLES

(Each links back to this pillar)

• Scraping for RAG: chunking and embeddings — hands-on companion to Section 9.
• HTML-to-Markdown extraction patterns — preserving tables, code, and headings faithfully.
• PDF/DOCX scraping for knowledge bases — handling non-HTML sources at scale.
• Freshness and lastmod auditing — detecting and scoring stale content.
• Citation readiness and schema alignment — making every chunk traceable to its source.
• Observability for scraping pipelines — metrics, alerts, and drift detection.
• Enterprise governance and compliance — robots policy, licensing, and PII handling.

INTERNAL LINKING RULE

The pillar links DOWN to every cluster with descriptive anchor text (not "click here"), and each cluster links UP to the pillar and SIDEWAYS to 1–2 sibling clusters. This creates a tight semantic graph that both users and crawlers can navigate.

5: SEARCH INTENT AND CONTENT COVERAGE

Ranking under 2026 systems means satisfying the DOMINANT intent while also covering the adjacent questions a genuinely useful page would answer.

PRIMARY INTENT

"How do I build a pipeline to feed live web data into an LLM?" — an implementation-and-architecture query.

SECONDARY INTENTS THIS GUIDE DELIBERATELY COVERS

1. "What tools should I use?"

Managed RAG API → Fast start, less control (prototypes)
Open-source frameworks → Full control, self-hosted (production)
Crawler: Scrapy
Extractor: Readability, BeautifulSoup
Vector DB: Pinecone, Weaviate, Qdrant
Orchestration: Airflow, Prefect, Dagster
Custom in-house → Tailored, expensive (enterprise)
Fine-tuning → No retrieval, but stale (not for live data)
Key settings:
Render JS: Auto (only when needed)
Chunk size: 512 tokens
Extract format: Markdown
Indexing: Reindex on hash change only
Retrieval: Use hybrid (vector + keyword) for technical content.

Q2: HOW DO I AVOID BAD EXTRACTION?

What to preserve:

• Heading hierarchy
• Tables (with row/column structure, not prose)
• Links with anchor text
• Metadata (title, author, date, URL)
• Provenance (source URL + fetch time)

Best practices:

• Use Readability algorithm, not tag stripping
• Output to Markdown format (consistent)
• Validate structure (check headings, tables survived)
• Catch template drift early

Common mistakes:

• Flattening tables into text ❌
• Losing publish dates ❌
• Inconsistent output format ❌

Q3: HOW DO I KEEP THE DATA FRESH?

Use adaptive scheduling:

• Frequently changing (news, prices) → crawl hourly
• Rarely changing (archived docs) → crawl monthly
• Automatic promotion/demotion based on actual change rate

Use conditional requests:

• Check ETag / If-Modified-Since before fetching (free)
• Only download if changed

Capture freshness signals:

• Publish date
• Last-modified date
• Fetch timestamp
• Content hash (for change detection)

At retrieval time:

• Boost recent chunks, demote stale ones
• Set freshness SLA per source (e.g., "prices ≤ 1 hour old")

Q4: HOW DO I MEASURE QUALITY?

Build an eval harness with these metrics:

• Retrieval: Recall@k, Precision@k (did right chunk get retrieved?)
• Groundedness: % of claims supported by context (90%+ target)
• Relevance: Does answer address the question? (85%+ target)
• Freshness: Does answer use current version? (95%+ for time-sensitive)

Per-stage monitoring:

• Fetch: success rate, soft-block rate
• Extraction: main-content found %, table preservation %
• Chunking: oversized/undersized chunk counts
• Retrieval: recall@k, latency
• Grounding: groundedness, hallucination rate

Alert on: Template drift (sudden drop in "main content found")

Run eval harness on every pipeline change — if quality drops, don't ship

COVERAGE PRINCIPLE

Each intent is answered with something concrete — an architecture decision, a configuration pattern, or a troubleshooting step — not a vague paragraph. Depth on the primary intent plus honest coverage of the secondary intents is what produces INFORMATION GAIN over thinner competitor articles.

6: DATA ACQUISITION

Discovery is where most pipelines quietly waste time and money. Crawling everything is expensive, slow, and often forbidden. Crawling THE RIGHT THINGS ON THE RIGHT CADENCE is the whole game.

STRATEGY 1: BUILD A URL STRATEGY, NOT A SPIDER

Instead of turning a crawler loose, define WHERE URLS COME FROM and WHY EACH ONE QUALIFIES:

START FROM KNOWN SOURCES

XML sitemaps, RSS/Atom feeds, internal links, and partner or API feeds. These are the cheapest, most reliable entry points and usually carry lastmod hints.

USE TOPIC-BASED URL SELECTION

Only crawl pages relevant to your domain. A URL classifier (rules or a small model over the URL path, title, and breadcrumb) prevents you from indexing the entire internet by accident.

RESPECT ROBOTS.TXT AND CRAWL POLICIES

Honor disallow rules, crawl-delay, and any explicit AI/usage terms. This is both an ethical baseline and a legal risk control.

SET CRAWL FREQUENCY FROM FRESHNESS REQUIREMENTS

Not habit. A page you re-crawl hourly that changes yearly is pure waste.

STRATEGY 2: FETCHING CORRECTLY

SEND HONEST USER-AGENT HEADERS

Include a descriptive User-Agent and contact URL. Don't impersonate a browser to evade blocks you've been asked to respect.

APPLY PER-HOST RATE LIMITS AND CONCURRENCY CAPS

Never degrade a source site.

DETECT JAVASCRIPT-RENDERED PAGES

Route only those through a headless browser — rendering everything is slow and costly.

TREAT SOFT FAILURES AS FAILURES

A 200 response with an empty body, a CAPTCHA, or a login wall must be flagged, not ingested.

STRATEGY 3: ADAPTIVE SCHEDULING

Scheduling should be ADAPTIVE, NOT FIXED. Track how often each source actually changes and adjust:

FREQUENTLY CHANGING PAGES

News, prices, docs → shorter recrawl intervals.

RARELY CHANGING PAGES

Archived references → longer intervals, validated with cheap HEAD / If-Modified-Since / ETag checks before a full fetch.

AUTOMATIC PROMOTION/DEMOTION

Use observed change rate to promote or demote a URL's crawl tier automatically.

RULE OF THUMB

Spend your crawl budget where change happens. A conditional request that returns 304 Not Modified costs almost nothing and protects both your budget and the source server.

7: EXTRACTION

Extraction is NOT "strip the HTML tags." It is LAYOUT-AWARE TRANSFORMATION — turning a rendered page into structured facts while preserving the relationships that give those facts meaning.

WHAT HIGH-QUALITY EXTRACTION OUTPUT CONTAINS

1. CLEAN MAIN CONTENT
   Heading hierarchy, paragraphs, and lists intact.
2. TABLES PRESERVED WITH ROW/COLUMN SEMANTICS
   Not flattened into prose. A price–tier table must remain queryable as structured data or faithful Markdown.
3. LINKS PRESERVED WITH ANCHOR TEXT
   Anchor text carries intent and enables citation and entity resolution later.
4. METADATA
   Title, author (if present), publish date and last-modified (if present), canonical URL, language.
5. PROVENANCE
   The source URL and the fetch timestamp, attached at extraction time so it can never be lost downstream.

WHY STRUCTURE MATTERS FOR AI

For an LLM pipeline, the extraction output should be LLM-READY AND CONSISTENT. Consistency is the point: if every page produces the same shape of output (e.g., normalized Markdown plus a metadata object), then chunking, embedding, and retrieval behave PREDICTABLY. Inconsistent extraction is the root cause of most "why did retrieval get worse?" incidents.

8: CLEANING

Removing Noise Without Losing Meaning

Cleaning is where you remove the boilerplate that harms retrieval:

• Navigation menus and breadcrumbs used purely for site chrome
• Cookie and consent banners
• Footer link farms and legal micro-links
• Repeated "related articles," "you may also like," and share widgets
• Ad slots and injected promotional blocks

But cleaning is a double-edged tool. Over-aggressive cleaning strips context that retrieval genuinely needs — a disclaimer that changes the meaning of a fact, a caption that identifies a table, a byline that establishes authority.

THE AUDIT-FIRST APPROACH

1. DEFINE WHAT "MEANING" MEANS FOR YOUR DOMAIN FIRST
   Before you write a single rule. For a docs bot, code samples and version notes are sacred. For a news RAG, datelines and bylines are sacred.
2. PREFER STRUCTURAL SIGNALS OVER KEYWORD BLOCKLISTS
   Removing an element because it's in <footer> or a repeated cross-page block is safer than deleting any paragraph containing the word "cookie."
3. NORMALIZE, DON'T MUTILATE
   Collapse whitespace, standardize quotes and dashes, and de-duplicate repeated blocks — without rewriting the author's actual sentences.
4. KEEP A BEFORE/AFTER DIFF SAMPLE IN REVIEW
   So a human can confirm you removed chrome, not substance.

THE TEST

If a knowledgeable reader would say "you deleted something I needed," your cleaning is too aggressive. If they'd say "why is the cookie banner in here," it's too soft. Tune against real samples, not assumptions.

9: CHUNKING

Token-Aware, Citation-Aware Segmentation

Chunking is the single biggest lever on retrieval quality. Chunk badly and even a perfect model retrieves the wrong context; chunk well and a modest model answers precisely.

WHAT GOOD CHUNKING DOES

• USES SEMANTIC BOUNDARIES
  Split on headings and section breaks, not blindly every N characters. A chunk should be a coherent idea, not an arbitrary slice.
• KEEPS RELATED FACTS TOGETHER
  A definition and its example, a step and its warning, a metric and its unit — these belong in the same chunk.
• NEVER SPLITS A TABLE MID-ROW
  Keep a table (or a labeled sub-section of it) whole, so a retrieved chunk is self-contained.

IS TOKEN-AWARE
Size chunks to your embedding model's sweet spot and your prompt budget — large enough to carry context, small enough to stay specific. Add modest overlap only where ideas span boundaries.

ATTACHES STABLE METADATA TO EVERY CHUNK
Source URL, section heading path, position, and IDs.

CITATION-AWARE CHUNKING

• Citation-aware means every chunk can be mapped back to a precise source location — URL plus heading path or character offsets. Even if you never show citations to end users, you need this traceability to debug: when an answer is wrong, you follow the chunk ID straight to the source and see whether the failure was extraction, cleaning, or retrieval.

THE ANTI-PATTERN

Fixed-size character chunking with no structure awareness. It's trivial to implement and reliably splits tables, separates headings from their content, and severs facts from their qualifiers.

Start from structure; fall back to size only within a section.

10: ENRICHMENT

Entities, Metadata, and Freshness Signals

Enrichment is the stage that turns clean chunks into retrievable, evaluable, trustworthy units. It adds the signals that improve both retrieval precision and downstream evaluation.

WHAT TO ADD

ENTITIES
People, organizations, products, locations, versions. Extracted entities enable filtering ("only chunks about Product X"), disambiguation, and knowledge-graph linking.

FRESHNESS SIGNALS
Publish date and last-modified, normalized to a single timezone and format. This is the most important enrichment for live web data.

TOPIC TAGS / TAXONOMY
A controlled vocabulary so retrieval and analytics can slice by subject.

PROVENANCE (CARRIED FORWARD)
Source URL, fetch timestamp, extractor version, and content hash for change detection.

WHY FRESHNESS IS NON-NEGOTIABLE

For live web data, freshness is a first-class ranking signal inside your own retriever. Without it, two chunks that contradict each other look equally valid, and your system may confidently return the outdated one.

With it, you can:

• Boost recent chunks and demote stale ones at retrieval time.
• Expire or re-validate content past a domain-specific TTL.
• Surface "last verified on" context to the model so it can hedge appropriately.

CONTENT HASHING TIP

Store a hash of each chunk's cleaned text. On recrawl, unchanged hashes skip re-embedding (saving cost), while changed hashes trigger re-indexing and a freshness-timestamp update — giving you cheap, precise change detection.

11: GROUNDING AND EVALUATION

Proving the Data Helps

Grounding is what separates a RAG system from a confident guess. Grounded means the model's answer is supported by, and traceable to, the retrieved chunks — not invented from parametric memory.

DESIGN FOR GROUNDING

• PASS RETRIEVED EVIDENCE EXPLICITLY
  Instruct the model to answer only from it, flagging when the context is insufficient rather than filling gaps.
• ATTACH CHUNK IDS TO CLAIMS
  So every sentence in an answer can be traced to a source. Even internal-only citations pay for themselves the first time you debug a hallucination.
• PREFER "I DON'T KNOW" OVER FABRICATION
  A system that abstains when evidence is missing is more trustworthy than one that always answers.

EVALUATE WITH A REAL HARNESS
You cannot improve what you don't measure. Build an evaluation set of representative queries with known-good answers and track:

RETRIEVAL METRICS
Recall@k and precision@k: did the right chunk make it into the context window at all?

GROUNDEDNESS / FAITHFULNESS
Is each claim actually supported by the retrieved text? (LLM-as-judge plus human spot-checks.)

ANSWER RELEVANCE
Does the response address the user's actual question?

FRESHNESS CORRECTNESS
When facts changed, did the answer reflect the current version?

AUDIT-FIRST RULE
Run the eval harness on every pipeline change — a new extractor, a new chunk size, a new embedding model. If an "improvement" drops groundedness, it isn't an improvement.

12: SECURITY, COMPLIANCE, AND SAFETY

Managing the Legal, Ethical, and Security Boundaries

Web scraping for AI touches legal, ethical, and security boundaries at once. Treat these as design requirements, not afterthoughts.

LEGAL AND ETHICAL

• RESPECT ROBOTS.TXT, TERMS OF SERVICE, AND RATE LIMITS
  Being technically able to fetch a page does not make it permitted.
• HONOR LICENSING AND COPYRIGHT
  Store provenance so you can prove where every fact came from and remove sources on request.
• HANDLE PERSONAL DATA (PII) LAWFULLY
  Detect and redact or exclude PII where you have no lawful basis to process it; align with GDPR/CCPA and your own data-processing agreements.

SECURITY — THE UNDERRATED RISK

Scraped content is untrusted input. In agent and RAG systems it becomes a live attack surface:

• PROMPT INJECTION
  A page can contain hidden instructions ("ignore previous instructions and…"). Never let retrieved text be treated as system instructions. Separate data from directives, and constrain what tools an agent can call.
• DATA EXFILTRATION
  Injected content may try to make an agent leak secrets or call attacker URLs. Sandbox tool use and allowlist outbound calls.
• POISONING
  Adversaries can seed misleading content to corrupt your index. Weight sources by trust and monitor for anomalous new claims.

THE PRINCIPLE

Treat every scraped byte as hostile until proven otherwise. Isolation, least privilege, and output validation are non-negotiable in agent pipelines.

13: PERFORMANCE AND COST ENGINEERING

Optimizing for Scale

At scale, ingestion cost is dominated by three things: fetching, embedding, and storage. Engineer each deliberately.

FETCH EFFICIENTLY

• Use conditional requests (ETag/If-Modified-Since) to skip unchanged pages
• Cache aggressively
• Render with a headless browser only when a page truly needs JS

EMBED ONLY WHAT CHANGED

Content hashing (from §10) means you re-embed a fraction of your corpus per cycle instead of all of it — often a 10×+ cost reduction on large sites.

RIGHT-SIZE STORAGE

Vector dimensions, quantization, and index type (HNSW vs. IVF) trade recall for memory and cost. Measure recall at each setting rather than defaulting to the largest.

BATCH AND PARALLELIZE

Embedding and indexing, but cap concurrency to respect source sites and downstream rate limits.

TIER YOUR COMPUTE

Cheap conditional checks first; expensive rendering and embedding only for content that passed the change filter.

THE COST LENS

The cheapest request is the one you don't make. Most savings come from not re-processing unchanged content, not from a faster model.

14: OBSERVABILITY

Debugging the Pipeline Like an Engineer

A scraping pipeline that can't be observed will fail silently. Instrument every stage so you can answer "what broke, where, and when?"

METRICS TO TRACK PER STAGE

DISCOVERY/FETCH

• URLs discovered
• Fetch success rate
• Soft-block rate
• Average latency
• 304 ratio

EXTRACTION

• % pages with a main-content block found
• Table-preservation rate
• Missing-metadata rate

CLEANING

• Average content-length delta (spikes = over/under-cleaning)

CHUNKING

• Chunks per document
• Oversized/undersized chunk counts

INDEXING/RETRIEVAL

• Index size
• Embedding failures
• Retrieval latency
• Recall@k on the eval set

GROUNDING

• Groundedness score
• Abstain rate
• Hallucination reports

DRIFT DETECTION

The key failure mode is template drift — a site redesign that quietly breaks extraction. Detect it by alerting on distribution shifts:

• Sudden drop in "main content found"
• Jump in content-length delta
• Spike in empty chunks

These fire before users notice bad answers.

TRACEABILITY PAYOFF

Because every chunk carries its source URL, fetch time, and extractor version (§10), any bad answer can be traced back through:

retrieval → chunk → extraction → fetch

In minutes, not days.

15: CONFIGURATION PATTERNS

Practical Implementation

You don't need exotic tooling — you need clear contracts between stages. Below is a representative, tool-agnostic configuration shape.

DISCOVERY CONFIGURATION

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Sources:
  - Type: sitemap
    URL: https://example.com/sitemap.xml

  - Type: feed
    URL: https://example.com/rss

URL Filter:
  Include paths: ["/docs/", "/blog/"]
  Exclude paths: ["/tag/", "/author/"]
  Respect robots.txt: YES

FETCH CONFIGURATION

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
User-Agent: "OllagraphBot/1.0 (+https://ollagraph.com/bot)"
Max concurrency per host: 2
Conditional requests: YES (ETag / If-Modified-Since)
Render JavaScript: Auto (only when needed)

EXTRACTION CONFIGURATION

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Main content strategy: Readability
Preserve: [headings, tables, links, code]
Output format: Markdown
Capture metadata: [title, author, published, modified, canonical, lang]

CLEANING CONFIGURATION

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Remove: [nav, footer, cookie_banner, related_posts, ads]
Strategy: Structural (prefer DOM structure over keyword blocklists)
Normalize whitespace: YES

CHUNKING CONFIGURATION

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Strategy: Semantic (split on headings first)
Max tokens: 512
Overlap tokens: 64
Keep tables intact: YES
Attach metadata: [source_url, heading_path, position, content_hash]

ENRICHMENT CONFIGURATION

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Extract entities: YES
Freshness signals: [published, modified]
Topic tags: Controlled vocabulary

INDEXING CONFIGURATION

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Vector store: [Your vector DB]
Reindex on: Content hash change (skip unchanged chunks)

SCHEDULING CONFIGURATION

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Strategy: Adaptive
Minimum interval: 6 hours
Maximum interval: 30 days

WHY THIS SHAPE WORKS

• Each block maps 1:1 to a pipeline stage
• Every stage declares what it preserves for the next
• Change detection (content_hash) is wired through indexing and scheduling
• You only pay for what actually changed

16: TROUBLESHOOTING DECISION TREE

When Answer Quality Drops, Diagnose Backward Through the Pipeline

BAD OR WRONG ANSWER?
│
├─ WAS THE CORRECT CHUNK RETRIEVED? (check recall@k)
│   │
│   ├─ NO → RETRIEVAL PROBLEM
│   │       │
│   │       ├─ Chunk exists in index?
│   │       │   │
│   │       │   ├─ NO → EXTRACTION/CLEANING DROPPED IT
│   │       │   │        → Inspect source document
│   │       │   │
│   │       │   └─ YES → EMBEDDING/QUERY MISMATCH
│   │       │            → Tune model, chunk size, or hybrid search
│   │
│   └─ YES → GO TO GROUNDING
│
├─ CHUNK RETRIEVED BUT ANSWER IGNORES IT?
│   → GROUNDING/PROMPT PROBLEM
│   → Tighten "answer only from context"
│   → Reduce distractor chunks
│
├─ ANSWER IS OUTDATED?
│   → FRESHNESS PROBLEM
│   → Check recrawl schedule
│   → Check TTL
│   → Check content-hash change detection
│
└─ WHOLE SITE RETURNS GARBAGE?
    → TEMPLATE DRIFT
    → Check "main content found" + content-length delta alerts
    → Fix extractor

17: ENTERPRISE DEPLOYMENT PLAYBOOK

Running This at Organizational Scale

Running a pipeline at organizational scale adds governance, reliability, and cross-team concerns.

DATA GOVERNANCE

Central catalog of sources with:

• Owner
• License
• Robots status
• PII classification

Nothing enters the index without a governance record.

ACCESS CONTROL

• Role-based access to the index
• Segment sensitive corpora so a general chatbot can't retrieve restricted content

REPRODUCIBILITY

• Version your extractors, chunkers, and embedding models
• Store the pipeline version alongside each chunk
• Reproduce or re-run any past state

HUMAN-IN-THE-LOOP REVIEW

Sample outputs for high-stakes domains before they reach users:

• Legal
• Medical
• Financial

SLAs AND OWNERSHIP

• Define freshness SLAs per source

Example: "pricing pages ≤ 1 hour stale"

• Assign a team that owns each source

AUDIT TRAIL

Retain provenance and eval history so you can answer a compliance question about any past answer.

18: CLOUD DEPLOYMENT PLAYBOOK

Mapping the Pipeline to Managed, Elastic Components

A cloud-native deployment maps the pipeline to managed, elastic components:

ORCHESTRATION

A workflow engine (e.g., a managed DAG/queue service) drives:

• discovery → fetch → extract → index as decoupled, retryable steps

QUEUES AND WORKERS

Decouple stages with message queues so:

• A slow render step never blocks indexing
• Failures retry with backoff

SERVERLESS FOR BURSTY WORK

Run fetch/extraction as scalable functions or containers:

• Scale to zero between crawl cycles
• Pay only for what you use

MANAGED VECTOR STORE + OBJECT STORAGE

• Keep raw HTML and intermediate Markdown in object storage (cheap, replayable)
• Keep chunks in a managed vector database

SECRETS AND NETWORK ISOLATION

• Store credentials in a secrets manager
• Run agent tool-calls inside a sandboxed, egress-restricted network

COST CONTROLS

• Autoscaling caps
• Per-host rate limits
• Conditional fetching keeps spend proportional to actual change

DESIGN GOAL

Every stage is independently scalable, retryable, and replayable from stored raw input — so reprocessing the corpus with a better extractor never requires re-crawling the web.

19: ALTERNATIVES AND COMPARISONS

Choosing Your Stack

There is no single "right" stack — choose based on scale, control, and compliance needs.

MANAGED SCRAPING/RAG API

Pros:

• Fast to start
• Handles rendering & blocks

Cons:

• Less control
• Ongoing cost
• Data leaves your perimeter

Best for: Prototypes, small teams

OPEN-SOURCE FRAMEWORKS

(Crawler + extractor + vector DB)

Pros:

• Full control
• Self-hosted
• Auditable

Cons:

• You own maintenance and drift handling

Best for: Most production systems

CUSTOM IN-HOUSE PIPELINE

Pros:

• Tailored to your domain and compliance

Cons:

• Highest build/maintenance cost

Best for: Large scale, strict governance

FINE-TUNING INSTEAD OF RAG

Pros:

• No retrieval at query time

Cons:

• Stale
• Expensive to update
• Hard to trace

Best for: Stable tone/format, not live facts

RETRIEVAL CHOICES

Vector-only search is simple but misses exact-match terms.

Hybrid (vector + keyword) retrieval is the pragmatic default for technical content where identifiers, versions, and error codes matter.

20: FAQs

Common Questions Answered

1. IS WEB SCRAPING FOR AI LEGAL?

It depends on the source's terms, the data type, and jurisdiction.

Best practices:

• Respect robots.txt and ToS
• Honor copyright and licensing
• Handle PII lawfully
• Keep provenance

When in doubt: Get legal sign-off per source.

RAG OR FINE-TUNING FOR LIVE DATA?

RAG, almost always.

Live facts change; RAG lets you:

• Refresh the index without retraining
• Give citations and freshness signals

Fine-tune for: Style and format, not volatile facts.

2. WHAT CHUNK SIZE SHOULD I USE?

Start around 300–512 tokens with semantic boundaries and small overlap, then tune against your eval harness.

There is no universal number — it depends on your embedding model and content structure.

3. HOW DO I STOP HALLUCINATIONS?

You reduce them through:

• Strong retrieval (right chunk in context)
• Strict "answer only from provided context" prompting
• An abstain path
• Groundedness evaluation on every change
• Traceable chunks let you diagnose the ones that slip through

4. HOW OFTEN SHOULD I RE-CRAWL?

As often as the content changes — no more.

Use:

• Adaptive scheduling driven by observed change rate
• Conditional requests
• Freshness SLAs per source

5. HOW DO I HANDLE JAVASCRIPT-HEAVY SITES?

Render only those pages with a headless browser.

Best approach:

• Detect the need automatically
• Rendering everything is slow and expensive
• Cache rendered output

21: CONCLUSION

Feeding live web data into LLMs is an engineering discipline, not a script. The teams that succeed treat it as an end-to-end system where every stage — discovery, fetch, extraction, cleaning, chunking, enrichment, grounding, and monitoring — preserves the meaning the next stage depends on.

THREE CORE IDEAS

1. INFORMATION GAIN OVER VOLUME Keep the facts that matter, drop the noise that hurts retrieval.

2. TRACEABILITY OVER TRUST Every chunk, and every answer, should be explainable back to a source and a timestamp.

3. CONTINUOUS EVALUATION OVER ONE-TIME SETUP Sites drift, facts change, and models update — your quality gate