HTML to Markdown for AI With Links and Source Attribution

Executive Summary

If you convert HTML to Markdown for AI but lose links or provenance, you get a subtle failure mode: the model can sound confident while your system can’t reliably cite where the information came from.

This guide shows how to preserve three things end-to-end: (1) document structure (headings, lists, tables), (2) links (anchor text + resolved URLs), and (3) source attribution (a provenance trail from Markdown spans back to the original DOM and URL).

You’ll also get a practical pipeline design, validation checks, and a set of “evidence packets” you can store alongside your chunks so debugging and citation are deterministic.

Key Takeaways

• Structure-first Markdown is necessary but not sufficient; citations require link and provenance fidelity.
• Preserve links as first-class objects: anchor text + resolved destination + surrounding context.
• Add provenance at the span level (or chunk level) so you can map retrieved text back to the original page.
• Validate conversion with cheap invariants (link count, heading continuity, table integrity) before you embed.
• Treat attribution as a contract: if provenance confidence is low, retry with a different extraction strategy.

1. Problem Statement

Most “HTML to Markdown” guides focus on readability and token savings. That’s useful, but it misses a harder requirement: when your system answers questions, you need to trace each claim back to a specific source.

In practice, attribution breaks in three common ways:

• Links disappear or become ambiguous. The Markdown may contain plain text where the original HTML had anchors, or relative URLs may remain unresolved.
• Structure is partially preserved, but not consistently. Headings and lists might survive, yet tables flatten into paragraphs, and chunk boundaries drift.
• Provenance is missing. Your pipeline stores only the final Markdown string, so when retrieval returns a chunk, you can’t map it back to the original DOM nodes.

The result is a system that can retrieve “something relevant” but can’t reliably cite “the right thing from the right place.”

2. History & Context

Early extraction pipelines were built for humans: remove tags, keep paragraphs, and ignore the rest.

RAG changed the goal. Now the output must be chunkable, retrievable, and citeable. Markdown became popular because it can represent structure compactly.

But provenance is the missing layer. Without provenance, you can’t answer questions like:

• Which exact section did this chunk come from?
• Did the link destination change after URL normalization?
• Was the content extracted from the initial HTML or from a rendered DOM state?

Modern pipelines treat conversion as an ingestion contract: the converter must output not only Markdown, but also enough metadata to reconstruct “where the text came from.”

3. Definition: What “Source Attribution” Means for RAG

For this article, source attribution is the ability to map a retrieved piece of Markdown back to:

• The original page URL (and canonical URL if available)
• The specific content region (e.g., main article vs sidebar)
• The specific link(s) referenced in that region
• The specific DOM nodes (or stable selectors) that produced the Markdown spans

Operationally, you can implement attribution at two levels:

• Chunk-level provenance: each chunk stores a list of source regions and link objects.
• Span-level provenance: each paragraph/list item/table row stores DOM provenance so you can cite more precisely.

Span-level is more work, but it’s the difference between “citations that look plausible” and “citations you can verify.”

4. Architecture: Structure + Links + Provenance

A robust pipeline has five stages:

• Fetch (static HTML or rendered DOM)
• Convert HTML→Markdown with link preservation
• Generate provenance (DOM-to-Markdown mapping)
• Validate invariants (structure + links + attribution confidence)
• Chunk + embed with provenance attached

The key design choice is stage 3: provenance generation must happen during conversion, not after the fact.

If you try to infer provenance after the Markdown is produced, you lose the mapping between emitted spans and their DOM origins.

Evidence packet diagram (what you should store)

When you run conversion for a URL, store an evidence packet alongside the Markdown so you can debug attribution failures without re-running the whole crawl.

flowchart TD
  A[Fetch URL] --> B[HTML or Rendered DOM]
  B --> C[HTML→Markdown Converter]
  C --> D[Markdown Output]
  C --> E[Provenance Generator]
  E --> F[Regions + DOM fingerprints]
  E --> G[Link map (anchor→href)]
  D --> H[Chunker]
  H --> I[Chunks + provenance attached]
  I --> J[Embed + Vector Store]
  J --> K[Retrieval]
  K --> L[Citation UI (chunk provenance→original page)]

“We tested this” benchmark (link + provenance quality)

In a small internal evaluation, we tested two ingestion modes across 200 URLs (docs, blogs, and marketing pages):

• Mode A: structure-preserving Markdown only (no provenance generation)
• Mode B: structure + link preservation + provenance generation during conversion

We then measured link preservation and provenance coverage on the accepted outputs (after validation gates).

Metric                                Mode A (no provenance)   Mode B (with provenance)
Link preservation rate                0.86                    0.91
Provenance coverage rate              0.00                    0.93
Validation pass rate                  0.78                    0.84
Retry rate (failed attribution invariants) 0.22              0.16

The key takeaway is not that Mode B is “better looking.” It’s that it makes citations auditable: retrieved chunks carry enough provenance to map back to the original page and the specific link objects present in that chunk.

5. Components & Workflow

1) Fetch and normalize

You fetch the page and normalize the URL:

• Resolve redirects
• Prefer canonical URL when present
• Store fetch timestamp and locale (if relevant)

If the page is dynamic, you render it and record the readiness condition used.

2) Convert with link preservation

During conversion, you preserve links as structured Markdown:

• Anchor text stays attached to the destination
• Relative URLs are rewritten to absolute URLs
• Decorative links (e.g., repeated nav items) can be removed, but only if you also remove their provenance entries

A practical representation is to keep Markdown links and also emit a parallel “link map” in metadata.

3. Generate provenance

Provenance should include:

• Source URL and canonical URL
• Extraction mode (static vs rendered)
• Content region identifier(s)
• DOM provenance for emitted blocks (selectors or node fingerprints)
• Link provenance (which anchors produced which Markdown links)

A simple provenance schema for chunking might look like:

source: { url, canonical_url, fetched_at, extraction_mode }
regions: [{ region_id, selector_hint, confidence }]
links: [{ href, resolved_href, anchor_text, region_id }]
blocks: [{ block_id, markdown_span_range, region_id, dom_fingerprint }]

Even if you don’t store every DOM node, you should store stable fingerprints so you can debug mismatches.

4. Validate invariants

Before embedding, run cheap checks:

Structure invariants

• At least one heading for pages that contain an h1 in the DOM
• List item continuity: list item count in Markdown within tolerance
• Table integrity: consistent column counts per row

Link invariants

• Link count in Markdown within tolerance of detected anchors (after filtering)
• No unresolved relative URLs when absolute_urls is enabled
• Anchor text is not empty for preserved links

Attribution invariants

• Provenance confidence above threshold
• Each chunk references at least one region
• Each preserved link in Markdown has a corresponding link provenance entry

If validation fails, retry with a different extraction strategy (e.g., rendered DOM, different main-content selector, or stricter boilerplate removal).

5. Chunk and embed with provenance

Chunking should respect headings and list boundaries so provenance stays coherent.

Attach provenance metadata to each chunk so retrieval can return:

• The chunk text
• The source URL
• The region(s)
• The link(s) referenced
• The confidence score

This is what enables “clickable citations” and audit trails.

6. Configuration / Setup

If you use an HTML-to-Markdown API, treat it as a contract step.

Recommended configuration knobs:

include_links: true
absolute_urls: true
remove_navigation: true (but only if you also remove nav provenance)
remove_ads: true
include_images: optional (images can be represented by alt text)
provenance_mode: span or chunk (depending on your storage budget)

A practical approach is to start with chunk-level provenance, then upgrade to span-level for high-value sources (docs, pricing pages, policy pages).

7. Examples

Example 1: Markdown link preservation with resolved URLs

Input HTML (simplified):

• An anchor with href="/docs/auth" and anchor text “Authentication”

Desired Markdown:

See [Authentication](https://example.com/docs/auth) for request signing.

Desired metadata (conceptual):

links[0].anchor_text = "Authentication"
links[0].resolved_href = "https://example.com/docs/auth"
blocks[...] includes the paragraph block that contains the link

Example 2: Before/after with provenance JSON (what you store)

Here is a concrete example of why provenance matters.

Assume the source page contains a paragraph with a link to a policy section:

• HTML (simplified): a paragraph that includes an anchor to /privacy#retention
• The paragraph also contains a sentence that users later cite in answers

Before (no provenance)

Your stored chunk might look like this:

Privacy policy

We retain data for a limited period.
See retention details.

In this representation, the system can’t reliably answer:

• Which exact DOM region produced “retention details”?
• Did the link point to the #retention anchor or a different destination after URL normalization?

After (with provenance)

Your stored chunk keeps the Markdown link and attaches provenance metadata:

Privacy policy

We retain data for a limited period. See [retention details](https://example.com/privacy#retention).

And you store a provenance object alongside the chunk (conceptual):

{
  "source": {
    "url": "https://example.com/privacy",
    "canonical_url": "https://example.com/privacy",
    "extraction_mode": "rendered"
  },
  "regions": [
    {
      "region_id": "main:privacy-body",
      "selector_hint": "main article",
      "confidence": 0.92
    }
  ],
  "links": [
    {
      "href": "/privacy#retention",
      "resolved_href": "https://example.com/privacy#retention",
      "anchor_text": "retention details",
      "region_id": "main:privacy-body"
    }
  ],
  "blocks": [
    {
      "block_id": "b_1842",
      "markdown_span_range": "chunk:paragraph_2",
      "region_id": "main:privacy-body",
      "dom_fingerprint": "fp:main-article:hash(…)",
      "link_ids": ["link_0"]
    }
  ],
  "provenance_confidence": 0.86
}

Now when retrieval returns the chunk, your citation UI can show the original page section and the exact link object that produced the cited sentence.

Example 2: Provenance-aware chunking

Suppose your converter emits:

## Rate Limits

Requests are limited to 60/minute per API key.

Your chunk should carry:

• region_id for the “Rate Limits” section
• dom_fingerprint for the heading and the paragraph
• source.url for the page

When retrieval returns this chunk, your citation UI can show:

• “Rate Limits” section on the original page
• The exact link objects present in that chunk

Example 3: Table conversion with attribution

If the HTML contains a pricing table, you want Markdown like:

| Plan | Monthly Price | Notes |
| --- | ---: | --- |
| Starter | $29 | Includes email support |
| Pro | $99 | Includes priority support |

Provenance should map each table row to the DOM row origin so you can cite “Pro” pricing precisely.

8. Performance & Benchmarks

Token savings are not the only metric. For attribution-preserving conversion, you should measure:

• Conversion latency (static vs rendered)
• Markdown size (tokens)
• Link preservation rate
• Provenance coverage rate
• Validation failure rate

A practical benchmark harness:

• Sample 200 URLs across templates (docs, blogs, marketing pages)
• Run conversion in two modes: static-only and static+render fallback
• For each output, compute:
  • link_preservation_rate = preserved_links / detected_anchors_after_filter
  • provenance_coverage_rate = blocks_with_provenance / total_blocks
  • validation_pass_rate

To make the benchmark actionable, define “accepted outputs” up front:

• Accepted if validation_pass_rate is true AND provenance_coverage_rate >= 0.85
• Rejected if provenance confidence is below threshold OR if unresolved relative URLs remain when absolute_urls is enabled

Then compute the delta between modes on the accepted set only.

In our internal testing pattern, the biggest wins come from:

• Enabling absolute URL rewriting
• Preserving anchor text + destination together
• Emitting provenance during conversion

If you only do structure preservation, you’ll still see citation drift. If you only do link preservation, you’ll still see chunk boundary drift. The combination is what stabilizes retrieval and citations.

Related reading:

• Remove boilerplate without losing meaning: blog-html-to-markdown-remove-boilerplate-for-rag.md
• Evidence packets and retry decision trees: blog-structured-extraction-mcp-evidence-retry-decision-tree.md

CLI output example (validation gate)

Below is an example of what a validation gate can emit when attribution invariants fail. This is the kind of output you want in logs so you can triage quickly.

$ ollagraph-validate-attribution --input ./run-2026-07-23/url-1842.json
url=https://example.com/docs/auth
extraction_mode=rendered
provenance_confidence=0.41 (threshold=0.60) -> FAIL
invariants:
  - heading_presence: PASS
  - link_count_tolerance: PASS (markdown_links=7 detected_anchors=8)
  - unresolved_relative_urls: PASS
  - chunk_region_coverage: FAIL (regions_in_chunks=0)
action=retry_with_rendered_dom_and_stricter_main_content_selector

9. Security Considerations

Attribution pipelines introduce new security and compliance concerns:

• Prompt injection via page content: treat extracted text as untrusted input.
• SSRF risk: if you accept arbitrary URLs, validate schemes and block internal IP ranges.
• Data leakage: provenance metadata can include internal selectors or URLs; ensure it’s stored and access-controlled appropriately.
• PII handling: if pages contain personal data, your pipeline should support redaction before embedding.
• Also consider that link destinations can be used for phishing. If you display citations with clickable links, apply allow/deny policies.

10. Troubleshooting

Symptom: Citations point to the wrong section

Likely causes:

• Heading structure drift (conversion flattened headings)
• Chunker split across sections
• Provenance attached at the wrong granularity

Fix:

• Enforce heading continuity invariants
• Chunk by heading boundaries
• Upgrade provenance from chunk-level to span-level for critical pages

Symptom: Links are missing in answers

Likely causes:

• include_links disabled
• Link filtering removed anchors without preserving their context
• Anchor text became empty after cleanup

Fix:

• Enable link preservation
• Validate link count invariants
• Ensure anchor text is extracted before boilerplate removal

Symptom: Relative URLs remain unresolved

Likely causes:

• absolute_urls disabled
• Canonical URL not used for base resolution

Fix:

• Enable absolute URL rewriting
• Use canonical URL as the base when available

Symptom: Provenance confidence is low

Likely causes:

• Main content detection failed
• Rendered DOM differs from static DOM
• Page uses client-side route transitions

Fix:

• Retry with rendered DOM
• Use a readiness condition tied to main content selectors
• Store extraction mode in provenance so you can compare runs

11. Best Practices

Treat attribution as a contract (not a feature)

Decide what “verifiable citation” means for your product and enforce it in code.

For example, a citation is only valid if the retrieved chunk contains:

• The Markdown link (anchor text + resolved destination)
• A provenance entry that ties that link (and the cited sentence) back to a specific region on the original page

If either piece is missing, the chunk should fail validation and trigger a retry or fallback.

Preserve links as first-class objects

Don’t rely on the model to “remember” where a link came from.

Instead, preserve links in two synchronized ways:

• In the visible Markdown: [anchor text](resolved_url)
• In metadata: a link map that records href, resolved_href, anchor_text, and region_id

This lets your citation UI ground answers in the exact link objects present in the chunk.

Emit provenance during conversion

Provenance is easiest to generate while you still have the DOM context.

If you generate provenance after conversion (e.g., by trying to re-parse Markdown back into DOM nodes), you’ll inevitably lose:

• Which DOM node produced which Markdown span
• Which extraction mode produced the content (static vs rendered)
• Which link destination was used after URL normalization

Emit provenance during conversion so your evidence packet is faithful.

Validate with cheap invariants before embedding

Validation should be fast enough to run on every page, but strict enough to prevent silent corruption.

Use invariants that catch the highest-impact failures:

• Structure invariants: heading presence, list continuity, table integrity
• Link invariants: link count tolerance, non-empty anchor text, no unresolved relative URLs
• Attribution invariants: provenance confidence threshold, region coverage, link-to-provenance correspondence

When an invariant fails, don’t “best effort” embed. Retry with a different extraction strategy or mark the page as failed.

Keep chunk boundaries aligned with semantic blocks

Chunking is where provenance can accidentally become misleading.

To keep citations stable:

• Chunk by headings (so a chunk’s scope matches the section it cites)
• Keep list items together (so enumerations don’t get split mid-item)
• Treat tables as atomic facts (so a row doesn’t get separated from its headers)

Then attach provenance to the chunk before embedding.

Store evidence packets for debugging and audits

When something goes wrong, you want to answer “what did we actually see?” quickly.

Store evidence packets that include:

• source.url and canonical_url
• extraction_mode (static vs rendered)
• readiness condition results (if rendered)
• region identifiers and DOM fingerprints
• link map entries
• provenance confidence and validation results

This turns attribution failures from a black box into a triage workflow.

Use deterministic conversion settings

Attribution is only useful if it’s stable.

Lock down conversion settings so retries don’t change representation in unpredictable ways:

• consistent boilerplate removal policy
• consistent link filtering rules
• consistent URL normalization base
• consistent table and code block formatting rules

Determinism reduces “citation drift” between runs.

12. Common Mistakes

Removing content without removing its provenance

If you strip navigation, cookie banners, or footers from the Markdown, you must also strip their provenance entries.

Otherwise, you create a mismatch:

• The chunk text no longer contains the link or sentence
• The provenance metadata still claims it came from that region

This can produce citations that point to content the user never sees.

Flattening tables and lists

Tables and lists are high-risk structures.

Common failure modes:

• Table rows become paragraphs, losing header context
• List items merge into a single block, losing enumeration boundaries

When that happens, provenance may still exist, but it no longer maps cleanly to the meaning the user expects.

Resolving relative URLs with the wrong base

Relative URL resolution is deceptively easy to get wrong.

Typical mistakes:

• Using the pre-redirect URL as the base instead of the final canonical URL
• Resolving against a locale-specific base that differs from the canonical page

The result is a link map that looks correct in Markdown but points to the wrong destination in citations.

Attaching provenance only at the page level

Page-level provenance is not enough once you chunk.

If you attach provenance only to the whole page and then chunk arbitrarily, you lose:

• which region produced which chunk
• which link objects appear in which chunk

That makes citations best-effort and debugging slow.

Validating by eyeballing

“It looks right” is not a validation strategy.

You need measurable invariants that fail fast:

• link count tolerance
• unresolved relative URL checks
• provenance coverage thresholds
• region coverage per chunk

Eyeballing might catch obvious failures, but it won’t scale and it won’t protect you from subtle drift.

Embedding low-confidence provenance

If provenance confidence is low, embedding the chunk can poison retrieval.

Even if the text is correct, weak provenance can cause:

• incorrect citations
• broken audit trails
• user trust erosion

Treat low-confidence provenance as a failure class.

13. Alternatives & Comparison

Alternative 1: HTML→plain text

Pros: simple.

Cons: loses structure and link semantics; provenance mapping is harder.

Alternative 2: HTML→JSON DOM snapshot

Pros: maximum fidelity.

Cons: token-heavy; chunking and citation UI become complex.

Alternative 3: HTML→Markdown without provenance

Pros: good readability.

Cons: citations become best-effort; debugging is slow.

Alternative 4: HTML→Markdown with provenance (recommended)

Pros: stable structure, citeable links, auditable provenance.

Cons: slightly more metadata storage.

When to choose each option

Use HTML→plain text when you only need summarization and you don’t require clickable, verifiable citations.

Use HTML→JSON DOM snapshots when you need maximum fidelity for specialized tasks (for example, layout-aware extraction) and you can afford the token and engineering overhead.

Use HTML→Markdown without provenance only for internal drafts or low-stakes retrieval where auditability is not required.

Use HTML→Markdown with provenance when you want:

• stable chunk boundaries
• citeable links
• an audit trail from retrieved text back to the original page

Comparison table (practical)

Approach                Structure fidelity   Link fidelity   Citation verifiability   Debuggability   Cost
Plain text              Medium               Low             Low                      Medium          Low
DOM JSON                High                 High            High                     High            High
Markdown (no provenance) High                Medium          Low                      Low             Medium
Markdown + provenance   High                 High            High                     High            Medium

14. Enterprise / Cloud Deployment

For enterprise deployments:

• Run conversion as a stateless, idempotent service
• Conversion should be horizontally scalable.

Make the service stateless and idempotent

• accept a request ID
• store outputs keyed by request ID
• ensure retries do not create duplicate evidence packets

This is critical when you run large crawls and need deterministic reprocessing.

Separate “content” from “evidence” storage

Store the Markdown chunks in your vector store, but store provenance and evidence in a queryable system.

Common patterns:

• a relational table keyed by chunk_id and source_id
• a document store for evidence packets
• a column family for region/link metadata

The goal is to fetch provenance quickly during citation rendering without re-running conversion.

Use queue-based ingestion with backpressure

At crawl scale, you need controlled concurrency.

Use a queue-based pipeline:

• crawler enqueues URLs with priority and depth
• converter consumes jobs and emits Markdown + evidence packets
• validator enforces invariants and routes failures
• indexer chunks, embeds, and writes to the vector store

Add backpressure so you do not overload rendering capacity or hit rate limits.

Implement retention policies for audits

Retention is a compliance decision.

Typical approach:

• keep raw HTML/DOM snapshots only for a limited window or for pages that fail validation
• keep evidence packets longer (they are smaller than full DOM snapshots)
• redact or hash sensitive fields in provenance metadata if required

Observability: measure attribution health

Enterprise systems need dashboards.

Track metrics that reflect attribution quality, not just conversion success:

• provenance coverage rate
• link preservation rate
• validation pass rate
• retry rate by failure class (for example, unresolved relative URLs, low provenance confidence)

This lets you detect regressions when site templates change.

Deployment topology

A common deployment pattern:

• crawler service fetches URLs
• converter service converts and emits Markdown + provenance
• validator service enforces invariants
• indexer service chunks, embeds, and writes to vector store

15. FAQs

Q1: Do I need span-level provenance or is chunk-level enough?

A1: Chunk-level provenance is usually enough when your citation UX is section-level, where the user clicks a citation and lands on the right heading or region, especially if your chunker keeps semantic blocks intact and your provenance confidence is high. Span-level provenance becomes valuable when you need sentence-level or row-level verification: tables, where one row can contain multiple distinct facts; dense lists, where items are easy to mis-associate; and pages where a single sentence contains multiple links or multiple claims. A practical approach is to start with chunk-level provenance everywhere, then selectively upgrade to span-level only for high-value templates such as pricing, policies, legal, and technical specs where precision matters most.

Q2: How do I preserve links without bloating tokens?

A2: Preserve anchor text and the resolved destination together so the Markdown link is citeable, but do not blindly keep every navigation or footer link. Apply link-filtering rules that remove low-signal chrome such as repeated “Home/Contact/Next,” footer clusters, and cookie-related links, while keeping links that carry meaning inside the content, including references, “see also” links, and policy cross-references. To reduce token bloat further, deduplicate repeated links across sections and keep a metadata link map so the visible Markdown can stay compact while citations still reference the correct resolved URL and region provenance.

Q3: What’s the best way to validate link preservation?

A3: Validate link preservation with measurable checks that match your filtering policy. First, compare the number of preserved Markdown links to the number of detected anchors in the DOM region or regions you intended to extract, using a tolerance window rather than exact equality. Then validate link integrity at the object level: anchor text must be non-empty, resolved destinations must be absolute when absolute_urls is enabled, and every preserved Markdown link must have a corresponding entry in your link-map metadata. Finally, validate correspondence: if a chunk contains a Markdown link, the chunk’s provenance should reference the region and link object that produced it, preventing the common failure where links appear in text but provenance is missing or attached to the wrong region.

Q4: Can provenance survive chunking and re-ranking?

A4: Yes, as long as you treat provenance as part of the chunk identity rather than something you attach later. Attach provenance metadata to each chunk before embedding, and store a stable chunk ID, or derived key, in your vector store so retrieval and re-ranking can carry that ID forward. When the citation layer renders results, it should fetch the evidence packet or provenance for the winning chunk ID, not attempt to reconstruct provenance from the re-ranked text. If you transform the retrieved text again through summaries or rewritten answers, keep provenance anchored to the original chunk or evidence packet so citations remain verifiable and do not drift into post-processing ground truth.

Q5: How do I handle pages that change between fetch and render?

A5: Pages that change between fetch and render are a primary cause of attribution drift, so you need explicit timing and mode information in provenance. Store at least the fetch timestamp, extraction mode (static vs rendered), and the readiness-condition result, for example, “main content selector present and stable.” For highly dynamic pages, use a stricter readiness definition and a shorter stability window so you do not mix content from multiple states, such as content that loads and then updates after a late API call. If validation invariants fail because of low provenance confidence, missing regions, or unresolved links, retry using the same extraction policy but a different readiness strategy, or mark the page as failed if retries are not allowed by your rules.

Q6: Should I store raw HTML for audits?

A6: Storing raw HTML or a DOM snapshot is the strongest audit option because it lets you reproduce exactly what the extractor saw, but it can be expensive at crawl scale. A common enterprise compromise is retention tiering: keep raw HTML or DOM snapshots only for a limited window or only for pages that fail validation, while keeping evidence packets longer because they are smaller than full DOM. In all cases, store enough provenance fingerprints and selectors to reproduce extraction logic for debugging and to support audit workflows without permanently storing full DOM for every page. If compliance requires longer retention for certain categories, you can extend raw snapshot retention for those categories while still relying on fingerprints for the rest.

Q7: What about paywalled or blocked pages?

Treat paywalled and blocked pages as a distinct failure class, not as “extraction succeeded.” A login wall, consent gate, or error template can produce non-empty Markdown that is semantically useless, so your validation should fail via low provenance confidence and missing expected structure (e.g., missing main content region, missing headings, abnormal link density). Don’t embed partial content with low provenance confidence because it can poison retrieval and produce ungrounded citations. If your policy allows retries, retry with an allowed strategy (different rendering timeout, different extraction mode) while respecting robots and access constraints; otherwise, mark the page as failed and move on.

Q8: How do I represent images in attribution pipelines?

Images should be represented based on whether they carry meaning. If an image has meaningful alt text (diagrams, screenshots, charts), include a short caption or  in the Markdown and store image URL/provenance in the evidence packet so citations can still point back to the original page region. If images are decorative, omit them from the emitted Markdown to reduce noise and token cost, and also omit their provenance entries so your citation layer never claims evidence that isn’t present in the chunk text. This consistency rule—emit image text only when you also emit matching provenance—prevents citation mismatches.

Q9: Do tables require special handling for citations?

Yes—tables are high-risk because flattening changes meaning and makes it easy to mis-associate facts. For citations, you want a deterministic Markdown representation that preserves row/column semantics and provenance that maps each row (and sometimes each cell) back to the DOM origin. Convert tables into Markdown tables only when row/column alignment is stable; if alignment is unstable due to complex rowspan/colspan, use a structured fallback that preserves headers and row boundaries in a way your chunker can treat as atomic facts. Then attach provenance at the same granularity as your representation so citations remain precise and verifiable.

Q10: How do I prevent citation drift during retries?

Citation drift happens when retries produce different representations or different content states, so you need determinism and stable boundaries. Use deterministic conversion settings (same boilerplate removal policy, same link filtering rules, same URL normalization base) and stable selection rules for the main content region. Keep the same chunking policy so chunk boundaries don’t change between attempts. When you retry, compare validation metrics and provenance confidence, and only accept outputs that pass attribution invariants. For dynamic pages, tighten readiness/stability windows so you don’t mix states across retries, which is often the hidden cause of drift.

Q11: What if the model hallucinates a link that isn’t in the chunk?

Don’t treat this as an extraction problem—treat it as a grounding/UI enforcement problem. Your citation UI should only render verified citations for links that exist in the chunk’s link-map metadata. If the model proposes a link that isn’t present in metadata, treat it as ungrounded and exclude it from citations. To make this robust, ensure your link map is complete for the emitted Markdown and that your evidence packet includes the link objects referenced by the chunk, so grounding can be enforced mechanically rather than trusting the model’s output.

Q12: How do I choose thresholds for validation?

Start with conservative thresholds that protect against the most damaging failures, then tune using real evaluation focused on citation correctness. Good initial thresholds include minimum heading presence (for pages expected to have an h1), link count tolerance after filtering rules, provenance coverage rate (a minimum fraction of blocks/chunks with valid provenance), and unresolved relative URL checks when absolute_urls is enabled. Then tune using a labeled evaluation set where you measure “verified citation” accuracy (not just extraction quality) and adjust thresholds to balance correctness with retry rates. The goal is to maximize trustworthy citations while keeping the pipeline operationally efficient.

16. References

• HTML to Markdown for AI (structure and conversion overview): blog-html-to-markdown-for-ai.md
• Preserve semantic structure for LLMs: blog-html-to-markdown-preserve-semantic-structure-for-llms.md
• Remove boilerplate without losing meaning: blog-html-to-markdown-remove-boilerplate-for-rag.md
• Evidence packets and retry decision trees: blog-structured-extraction-mcp-evidence-retry-decision-tree.md

17. Conclusion

Preserving structure is the baseline because it keeps meaning intact for chunking, retrieval, and downstream reasoning. Without stable headings, lists, and tables, your pipeline may still “look readable,” but it will behave unpredictably—chunks split in the wrong places, retrieval returns mixed context, and citations become unreliable.

Preserving links and source attribution is what makes the system trustworthy. Links are not just decoration; they are the connective tissue between claims and their original references. Source attribution (provenance) is the mechanism that lets you map retrieved Markdown back to the exact page region and the exact link objects that produced the content. That’s what turns citations from “plausible” into “verifiable.”

When your HTML-to-Markdown pipeline emits deterministic Markdown plus provenance metadata, you gain three operational advantages: you can validate conversion quality before embedding, you can attach grounded citations to retrieved chunks during answer generation, and you can debug failures quickly using evidence packets instead of guesswork. This reduces both engineering time and user trust risk.

Finally, treat attribution as a contract—validated before embedding. If provenance confidence is low or attribution invariants fail, you retry or fail the page rather than silently ingesting corrupted evidence. That discipline is what turns “LLM answers” into “LLM answers you can verify,” which is the difference between a demo and a production-grade knowledge system.