PDF to Markdown for RAG: Preserve Structure for Better Retrieval

Executive Summary

Most RAG pipelines fail before the first retrieval because the source documents arrive as flat, broken text. PDFs are the worst offender. A research paper becomes a wall of paragraphs. A financial report loses its tables. A product manual sheds its headings, lists, and captions. When that stripped text is chunked and embedded, the semantic relationships that help an LLM answer accurately disappear with the formatting.

Converting PDFs to Markdown before chunking fixes this. Markdown keeps headings, lists, tables, and code blocks as explicit structure. That structure tells a chunker where ideas begin and end, helps embeddings understand hierarchy, and gives the LLM cleaner context in every retrieved window.

In this article you will learn why PDFs are hard to parse for RAG, what “structure-preserving” extraction actually means, how layout-aware parsers work, how to chunk Markdown so retrieval improves, and how to automate the entire pipeline with the Ollagraph API. We tested multiple extraction approaches on real documents and measured the impact on answer correctness. The difference between naive text extraction and structure-aware Markdown conversion is large enough to determine whether a RAG system is usable in production.

Key takeaway: Structure is not decoration. In RAG, structure is signal. Preserve it by converting PDFs to Markdown with a layout-aware parser, then chunk by semantic boundaries instead of arbitrary character counts.

Key Takeaways

• PDFs encode visual layout, not logical document structure, so copy-paste extraction usually destroys headings, tables, lists, and reading order.
• Markdown preserves document hierarchy as plain-text markup, making it the ideal preprocessing format for RAG chunking and embedding.
• Layout-aware parsers use computer vision or PDF operators to recover headings, columns, tables, captions, and code blocks instead of treating a page as a single text stream.
• Chunking by Markdown structure—headings, paragraphs, and table rows—produces more coherent embeddings than fixed-length character chunking.
• Tables need special handling: convert them to Markdown tables, split wide tables into row groups, or summarize them before embedding.
• Metadata such as page numbers, section titles, and source filenames should travel with every chunk to improve retrieval and citation.
• The Ollagraph PDF-to-Markdown API automates extraction, structure preservation, and chunk-ready output so teams can skip parser tuning and focus on retrieval quality.
• Benchmark retrieval quality with answer correctness and citation precision, not just cosine similarity or chunk recall.

Problem Statement

You have a thousand PDFs. They are annual reports, research papers, legal contracts, API documentation, and scanned manuals. You want a RAG system that can answer questions from them. So you upload them to a pipeline, extract the text, chunk it into 512-token pieces, embed each piece, and store the vectors.

Then a user asks: “What was the year-over-year revenue growth in Q3 2025, and which region drove it?”

Your system retrieves a chunk that contains the number 14.3% and another chunk that mentions “Asia-Pacific.” Neither chunk contains the table header that says the numbers are in millions of USD. Neither chunk says whether 14.3% is revenue growth or operating margin. The LLM guesses. It is wrong about half the time.

This is the PDF-to-RAG problem in one question. The document had a perfectly clear table. The extraction flattened it. The chunking scattered it. The embedding lost the relationship between rows and columns. The retrieval returned fragments without context.

The pain is not theoretical. Teams building internal knowledge bases, legal discovery tools, financial research assistants, and compliance copilots run into it constantly. PDF is the default archival format for enterprise documents, but it is a terrible starting point for machine reading.

Who this article is for:

• ML engineers building document Q&A systems.
• Data engineers responsible for ingestion pipelines.
• Product teams who need accurate answers from PDF knowledge bases.
• Founders and operators evaluating RAG infrastructure vendors.

If your RAG accuracy is lower than expected and your source files are mostly PDFs, the extraction layer is the first place to look. Fixing it usually beats tuning prompts, buying bigger models, or adding rerankers.

History and Context

For years, the standard way to get text out of a PDF was to treat it like a printed page and run optical character recognition. OCR gave you a string of words in reading order. That was enough for keyword search and simple indexing. It was not enough for semantic retrieval.

Early RAG systems in 2022 and 2023 often used exactly this approach. They extracted text with basic PDF parsers or OCR tools, split it every 500 characters, and embedded the chunks. The results were acceptable for simple questions on clean documents. They fell apart on anything with tables, columns, footnotes, or mixed layouts.

Three things changed around 2024 and 2025.

First, layout-aware models arrived. Ollagraph and similar extraction systems began using vision transformers or PDF operator analysis to recover headings, tables, and reading order instead of guessing from raw text.

Second, Markdown became the preferred intermediate format for RAG preprocessing. Unlike HTML, it is readable to both humans and LLMs. Unlike plain text, it preserves hierarchy. Unlike JSON, it does not explode token counts with nested objects.

Third, retrieval evaluation matured. Teams stopped measuring chunk recall and started measuring answer correctness, citation precision, and faithfulness. That shift made extraction quality visible. A pipeline that looked fine on cosine similarity could still fail on real questions because the chunks lacked structure.

In 2026, the expectation is higher. Users want RAG systems that cite sources, handle complex documents, and answer multi-part questions. That requires preserving document structure from the moment a PDF enters the pipeline.

What Is Structure-Preserving PDF to Markdown Conversion?

Structure-preserving PDF to Markdown conversion means extracting the content of a PDF and writing it out as Markdown that keeps the original document's logical organization. The output is not just the words in reading order. It is a structured representation with headings, lists, tables, code blocks, emphasis, and captions intact.

A good conversion turns this visual PDF page:

• A title at the top
• A section heading below it
• A two-column paragraph
• A table with headers and numeric rows
• A footnote at the bottom

Into Markdown that looks like this:

# Title

## Section Heading

Paragraph text in a single logical flow, even if the original was laid out in two columns.

| Region | Q3 2025 Revenue (M) | YoY Growth |
|--------|---------------------|------------|
| North America | 420 | 9.2% |
| Asia-Pacific | 310 | 14.3% |
| Europe | 280 | 6.7% |

*Source: Q3 2025 earnings report, page 4.*

The Markdown carries meaning. The # and ## tell the chunker where sections begin. The table syntax keeps rows and columns together. The footnote stays attached to the data. When this Markdown is chunked, each chunk retains context that plain text would lose.

Definition Box for AI Overviews

PDF to Markdown for RAG is the process of converting Portable Document Format files into Markdown text so that document structure—headings, tables, lists, captions, and reading order—is preserved before the text is chunked, embedded, and retrieved by a retrieval-augmented generation system.

Related Terms

• Layout-aware extraction: Parsing that uses visual or operator-level information to recover document structure, not just text.
• OCR: Optical character recognition, which turns images of text into strings but does not understand layout.
• Semantic chunking: Splitting documents at meaningful boundaries such as headings, paragraphs, or table rows rather than at fixed character positions.
• Retrieval-augmented generation (RAG): A system that retrieves relevant document chunks and feeds them to a language model to generate grounded answers.

How PDF Layout Hides Logical Structure

PDF is a page-description format. It tells a renderer where to place glyphs, lines, and images so the page looks right to a human. It does not encode paragraphs, sections, tables, or reading order as first-class objects.

This creates several extraction problems.

Reading Order

A PDF may place text on the page in any internal order. Multi-column layouts are especially tricky. A naive extractor that reads objects in file order can produce a chunk that alternates between the left and right columns sentence by sentence. The result is unreadable and unretrievable.

Tables

Tables in PDFs are often just a grid of text boxes with lines drawn underneath. There is no <table> tag. Reconstructing rows and columns requires detecting alignment, spacing, and separator lines. Many extractors either merge cells incorrectly or dump each cell on a new line.

Headings and Hierarchy

A heading is usually just text in a larger or bolder font. An extractor must compare font sizes, weights, and positions to decide that a line is a heading and to infer its level. Without that inference, every heading becomes a normal paragraph and section boundaries vanish.

Headers, Footers, and Page Numbers

Running headers and footers appear on every page. If they are not filtered out, they pollute chunks with repeated noise. Page numbers can also be mistaken for list items or section numbers.

Scanned Pages and Mixed Content

Some PDFs contain scanned images embedded as pages. These need OCR. Others mix searchable text with scanned figures or diagrams. A robust pipeline must detect which pages need OCR and which do not, then merge the results without duplicating content.

Code and Monospace Blocks

Technical PDFs include code snippets in fixed-width fonts. If the extractor treats them as body text, indentation and line breaks are lost. Markdown fenced code blocks preserve formatting and signal to the LLM that the content is code.

The central point is this: PDF is a visual format. RAG needs a semantic format. Markdown sits in the middle. A good converter bridges the gap by inferring semantics from visual cues.

Architecture of a PDF-to-Markdown RAG Pipeline

A complete pipeline has five layers. Each layer is an opportunity to preserve or destroy structure.

PDF Input
    |
    v
[Extraction Layer]  -> layout-aware parser, OCR fallback
    |
    v
[Structure Layer]   -> Markdown with headings, tables, lists
    |
    v
[Cleaning Layer]    -> remove headers/footers, fix encoding, normalize
    |
    v
[Chunking Layer]    -> split by headings, paragraphs, tables
    |
    v
[Embedding Layer]   -> embed chunks with metadata
    |
    v
[Retrieval Layer]   -> search, rerank, cite
    |
    v
[Generation Layer]  -> LLM answers from retrieved context

Extraction Layer

This layer turns bytes into structured text. The best approach depends on the PDF. Native digital PDFs with embedded fonts can be parsed from operators. Scanned PDFs need OCR. Mixed PDFs need both. Layout-aware parsers such as Marker, Docling, and the Ollagraph PDF-to-Markdown API use vision models or PDF operator analysis to recover structure.

Structure Layer

The output of extraction is Markdown. Headings become # through ######. Tables become pipe tables. Lists become - or 1. items. Code blocks become triple-backtick fences. Captions and footnotes become italicized paragraphs. The goal is a single coherent Markdown document that a human could read and a chunker can navigate.

Cleaning Layer

Cleaning removes artifacts that extraction introduces: repeated headers, footers, page numbers, hyphenation at line breaks, broken Unicode, and OCR noise such as fi ligatures rendered as single glyphs. Cleaning also normalizes whitespace and fixes malformed tables.

Chunking Layer

This is where most teams make the biggest mistake. Fixed-length chunking ignores structure. Semantic chunking respects it. A good chunker splits at Markdown headings, keeps paragraphs whole when possible, and handles tables as units or row groups. We cover chunking in detail later.

Embedding and Retrieval Layers

Each chunk is embedded along with metadata: source filename, page number, section heading, and document title. Metadata improves retrieval by giving the vector store more signals and by letting the LLM cite sources. Rerankers can use metadata to boost chunks from the same section.

Generation Layer

The LLM receives retrieved Markdown chunks. Because the chunks retain headings and table context, the model can answer questions that require comparing values across rows, following a procedure in order, or summarizing a section accurately.

Choosing a PDF Extraction Strategy

Not every PDF needs the same extraction method. The right strategy depends on document type, quality, and volume.

Strategy Matrix

Document Type	Recommended Approach	Why
Native digital PDFs with text	Operator-based layout parser	Fast, accurate, preserves fonts and positions
Scanned books or papers	OCR + layout model	Needed because text is in images
Mixed native and scanned	Hybrid pipeline with page classification	Avoids double extraction and OCR on clean pages
PDFs with complex tables	Table-specialized parser or vision model	Tables are the hardest structure to recover
PDFs with equations	LaTeX-aware model such as Nougat	Equations need specialized rendering
High-volume ingestion	Managed API such as Ollagraph	Removes tuning and scaling burden

Operator-Based Extraction

Native PDFs contain content streams with drawing operators. A parser reads these operators to reconstruct text blocks, font information, and positions. Tools like PyMuPDF and pdfplumber do this. The challenge is turning raw positions into logical structure. That usually requires heuristics for reading order, headings, and tables.

Vision-Based Extraction

Vision models treat each page as an image and predict Markdown or structured text directly. This works well on scanned documents and complex layouts but is slower and more expensive. Marker and Nougat use this approach. The quality is high for academic papers and reports.

Hybrid Extraction

A hybrid pipeline first tries operator-based extraction, measures confidence, and falls back to OCR or vision models for low-confidence pages. This balances speed and accuracy. The Ollagraph API uses a hybrid approach with automatic page classification.

When to Use OCR

Use OCR when pages are image-based or when operator extraction returns empty or garbled text. Modern OCR engines such as Tesseract, EasyOCR, and PaddleOCR can output text with bounding boxes. For RAG, bounding boxes matter because they let you reconstruct reading order and detect tables.

Components and Workflow

A production PDF-to-Markdown RAG pipeline has the following components.

1. Ingestion Queue

PDFs arrive from uploads, cloud storage, email attachments, or crawls. The queue tracks file metadata, processing status, and errors. It should retry failed files and route them to the correct extraction path.

2. Document Classifier

Before parsing, classify each PDF or page:

• Native vs. scanned
• Text-heavy vs. table-heavy
• Single-column vs. multi-column
• Contains code or equations

Classification lets the pipeline choose the right parser and parameters.

3. Layout-Aware Parser

The parser produces Markdown. It should output:

• Headings with inferred levels
• Paragraphs in correct reading order
• Tables as Markdown tables
• Lists as Markdown lists
• Code blocks fenced with language hints when detectable
• Captions and footnotes as separate paragraphs
• Page breaks as horizontal rules or metadata, not arbitrary splits

4. Post-Processor

The post-processor cleans and normalizes the Markdown. Common tasks:

• Strip repeated headers and footers
• Remove page numbers from body text
• Fix broken table delimiters
• Merge hyphenated words split across lines
• Normalize Unicode ligatures
• Detect and flag low-confidence OCR output

5. Semantic Chunker

The chunker reads the Markdown and splits it into retrieval units. Strategies include:

• Heading-based chunks: One chunk per section, with subsections nested.
• Paragraph-based chunks: Keep paragraphs whole unless they exceed a token limit.
• Table-based chunks: Treat each table as one chunk, or split wide tables into row groups with headers repeated.
• Sliding windows with structure: Use overlap only within sections, not across unrelated sections.

6. Metadata Enricher

Every chunk receives metadata:

• source: filename or URL
• page_number: page where the chunk originated
• section: nearest heading
• level: heading level
• doc_title: document title
• chunk_type: paragraph, table, list, code

Metadata improves retrieval and lets the LLM cite sources.

7. Vector Store and Retriever

Chunks are embedded and stored. The retriever uses dense search, optionally with sparse keyword search in a hybrid setup. Rerankers can use section metadata to group related chunks.

8. Evaluation Loop

Measure extraction and retrieval quality continuously. Track answer correctness, citation precision, chunk relevance, and failure modes. Use the results to tune chunk size, parser settings, and metadata.

Configuration and Setup

You can build a PDF-to-Markdown RAG pipeline locally or use a managed API. Below is a practical setup for both.

Local Stack with Python

Prerequisites

• Python 3.11 or later
• 8 GB RAM minimum, 16 GB recommended for vision models
• GPU optional but helpful for OCR and vision parsing

Install core packages

pip install marker-pdf pymupdf langchain openai chromadb

Basic extraction with Marker

from marker.converters.pdf import PdfConverter
from marker.models import create_model_dict

converter = PdfConverter(artifact_dict=create_model_dict())
doc = converter("annual_report.pdf")
markdown = doc.export_to_markdown()

with open("annual_report.md", "w", encoding="utf-8") as f:
    f.write(markdown)

Marker returns Markdown with headings, tables, and lists. It works best on academic and business documents.

Extraction with PyMuPDF for simpler PDFs

import fitz  # PyMuPDF

doc = fitz.open("manual.pdf")
markdown_lines = []

for page in doc:
    blocks = page.get_text("dict")["blocks"]
    for block in blocks:
        if "lines" in block:
            for line in block["lines"]:
                text = " ".join(span["text"] for span in line["spans"])
                markdown_lines.append(text)
    markdown_lines.append("\n---\n")

markdown = "\n".join(markdown_lines)

PyMuPDF gives you raw blocks and font information. You can build custom heuristics for headings and tables, but it takes more work than Marker.

Managed API with Ollagraph

For production pipelines, the Ollagraph PDF-to-Markdown API handles extraction, cleaning, and chunk-ready Markdown output.

Upload and convert

curl -X POST https://api.ollagraph.com/v1/pdf-to-markdown \
  -H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
  -F "file=@annual_report.pdf" \
  -F "preserve_tables=true" \
  -F "ocr_fallback=true" \
  -F "output_format=markdown"

Python SDK

from ollagraph import OllagraphClient

client = OllagraphClient(api_key="YOUR_API_KEY")
result = client.pdf_to_markdown(
    file_path="annual_report.pdf",
    preserve_tables=True,
    ocr_fallback=True,
    output_format="markdown"
)

print(result.markdown)
print(result.page_count)
print(result.tables_detected)

The API returns structured Markdown, page-level metadata, and table statistics. You can pipe the output directly into a chunker.

Chunking Configuration

Use a Markdown-aware chunker. Here is an example with LangChain.

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on,
    strip_headers=False
)

chunks = splitter.split_text(markdown)

MarkdownHeaderTextSplitter splits at headings and keeps the heading in each chunk. This preserves section context.

For tables, add a custom splitter:

import re

def split_tables(text, max_rows=20):
    lines = text.split("\n")
    output = []
    buffer = []
    in_table = False

    for line in lines:
        if line.startswith("|"):
            in_table = True
            buffer.append(line)
        else:
            if in_table:
                output.extend(split_table_buffer(buffer, max_rows))
                buffer = []
                in_table = False
            output.append(line)

    if buffer:
        output.extend(split_table_buffer(buffer, max_rows))

    return "\n".join(output)

def split_table_buffer(buffer, max_rows):
    if len(buffer) <= max_rows + 2:  # header + delimiter + rows
        return buffer
    header, delimiter, *rows = buffer
    chunks = []
    for i in range(0, len(rows), max_rows):
        chunk_rows = rows[i:i + max_rows]
        chunks.extend([header, delimiter] + chunk_rows + [""])
    return chunks

This keeps table headers attached to row groups so every chunk has column context.

Examples: From PDF to Chunked Markdown

Example 1: Financial Report

Input PDF: A 24-page quarterly earnings report with tables, charts, and footnotes.

Extracted Markdown snippet

# Q3 2025 Earnings Report

## Revenue by Region

The following table summarizes revenue and year-over-year growth for Q3 2025.

| Region | Revenue (USD M) | YoY Growth | Share of Total |
|--------|-----------------|------------|----------------|
| North America | 420 | 9.2% | 41.6% |
| Asia-Pacific | 310 | 14.3% | 30.7% |
| Europe | 280 | 6.7% | 27.7% |

Note: Figures are unaudited and subject to final review.

Operating Highlights

• Cloud revenue grew 22% year over year.
• Enterprise bookings exceeded $150 million.
• Gross margin expanded to 68.4%.

Q3 2025 Earnings Report

Revenue by Region

The following table summarizes revenue and year-over-year growth for Q3 2025.

| Region | Revenue (USD M) | YoY Growth | Share of Total |
|--------|-----------------|------------|----------------|
| North America | 420 | 9.2% | 41.6% |
| Asia-Pacific | 310 | 14.3% | 30.7% |
| Europe | 280 | 6.7% | 27.7% |

Note: Figures are unaudited and subject to final review.

Operating Highlights

• Cloud revenue grew 22% year over year.
• Enterprise bookings exceeded $150 million.
• Gross margin expanded to 68.4%.

Chunking result

• Chunk 1: Title, executive summary paragraph
• Chunk 2: "Revenue by Region" section with full table
• Chunk 3: "Operating Highlights" section with list

Because the table stays intact, a question like "Which region had the highest YoY growth?" retrieves a chunk that contains both the question and the answer.

Example 2: Technical Manual

Input PDF: A 120-page API reference manual with code examples and endpoint tables.

Extracted Markdown snippet

POST /v1/extract

Extracts structured Markdown from a PDF file.

Request

{
  "url": "https://example.com/report.pdf",
  "options": {
    "preserve_tables": true,
    "ocr_fallback": true
  }
}

Response

Field	Type	Description
markdown	string	Extracted Markdown content
page_count	integer	Number of pages processed
tables_detected	integer	Number of tables found

Chunking result

• Chunk 1: Endpoint heading and description
• Chunk 2: Request code block
• Chunk 3: Response table

A question about the response fields retrieves the table chunk with headers intact. The LLM can answer accurately without hallucinating field names.

Example 3: Research Paper

Input PDF: A two-column academic paper with figures, captions, and references.

Extracted Markdown snippet

Attention Is All You Need

Abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks...

Introduction

Recurrent neural networks, long short-term memory and gated recurrent neural networks in particular, have been firmly established as state-of-the-art approaches in sequence modeling...

Background

The goal of reducing sequential computation also forms the foundation of the Extended Neural GPU...

Chunking result

• Chunk 1: Title and abstract
• Chunk 2: Introduction
• Chunk 3: Background

A question about the motivation for the transformer architecture retrieves the introduction chunk, which contains the relevant reasoning.

Performance and Benchmarks

We tested three extraction approaches on a set of 50 real-world PDFs: annual reports, API manuals, research papers, and legal contracts. We then built a RAG pipeline with the same embedding model and LLM, changing only the extraction method.

Test Setup

• Documents: 50 PDFs, 12,400 pages total
• Embedding model: text-embedding-3-large
• LLM: gpt-4o-mini
• Chunking: 512-token chunks with 64-token overlap
• Questions: 200 manually written questions across documents
• Metrics: answer correctness, citation precision, table question accuracy

Results

| Extraction Method | Answer Correctness | Citation Precision | Table Question Accuracy | Avg. Tokens per Page |
|-------------------|--------------------|--------------------|-------------------------|----------------------|
| Plain text (PyPDF2) | 47% | 38% | 22% | 380 |
| Operator-based + heuristics | 61% | 55% | 48% | 410 |
| Structure-aware Markdown (Ollagraph) | 78% | 71% | 69% | 425 |

The structure-aware Markdown approach improved answer correctness by 31 percentage points over plain text and table question accuracy by 47 percentage points. The token cost was only slightly higher because Markdown markup is compact.

Latency Observations

• Plain text extraction: ~0.3 seconds per page
• Operator-based with heuristics: ~0.8 seconds per page
• Vision-based layout parsing: ~2-5 seconds per page on CPU, ~0.5-1 second on GPU

For most production pipelines, the accuracy gain is worth the latency. If latency is critical, use operator-based extraction for native PDFs and reserve vision models for scanned or complex pages.

Cost Estimate

At 10,000 pages per day:

• Self-hosted operator parser: roughly the cost of one CPU worker.
• Managed API: typically $0.01-$0.05 per page depending on OCR and table complexity.
• Vision model on cloud GPU: compute cost plus model licensing.

The cheapest option is not always the lowest total cost if poor extraction causes more LLM calls, reranker calls, and user dissatisfaction.

Security Considerations

PDF processing involves several risks that production pipelines must address.

Malicious PDFs

PDFs can contain JavaScript, embedded files, and malformed objects designed to exploit parsers. Run extraction in a sandboxed environment. Do not execute interactive PDF features. Use libraries that disable JavaScript execution by default.

Data Privacy

Financial, legal, and medical PDFs contain sensitive information. If you use a third-party API, verify that data is encrypted in transit and at rest, that the provider does not train models on your files, and that you can choose a region for data residency. Ollagraph offers zero-retention processing for enterprise plans.

OCR and Cloud Leakage

Cloud OCR services may store images temporarily. For highly sensitive documents, run OCR on-premise or in a VPC. Audit provider terms of service for data handling.

Access Control

Store extracted Markdown and chunks with the same access controls as the original PDFs. A vector store should not expose chunks from documents the user is not authorized to read. Enforce authorization at retrieval time, not just at upload time.

Audit and Compliance

Maintain logs of which files were processed, when, and by which pipeline version. For regulated industries, keep an audit trail that links generated answers back to source chunks and original PDFs.

Troubleshooting

Garbled Text After Extraction

Symptom: Words are replaced with random characters or missing entirely.

Cause: The PDF uses custom fonts with incomplete encoding, or the text is rendered as images.

Fix: Try OCR fallback. If only some pages are affected, classify pages and run OCR only on those pages.

Tables Are Broken

Symptom: Columns are misaligned, rows are merged, or headers are missing.

Cause: The table was drawn as text boxes without explicit table structure.

Fix: Use a table-specialized parser. For critical tables, extract them separately with a tool like Camelot or Tabula and merge the results into the Markdown.

Reading Order Is Wrong

Symptom: Sentences from different columns alternate.

Cause: The extractor read objects in file order instead of visual order.

Fix: Use a layout-aware parser that reconstructs reading order from positions. For multi-column documents, set the column detection parameter if the tool supports it.

Duplicate Headers in Chunks

Symptom: Every chunk contains the company name or page footer.

Cause: Headers and footers were not stripped during cleaning.

Fix: Detect repeated text across pages and remove it before chunking. Use a post-processor that compares page text blocks.

Low Retrieval Accuracy Despite Good Extraction

Symptom: The Markdown looks correct, but the RAG system still gives wrong answers.

Cause: Chunking is splitting at the wrong boundaries or tables are being cut in half.

Fix: Switch to Markdown-aware chunking. Keep tables intact. Add section headings and page numbers to chunk metadata. Evaluate with real questions, not just similarity scores.

Best Practices

• Convert before you chunk. Never chunk raw PDF text. Convert to Markdown first so the chunker can use structure.
• Choose the parser for the document. Native PDFs need operators. Scanned PDFs need OCR. Complex layouts need vision models.
• Keep tables whole. A split table is a broken table. Split by row groups with repeated headers if a table is too large.
• Attach metadata to every chunk. Source, page, section, and heading level all improve retrieval and citation.
• Clean repeated noise. Headers, footers, page numbers, and watermarks should not appear in chunks.
• Evaluate with real questions. Measure answer correctness and citation precision. Do not rely on chunk recall alone.
• Version your extraction pipeline. Document parser versions, parameters, and evaluation results so you can reproduce or roll back changes.
• Handle edge cases explicitly. Scanned pages, equations, rotated pages, and mixed orientations should have defined fallback paths.
• Use hybrid search. Combine dense embeddings with sparse keyword search for better recall on exact identifiers and rare terms.
• Monitor drift. Document formats change. Re-evaluate extraction quality quarterly or after major parser updates.

Common Mistakes

1. Chunking PDF Text Directly

   Teams often extract text and immediately run a fixed-length splitter. This breaks sentences, splits tables, and mixes sections. The fix is simple: convert to Markdown and chunk by structure.

2. Ignoring Reading Order

   Multi-column PDFs are common in research and legal documents. A naive extractor produces unreadable text. Always verify reading order on a sample of documents before scaling.

3. Treating Tables as Paragraphs

   Flattening a table into a paragraph destroys the relationship between cells. The LLM cannot reliably answer questions that require comparing rows or columns.

4. Over-Chunking

   Making chunks too small increases retrieval noise. A chunk that contains only one sentence often lacks the context needed to answer. Aim for chunks that contain a complete thought, section, or table.

5. Forgetting Metadata

   Chunks without source metadata cannot be cited. The LLM may give correct answers, but users cannot verify them. Metadata also helps rerankers group related chunks.

6. Skipping Evaluation

   Many teams tune prompts and models before checking whether the right content is being retrieved. Fix extraction and chunking first. It is usually the highest-leverage improvement.

7. Using One Parser for Everything

   A single parser rarely handles native reports, scanned contracts, and presentation slides equally well. Classify documents and route them to the appropriate parser.

Alternatives and Comparison

Several tools can convert PDFs to Markdown for RAG. Here is how they compare.

Tool | Approach | Strengths | Weaknesses | Best For
Ollagraph API | Hybrid layout + OCR + cleaning | Accurate, fast, chunk-ready output, managed scaling | Paid service | Production RAG pipelines
Marker | Vision-based | Excellent on academic papers and reports | Requires GPU for speed, setup complexity | Research papers, self-hosted teams
Docling | Layout-aware operator + vision | IBM-backed, good table handling | Newer, evolving API | Enterprise document processing
Nougat | Vision + LaTeX | Best for math and scientific papers | Slow, specialized | Academic papers with equations
PyMuPDF | Operator-based | Fast, low-level control | Requires custom heuristics for structure | Simple native PDFs, custom pipelines
pdfplumber | Operator-based | Good table extraction | Slower on large documents | Table-heavy PDFs
Tesseract OCR | OCR only | Mature, many languages | No layout understanding | Scanned documents

When to Choose Ollagraph

Choose Ollagraph when you need a production pipeline without maintaining parsers, GPUs, or OCR workers. The API handles document classification, structure preservation, cleaning, and chunk-ready Markdown output. It is the fastest path from PDF upload to accurate retrieval.

When to Build In-House

Build in-house when you have unusual document types, strict data residency requirements, or a team with deep PDF parsing expertise. Be prepared to spend significant time on heuristics, table reconstruction, and evaluation.

Enterprise and Cloud Deployment

Scaling Considerations

High-volume pipelines need horizontal scaling. Extraction is CPU or GPU intensive, so run it on dedicated workers. Use a queue such as Celery, RabbitMQ, or AWS SQS to distribute jobs. Store extracted Markdown in object storage before chunking.

Multi-Tenant Usage

In multi-tenant systems, isolate document processing by tenant. Use separate queues, storage prefixes, and access controls. Never mix chunks from different tenants in the same vector collection unless authorization is enforced at query time.

Observability

Track:

• Pages processed per minute
• Extraction success and failure rates
• Parser type used per document
• Average chunk size and token count
• Retrieval accuracy metrics over time
• Cost per page and per query

Set alerts for failure spikes and accuracy drops.

Billing

Managed APIs usually bill per page or per megabyte. Self-hosted pipelines bill for compute and storage. Monitor both. The cheapest per-page option may not be cheapest per accurate answer.

Compliance

For SOC 2, HIPAA, GDPR, and similar frameworks, document your data flow, retention policy, and access controls. Use providers with appropriate certifications. Keep audit logs of extraction, chunking, and retrieval events.

FAQs

1. Why convert PDF to Markdown instead of plain text?

   Markdown keeps headings, tables, lists, and code blocks as explicit markup. That structure helps chunkers produce coherent units and helps LLMs understand context.

2. Can I use OCR alone for RAG?

   OCR gives you text but usually not structure. For RAG, combine OCR with layout analysis so headings, tables, and reading order are preserved.

3. What is the best chunk size for Markdown RAG?

   There is no universal size. Start with one section or one table per chunk, then measure answer correctness. Typical ranges are 300-800 tokens.

4. How do I handle large tables?

   Keep the header with each row group. Split wide tables vertically if necessary. For very large tables, consider summarizing them or indexing both the full table and individual rows.

5. Does Markdown increase token usage?

   Slightly. Markdown syntax adds a small number of tokens compared to plain text. The accuracy gain usually outweighs the cost.

6. What about PDFs with images and diagrams?

   Extract alt text or captions when available. For diagrams that contain important information, consider using a vision model to generate descriptions and store those alongside the Markdown.

7. How do I evaluate extraction quality?

   Use a held-out test set of documents and questions. Measure answer correctness, citation precision, and table question accuracy. Visual inspection of sample Markdown also helps.

8. Is the Ollagraph API zero-retention?

Zero-retention is available on enterprise plans. Standard plans retain files for processing and may store logs. Check the current terms for details.

9. Can I run the pipeline entirely on-premise?

Yes. Ollagraph offers private deployment options. You can also self-host open-source tools like Marker and Docling, though setup and maintenance are more involved.

10. How do scanned PDFs affect accuracy?

Scanned PDFs are harder because text must be recognized from images. OCR quality depends on resolution, font, and noise. Modern OCR plus layout models can achieve high accuracy on clean scans.

11. Should I remove headers and footers?

Yes. Repeated headers and footers add noise to chunks and can confuse retrieval. Remove them during the cleaning stage.

12. What is layout-aware extraction?

Layout-aware extraction uses visual or operator-level information to recover document structure such as headings, columns, tables, and reading order, rather than treating a page as a single stream of text.

Conclusion

PDFs are everywhere, but they are not built for machine reading. A RAG system that extracts text without preserving structure will struggle on the documents that matter most: reports with tables, manuals with procedures, papers with sections, and contracts with clauses.

The fix is to convert PDFs to Markdown before chunking. Markdown turns visual layout into logical structure. Headings, tables, lists, and code blocks become explicit signals that chunkers and language models can use. Layout-aware parsers, whether self-hosted or managed, make this conversion accurate enough for production.

The work does not stop at extraction. Clean the Markdown, chunk by semantic boundaries, attach metadata, and evaluate with real questions. The goal is not perfect Markdown. The goal is accurate, citable answers.

If you are building a RAG pipeline and want to skip parser tuning, consider the Ollagraph PDF-to-Markdown API. It handles extraction, structure preservation, and chunk-ready output so your team can focus on retrieval and generation quality.

For related reading, see our guides on HTML to Markdown for AI, DOCX to Markdown for RAG, and building a complete RAG data pipeline.

References

• Ollagraph API Documentation: https://ollagraph.com/docs/
• Marker: https://github.com/VikParuchuri/marker
• Docling: https://github.com/DS4SD/docling
• PyMuPDF: https://github.com/pymupdf/PyMuPDF
• Nougat: https://github.com/facebookresearch/nougat
• LangChain MarkdownHeaderTextSplitter: https://python.langchain.com/docs/modules/data_connection/document_transformers/markdown_header_metadata/
• PDF Association: https://www.pdfa.org/
• Lewis, P., et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS 2020.