RAG Chunking for PDFs With Tables: An Ingestion Strategy That Holds Up

A user asks your assistant for the annual limit on a particular plan in last year's benefits schedule. The PDF is in the index, the answer is in a table on page 14, and the assistant still gets it wrong. When you open the retrieved chunks, the cause is obvious: the table was read line by line into a stream of numbers, the chunker cut it in half, and the header row that said which column was which ended up in a different chunk.

Retrieval cannot recover what ingestion broke. Better embeddings and hybrid search with reranking help only once the right content exists as a coherent chunk. This guide stays on ingestion: how to parse PDFs with tables, headers and multi-page layouts, how to chunk them, what metadata to attach, and how to test the result on your own questions.

Why do PDFs with tables break naive chunking?

A PDF stores positioned text and drawing instructions, not paragraphs or tables. Plain text extraction reconstructs reading order with heuristics, and a fixed-size splitter then cuts that text at arbitrary character counts. Together they produce a familiar set of symptoms.

• Numbers retrieved without their row or column labels, so the model guesses what they mean.
• Page headers and footers repeated in every chunk, crowding out the content that matters.
• A table that continues over two pages becomes two unrelated fragments, the second with no header.
• Two-column layouts read straight across, mixing sentences from both columns.
• Section headings separated from their text, so a question that names the section cannot find it.
• Footnotes and table notes attached to whichever paragraph happened to follow them.

These are some of the most common causes of the problems in RAG in production: six anti-patterns we keep finding, and they are invisible in a demo that only asks questions answered by prose.

What is the best chunking strategy for PDFs with tables in RAG?

Start with a layout-aware parser that labels titles, paragraphs, tables and page headers, not plain text extraction. Chunk along that structure: a new chunk at each section heading, filled up to a token limit, never spanning two sections. Keep each table in its own chunk with its caption and section heading; split large tables by rows and repeat the header row in every part. Index a plain-language version of each table for retrieval, keep the original for answering, attach page and section metadata, and test pipelines on your questions.

Step 1: Use layout-aware extraction

A layout-aware parser detects the elements on each page, such as titles, paragraphs, list items, tables, figures, page headers and footers, and returns them with a type, a position and often a hierarchy. Several tools do this, open source and managed. Described from their own documentation:

• Unstructured (open source): its PDF partitioner offers several strategies, and the hi_res strategy is the only one that currently extracts tables from PDFs. Elements carry metadata such as page number and parent element, and tables can include an HTML representation in a text_as_html field.
• Docling (open source, MIT-licensed): its technical report describes specialised models for layout analysis and for table structure recognition (TableFormer), and the library converts documents into a structured document object that its chunkers work from.
• Azure Document Intelligence (managed): the layout model assigns paragraph roles such as title, sectionHeading, pageHeader and pageFooter, and can return Markdown. In version 4.0 it represents tables in that Markdown as HTML so merged cells and multi-row headers survive.
• Amazon Textract (managed): table analysis returns cells, merged cells, column headers, table titles and footers, each with a confidence score.

Don't choose on a sample PDF from the vendor. Run each candidate on a set of your own hardest documents: scanned pages, rotated tables, multi-page schedules, footnotes. Check whether scanned pages go through OCR, how low confidence is signalled, and whether page headers and footers come back labelled so you can drop or tag them instead of writing fragile patterns to strip them.

Step 2: Chunk by structure first, then by size

With typed elements in hand, chunk along the document's own structure. Start a new chunk at each section heading, add paragraphs until you reach a token limit, and never let a chunk span two sections. Unstructured's by_title chunking works this way and always keeps a table element on its own. Docling's HybridChunker starts from the document hierarchy, splits only chunks that exceed a token limit set with a tokenizer matched to your embedding model, and merges small neighbouring chunks that share the same headings.

There is some published evidence for structure first. Jimeno Yepes and colleagues (2024) chunked financial reports by element type, such as titles and tables, rather than by paragraph, and reported that this largely improved RAG results on financial reporting. On the other side, Qu, Tu and Bao (Findings of NAACL 2025) found the extra computation of semantic chunking was not justified by consistent gains over fixed-size chunking. The practical reading: structure from a good parser tends to be worth more than clever splitting on embeddings, but neither result transfers automatically to your documents, so measure.

Step 3: Keep tables whole, or split them by rows

• Small and medium tables: one chunk per table, together with its caption or title and the heading of the section it sits in.
• Large tables: split by rows, never through a row, and repeat the header row at the top of every part. Docling's HybridChunker repeats table headers by default for exactly this reason.
• Tables continued across pages: detect the continuation (same columns, no new title) and join the parts before chunking, so the second page inherits the header.
• Merged cells and multi-row headers: keep the structure as HTML or Markdown rather than flattening it, because flattening loses which header governs which cell.
• Table notes and footnotes: keep them in the same chunk as the table they qualify.

Should you convert tables to text?

Often, yes, but alongside the table rather than instead of it. A grid of numbers in Markdown tends to match poorly against a question written in words. A common pattern stores two representations: a plain-language version used for retrieval, and the original table passed to the model when it answers.

• Row sentences: for lookup tables, turn each row into a sentence that repeats the headers, such as the plan name, the benefit and its limit. This is deterministic and easy to check.
• Table summaries: for complex tables, an LLM can write a short description of what the table covers. Summaries can introduce errors, so use them only to find the table, never as the text the answer quotes.
• Calculations: if users ask for totals, comparisons or trends across a table, retrieval is the wrong tool. Load the table into a structured store and let the system query it, then cite the source table.

What metadata should every chunk carry?

• Document identity: ID, title, version and effective date, so outdated editions can be filtered out.
• Page range: start and end page, for citations and for opening the source at the right place.
• Section path: the chain of headings above the chunk, from the document title down to the subsection.
• Element type: paragraph, list, table, or part of a table, with a table ID and part number for split tables.
• Access labels inherited from the source document, so per-user access control can be enforced at retrieval time.
• Pipeline version: the parser, chunker and settings that produced the chunk, so you can re-ingest and compare.

Consider adding the section path and table caption to the text you embed, while storing the raw chunk separately for display. Docling's contextualize method produces this kind of metadata-enriched text for embedding. A chunk that says which document and section it came from is much easier to retrieve than one that starts mid-thought.

How much overlap do you need?

Overlap exists to protect sentences cut at an arbitrary boundary in fixed-size splitting. Once chunks follow section and element boundaries, it matters less. Unstructured, for example, defaults overlap to zero and applies it only when splitting an oversized element, unless you ask for it everywhere, and its documentation notes that overlap between ordinary chunks can mix content across boundaries. For tables, repeating the header does the job overlap would do; overlapping rows only creates duplicates that may be retrieved twice.

How do you evaluate a chunking strategy on your own questions?

1. Build a question set from real user questions, deliberately including ones answered by tables, by multi-page tables and by footnotes. Record the page and table that holds each answer.
2. Ingest your documents once per candidate pipeline (parser, chunker and settings), each into its own index.
3. Score retrieval first: is the chunk containing the answer in the top results? Recall at k and mean reciprocal rank are the usual measures, covered in RAG evaluation metrics.
4. Score answers to table questions separately, and require numbers to match exactly.
5. Read the failures. Open the retrieved chunks; most ingestion problems are visible to the eye.
6. Keep the question set and pipeline versions together, and re-run them whenever a parser or embedding model changes.

This comparison is usually cheaper than it sounds, and it settles arguments about chunk size with evidence instead of preference. It is also the first thing we look at in a RAG development review, because it explains so many downstream complaints.

Retrieval can only find what ingestion kept intact.

Where Octalcode fits

Octalcode is a studio in Lahore, with a US presence in Austin, that builds RAG systems, LLM applications and SI agents. For document-heavy work we start at ingestion: parse a sample of your hardest PDFs, compare pipelines on your own questions, and show you the chunks before anyone tunes a prompt.

If your assistant is fluent on prose but unreliable on tables, our RAG development team can review the pipeline with you and leave a written list of what to change. We sign an NDA before seeing any of your documents.