What's the best pipeline for turning unstructured documents into LLM-ready data?
Data as of Sep 21, 2026 · Based on 50 AI responses from ChatGPT Search and Google AI Mode · See how Parse measures this
Data as of Sep 21, 2026 · Based on 50 AI responses from ChatGPT Search and Google AI Mode · See how Parse measures this
Source | Citation share |
|---|---|
 Best Practices for Implementing RAG Systems in Production | Unstructuredhttps://unstructured.io/insights/rag-systems-best-practices-unstructured-data-pipeline | 2% |
What Matters for LLM Data Ingestion and Preprocessinghttps://unstructured.io/blog/understanding-what-matters-for-llm-ingestion-and-preprocessing | 2% |
Common Challenges in RAG and How to Solve Them in Production | Unstructuredhttps://unstructured.io/insights/rag-pipeline-challenges-from-data-ingestion-to-retrieval | 2% |
Ingesting Unstructured Data at Scale: Best Practices | Unstructuredhttps://unstructured.io/insights/unstructured-data-ingestion-at-scale-enterprise-best-practices | 2% |
 Build an unstructured data pipeline for RAG | Databricks on AWShttps://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/quality-data-pipeline-rag | 2% |
Unstructured Data Platform for GenAI | Unstructuredhttps://unstructured.io/ | 2% |
 Best AI for Unstructured Data - LlamaIndexhttps://www.llamaindex.ai/insights/best-ai-for-unstructured-data | 2% |
 From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answeringhttps://arxiv.org/abs/2604.04948 | 1% |
Build an unstructured data pipeline for RAG | Databricks on AWShttps://docs.databricks.com/aws/en/generative-ai/tutorials/ai-cookbook/quality-data-pipeline-rag | 1% |
Build an unstructured data pipeline for RAG | Databricks on AWShttps://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/quality-data-pipeline-rag | 1% |
The best approach is **not “PDF → plain text → chunks.”** Treat document processing as a data-engineering pipeline whose output preserves **structure, provenance, and semantics**. A strong production pipeline looks like this: ```
The best approach is not “PDF → plain text → chunks.” Treat document processing as a data-engineering pipeline whose output preserves structure, provenance, and semantics.
A strong production pipeline looks like this:
Documents
↓
Ingest + fingerprint
↓
Format-aware parsing / OCR
↓
Layout + reading-order reconstruction
↓
Structured canonical representation
↓
Cleaning + normalization
↓
Semantic enrichment
↓
Structure-aware chunking
↓
Metadata + provenance
↓
Quality checks
↓
Embeddings / search index / LLM context
Keep the original files immutable.
For every document, capture:
document_idThe hash lets you make ingestion idempotent: unchanged documents don't get processed again.
This is where many pipelines go wrong.
Don't treat a PDF as a stream of characters. You want to recover:
For a general-purpose open-source stack, Docling is a particularly useful starting point: it handles PDFs, DOCX, PPTX, XLSX, HTML and images and can preserve layout, reading order, tables, formulas and provenance in a unified representation.
For scanned documents, add OCR rather than assuming the PDF's text layer is reliable. Docling, for example, supports multiple OCR engines and explicitly distinguishes OCR from its higher-level layout/document understanding.
This is arguably the most important architectural decision.
Don't make Markdown your only source of truth. Store something structurally closer to:
{
"document_id": "abc123",
"title": "Annual Report",
"sections": [
{
"heading": "Revenue",
"level": 2,
"blocks": [
{
"type": "paragraph",
"text": "Revenue increased...",
"page": 17,
"bbox": [72, 144, 520, 230]
},
{
"type": "table",
"rows": [
["Year", "Revenue"],
["2025", "$10M"],
["2026", "$13M"]
],
"page": 17
}
]
}
]
}
That gives you the ability to generate different downstream representations without reparsing the original: Markdown for an LLM, JSON for extraction, chunks for RAG, HTML for visualization, etc. This is also the architecture Docling's unified document model is designed around.
Remove things that genuinely aren't information:
But don't aggressively summarize or rewrite the source at this stage.
Preserve the original extracted text alongside the normalized version. Information loss during preprocessing is difficult to recover.
A table should not become:
Q1 100 Q2 120 Q3 140 Q4 160
It should retain its row/column relationships.
Likewise, for figures and diagrams, retain:
This matters enormously for financial, scientific, legal and technical documents.
Recent evaluation work found that hierarchy-aware splitting and metadata enrichment can have a larger effect on downstream RAG accuracy than simply changing the PDF conversion framework. In one 2026 benchmark, hierarchical splitting plus image descriptions produced substantially better results than naïve PDF extraction.
Avoid:
text[i:i+1000]
Prefer:
Document
└── Section
└── Subsection
├── paragraph
├── paragraph
└── table
Then form chunks around those boundaries.
A chunk should ideally be understandable without requiring the retriever to reconstruct half the document.
Include contextual metadata such as:
{
"chunk_id": "...",
"document_id": "...",
"section": "Revenue > North America",
"page_start": 17,
"page_end": 18,
"content": "...",
"content_type": "text",
"source_hash": "...",
"bbox": [...]
}
Structure-aware chunking is preferable to blind token splitting; tools such as Docling's HybridChunker explicitly operate on document structure rather than merely slicing characters.
Useful enrichment includes:
Be careful with LLM-generated metadata: store it as derived data, never as a replacement for the source.
I'd make ingestion produce a quality report:
Document: annual_report.pdf
Pages: 147
OCR pages: 23
Tables: 41
Figures: 18
Sections: 62
Chunks: 384
Warnings:
⚠ 3 pages have low OCR confidence
⚠ 2 tables have inconsistent column counts
⚠ 1 page has unusual reading order
Status: REVIEW
This is much better than discovering six months later that your RAG system has been answering questions from corrupted tables.
Every chunk should be traceable:
answer
↓
retrieved chunk
↓
section
↓
page 47, bbox(...)
↓
original document
That enables citations, debugging, human verification and auditability. Layout-aware systems can preserve page and spatial information specifically for this purpose.
I would make the architecture explicitly two-stage:
Offline
Files
→ Parse
→ Normalize
→ Enrich
→ Chunk
→ Embed
→ Index
Online
Question
→ Query understanding
→ Retrieval
→ Reranking
→ Context assembly
→ LLM
→ Citations
This separation makes the ingestion pipeline independently testable and lets you re-chunk/re-index without touching your answering logic. This offline/online separation is also a standard pattern in production RAG architectures.
For a new system, I'd start roughly here:
| Layer | Default |
|---|---|
| Storage | S3/GCS/Azure Blob |
| Parsing | Docling |
| OCR | Docling + appropriate OCR backend |
| Canonical format | Structured JSON/DoclingDocument |
| Human-readable representation | Markdown |
| Chunking | Hierarchy/structure-aware |
| Metadata | JSON alongside each chunk |
| Embeddings | Your chosen embedding model |
| Retrieval | Hybrid BM25 + vector |
| Reranking | Cross-encoder/reranker |
| Index | Postgres/pgvector, Elasticsearch/OpenSearch, or a vector DB |
| Evaluation | Curated question/answer + retrieval benchmarks |
| Observability | Per-document extraction and retrieval metrics |
The key principle is:
Preserve information first; transform it for the LLM second. If you preserve layout, hierarchy, tables, provenance and metadata in an intermediate representation, you can subsequently optimize chunking, embeddings and retrieval without repeatedly fighting the original PDFs. Recent benchmarking supports this general strategy: preprocessing quality and hierarchy/metadata enrichment materially affect RAG performance.
If you're building this for 10k–10M documents, the architecture changes somewhat—especially around queues, incremental processing, deduplication, OCR throughput, and evaluation—and I can lay out that production architecture too.
Then form chunks around those boundaries.
A chunk should ideally be understandable without requiring the retriever to reconstruct half the document.
Include contextual metadata such as:
{
"chunk_id": "...",
"document_id": "...",
"section": "Revenue > North America",
"page_start": 17,
"page_end": 18,
"content": "...",
"content_type": "text",
"source_hash": "...",
"bbox": [...]
}
Structure-aware chunking is preferable to blind token splitting; tools such as Docling's HybridChunker explicitly operate on document structure rather than merely slicing characters.
Useful enrichment includes:
Be careful with LLM-generated metadata: store it as derived data, never as a replacement for the source.
The "best" pipeline for turning unstructured documents (PDFs, scans, Word docs, slides) into LLM-ready data isn't a single monolithic tool—it is a **modular, multi-stage architecture** . If your parser mangles a table or drops a header, your embeddings and generations fail…
The "best" pipeline for turning unstructured documents (PDFs, scans, Word docs, slides) into LLM-ready data isn't a single monolithic tool—it is a modular, multi-stage architecture . If your parser mangles a table or drops a header, your embeddings and generations fail downstream.
A battle-tested, modern pipeline consists of five distinct stages:
author, date, document_category, summary).To help tailor this pipeline, tell me:
The strongest general-purpose pipeline is: **Ingest → layout-aware parse/OCR → normalize → enrich metadata → structure-aware chunk → validate → embed + keyword index → hybrid retrieve/rerank.** [Databricks…
The strongest general-purpose pipeline is:
Ingest → layout-aware parse/OCR → normalize → enrich metadata → structure-aware chunk → validate → embed + keyword index → hybrid retrieve/rerank.
A useful canonical record is:
{
"text": "...",
"metadata": {
"document_id": "...",
"version": "...",
"section": "...",
"page": 12,
"source": "...",
"document_type": "..."
}
}
Recent evaluation work suggests hierarchy-aware chunking and metadata can matter more than the specific PDF conversion tool, so optimize the pipeline end-to-end rather than obsessing over one parser.
The "best" pipeline for turning unstructured documents (PDFs, scanned invoices, multi-column reports, PowerPoints, and messy web pages) into LLM-ready data follows a modern **parse → clean → chunk → enrich → load** architecture. Because traditional text-extraction tools (like basic `pypdf` or `pdfminer` ) destroy…
The "best" pipeline for turning unstructured documents (PDFs, scanned invoices, multi-column reports, PowerPoints, and messy web pages) into LLM-ready data follows a modern parse → clean → chunk → enrich → load architecture.
Because traditional text-extraction tools (like basic pypdf or pdfminer ) destroy tables, reading order, and multi-column layouts, the modern consensus shifts heavily toward vision-based parsing and layout-aware engines.
To help narrow down the ideal architecture for you, tell me:
The best pipeline for turning unstructured documents (PDFs, Word docs, scanned images, HTML) into LLM-ready data follows a modular, multi-stage architecture often referred to as **Ingestion → Parsing → Enrichment → Chunking →…
The best pipeline for turning unstructured documents (PDFs, Word docs, scanned images, HTML) into LLM-ready data follows a modular, multi-stage architecture often referred to as Ingestion → Parsing → Enrichment → Chunking → Embedding/Storage.
Instead of relying on a single "magic" tool, modern production systems combine specialized parsers with semantic chunking.
If you want to tailor this stack, tell me:
I can give you a specific code pattern or architecture recommendation.
The best pipeline is usually **not “PDF → embeddings → vector DB.”** High-quality LLM systems treat document preparation as a data engineering problem: preserve structure, add metadata, create good retrieval units, and continuously evaluate. [Databricks…
The best pipeline is usually not “PDF → embeddings → vector DB.” High-quality LLM systems treat document preparation as a data engineering problem: preserve structure, add metadata, create good retrieval units, and continuously evaluate.
A strong production pipeline looks like this:
Raw documents
↓
1. Ingestion + versioning
↓
2. Parsing + structure extraction
↓
3. Cleaning + normalization
↓
4. Metadata enrichment
↓
5. Semantic chunking
↓
6. Embedding generation
↓
7. Hybrid indexing
↓
8. Retrieval + reranking
↓
9. Context assembly
↓
10. LLM generation + evaluation loop
Inputs:
Store the original artifact unchanged. Create a document record:
{
"document_id": "abc123",
"source": "sharepoint",
"title": "Employee Handbook",
"created_at": "2026-01-10",
"modified_at": "2026-08-01",
"permissions": ["hr"],
"hash": "sha256..."
}
Important:
Avoid flattening everything into plain text too early.
Bad:
Section heading
paragraph
table
footer
paragraph
Better:
{
"type": "section",
"heading": "Refund Policy",
"content": [
{
"type": "paragraph",
"text": "Refunds are available within 30 days..."
},
{
"type": "table",
"rows": [...]
}
]
}
Preserve:
For PDFs, this is especially important because layout often carries meaning. Scanned documents may require OCR before downstream processing.
Remove noise:
Normalize:
Do not aggressively summarize here. You want retrieval-ready source material, not compressed information.
Metadata often improves retrieval more than changing embedding models.
Useful fields:
{
"department": "finance",
"document_type": "policy",
"author": "Jane Smith",
"date": "2026-03-01",
"security_level": "internal",
"entities": ["Acme Corp", "California"],
"section": "Expense Rules"
}
This enables:
Chunking is one of the biggest quality levers.
Weak:
Every 1,000 characters → split
Better:
Document
├── Chapter
├── Section
├── Paragraph group
└── Retrieval chunk
A chunk should answer a likely user question by itself.
Typical starting points:
Keep parent-child relationships:
Parent:
"Security Policy"
Children:
- Password requirements
- MFA rules
- Device policy
The retriever can fetch precise chunks while preserving broader context.
For each chunk:
{
"chunk_id": "abc123-07",
"text": "Employees must enable MFA...",
"embedding": [0.012, -0.034, ...],
"metadata": {...}
}
Store:
This makes future re-indexing possible.
A good production index usually combines:
Finds semantic matches:
"How do I get reimbursed?" matches:
"Expense claim submission procedure"
Finds exact terms:
"SOC-2 Type II" A common architecture:
Query
↓
Vector search
+
Keyword search
↓
Merge results
↓
Reranker
↓
LLM
Hybrid retrieval is especially useful for enterprise documents with names, IDs, regulations, and technical terms.
Initial retrieval might return 50 candidates.
A reranker asks:
"Which of these passages actually answers this question?" Pipeline:
Retriever → top 50 chunks
↓
reranker model
↓
top 5 chunks
↓
LLM
This often improves quality more than endlessly tuning prompts.
Before sending to the LLM:
Example:
Question:
"What is the expense approval limit?"
Sources:
[Policy.pdf, page 12]
Managers may approve expenses up to $5,000...
[Policy.pdf, page 13]
Expenses above $5,000 require director approval...
Do not judge the pipeline by whether demos look good.
Create a test set:
{
"question": "Who approves expenses over $5000?",
"expected_source": "policy.pdf",
"expected_answer": "Director approval required"
}
Measure:
Evaluation should happen after every major change:
For many teams:
| Layer | Options |
|---|---|
| Storage | S3, Azure Blob, GCS |
| Parsing | Unstructured, Apache Tika, custom parsers |
| OCR | Textract, Azure AI Vision, Tesseract |
| Chunking | semantic splitters + document-specific rules |
| Embeddings | OpenAI embeddings, BGE, E5, GTE |
| Vector DB | pgvector, Milvus, Pinecone, Elasticsearch |
| Keyword search | BM25 / Elasticsearch |
| Reranking | cross-encoder reranker |
| Evaluation | custom golden sets + RAG evaluation frameworks |
The biggest practical lesson: optimize the retrieval substrate before optimizing the LLM prompt. Most failures that look like “the model hallucinated” are actually “the right information never reached the model.”
A good first production version is usually:
parse → clean → metadata → semantic chunks → embeddings → hybrid search → rerank → cite → evaluate.
Initial retrieval might return 50 candidates.
A reranker asks:
"Which of these passages actually answers this question?" Pipeline:
Retriever → top 50 chunks
↓
reranker model
↓
top 5 chunks
↓
LLM
This often improves quality more than endlessly tuning prompts.
Before sending to the LLM:
The best pipeline for turning unstructured documents into LLM-ready data is usually **not “PDF → text → embeddings.”** The strongest systems treat document preparation as a data engineering pipeline that preserves structure, meaning, provenance, and permissions before retrieval. [Databricks…
The best pipeline for turning unstructured documents into LLM-ready data is usually not “PDF → text → embeddings.” The strongest systems treat document preparation as a data engineering pipeline that preserves structure, meaning, provenance, and permissions before retrieval.
A production-grade pipeline looks like this:
Raw documents
↓
1. Ingestion + inventory
↓
2. Parsing + layout extraction
↓
3. Cleaning + normalization
↓
4. Structure-aware chunking
↓
5. Metadata enrichment
↓
6. Embedding + indexing
↓
7. Retrieval optimization
↓
8. Evaluation + feedback loop
↓
LLM-ready knowledge layer
Goal: Create a reliable source-of-truth layer.
Inputs:
Store:
Why it matters:
Example output:
{
"document_id": "policy_123",
"source": "sharepoint",
"version": "7",
"created_at": "2026-01-10",
"access": ["hr-team"]
}
Avoid flattening everything into plain text.
A good parser produces:
[
{
"type": "heading",
"text": "Termination Policy"
},
{
"type": "paragraph",
"text": "Employees must provide..."
},
{
"type": "table",
"cells": [...]
}
]
Preserve:
For scanned documents, add OCR. For complex PDFs, layout understanding matters because columns, tables, and footnotes can otherwise be scrambled.
Typical transformations:
Remove:
Normalize:
Do not aggressively summarize at this stage. You want a faithful representation of the source.
Chunking is one of the biggest quality levers.
Bad:
Every 1000 characters
Better:
Document
├── Section
│ ├── Subsection
│ │ └── Paragraph chunks
Good chunk metadata:
{
"text": "Employees are eligible after 90 days...",
"document": "benefits.pdf",
"section": "Healthcare Benefits",
"page": 12,
"access": "employees"
}
Rules of thumb:
Structure-aware chunking generally performs better than blind splitting because retrieval depends on receiving coherent context.
Add information that improves retrieval:
Example:
{
"chunk_id": "abc123",
"topic": "employee benefits",
"entities": [
"Health Insurance"
],
"security": "internal"
}
A strong system often stores more than embeddings.
Good for:
Example:
"How much vacation do new hires receive?" Matches:
"Annual leave entitlement for employees in their first year..."
Good for:
Example:
Many production systems use hybrid retrieval:
Retrieval often returns "related" documents, not necessarily the best ones.
A common pipeline:
User query
↓
Retrieve top 50 chunks
↓
Reranker scores relevance
↓
Keep top 5-10 chunks
↓
LLM generates answer
This improves precision without forcing the LLM to read huge contexts.
Track:
A useful production metric is:
Question
↓
Expected evidence
↓
Retrieved evidence
↓
Final answer
You want visibility into every step.
A modern stack often looks like:
┌─────────────┐
│ SharePoint │
│ Drive │
│ S3 │
└──────┬──────┘
↓
Document loader
↓
Parser + OCR + layout
↓
Structured document JSON
↓
┌──────────────┴──────────────┐
↓ ↓
Chunk database Search indexes
↓ ↓
Embeddings Vector + keyword
↓ ↓
Retrieval + reranking
↓
LLM
↓
Answer + citations
For most enterprise RAG systems, the highest-impact order of investment is:
The LLM is usually not the bottleneck—the quality of the context you feed it is.
┌─────────────┐
│ SharePoint │
│ Drive │
│ S3 │
└──────┬──────┘
↓
Document loader
↓
Parser + OCR + layout
↓
Structured document JSON
↓
┌──────────────┴──────────────┐
↓ ↓
Chunk database Search indexes
↓ ↓
Embeddings Vector + keyword
↓ ↓
Retrieval + reranking
↓
LLM
↓
Answer + citations
The "best" pipeline for turning unstructured documents (PDFs, DOCX, scanned images, financial filings) into LLM-ready data follows a modern **Ingest → Parse → Chunk → Enrich → Load** architecture. Rather than relying on basic text-scraping (which mangles tables, multi-column layouts, and charts), the state-of-the-art…
The "best" pipeline for turning unstructured documents (PDFs, DOCX, scanned images, financial filings) into LLM-ready data follows a modern Ingest → Parse → Chunk → Enrich → Load architecture.
Rather than relying on basic text-scraping (which mangles tables, multi-column layouts, and charts), the state-of-the-art stack relies on vision-based/agentic parsers coupled with semantic chunking.
Standard libraries (like PyPDF or basic pdfminer ) drop layout context, turning multi-column text and complex tables into unreadable gibberish. Modern AI-native parsing tools use vision models or specialized layout-aware OCR to preserve the document's structure.
#, ## ), bullet points, and pipe-tables retain hierarchical relationships natively, which drastically improves retrieval accuracy later.Instead of cutting documents arbitrarily by character count (e.g., 500 tokens with 50 overlap), use context-aware chunking:
An LLM performs significantly better when a chunk carries context about where it came from. Enrich every chunk with programmatic metadata during the pipeline run:
Financials > Q3 Revenue > North America).Store your enriched chunks into a dual-retrieval setup:
To help tailor this pipeline to your exact stack, tell me:
The best pipeline is **not “PDF → text → chunks → embeddings.”** For high-quality LLM/RAG systems, treat document processing as a **lossless, structure-preserving ETL pipeline** and delay irreversible transformations until you know why you need them. ## Recommended pipeline ```
The best pipeline is not “PDF → text → chunks → embeddings.” For high-quality LLM/RAG systems, treat document processing as a lossless, structure-preserving ETL pipeline and delay irreversible transformations until you know why you need them.
Raw documents
│
▼
1. Ingest + fingerprint
│
├── MIME/type detection
├── SHA-256 / version ID
├── source + ACL metadata
└── deduplication
│
▼
2. Parse / OCR
│
├── Native text extraction
├── Layout-aware parsing
├── OCR for scans
└── Image / table detection
│
▼
3. Structural representation
│
├── Title
├── headings
├── paragraphs
├── lists
├── tables
├── captions
└── page / bounding-box provenance
│
▼
4. Normalize + clean
│
├── remove headers/footers
├── fix OCR artifacts
├── normalize whitespace
├── resolve reading order
└── preserve original text
│
▼
5. Enrich metadata
│
├── document / section / page IDs
├── dates, authors, entities
├── access permissions
└── hierarchy / breadcrumbs
│
▼
6. Semantic chunking
│
├── section-aware
├── paragraph/list-aware
├── tables treated specially
└── token/size limits
│
▼
7. Contextualize
│
└── add section/document context where useful
│
▼
8. Quality checks
│
├── extraction quality
├── missing pages
├── table integrity
└── chunk/retrieval tests
│
▼
9. LLM-ready records
│
├── text
├── metadata
├── provenance
├── embeddings
└── searchable indexes
This is probably the most important design decision.
Instead of producing:
Here is a giant string containing the entire PDF...
produce something closer to:
{
"document_id": "abc123",
"source": "annual_report.pdf",
"elements": [
{
"type": "title",
"text": "Annual Report 2026",
"page": 1
},
{
"type": "heading",
"text": "Revenue",
"page": 12
},
{
"type": "paragraph",
"text": "...",
"page": 12
},
{
"type": "table",
"content": "...",
"page": 13
}
]
}
Layout-aware parsers such as docs.unstructured.io explicitly model elements such as titles, narrative text, lists, tables, images, headers and footers, rather than treating everything as undifferentiated text.
For PDFs, I'd use a fast native-text path first, escalating to layout analysis/OCR when necessary. For scanned PDFs, OCR is unavoidable; for complicated layouts, layout-aware extraction is generally worth the additional cost.
Every chunk should be traceable back to:
chunk
└── section
└── page(s)
└── source document
└── original file/version
I'd store at least:
document_iddocument_versionchunk_idsource_uripage_numbersection_pathelement_idscreated_atparser_versioncontent_hashThis makes citations, debugging, reprocessing, and incremental indexing dramatically easier. Element-level metadata is specifically useful for mapping retrieved content back to its source page and filtering retrieval results.
Don't aggressively "improve" the text.
Good:
Dangerous:
Keep the original extraction alongside the cleaned representation.
That gives you a reversible pipeline.
A naïve:
text[i:i+1000]
chunker is usually a poor starting point.
Instead:
Document
→ Section
→ Paragraphs
→ Lists
→ Tables
and combine adjacent semantic units until you approach your target size.
For example:
Chunk 1
"3. Revenue"
paragraph
paragraph
bullet list
Chunk 2
"4. Expenses"
paragraph
paragraph
Chunk 3
table
This preserves relationships that arbitrary character splitting destroys. Unstructured's current chunking approach similarly operates on parsed document elements and preserves those semantic boundaries where possible. Its by_title strategy explicitly prevents a chunk from crossing section boundaries.
I would not blindly optimize for a particular chunk size. Tune it against your actual retrieval benchmark and embedding model.
Tables are where many document pipelines quietly fail.
Don't turn:
| Product | Q1 | Q2 |
|---|---|---|
| A | 10 | 15 |
| B | 20 | 25 |
into a flat sentence without preserving the row/column relationships.
Depending on your application, maintain:
{
"type": "table",
"caption": "...",
"headers": ["Product", "Q1", "Q2"],
"rows": [
["A", 10, 15],
["B", 20, 25]
],
"page": 17
}
and potentially create a textual representation for embedding/search.
Tables should generally be isolated from ordinary prose during chunking; Unstructured does this explicitly.
A surprisingly powerful representation is:
Document: ACME 2026 Annual Report
Section: Financial Results > Revenue > Geographic Revenue
Page: 42
[actual chunk text]
rather than embedding only:
Revenue increased 14% year over year...
The additional context helps a retriever understand what the otherwise ambiguous passage is about.
Contextual chunking—adding chunk-specific explanatory context before embedding—is now an established technique for improving retrieval accuracy.
I'd maintain two layers.
Canonical document store
raw file
parsed elements
cleaned elements
metadata
provenance
Retrieval store
chunk_id
chunk text
embedding
sparse/BM25 representation
metadata filters
Then you can change:
without having to re-OCR/re-parse thousands of documents.
For a production system, my default stack would look like:
S3 / blob storage
↓
Document ingestion queue
↓
File type detection
↓
Parser router
┌───┼──────────────┐
PDF DOCX HTML/Excel/etc.
│
native extraction
↓
OCR/layout fallback
↓
Canonical Element JSON
↓
Cleaning + normalization
↓
Metadata/entity enrichment
↓
Section-aware chunking
↓
Contextualization
↓
┌─────┴─────────┐
↓ ↓
BM25 Embeddings
↓ ↓
└──────┬────────┘
↓
Hybrid retrieval
↓
Reranker
↓
LLM
The key architectural principle is:
Parse once, preserve everything, derive many representations. Don't make your vector database the source of truth.
And don't evaluate the pipeline solely by asking “did we extract all the text?” The real metric is end-to-end retrieval quality: can the system retrieve the exact evidence needed to answer representative questions, with correct provenance?
If you're building this today, I'd spend disproportionately more engineering effort on parsing quality, metadata/provenance, tables, chunk boundaries, and retrieval evaluation than on fancy embedding models.
There is no single "magic bullet" tool that works universally for every file type, but the industry has largely converged on a **modern 5-stage architecture** standard for turning messy unstructured documents (PDFs, scans, DOCX, HTML) into pristine LLM-ready data.[](https://www.youtube.com/watch?v=PZDgQJSpElc&t=36)…
There is no single "magic bullet" tool that works universally for every file type, but the industry has largely converged on a modern 5-stage architecture standard for turning messy unstructured documents (PDFs, scans, DOCX, HTML) into pristine LLM-ready data.
The gold standard pipeline breaks down into these sequential phases:
Connect to your data source (S3, SharePoint, local directories, or web scrapers) and pull raw documents.
This is where the heavy lifting happens. Traditional text scrapers fail on complex layouts, multi-column text, headers, and especially tables. Modern pipelines use Vision Language Models (VLMs) or layout-aware parsers to preserve document topology (markdown or structured JSON).
Strip out repetitive artifacts (page numbers, headers/footers, watermarks) and redact sensitive information. Doing PII masking before chunking and embedding is critical—otherwise, sensitive data is permanently baked into your vector database.
Never split documents by arbitrary character counts alone (e.g., rigid 500-token blocks), which cuts sentences and tables in half. Use structure-aware chunking that respects natural boundaries like headers, paragraphs, and markdown table blocks.
Generate vector embeddings for semantic search and enrich chunks with contextual metadata (file name, section title, date, hierarchical breadcrumbs) to boost retrieval accuracy.
To help narrow down the ideal stack for you, tell me:
