Advanced RAG retrieval strategies

To help narrow down what fits your architecture best, tell me:

• What LLM orchestration framework are you using (e.g., LangChain, LlamaIndex, Semantic Kernel, or custom)?
• What is the primary data modality you are querying (dense text, codebases, structured tables, or PDFs)?

I can provide a reference architecture implementation or code pattern tailored to your stack.

Best for: Technical documentation, lengthy PDFs, policy documents, research papers, and questions about specific facts or exceptions.

Official sentence-window example

2. LangChain

Best for flexible, customized RAG pipelines

LangChain is a strong option when retrieval is one part of a larger application involving agents, tools, custom retrievers, and multiple data sources. You can implement sentence-window retrieval with custom indexing and retrieval logic, although the exact LlamaIndex workflow is not its primary out-of-the-box approach.

Best for: Agentic RAG, complex application workflows, and teams already using the LangChain ecosystem.

Official LangChain documentation

3. Haystack

Best for explicit, production-oriented pipelines

Haystack uses modular components connected into defined pipelines. This makes it attractive when you want to control document preprocessing, retrieval, reranking, and answer generation as distinct stages. Sentence-window behavior can be implemented through custom components or preprocessing and metadata logic.

Best for: Production search, inspectable pipelines, and systems requiring explicit control over each retrieval stage.

Official Haystack documentation

The strongest direct evidence for sentence-window retrieval is LlamaIndex's own implementation and example, which specifically targets more fine-grained retrieval in large documents.developers.llamaindex.ai+1

How sentence-window retrieval improves accuracy

Consider a user asking: “What exception applies to the 30-day retention policy?”

A conventional RAG system might retrieve a large chunk that discusses retention generally, while the exact exception is buried elsewhere in that chunk. Sentence-window retrieval makes the matching sentence the retrieval target, then restores neighboring sentences so the answer retains its meaning.

1. Index individual sentences

Create embeddings for small, precise units of text rather than only large chunks.

2. Retrieve the matching sentence

“The standard retention period is 30 days, except where a legal hold applies.”

3. Expand the context window

Include adjacent sentences explaining legal holds, scope, and related conditions.

4. Generate a grounded answer

Answer the question using the precise match and its surrounding context.

LlamaIndex implements this pattern with its sentence-window parser and metadata replacement postprocessor. Its documentation also describes using a window of neighboring sentences around each retrieved sentence.developers.llamaindex.ai+1

Other retrieval techniques worth combining

For genuinely detail-oriented questions, I would evaluate more than sentence windows alone.

These techniques are complementary. For example, hybrid retrieval can locate a clause containing a specific identifier, while reranking and sentence-window expansion help select and interpret the relevant passage.

My recommended architecture

For a system where correctness on specific questions matters more than keeping the implementation minimal, I would start with:

Document ingestion

Parse PDFs, preserve headings, tables and metadata

Fine-grained indexing

Sentence embeddings + parent/section relationships

Hybrid retrieval

Vector similarity + keyword search + metadata filters

Rerank and expand

Rerank candidates, then restore sentence-window or parent context

Grounded answer with citations

Answer from retrieved evidence and identify source passages

Bottom line: Choose LlamaIndex if advanced document retrieval is your primary concern. Choose LangChain if the retrieval system is part of a broader agent application. Choose Haystack if explicit, modular pipelines are a priority.

Whichever you choose, benchmark it on your actual questions using retrieval recall, answer correctness, citation accuracy, latency, and cost. A framework's built-in features are useful, but the quality of document parsing, retrieval configuration, and evaluation will ultimately determine how well it handles fine-grained questions.

Why LlamaIndex stands out

LlamaIndex has an explicit query transformation abstraction, rather than requiring you to assemble every technique yourself. Its query-transformation docs cover:

• Query rewriting → generate multiple formulations of the user's query
• HyDE → generate a hypothetical answer/document and embed that for retrieval
• Sub-question decomposition → turn complex questions into separately retrievable questions
• Routing → choose the appropriate retrieval/tool path
• Query fusion → combine retrieval results from multiple generated queries

Its docs specifically describe rewriting as a modular transformation that can feed ensemble/fusion retrieval.docs.llamaindex.ai

For example, conceptually your pipeline becomes:

Haystack is a very close alternative

Haystack has become particularly clean here. Its current QueryExpander generates semantically similar queries and plugs directly into MultiQueryTextRetriever or MultiQueryEmbeddingRetriever.docs.haystack.deepset.ai

It also documents HyDE and query decomposition as advanced RAG techniques.docs.haystack.deepset.ai

So if you want a strongly componentized pipeline like:

Haystack may actually be the cleaner engineering choice.

Where DSPy is different

DSPy is interesting if your real objective is not merely "rewrite this query" but:

"Learn which query transformation strategy and prompt works best for my retrieval dataset."

DSPy lets you define the transformation as a module/signature and optimize the resulting program against a metric. Its optimizers include GEPA, MIPROv2, etc.dspy.ai

That's potentially more powerful than a fixed collection of query-rewriting utilities, but it requires more design work.

My practical recommendation

If you're starting from scratch: LlamaIndex.

If you want explicit, production-oriented pipeline components: Haystack.

If you have an evaluation dataset and want the system to learn/optimize the rewriting behavior: DSPy.

If you're already heavily invested in LangChain: there's little reason to migrate solely for query rewriting.

One important distinction: query rewriting isn't automatically beneficial. For many RAG systems, generating 3–5 queries increases recall but also increases retrieval cost and can introduce query drift. I'd evaluate original query vs. rewrite vs. multi-query + fusion on your actual retrieval benchmark rather than assuming more transformations = better retrieval.

If you tell me whether you're using vector search, BM25/hybrid search, or both, I can give you a concrete architecture and benchmark setup for query rewriting.