Why Your RAG Pipeline Is Hallucinating (And How Docling Fixes It)

If you have built a RAG system over clinical documents and wondered why the model keeps missing answers that are clearly in the document, the problem is probably chunking. Not the model, not the embeddings, not the retrieval threshold — the chunking. And there is now a mature, open-source, fully local approach to fixing it that is worth understanding before you build your next pipeline.

This post covers Chunkless RAG — what it is, how IBM's Docling makes it practical, when to use it over traditional chunking, and what it means specifically for healthcare AI pipelines that need to operate over structured clinical documents without sending data to a cloud service.

The Problem With Chunking Clinical Documents

Traditional RAG splits documents into fixed-size text segments — typically 512 or 1,024 tokens — embeds each chunk independently, and retrieves the most semantically similar chunks at query time. The approach works well for large document collections where you need broad, fuzzy retrieval across millions of records. It works poorly for documents where the answer depends on understanding how sections relate to each other.

Clinical documents are almost entirely in that second category. A discharge summary has a structured hierarchy: chief complaint, history of present illness, medications, allergies, assessment, plan. A clinical policy document has numbered sections with cross-references. A formulary has tables where column headers two pages back determine what a cell means. When a chunker splits these documents into 512-token blobs, it flattens the hierarchy. The chunk that contains the answer may no longer contain the heading that tells the model what the answer is answering. The table cell lands in one chunk; the column header that identifies it lands in another. The cross-reference points to a section number that isn't in the retrieved chunk.

The model isn't hallucinating because it is confused. It is hallucinating because the retrieval step handed it fragments with missing context, and it is doing its best to fill in the gaps.

What Chunkless RAG Does Instead

Chunkless RAG keeps the document whole and gives an AI agent the ability to navigate it structurally — reading a table of contents, walking into the relevant section, reading only what it needs, and maintaining the full hierarchical context while doing so. Rather than retrieving fragments and asking the model to synthesize them, it asks the agent to navigate to the answer the way a human expert would navigate a document they know well.

The tradeoff is compute and scale. Navigating a document tree takes more reasoning steps than retrieving pre-embedded chunks. For a collection of millions of documents, traditional chunking wins on throughput. For a small number of long, highly organized documents where precision matters — which describes most of the reference material a healthcare AI system needs to reason over — chunkless navigation produces more accurate, more grounded answers with fewer fabricated connections between fragments.

Docling: The Infrastructure That Makes This Practical

The reason chunkless RAG has moved from an interesting idea to a practical option is Docling. IBM Research open-sourced Docling in late 2024, and it has become the most capable open-source document parser built specifically for AI pipelines, with 61,000 GitHub stars and native integrations with LangChain, LlamaIndex, Crew AI, and Haystack.

What Docling does that generic PDF parsers do not is reconstruct the document's logical structure rather than extracting its text. It produces a unified DoclingDocument representation that captures layout, reading order, table cell boundaries with their headers, figure placement, formula positions, and the hierarchical relationship between headings and their content. That structured representation exports to Markdown, HTML, JSON, or its own DocTags format — all preserving the structure that chunking would destroy.

The January 2026 release of Granite-Docling-258M added a production vision-language model under Apache 2.0 that handles charts, tables, forms, code blocks, equations, footnotes, and captions in a single pass, avoiding the error accumulation that multi-stage parsing pipelines introduce. The result is a document representation where an AI agent can actually walk the tree — read the table of contents, navigate to a section, read the table with its headers intact, and follow a cross-reference — rather than receiving disconnected fragments.

Three details matter specifically for healthcare pipelines.

First, Docling runs entirely locally. No API key, no per-page charge, no cloud upload. The first call downloads the models to a local cache; every subsequent call is offline. For healthcare organizations where PHI cannot leave the host, this is not a nice-to-have — it is a requirement, and Docling meets it without architectural compromise.

Second, Docling ships an MCP server. This means it plugs directly into agentic workflows without wrapper code — an agent that needs to navigate a clinical document can call Docling's MCP tools natively, the same way it would call any other tool in its arsenal.

Third, the license is MIT. There are no usage restrictions, no commercial licensing questions, and no vendor dependency to manage.

How This Fits Into a Healthcare AI Pipeline

The practical implementation is a preprocessing step, not a wholesale replacement of your existing RAG infrastructure. You run documents through Docling first, producing a structured representation. For documents where chunkless navigation is the right approach — long clinical guidelines, formularies, policy documents, structured protocols — you pass the Docling output to an agent that navigates it. For large-scale fuzzy retrieval across document collections — searching a corpus of clinical notes, retrieving relevant studies from a literature database — traditional chunking with embeddings remains the right tool.

The two approaches are complementary, not competing. The question for each document type in your pipeline is which retrieval pattern fits: precise structural navigation over a well-organized document, or broad semantic retrieval across many documents.

AnythingLLM Integration: What to Expect

If you are running AnythingLLM locally, Docling does not plug in natively as a connector. AnythingLLM handles its own document ingestion and chunking. The practical workflow is to preprocess documents through Docling first — exporting to Markdown or structured JSON — then ingest the Docling output into AnythingLLM as the source document. You get Docling's structure-preserving parsing at the ingestion stage, and AnythingLLM handles the retrieval layer from there.

For a full chunkless agentic navigation pattern, you would need to build outside AnythingLLM — using LangChain or LlamaIndex with their native Docling integrations, or calling Docling's MCP server directly from a Claude Code or similar agentic workflow. AnythingLLM is an excellent RAG platform for local operation, but the chunkless pattern specifically requires an agent that can walk the document tree, which is an agentic capability rather than a retrieval one.

Where to Start

Docling installs as a standard Python package and runs on a laptop. The quickest way to evaluate whether it improves your pipeline is to run a document you know is causing retrieval problems through Docling's converter and inspect the structured output before you touch your retrieval layer at all.

pip install docling

from docling.document_converter import DocumentConverter
converter = DocumentConverter()
doc = converter.convert("your-document.pdf").document
print(doc.export_to_markdown())

If the Markdown output preserves the table headers, heading hierarchy, and section relationships that your current parser is flattening, that is the signal that Docling belongs in your pipeline. The LlamaIndex and LangChain integrations both have Docling readers that take the conversion step from there directly into your retrieval layer.

IBM has also released OpenRAG — a single-package RAG platform built on Docling, Langflow, and OpenSearch — if you want a more complete reference architecture to evaluate rather than building from individual components.

The Practical Takeaway

For healthcare AI practitioners building RAG over structured clinical reference documents, Chunkless RAG with Docling addresses a real problem that traditional chunking creates and that model quality alone cannot fix. The infrastructure is mature, open source, MIT licensed, runs locally with no cloud dependency, and integrates with the major LLM orchestration frameworks your pipeline is likely already using.

The decision framework from the IBM video is the right one: chunkless navigation for long, highly organized documents where precision and logical connections are essential; traditional chunk-based retrieval for large-scale fuzzy search across document collections. Most healthcare AI pipelines need both. Docling makes the first option practical without sacrificing the local, private operation that regulated environments require.

For related coverage, see Running RAG Locally: AnythingLLM for Healthcare AI Workflows and Ollama and Local LLMs: Keeping PHI Off the Cloud.