When Vector Search Isn't Enough: What 11,794 Pages Taught Me About the Limits of AI Search

I asked a working AI search system to list every organization that had ever touched a set of records. It gave me seven, with confidence, in about nine seconds. The real number was twenty-two.

Nothing in that answer was wrong. Every one of the seven was real. The system did not hallucinate, did not glitch, did not need a better model. It simply stopped at the edge of what it had looked at, and it had no way to tell me where that edge was.

That gap is the subject of this piece. It is not a bug in anyone's product. It is a structural property of how nearly all AI document search works right now, and if the questions that matter in your business are about completeness, it will bite you on exactly the questions you care about most.

I spent the last stretch of this year proving it out on a real archive: 296 PDFs, 11,794 scanned pages, twenty years of maintenance records for a business aircraft. I built a knowledge graph alongside the vector search that was already running, benchmarked the two head to head on seven questions, and instrumented every single API call so I could report costs instead of estimating them. Total spend to build the graph layer: about $53.

What follows is the plain-English version. No prior background assumed. If you want the full instrumented write-up with the token counts, the cost ledger and the engineering failures, the white paper is linked at the end.

First, what vector search actually is

Almost every "chat with your documents" product you have used in the last three years works the same way. It is worth understanding, because the limits fall directly out of the mechanism.

Step one: turn text into numbers. An embedding converts a chunk of text into a long list of numbers that captures its meaning. Chunks about similar topics end up numerically close together, even when they share no words at all. Ask about "turbine damage" and it will find a page that says "blade erosion," because the numbers land near each other. This is genuinely clever and it is why these systems feel like magic the first time.

Step two: chop the archive up. Your documents get split into thousands of small chunks, each one embedded and stored.

Step three: answer by similarity. When you ask a question, your question gets converted into numbers the same way, and the system returns the handful of chunks whose numbers sit closest. Usually ten or twenty. That handful is called the top-k.

Step four: write the answer from that handful. The AI model reads only those retrieved chunks and composes a response. This whole pattern has a name: RAG, short for retrieval-augmented generation.

Hold that last step in your head, because it is the whole ballgame. The model does not read your archive. It reads ten paragraphs someone else picked for it. Whatever those paragraphs contain becomes the answer. Whatever they leave out does not exist, as far as the model is concerned. And there is no error message for "the other thirty relevant pages were never retrieved."

The four things that break, and why more tuning will not fix them

These are not vendor problems. They follow logically from the design above, which is why swapping embedding models or raising k softens them but never removes them.

1. Truncation. Top-k physically caps every list. If the true answer is spread across forty pages and the system retrieves ten chunks that name seven vendors, you get seven vendors, delivered with total confidence. Raising k helps until it does not: context windows are finite and retrieval precision degrades as k grows. This one failure mode alone produced the three-times undercount I opened with.

2. Absence blindness. You cannot retrieve a chunk that does not exist. This is the big one, and the least discussed. Questions about what is missing are structurally unanswerable by similarity search. Which installed parts have no release paperwork. Which patients never got a follow-up. Which contracts lack a signed amendment. There is no page anywhere in the archive that says "this part has no paperwork," so there is nothing to retrieve. Answering requires knowing the full set of things that exist, the full set of documentation that exists, and subtracting. Similarity search cannot subtract. Asked one of these questions, a RAG system will either hedge honestly or guess fluently, and neither one gives you a punch list you can act on.

3. Fragmentation. Multi-document answers arrive in pieces. Real facts are scattered. One event might live partly in a logbook entry, partly in a work order, partly in a regulatory release tag, tied together by a shared serial number but stored in different boxes filled years apart. Top-k retrieval sees each fragment independently. It has no mechanism for recognizing that three excerpts describe one event, or that the entity in chunk 4 is the same entity as in chunk 17. The answer stitches together whatever arrived.

4. Contradiction invisibility. Your archive can disagree with itself forever. If two documents record the same serial number differently, because of a scanning error or an OCR misread of handwriting, a vector store holds both versions peacefully in separate chunks. Each bad reading only affects the queries that happen to retrieve it. Most questions see only the good pages. The system has no concept of "the same real-world thing," so it can never notice that its own data is corrupt. This can sit undetected for years, and in this project, it had.

The pattern behind all four: vector search operates on text that sounds relevant. The hard questions operate on things and their relationships. Those are different data structures, and no amount of retrieval tuning converts one into the other.

So what is a knowledge graph, in plain English

A different way of storing what your documents say.

Instead of keeping text chunks, you pull out the entities: every part, serial number, organization, work order, date, person. Each becomes a node, a dot in a network. Then you pull out the relationships between them, and each becomes an edge, a line connecting two dots.

Read as a sentence, one path through the network looks like this: fuel pump serial 0768 was installed on the left engine, which was overhauled by a repair station, under a specific work order, on a specific date.

Once facts live as a network instead of as prose, questions stop being searches and become walks. "Trace this serial number" literally means "start at this dot and follow every line touching it." And critically, you can now ask structural questions that have no textual answer. "Which parts have no paperwork" becomes "find every part node with zero paperwork edges." That is a query a database can run. It is not a search at all.

This pairing of a knowledge graph with an AI model has a name now: GraphRAG, popularized by Microsoft Research in 2024. The tooling has matured fast. I used cognee, an open-source Python library under Apache 2.0 with roughly 25,000 GitHub stars, which runs entirely on infrastructure you control. No new account, no subscription, no vendor holding your data. Its only outbound calls go to whatever AI API key you already have.

One subtlety worth stating, because it gets misreported constantly: a knowledge graph does not replace vector search. It is built on top of it. The graph still uses embeddings to find its starting points. The difference is what gets stored and what the AI reasons over at question time: connected facts instead of loose excerpts.

The test case: six boxes and twenty years

The corpus is real. Six boxes of maintenance documents for a midsize business jet in service more than two decades. 296 PDFs. 11,794 scanned pages. Logbooks, work orders, regulatory release tags, traceability bundles, manufacturer manuals. A mix of clean print, dot-matrix output, fax artifacts, rubber stamps, and twenty years of handwriting. The aircraft, its serial numbers and its vendors are anonymized here and in the paper, but every count, cost and result is reported exactly as measured.

I picked this archive because it is a graph-shaped dataset wearing a document costume. The same physical part shows up in a logbook entry, a release tag, a work-order envelope and a traceability bundle, connected by part number and serial number, but physically scattered across boxes filled over two decades. The questions that carry commercial weight are almost all join questions:

• Trace this serial number across every document it touches.
• Which installed parts have release paperwork, and which do not?
• What did this repair station do to this engine, ever?
• Is the maintenance timeline continuous, or are there gaps?

If that shape sounds familiar, it should. A litigation document set is the same structure: parties, agreements, amendments, correspondence, all cross-referencing. So is a patient's medical history. So is a supply chain certification trail, an M&A data room, a construction submittal log. Aviation records are simply an unusually honest specimen, because the joins are explicit and the ground truth is checkable against a physical scan.

Do the cheap thing first: OCR

Before any of the interesting work, one unglamorous step. OCR, optical character recognition, is software that looks at a picture of a page and converts it into machine-readable text. A scanned archive is a pile of pictures. Nothing downstream, not search, not AI, not a database, can use a picture directly.

The 9,995 pages that had never been through OCR went through a commercial service in 2.3 hours for $14.99, with zero failed pages. The result is 19 million characters of verbatim text sitting in a database, with a per-page confidence score attached.

If you take one tactical thing from this article, take this one. OCR your archive once, store the text and the confidence scores, and every AI experiment you run afterward gets an order of magnitude cheaper. That text layer is a permanent asset, independent of any tool or vendor you choose later. It cost fifteen dollars and it is the reason everything after this point was affordable.

The pilot: never scale what you have not measured

Before spending real money, I ran a deliberately small slice: 611 pages covering one engine's interlocking paper trail, chosen specifically because those documents should cross-reference each other, which makes the linking testable.

Measured, not estimated: 24 minutes of wall time, 1,345 AI calls on a cheap fast model tier, 2.46 million tokens in and 1.10 million out, $4.33 total, which is $0.0071 per page.

For contrast, an earlier approach on this same archive had a premium vision model look at page images directly. That ran $0.019 to $0.041 per page. Reading structure out of OCR text you already paid for is three to six times cheaper than making a model stare at pictures, and the model never touched a scanned image.

Seven questions, two systems, one clear result

I wrote seven questions, each one aimed at a specific failure mode above, and ran them against both systems on the full corpus.

The standout was the absence question: "Which parts or components in these records do NOT have any matching release paperwork?"

Vector search behaved exactly as the theory predicts. It hedged. It surfaced one component marked unknown and then conceded that the records do not provide matching tags for every sub-component listed. Honest, and completely useless as a to-do list.

The graph returned an enumerated list of 18 components lacking release paperwork, each with its part number and serial number. Disks, an impeller, a shaft, a seal plate, nozzle assemblies. That is a document-the-gaps punch list an auditor can work from on Monday morning. Vector search did not produce one, and never will, for the structural reason above.

The rest ran the same direction:

• Full maintenance timeline: the graph assembled roughly 55 events spanning 2002 to 2023, drawn from at least seven documents across multiple boxes, each tagged with its source. Vector search managed about 15 with visible holes.
• Enumerate every organization: 22 organizations with their certificate numbers and roles, versus 7. The additions were component manufacturers, an oil-analysis lab, a non-destructive testing shop, rotables shops. Vector search was not wrong. It just stopped at what its ten retrieved excerpts happened to mention.
• Per-organization rollup: complete across three work orders spanning seven years, each with its parts list. Each system had previously held half the picture. The graph had all of it.
• Temporal gaps: this one is the punchline. Vector search confidently reported a five-year maintenance gap. The graph's own timeline refutes it, showing documented activity in every one of those years that retrieval simply never surfaced. The graph instead found a single quieter stretch of about 3.6 years, which lines up with a change of ownership.

Read that last one carefully, because it is the whole argument in one result. The vector system did not just come up short. It manufactured a finding that was not true, from incompleteness it had no way to detect, and it delivered that finding in the same confident tone as everything else.

The part where I was wrong

Round one of this benchmark was not a fair fight, and the graph lost a question because of it. At that point vector search had the whole archive indexed and the graph only had the 611-page pilot slice. Asked about maintenance gaps, the graph reported a 7.8-year gap. Wrong. The work existed, in documents the graph had never been given.

The lesson is not that graphs are bad at time. It is that a graph is confidently wrong about exactly the material it was never given. An incomplete graph does not say "I don't know." It says "there's a gap," and it sounds authoritative doing it.

That is why the full-library build had to happen before any production judgment. It also cuts against the easy version of this article. Both architectures can be confidently wrong. The difference is that the graph's incompleteness was a one-time corpus problem that about $48 permanently fixed, while vector search's incompleteness is per-question, structural, and permanent.

The bonus nobody promised: the graph audits your data

This was the genuine surprise of the project.

During testing I asked a boring sanity question: list the engines and the auxiliary power unit with their serial numbers. The graph answered, then hedged. It reported the unit's serial as ending in 690, "also identified as" ending in 609.

The registry and the logbook covers agree on 690. So why the hedge? Because somewhere in 11,794 pages, at least one document records that serial with the final digits transposed. A handwritten 9 read as a 0, a smudged stamp, a worn carbon copy. Ordinary handwriting-digitization noise.

A vector system would never notice. The bad reading just sits in one chunk, waiting to mislead whichever question happens to retrieve it, while every other question sees the good pages. The graph cannot ignore it, because the entire point of a knowledge graph is one node per real-world thing. When mentions disagree about an identifier, the conflict becomes visible in the structure. Both serials end up hanging off the same node, and the answer surfaces both instead of silently picking one.

Vector search hides data-quality problems. A knowledge graph concentrates them. It does not just answer questions, it audits the archive as a side effect of existing.

The follow-on to that was a review queue that sorts pages by OCR confidence, worst first, so a human only ever looks at pages the machine is already unsure about. Nobody proofreads 11,794 pages and nobody should. The first item it surfaced was the single worst page in the archive at 54% confidence, whose text began 2NLEKCEDED · BICKED. Look at that for a second. It is "SUPERSEDED" and "AIRPLANE" read upside down. The page had been scanned rotated 180 degrees, the confidence score collapsed exactly as it should have, and the queue put it at position one. No AI heroics. The scoring did the finding and a human does the two-minute fix.

What it cost, measured rather than guessed

Every figure here comes from token counting hooks on live calls, not from a pricing calculator.

• Vector search indexing, whole archive: ~$3
• Graph pilot, 611 pages: $4.33
• OCR backfill, 9,995 pages: $14.99
• Full-corpus graph build, including the waste from crashed runs: ~$33
• Probes, smoke tests, queries: ~$1
• Total: about $53. The originally planned alternative, running a premium vision model over all six boxes, was estimated at $215 to $425 and was avoided.

The number worth internalizing is not $53. It is $0.003 to $0.007 per page for graph extraction over OCR text you already have, versus $0.02 to $0.04 per page for vision-model extraction. At archive scale, that ratio is the difference between a project that gets approved and one that does not.

Three engineering traps that cost real money

I am including these because a build-versus-buy decision depends on them as much as on any benchmark, and because two of them will save you money on your very next AI project regardless of what you are building.

1. Hidden thinking-token billing. This is the expensive one. Modern AI models can privately "think" before answering, generating internal scratch-work you never see. That scratch-work is billed to you as output tokens. A model that thinks for 700 tokens and answers in 180 charges you for 880. On this project I measured 75% of all billed output going to invisible thinking before I explicitly turned it off. Disabling it cut costs by 3.2 times with no drop in extraction quality. The standing rule I now apply everywhere: never run bulk work on a reasoning model without explicitly configuring the thinking budget, and always meter actual token usage with a counting hook instead of trusting an estimate.

2. Turning thinking off has its own failure mode. With reasoning fully disabled, roughly 10% of extraction calls came back malformed. The model would echo the format definition back at me instead of filling it in, or append garbage after valid data. Demand rigid machine-readable output from a model that is not allowed to think and it occasionally sleepwalks. The fix was a small thinking budget for just that call type, retries, and per-document checkpoints so one bad response cannot kill an hours-long run.

3. Batch tools have a preferred grain, and it is not obvious. Feeding the engine 11,794 individual page items stalled its loader for three hours, because the bookkeeping overhead is per item. Regrouping the identical text into 296 per-document files with inline page markers loaded in 45 seconds. Same content, roughly 200 times the difference. Find that grain experimentally on a small batch before you scale.

If that pattern of AI tools quietly doing the expensive or lazy thing while you are not watching sounds familiar, I wrote about it at length in The Lazy Genius in the Room.

The architecture that actually shipped: both, not either

The honest conclusion is not that graphs beat vector search. It is that they answer different question classes, fail in different ways, and are both cheap enough to run side by side.

• Search and cite: vector search, unchanged. "Find the page that says X." Narrative retrieval with citations, 7 to 20 second answers.
• Reason and audit: the knowledge graph. Traces, joins, rollups, enumerations, completeness and absence checks. 30 to 120 second answers.
• Verbatim record: the OCR text and confidence scores in plain SQL. Full-text search, future extractions, ground truth.

In production the chat layer routes by question shape. "Where does it say..." goes to the fast path. "Trace, list all, what is missing..." goes to the graph. Voice interfaces stay on the fast path permanently, because a spoken conversation cannot absorb a minute of dead air.

The interface has to teach that choice, which turned out to matter as much as the retrieval work. The deep mode is a visually distinct color, the input placeholder changes to nudge toward the questions the graph is good at, and the loading state shows a live elapsed counter with rotating hints that narrate the real mechanics. A 60-second spinner with no explanation reads as broken. A 60-second spinner that says "following serial numbers between logbooks and release tags" reads as work being done, which is the difference between someone using the feature twice and never using it again.

Should you build one? A short honest test

A knowledge graph is not a default upgrade. It is a real pipeline with an extraction budget, retry logic, failure modes and an artifact you own and rebuild. Plenty of RAG deployments genuinely do not need one.

Your archive is probably graph-shaped if: the same real-world entities recur across many documents under shared identifiers; the questions that matter are joins, traces, rollups or enumerations rather than "find the passage"; absence questions carry commercial or regulatory weight; every claim needs a citation trail back to a source page; or the archive accumulated over years from multiple sources, which makes internal contradictions likely and worth surfacing.

You probably do not need one if: your questions are overwhelmingly narrative and local; your corpus is small enough that generous retrieval into a large context window effectively reads everything; nobody will act differently on a completeness guarantee; or you cannot commit to operating a pipeline.

If you do proceed, in this order: OCR everything once and store the confidence scores. Pilot on a deliberately interlocking slice and meter every call. Write your benchmark questions before the build and run them against your existing search first, so the comparison is honest. Cap model thinking for bulk work. Only scale after a fair test on equal data. Then ship it as a mode beside your fast path, not as a replacement.

The part that generalizes

Strip out the aircraft and the finding is architectural.

Text that sounds relevant and things with relationships are different data structures. A system built on the first cannot be tuned into answering questions about the second, no matter how good the underlying model gets. If the questions that matter in your domain are about completeness, connection and absence, a graph is not an optimization of your AI search stack. It is the missing half of it.

And the cost of finding out is now small enough that not testing it is the expensive choice. Fifty-three dollars, on an archive that had been sitting in boxes for twenty years, produced a completeness audit, a missing-paperwork punch list, a corrected timeline, and two data-quality defects nobody knew were there.

This is the kind of work I do: taking an AI capability that demos beautifully and finding out where it actually breaks before a client's money is on it. If you want to see more of it, the A.I. portfolio covers the production systems, and How I Work covers the process. If you have an archive and a question you suspect your current search cannot answer, get in touch and I will tell you honestly whether this is worth your money.