How to get good answers from your own documents, without over-building it

Here's the point up front, because it's the thing I most want you to leave with. If you want an AI to answer questions from your own documents, start with the simplest setup that works, and only add machinery when the size of the problem actually forces you to. Building the complicated version early is one of the most common and expensive mistakes I see, and it often makes the answers worse, not better.

If you already know your embeddings from your rerankers, skip to the summary. Otherwise, stay with me and I'll define everything as it comes up.

The problem, and the three things you're trading off

You have a body of information. Notes, docs, a folder of PDFs, a knowledge base. You want to hand it to an AI and get accurate answers back. The moment people frame it that way, the next sentence is almost always "so we need a vector database." Hold on.

Before reaching for that, it helps to know what you're actually choosing between, because only three things really matter here:

• How accurate the answers are. This is the whole point.
• How complicated the plumbing is to build and to keep running.
• How much it costs.

Every option below is a different trade-off between those three. The skill is picking the simplest one that gets you the accuracy you need, and stopping there. I'll walk them from simplest to most complex.

Simplest: just give the AI the documents

Modern models can read a lot of text at once. The amount they can hold in mind for a single question is called the context window, and it's now large enough that a surprising number of real document sets fit inside it whole. If yours does, you don't need a search system at all. You give the model everything and ask your question.

You can make this cheap with prompt caching, which means the model keeps your documents ready after the first read, so you aren't paying to re-read them on every question. Cached reads cost roughly a tenth of the normal price. A modest set of documents, asked a hundred questions, can cost well under a dollar this way, with nothing to build and no loss of detail. For a lot of real cases this is the entire answer, and people skip past it because it feels too easy to be the sophisticated choice.

When it's too big to hand over: retrieval

Once your documents no longer fit in the window, you need retrieval, the practice usually called retrieval-augmented generation, or RAG. It just means pulling out the handful of relevant pieces and showing the AI only those. This is where people jump to a vector database, and this is where the over-building tends to start.

Two things are worth knowing before you do. First, vector search (matching by meaning rather than by exact words, using numerical fingerprints called embeddings) on its own is usually not the best option. The setup that consistently wins in practice is hybrid search: run keyword matching (the classic algorithm is BM25) and meaning-based matching side by side, combine the two result lists (a step called reciprocal rank fusion), then add a reranking pass where a slower, more careful model called a cross-encoder reorders the shortlist so the strongest answers rise to the top. Keyword matching alone misses anything phrased differently. Meaning-based matching alone fumbles exact things like product names and reference numbers. Put together they cover each other, and on standard retrieval benchmarks the pair beats either one alone by a clear margin, measured in NDCG, a ranking-quality score. (Benchmark source link to add.) If you do add the meaning-based layer, Anthropic's Contextual Retrieval is the highest-value, low-effort upgrade to reach for first.

Second, and this is the part that genuinely surprised me: the simpler approach can be more accurate, not only cheaper. When Anthropic built Claude Code, they dropped the whole vector-database approach in favour of plain text search, the same grep command that's been on every computer for decades. It scored higher on correctness, 8.4 out of 10 against 6.4 for the vector-database version, in one independent benchmark. (Source link to add; the score is that benchmark's own correctness rating.) A coding assistant searching a codebase is exactly the case you'd expect to need the fancy version, and the simple one won on the thing that matters. Cursor, Windsurf and others made the same move. Anthropic's own guidance now is to start with simple search and add the meaning-based layer only if you actually need it.

Only at real scale: the heavy machinery

Below a few hundred thousand items you do not need dedicated vector-database software (products like Qdrant or Milvus, whose only job is storing and searching by meaning). A normal database with a search add-on, such as Postgres with the pgvector extension, does the job for far longer than people assume, and it's one less thing to run.

The heaviest option is GraphRAG, a Microsoft approach that first builds a map of how everything in your documents connects, so it can answer questions that need joining many pieces together ("what are the themes across all of this?"). Building that map costs several times more, and it only pays off when your questions genuinely need that kind of joining. Most questions are answered from one or two places in the documents, and for those the map is pure cost.

A shortcut if your data is already questions and answers

If your information is already in question-and-answer pairs (a support archive, an FAQ, past RFP responses), you have a real advantage, so use it. Match the incoming question against your stored questions, not against the stored answers (practitioners call this question-to-question, or q2q, matching). It is far easier to accurately match two short questions to each other than to match a short question against a long answer, and it works out of the box with no training. Then clean up near-duplicate questions down to one good answer each, so you aren't surfacing five slightly different old answers to the same thing. That cleanup, done by a person, does more for accuracy than almost anything you can do on the AI side.

What I run, and how I know when to upgrade

My own system is a few hundred markdown files. The right search tool for that is plain text search, plus filtering on the frontmatter, the small labelled header at the top of each file (topic, date, status). Filtering on that narrows the field before I even search, which is most of the accuracy right there.

I copied this deliberately, not from first principles. The teams building the best coding agents all landed on search-first over vector databases, and Andrej Karpathy is the person I try to write and think like on this: explain the complicated thing simply, and don't add machinery you can't justify. The part that keeps me honest is that I know my upgrade thresholds. Around a thousand files I'd move to a proper local search index. Around ten thousand, the meaning-based layer starts to earn its place. I'm nowhere near either, so I've built neither. Watching a real threshold instead of guessing is what stops me over-building on a hunch.

The downside of over-building early

It's worth being explicit about the cost of the mistake, since avoiding it is the whole point. Every piece of machinery you add is something you now own forever. You maintain it, you pay to run it, and you debug it when it breaks. Worst of all, the most error-prone step in these systems, chunking (chopping documents into pieces small enough to search over, and hoping each piece still makes sense on its own), is a step the simplest setups skip completely. Over-build early and you take on all of that for a capability you often didn't need, and, as the coding-agent example shows, sometimes for worse answers than you started with.

Summary

Match the tool to the size of the problem.

• Documents fit in the model's context? Give them to the AI directly and cache them. Cheapest, simplest, full detail.
• Too big for that? Add hybrid search (keyword plus meaning) with a reranking pass.
• Genuinely large? Then, and only then, a dedicated vector database, or a graph if your questions need joining lots of pieces together.
• Data already in question-and-answer form? Match question to question, and dedupe to one good answer each.

Pick the rung you're standing on, not the one at the top. The person selling you the top rung isn't the one who has to keep it running.