How to build an AI knowledge base

By Dylan Bailey, Certified Claude Expert · Updated 4 August 2026 · 6 min read

"Can we get AI to answer questions from our own documents?" is one of the most common requests, and one of the most commonly disappointing projects. The technology is the easy part. The reason it underdelivers is almost always the documents.

Why pointing it at everything fails

A typical shared drive contains the current procedure, two earlier versions, a draft nobody finished, a client-specific variation, and someone's notes from a meeting in 2019 — all as plain files with no indication of which is authoritative.

A person navigates this using context: they know Sharon left, that project finished, that folder is dead. The AI has none of that. Every document is equally true, and it will answer from whichever is textually closest to the question.

The result is an assistant that confidently states a policy you replaced two years ago — and does it in the same tone as everything else it says.

Start with questions, not documents

Before gathering anything, write down the twenty questions people actually ask. Not what you think they should ask — what lands in inboxes and gets asked across the desk.

This list does three things:

• Tells you which documents you actually need
• Reveals answers that don't exist in writing anywhere — often the most valuable finding
• Becomes your test set, so you can tell whether it works

Curate hard

• One current version of each thing. Archive the rest somewhere the system can't see.
• Give each document an owner and a review date. Unowned documents go stale silently.
• Write the missing answers down. If three of your twenty questions have no document, that's the highest-value writing you'll do.
• Prefer clear prose. Headings, short sections, plain language. Material written for humans in a hurry works well for retrieval too.
• Watch the exceptions. If a rule has carve-outs, keep them in the same section. Split apart, retrieval may surface one without the other.

An Australian small-business example

A Brisbane logistics company connects an assistant to their entire operations folder — around 1,200 files. Early testing looks impressive.

Then a dispatcher asks about dangerous goods procedure and gets an answer from a 2021 document superseded after a regulatory change. It's confident, detailed, and wrong in a way that matters.

They rebuild with 34 documents: current procedures only, each with a named owner, plus six newly written to cover gaps the question list exposed. Answers now cite the document they came from.

It covers less ground and is trusted far more — which is the trade worth making, because an assistant people don't trust gets abandoned regardless of coverage.

Make it show its working

Whatever tool you use, answers should cite which document they came from, and ideally link to it. This matters more than it sounds:

• Staff can verify without asking anyone
• Wrong answers become traceable to a specific stale document you can then fix
• It sets the right expectation — this is a finding aid, not an oracle

Keeping it alive

A knowledge base is a living thing. Set a quarterly review: check the owner list, remove anything superseded, and re-run your twenty test questions.

That last step is the one that gets skipped, and it's the one that catches slow decay. A knowledge base that was accurate at launch and unmaintained for a year is worse than none, because people have learned to trust it.