Your documents aren't ready for AI yet. Here's how to fix that.
An assistant on your own documents sounds simple: drop in the folder and ask a question. In practice you get answers that sound plausible and are just slightly wrong. Almost always that isn't the model, it's what you put in.
The problem is retrieval, not answering
This kind of assistant works in two steps. First it finds the passages that appear to match your question, then it has the model write an answer based on exactly those passages. If step one goes wrong, step two is hopeless: the model does its best with the wrong source and sounds as confident as ever. All the gains sit in retrieval.
Intervention 1: split on meaning, not character count
The default approach slices documents into blocks of a fixed number of characters. A table gets cut in half, a list is separated from its introduction, and the heading the rule belongs to disappears. Split on the document's structure instead: per heading, per article, per procedure step. For policy terms that's per article; for a manual, per section.
Intervention 2: give every passage its context back
A loose paragraph reading “this does not apply to participants under 18” is worthless. Record where each passage comes from: document name, version, date, heading and the path above it. Two things then become possible. You can filter on validity, and you can show the source with every answer. That citation isn't a nice extra, it's the difference between a tool people dare to use and a tool nobody trusts.
Intervention 3: clean the archive before you start
The most underestimated step. Almost every shared folder holds three versions of the same protocol, two of them outdated. A human sees from the filename which one applies; the search system doesn't, it finds the old one just as relevant. One afternoon of cleaning and one agreement about where the current version lives does more for quality than a more expensive model.
Intervention 4: write down what “good” means
List thirty questions your team actually asks, with the right answer and the document it's in. That list is your measuring stick: every change gets tested against it. Measure two things separately, did the system surface the right document, and was the answer correct. Then a bad result tells you which side to work on. And explicitly instruct the system to say “that isn't in my documents”; an assistant that never says “I don't know” is inventing.
The short version
Split on structure. Keep provenance and show it as a source. Clear out outdated versions. And write down thirty question-answer pairs before you start. Four interventions, none of which needs a bigger model.
Want to know whether your documents are ready?
Book your free intro call →