Building an AI knowledge base: RAG choices, documents, and answer checks
Build a knowledge base around a real class of questions. Inspect document extraction and retrieval before checking whether answers are supported. This guide covers when RAG is useful, missing evidence, stale material, and access control.
First establish which question the sources can answer
A knowledge-base answer needs retrievable supporting material before the model writes a response. For an initial trial, choose documents with clear provenance and identifiable versions, then record the supporting passages for common questions. After importing files, inspect the chat answer separately from the retrieved evidence, and record how it cites the source.
Decide between one-off reading and an ongoing knowledge base
For a one-off analysis of one short document, supplying the text directly may be enough. Retrieval becomes useful for repeated questions over changing material or for collections with user-specific access. Fitting a document into the context window does not establish that every relevant detail will be used. Evaluate the choice with your own questions.
RAG and fine-tuning address different needs. If an answer remains stale after company documents change, first inspect document versions and the index rather than immediately training a model. Consistent output style may require separate work on prompts or training. Retrieval is not a universal fix.
Inspect extracted text before indexing
A document can look correct in a reader while extracted text loses its row and column relationships. Sample pages with tables and compare which value belongs to which field. Scanned PDFs without a usable text layer need OCR. Changing an image’s extension to PDF creates neither readable text nor reliable table structure.
Keep the original when cleaning text so that line breaks and numbered clauses are not removed indiscriminately. They can distinguish conditions or records. When using the HotAI text-cleaning tool, first try the selected operation on a short anonymous sample and compare the output. Neat formatting does not establish correct meaning.
Keep each passage with its provenance
After chunking, each passage should remain traceable to its source page or paragraph. If an exception is separated from a general rule, the model may receive only the rule. Include conditional passages when inspecting boundaries. Preserve document versions and accessible source locations so disputed answers can be checked against the same original.
Inspect the returned passages before changing their count. A larger top-k can add relevant evidence but also stale or similar-looking irrelevant passages. Test exact matches separately for product identifiers, and test semantic retrieval for natural-language descriptions. Compare hybrid retrieval when both needs occur.
Record retrieval and answer support separately
Keep the expected supporting passage for each trial question and record whether it was retrieved. If the passage was found but the answer is wrong, the generation step may have ignored a condition. If it was not found, rephrasing may merely hide the failure. Retain the same questions to see whether a chunking change actually addresses the previous errors.
Reserve questions that were not used for tuning. When the collection has no answer, an appropriate response may state that evidence is missing. Record the failing stage rather than hiding retrieval failures in a single overall score. This guide does not measure the accuracy of any knowledge-base product.
| Trial question | Evidence to inspect | Next step |
|---|---|---|
| Answer exists in the collection | The supporting passage is returned and the answer agrees with it | Check whether exceptions were retained |
| No answer exists in the collection | The system identifies the evidence gap | Add the relevant source and retrieve again |
| An older version also exists | The citation points to the current applicable version | Fix updates or version filtering |
Check access and updates before release
Test a shared knowledge base with accounts that have different access rights. Documents a user cannot access should not enter that user’s answer context as retrieval results. A promise in the model’s instructions is not an access control. For private material, verify the platform’s data handling and retention terms before importing it. An anonymous trial establishes only its own result.
Update a known source, repeat the same question, and inspect whether the citation now reflects the change. Run the same check for removed or retired material. Retain failures after fixing them so they can be replayed when changing the model or retrieval strategy. A plausible-looking answer should not be the only acceptance criterion.
Method reference
These sources explain methods and technical conditions. This is a task guide; samples and results from HotAI tests are documented separately in the linked tested guides.