Define what a supported answer looks like
A retrieval-augmented generation system has at least two opportunities to fail: retrieving unsuitable evidence and generating an unsuitable answer from the evidence it receives. For an employee policy assistant, success means more than fluent text. The answer should use current, authorized policy documents, address the question, cite the relevant passage, and acknowledge when evidence is insufficient.
Write a rubric before changing chunk sizes or prompts. Define acceptable completeness, factual support, citation quality, and refusal behavior. Include business constraints such as latency and permitted document collections. OpenAI's evaluation guidance recommends task-specific tests and combining metrics with human judgment rather than relying on a single general benchmark. [1](https://developers.openai.com/api/docs/guides/evaluation-best-practices)
Create a small but difficult question collection
Start with questions drawn from authorized support requests or realistic scenarios. Include direct lookups, questions requiring two passages, ambiguous wording, outdated terminology, and questions the collection cannot answer. Keep development questions separate from a held-out set used to compare final candidates.
For each item, record the question, expected answer characteristics, acceptable evidence passages, document version, and user access role. A benefits question may have different correct evidence for different countries. Store those distinctions rather than pretending every question has one universal answer. Have a domain reviewer check a sample, especially cases where policies conflict or where several differently worded answers would be valid.
Evaluate retrieval without the generator
Run each question through retrieval and save the ranked document identifiers and passage text. Measure whether the evidence needed to answer appears within the retrieved context. Recall at a chosen retrieval depth is useful when you have sufficiently complete relevance labels; missing labels can otherwise make a useful passage look incorrect.
Inspect misses manually. Was the document absent, indexed badly, split across chunks, filtered out, or ranked too low? Try one change at a time, such as preserving section headings or adding a reranker. Increasing retrieval depth may improve evidence coverage while also increasing cost, latency, and distracting context. Record those trade-offs rather than treating more passages as automatically better.
Score answers against their actual evidence
Evaluate factual support separately from answer correctness. An answer might happen to be true while citing a passage that does not support it. Conversely, an answer can faithfully repeat an outdated document and still be wrong for the current policy. Reviewers need the question, generated answer, retrieved passages, and approved reference material to distinguish these failures.
Use clear labels such as supported, unsupported, incomplete, and appropriately abstained. If an automated grader helps triage results, calibrate it against human judgments and review disagreements. Do not ask the same unexamined model to generate answers and certify their quality without checks. Treat grader prompts and model versions as part of the experiment configuration.
Test permissions and adversarial content
Create tests in which a user asks about a restricted policy, a retrieved document contains malicious instructions, or a query requests confidential information. Enforce access control before unauthorized passages reach the model. Prompt wording alone is not an access-control mechanism, and a convincing citation does not prove that a user was entitled to see the source.
OpenAI's safety guidance recommends adversarial testing and human oversight, especially when consequences are significant. [2](https://developers.openai.com/api/docs/guides/safety-best-practices) For this assistant, require escalation when policies conflict and prevent document text from changing application permissions. Preserve enough redacted diagnostic detail to investigate failures without logging private employee questions unnecessarily.
Release with a regression report
Compare candidate configurations on the same versioned questions. Report retrieval coverage, answer support, appropriate abstention, access-control failures, latency, and estimated operating cost separately. A combined score can conceal a serious regression, so identify non-negotiable release conditions such as no observed unauthorized-document exposure in the test suite.
After a release, collect reviewed failure examples and add them to future evaluations without endlessly tuning against the held-out set. Re-evaluate when documents, indexing, prompts, or models change. The practical deliverable is an evidence-backed release note explaining what improved, what worsened, and which question types still need a human, not a claim that hallucinations have been eliminated.
Sources & further reading
Last editorial review: 27 October 2026. Examples are educational, not live employer data or guaranteed outcomes.