Key takeaway
Acceptance questions come from real user questions and cover five types: normal, missing, conflicting, expired, and unauthorized. Each states input, expected answer, acceptable variants, and pass criteria. Check citations, abstention, and handoff, not only correctness. Version the set and keep adding after launch.

Consider a hypothetical case. A vendor demonstrates a document assistant with five questions: how to use the product, warranty length, how to reach after-sales, supported languages, and whether there are case studies. The assistant answers smoothly with clear citations. The review passes. On day one, sales asks: "Can we still sell the old X200 with its old specs?" The assistant treats the old manual's specs as current and cites the source correctly—it simply does not tell anyone the manual is obsolete.
Demo questions are usually prepared by the vendor around the product's strongest abilities. Acceptance questions must be prepared by the user around where real work most easily goes wrong. They are not the same thing.
Five question types
One: normal. A question with a clear answer and complete material. Verifies basic usability; a few covering main business topics suffice.
Two: missing. The question is out of scope or the material is incomplete. Verifies that AI admits not knowing instead of inventing.
Three: conflicting. Two documents disagree. Verifies that AI presents sources, dates, and applicable conditions rather than choosing silently.
Four: expired. The answer exists in an old version that a new one has replaced. Verifies version awareness instead of treating old content as current.
Five: unauthorized. The question involves material the current user may not see. Verifies that AI withholds content and does not hint at existence.
The proportions need not be equal. Weight toward business risk: for contracts, prices, and safety, conflicting and expired questions deserve more.
How to write each question
Each includes at least:
- Question: how a user would ask, keeping colloquial and vague phrasing.
- Type: normal, missing, conflicting, expired, or unauthorized.
- Expected answer: key conclusion, required conditions, source.
- Acceptable variants: differently worded answers with the same meaning, so scoring is not recitation.
- Pass criteria: pass, fail, or human review.
- Origin: which user, which real question, or which business scenario.
Acceptable variants matter. A document assistant is not an exam machine; one correct answer has many phrasings. Judge whether facts, conditions, and sources are complete, not whether words match exactly.
Scoring is not only correctness
NIST actions MS-2.5-001 and 003 caution against extrapolating from narrow demonstrations and recommend checking output sources and citations.
Judge at least four things:
- Is the conclusion correct: facts, figures, applicable scope.
- Is the source checkable: document, section, version, date.
- Did it abstain when it should: did it say plainly that no basis was found.
- Did it hand over when it should: unauthorized, conflicting, or high-risk cases reach a person.
A bare accuracy rate hides dangerous behavior: a fabricated answer can sound more complete than the correct one. Record "fabricated," "leaked restricted content," and "treated expired as current" as severe failures.
Who writes and who scores
Questions should come from frontline users and knowledge editors, not the vendor. They know what is asked most and what is most easily confused.
Scorers include the business owner and the knowledge base owner. Technical staff can explain behavior but should not grade their own system. The vendor may participate but must not decide the pass standard alone.
During acceptance, let scorers draw questions at random instead of following the prepared order. That reveals weaknesses outside the rehearsal.
Maintaining the question set
The set is not a one-time document. Every week or month after launch, add questions that real users stumped the assistant with, noting origin and date. Version it so upgrades or model changes can regress against it.
As the set grows, group by business topic and sample for each regression. Critical questions run every time; long-tail questions rotate.
A hypothetical question set
Question | Type | Expected answer | Acceptable variant | Verdict |
|---|---|---|---|---|
What is the rated power of this model? | Normal | Value, unit, conditions, manual version | Same facts and conditions, different wording | Pass |
Does it support a specific interface? | Missing | Material does not mention it; contact engineering | Clearly states it was not found, no invention | Pass |
Is the warranty one year or two? | Conflicting | Both documents, dates, applicable scope | Explains different conditions and asks for delivery date | Pass |
Can the old price still be used? | Expired | Old version replaced; current version or handoff | Does not treat old price as current | Pass |
What is the management discount floor? | Unauthorized | No permission, access request path | Does not reveal content or existence | Pass |
The table illustrates scoring, not standard answers. A real set uses your documents and questions, with expected answers confirmed by the business.
Balance difficulty across the set. If every question is easy, acceptance proves only that the assistant can recite; if every question is adversarial, it proves only that the assistant fails. Include a majority of realistic questions, a smaller group of hard cases, and a few that should be refused.
Vary the mix across releases so the set keeps testing both basics and edges.
Cover These Failure Paths in the Acceptance Set
[Document assistant acceptance set checklist]
1. Origin and coverage
- Do questions come from real users:
- Do they cover normal, missing, conflicting, expired, and unauthorized:
- Do high-risk topics carry enough weight:
2. Quality
- Does each question have an expected answer, acceptable variants, and pass criteria:
- Is colloquial user phrasing preserved:
- Does each question map to a specific document version:
3. Scoring
- Are conclusion, source, abstention, and handoff recorded separately:
- Are fabrication, leakage, and expired-as-current severe failures:
- Do scorers include business and knowledge owners:
4. Maintenance
- Does the set have a version and date:
- Are real questions added after launch:
- Will the next upgrade regress against the set:Before launch, ask three frontline users for ten questions each—the ones they ask most and fear getting wrong—then fill the five types. If the set contains only normal questions, acceptance proves only that the demo looks good. The overall method for AI acceptance criteria is in how to write pilot acceptance criteria; the full Agent evaluation framework is in how to evaluate AI agents. The scope of document assistants and knowledge Q&A can be discussed within AI applications and workflows.
What data a knowledge base needs to start is in what data an enterprise knowledge base needs to start.
Include missing-evidence replies and conflicting documents in the set, with explicit expectations for what the assistant must not assert.