How should teams test retrieval quality in a private knowledge base?

Decision: adopt gold questions and missing-document tests as a promotion gate for private retrieval—not as an open-ended quality program. The bounded use case is simple. Before you ship an index rebuild, re-chunk, ACL change, or connector refresh, run a fixed, versioned set of asks whose answers already live in the corpus, plus a paired set whose answers should not.

A gold question is a natural-language ask tied to a known source span you expect retrieval to surface, with failure when the system returns the wrong neighbor or invents a path around that span. A missing-document test is the inverse: withhold or remove the only authoritative source, ask for the fact it alone held, and require abstention or an explicit not-found path—especially when stale copies or over-broad permissions still leave adjacent material in scope.

The control that makes this real: store both sets with the same commit as the index snapshot, and block promotion if gold coverage slips or any missing-document case returns a grounded answer.

Draw the line there. Do not stretch the harness into open-ended analytical scoring, every role-permission edge, or continuous “freshness.” The non-obvious consequence is quieter: teams start writing gold asks that match what retrieval already rewards, and ranking and chunking decisions freeze under the appearance of rigor.

Retrieval quality is the wrong primary score for a private knowledge base. Teams keep polishing gold-question relevance while the real failure mode is quieter: a fluent answer that crossed a permission boundary, or a miss that cannot be reconstructed after the fact.

Relevance without access control is not quality. It is leakage with good ranking. Before you trust any gold set, run a dual-identity probe on the same query string under two authenticated principals—one entitled to the source, one not. Capture the full retrieval trace for both: doc IDs considered, ACL decision, rank, and final surface. The entitled path must surface the expected material; the denied path must return no fragment, no snippet, no citation from that material. Missing-doc and stale cases belong in the same harness: assert that absence is explicit and attributable, not a soft substitute answer drawn from adjacent open content.

If you only measure answer quality, you will ship systems that look smart on authorized gold questions and still fail audit when the wrong principal sees the right fact.

Ship nothing broad until a sealed, reversible pilot proves the retrieval change holds under the failure modes that actually break private knowledge bases.

Freeze a compact gold set before any index, chunker, or ranking tweak: known-answer questions, deliberately missing-doc queries that should return empty or refuse, the same topical question under two permission scopes, and at least one item whose authoritative source has been aged or superseded. Run the current stack once, record only ranked hits and cited sources, apply the proposal, re-run against the identical freeze, then do a side-by-side failure review with whoever owns the material. No dashboard theater—just pass/fail on answerability, access fidelity, and currency.

The non-obvious consequence is that relevance can look better while permission leakage and stale resurrection get worse; gold-only scoring rarely surfaces those. Reject the pilot—and the proposal with it—if the base cannot express access boundaries or document currency clearly enough to include the missing-doc and permission cases. Without those controls, the test will not hold under real operating conditions, and you should fix access-and-freshness plumbing first.