How should teams test retrieval quality in a private knowledge base?

Decision: adopt gold questions and missing-document tests as a promotion gate for private retrieval—not as an open-ended quality program. The bounded use case is simple. Before you ship an index rebuild, re-chunk, ACL change, or connector refresh, run a fixed, versioned set of asks whose answers already live in the corpus, plus a paired set whose answers should not.

A gold question is a natural-language ask tied to a known source span you expect retrieval to surface, with failure when the system returns the wrong neighbor or invents a path around that span. A missing-document test is the inverse: withhold or remove the only authoritative source, ask for the fact it alone held, and require abstention or an explicit not-found path—especially when stale copies or over-broad permissions still leave adjacent material in scope.

The control that makes this real: store both sets with the same commit as the index snapshot, and block promotion if gold coverage slips or any missing-document case returns a grounded answer.

Draw the line there. Do not stretch the harness into open-ended analytical scoring, every role-permission edge, or continuous “freshness.” The non-obvious consequence is quieter: teams start writing gold asks that match what retrieval already rewards, and ranking and chunking decisions freeze under the appearance of rigor.

Retrieval quality is the wrong primary score for a private knowledge base. Teams keep polishing gold-question relevance while the real failure mode is quieter: a fluent answer that crossed a permission boundary, or a miss that cannot be reconstructed after the fact.

Relevance without access control is not quality. It is leakage with good ranking. Before you trust any gold set, run a dual-identity probe on the same query string under two authenticated principals—one entitled to the source, one not. Capture the full retrieval trace for both: doc IDs considered, ACL decision, rank, and final surface. The entitled path must surface the expected material; the denied path must return no fragment, no snippet, no citation from that material. Missing-doc and stale cases belong in the same harness: assert that absence is explicit and attributable, not a soft substitute answer drawn from adjacent open content.

If you only measure answer quality, you will ship systems that look smart on authorized gold questions and still fail audit when the wrong principal sees the right fact.