What is the cleanest way to evaluate agent memory without trusting vendor demos?

The cleanest way to evaluate agent memory is not a vendor walkthrough and not a cross-product ranking. Build a harness around one bounded workflow you already operate—support triage, internal SOP lookup, or account-context answers—and freeze that scope for the full test window. Outside it, treat memory claims as unmeasured marketing.

Seed a small, versioned corpus with ground-truth pairs: what must be retrieved, what must stay hidden, and what should go stale after a deliberate cutover. One concrete control: write a fact, confirm the agent recalls it, supersede that fact with a corrected record under the same key, then re-run the identical query set with side channels closed. Pass only if the system prefers the new record and can export a trace naming source entry, timestamp, and access scope for the recall.

That audit trail is the real privacy gate. Without reconstructable provenance, retrieval accuracy is a stage effect. Use the harness only to compare candidates for that one workflow. Stop before you turn it into a general “memory score.” Stretching one bounded test into a universal grade manufactures false confidence and steers procurement toward narrative polish instead of operational fit.

The weak assumption is that “evaluating agent memory” means proving recall works. Vendor demos sell retention as a feature. From a privacy angle, that is the wrong success criterion. The cleanest evaluation is whether the agent can be forced to forget on schedule, refuse to rehydrate stale context, and still complete the task with less personal data than it held yesterday.

Run a retention stress test, not a retrieval beauty contest. Seed a short-lived fact with a hard TTL and a longer-lived preference that should never expire. After the TTL, ask for both under paraphrased prompts, tool-mediated fetches, and session resume. The pass condition is simple: the expired fact is unreachable through every surface, the durable preference remains, and an audit trail shows what was deleted, when, and under which policy—not a soft “we usually purge.”

Data minimization is the control that makes the test meaningful. If the product cannot demonstrate selective decay, every impressive memory demo is also a retention risk. Teams that only score retrieval accuracy will ship agents that hoard context because forgetting lowers their leaderboard score.

Polished demos prove presentation, not memory integrity. The cleanest evaluation is a small reversible pilot on your own write path: seed a fixed fixture set with paired current and deliberately stale facts about the same entities, then force multi-session retrieval, correction, and deletion under the same controls you would ship.

What you measure is narrow and observable. Does the agent surface the current fact after a correction, or does the stale one reappear? Can privacy redactions actually prevent re-injection of restricted fields? Can you export, for each answer, what was stored, what was retrieved, what was suppressed, and who could have seen it? A demo can hide all of that behind a clean transcript.

Run the pilot for a short window, keep the fixture set under version control, and treat rollback of the memory store as a first-class exit criterion. The non-obvious consequence is operational: if you cannot reconstruct why a decision used a particular memory entry, later accuracy fixes create untraceable drift. Teams then “improve” retrieval while losing the ability to audit or reverse the system that produced the answers.