Replaying a production memory distiller: a fidelity control that failed, and what it found

A data note on one deployment of an agent memory system. We pre-registered an offline comparison of two extraction criteria on the system's own production sessions: the deployed criterion against a revised one that had won on a public benchmark. The registered order ran in full, each step committed publicly before the next: pre-registration tag, frozen sample, question set, model runs, a blind human sample and the reading.

The reading stopped at its controls. The ceiling, positive and counterfactual controls passed. The fidelity control, which asks whether our replay of the deployed criterion reproduces what production stored, failed, so by the registered rule the two hypotheses are reported as not interpreted. A diagnosis written after the result found a defect that plausibly explains the failure, though it was not tested by rerunning the replay: production's distiller drops the prompt that opens a scheduled job before it distils, and our replay kept it. Most of the replay's input, and more than half of the primary questions, came from those prompts.

A follow-up census found that the distiller logged few of the eligible sessions, most of the missing ones from one week while it was reading an emptied session address, and that most of the logged sessions it did not send to its model were two-message sessions whose text is almost all on the user side. The note does not claim that either criterion is better on production sessions.

The sessions, the questions and the human grading sheet are private and not released. The archive carries the note, its provenance file, the pre-registration, the public commitments, the reading, the diagnosis, the censuses and the scripts.