Construction before search: a pre-registered test of what an LLM memory builder keeps

Several recent studies compare memory extracted by a language model with retrieval or search over the raw conversation, and in most of their settings the raw record wins. Their conclusions carry conditions: one limits its recommendation to the extraction designs it tested, one finds extraction more accurate at generous context budgets, and one finds designed memory helping only a weak reader. One of them, Fidelity before Structure (arXiv 2601.00821), also varies the extraction itself, down to a near-verbatim, full-coverage design that still trails verbatim chunks, and predicts that better extraction prompts can narrow the gap but cannot reach what a verbatim store retains. None of them varies how much of the conversation the builder is shown.

We pre-registered a test of two repairs to the construction step: how much of the conversation the builder is shown (the window), and the instructions it extracts under (the criterion). The first instrument failed its own registered floor. With the full window, the pipeline answered 21% of LongMemEval questions against a required 50%, so its primary contrast was not interpreted. Two more amendments followed, each fixed before its calibration calls. The first of them tried four readers, and none passed. An exploratory run then found the larger loss between the raw evidence and the units distilled from it. Units distilled from the evidence sessions scored 0.475, and the raw turns of the same sessions scored 0.830. A revised criterion, C2, asks the builder to keep facts said in passing, the event's own time and what the assistant said. It passed calibration on the development instances.

The confirmatory run used 242 instances with no prior exposure, defined by provenance and not by question id alone. All four registered controls passed. Repairing both factors against repairing neither raised accuracy by +66.5 pp [+60.6, +72.5] under judge 1 and +66.9 pp under a second judge family. At the truncated window, 120 of the 242 instances kept no annotated evidence session, and the truncated cells answered none of them. That pattern is consistent with evidence availability as a large part of the window contrast, though the design does not estimate a mediated share. The criterion contrast at the full window is +24.8 to +27.3 pp across judges. A length-matched arm rules out its own form of padding as the explanation. The result confirms a revised intervention, not the original hypotheses. A raw-evidence arm, registered as a further amendment before it ran, used the same 242. C2 units of the evidence sessions scored below the raw turns of those sessions. Its two judges fell on opposite sides of the decision rule (−5.4 pp [−10.3, −0.8] and −5.0 pp [−9.9, 0.0]), so a last amendment, written after that split and fixed before any call of the new judges, added two judges and a committee label. Under it the units lose to raw evidence by −5.8 pp [−10.4, −1.2], and non-inferiority at 10 pp is not shown. Both arms are restricted to the annotated evidence and differ in length and order as well as content, so the arm compares two served materials and does not test retention.