Construction before search: a pre-registered factorial test of two repairs to LLM-extracted agent memory

Four recent studies find that LLM-extracted memory loses to retrieval over raw turns, and conclude that the answer is better search rather than better construction. All four hold the extraction step at its defaults. On a production corpus we measured two defects in that step — how much of the conversation the builder is shown, and whether its criterion requires identifiers verbatim — and found that repairing both recovers 21.2 pp of accuracy, of which 13.0 pp is interaction: neither repair alone accounts for it.

This is the plan to test that on a public corpus, registered before any arm runs. The design is a 2x2 over LongMemEval s_cleaned: window (truncated to production's measured median seen-fraction, against the full haystack) crossed with criterion (the deployed extraction prompt, against the same prompt plus a verbatim-identifier paragraph). The primary endpoint is the total effect of repairing both, which 500 instances can resolve at 99.6% power; the interaction is a pre-registered secondary that no sample this corpus allows powers to the usual standard, and the plan says so in advance rather than after.

A positive interaction does not by itself identify a joint mechanism: two factors acting in series — one deciding whether evidence arrives, the other whether it is used — produce superadditivity with nothing jointly repaired. That serial model is therefore the plan's substantive null, computed from three of the four production cells, and it predicts +0.4 pp against the +13.0 pp measured, a factor of 33. The plan also states what it cannot do: no outcome refutes the repair thesis by showing the repair does nothing, because at this corpus size no confidence interval is narrow enough to bound the effect near zero. What remains refutable is the mechanism.

The document carries its own audit trail. Its first design was refuted by its own sampling code, and is kept rather than deleted. Three adversarial model families read the plan before deposit; each found defects the previous ones had introduced, including a power calculation that sized the interaction with the total effect, two controls that a pipeline returning a constant refusal would pass, and an equivalence verdict that was unreachable at every sample the corpus allows.