A registered horizon that outlived the intervention the trial ran: a pre-registered randomized trial of memory dosing in a production agent fleet

We report a pre-registered, fleet-wide interventional trial of dose-weighted memory promotion in a production agent-memory system. Epochs of 24 h were randomized to control or to one of three dose levels by constrained randomization seeded from a public drand beacon, with the assignment script and its hash registered before the seed was drawn (OSF yf7d2, 2026-08-18T07:56:44Z; Zenodo 10.5281/zenodo.22110203).

The registered design called for 234 epochs. Twenty were realized, of which nineteen served data and form the analysis set.

The trial ended before its registered 234-epoch horizon, after the fixed designation exhausted its eligibility. The designation could support dose exposure in only 20 trial epochs, a limitation fixed before Epoch 1. All 19 designated items share one created_at and sit in a sub-pool with a 30-day window, so they leave eligibility together at 2026-09-20 22:51:23 (20 of the 234 registered epochs, 8.5%). Past that instant no value of the dose reaches them, because the boost is addressed by id and they are no longer in the candidate list; treatment and control would have been the same intervention for 214 epochs. The deposited registration was not amended and still said 234 epochs; the trial adopted a fixed designation, chosen on 2026-08-26 after v1.12 was deposited and not deposited itself, that was eligible for 20 epochs, and it closed when that designation expired. sizing.py sized for 234 epochs on an intervention that existed for 20. This incompatibility was present before the trial began, and the under-powering of this study follows from it, not from the realized window.

The deposited registration (v1.12) names H1, repeated-failure density per session-hour, as the primary outcome and H1a–H1c as a Holm-corrected co-primary family. This paper reports H1c, the share of opportunities yielding a repeated failure, as its primary. The switch was decided on 2026-08-30 (DESIGN-REVISION-2026-08-30.md §3-ter; PROSPECTIVE-ESTIMAND-2026-08-30.md §3-bis), before the assignment seed was drawn (round 31774052, emitted 2026-08-30T21:32:04Z) and before any randomized epoch existed, in documents dated by our commit log and not deposited. It is therefore a deviation from the public registration, and we report the deposited primary beside it. In the registered analysis H1 does not reject under either reading: tested alone at α = 0.05, as the deposited reading has it, p = 0.3294; as a secondary in the Holm family, as the switch we adopt has it, adjusted 1.0 (§4.0.2). Under the deposited reading H1 did reject in versions up to rc9 of this analysis (p = 0.0302 in rc9, 0.0133 in rc8, 0.0127 in the sensitivity analysis); those versions counted a session that crosses an epoch boundary in every epoch it touches, where the registration counts it in the epoch of its start (§4.3). Under the deposited Holm correction no member of the H1a–H1c and H2 family rejects (§1): H1a's unadjusted p-value is 0.2599, and every Holm-adjusted p-value in the family is 1.0.

The reported analysis combines deposited analysis provisions, the undeposited analysis specification, and the post-trial substitution rule declared in DEVIATIONS §10.31: the fourth panel family only as a substitute below the three-verdict floor (the deposited registration retained three families and mentioned a fourth only as a mitigation not adopted there), the registered two-hour washout removed from the outcomes and from the session-hour denominator, every session counted in the epoch of its start, the analysis window cut at the designation's expiry with the specification's exposure offsets for the partial epochs, a BCa interval with a leave-one-epoch-out jackknife, and 19 epochs (the empty 09-02 excluded). The analysis of earlier versions, which departed from all six, is reported as a sensitivity. One verdict differs between the two: under the deposited reading the sensitivity analysis rejects H1 and the registered analysis does not (§4). Versions up to rc8 kept the washout in the denominator, and versions up to rc9 split boundary-crossing sessions across epochs; the second moves H1 and H1a by an order of magnitude, and H1c slightly (§2, §4).

The primary outcome is null: the registered re-randomization test does not reject the sharp null for H1c (p = 0.4006, §4.0.2). Under the planning assumptions and the fractional counting the analysis specification fixes, no reduction from p0 = 0.0782, including total elimination, reaches 80% power (§4.1.1). As pre-committed on 2026-08-30, the null does not distinguish a mechanism that does not work from lessons too generic to work (§4.5). This is a claim about H1c and not about every quantity the trial produced. In the registered analysis no interval of the H1 family excludes zero on any leg: H1 −1.27 [−28.36; +23.15] and H1a −15.22 [−373.70; +310.82] per session-hour (§4.2, §4.3), and neither excludes zero on the registered sensitivity without the sessions that cross an epoch boundary (§5). Earlier versions reported intervals excluding zero for both, and a rejection of H1 under the deposited reading; they rested on three long sessions split across the epochs they touched, one of them spanning 240 h, and they are reported as the sensitivity analyses they now are (§4.3). Two intervals of H2, the secondary confirmatory outcome, exclude zero in the direction of a treatment that costs more (winsorized time, raw tokens) under the registered BCa construction, by margins the percentile construction does not hold, and its registered test rejects neither (§5). The analysis specification, written on 2026-09-10 (ten days before the window closed and before any outcome was computed), dated by our commit log and not deposited with the registration, recorded that the minimum detectable effect for H1c was saturated at 100% under each of its inclusion cuts, among them the 19 clusters the calculation counted (11 treatment, 8 control): no reduction, including total elimination, was detectable at 80% power. Those 19 clusters are the registered analysis set. The spec counts partial epochs by fractional exposure, and that counting governs the power statement; counting whole epochs at the same 11 treatment and 8 control gives the same verdict (effective size 90.2 against a critical 95.26). Only at the 20 epochs of the sensitivity analysis, with 09-02 as a ninth control, does whole-epoch counting flip it (96.41, §4.1.1). The realized estimate is −0.0140 (treatment 0.0663, control 0.0803; 95% BCa interval [−0.0464; +0.0110], epoch as the resampling unit), containing zero on all three legs (locked, pre-committed sensitivity, post-hoc sensitivity) and in the sensitivity analysis (−0.0199 [−0.0560; +0.0086], p = 0.1603).

Our two statements about uncertainty, the planning power approximation and the realized bootstrap interval, rest on different variance constructions and assumptions; their difference does not establish a contradiction, and it does not quantify any under-coverage of the bootstrap. The interval's coverage has not been calibrated (a BCa bootstrap on 19 clusters, 11 treatment and 8 control, may under-cover; it does not explicitly propagate the stratum-B sampling or the variation between adjudication runs, whose labels did not all reproduce when re-run, and the direction of the coverage error that omission causes is not established; §4.1.2), and the power statement rests on an ICC estimated for a different quantity. Neither is a calibrated measure of this study's uncertainty. The null (non-rejection by the registered re-randomization test, p = 0.4006) does not rest on the interval; the point estimate, a 17% relative reduction, is not distinguishable from zero at this N.

Three further commitments made before the seed, besides the primary outcome and the horizon above, were not kept. One, a stopping rule, we found only after the trial, from the artifacts. Written before the assignment seed (PROSPECTIVE-ESTIMAND-2026-08-30.md §3-bis), it closes the interventional arm with a null if the coverage measured in Epoch 1 falls below 36.7%. Measured now from the locked artifacts by the definition the threshold was derived from, Epoch-1 coverage is 32.4% (45 of 139 opportunities; 33.7% over the active phase alone, while two other readings, after the washout or under the trial's weighted opportunity rule, are above it): the condition was met under the planning definition (seven signatures promotable at w = 2). Under the same definition applied to the dose Epoch 1 was served at (ten signatures promotable at w = 4) it was not: 37.4% (52 of 139), and every other reading is above the threshold too. We found no contemporaneous record of evaluating the rule, and the arm continued (§3.0). The second, the replacement of the declared assignment rule, was retracted in writing 24 min 38 s after the drand round was emitted, before Epoch 1 (ASSIGN-SEED-2026-08-30.md, retraction section; commit 3e1c259), and had not been reported in this manuscript until rc8, which quantifies the difference: the seed declaration named a rule other than the registered assign_arms.py, and run offline on that round the two rules assign a different arm to 11 of the 20 realized epochs (§1). The third is in the deposit: PREREG §3 locked census adjudication of the live study, and the reported analysis adjudicates a census of the is_error stratum and a hash-ordered sample of 800 of the 5 556 other episodes, weighted by 6.945, in the registered and the sensitivity analysis alike (§4). An attempt to restore the census after unblinding stopped at a test-retest gate declared before its first call: on 50 re-adjudicated episodes, two of the three panel families returned 47/50 and 46/50 identical labels against a declared 0.99, so the sample stays a declared deviation (§4, §7).

Given the small number of epochs and the uncertainty limitations above, the primary test's non-rejection of the sharp null neither establishes absence of effect nor supports a confident efficacy claim. We report the trial in full regardless, because the parts that carry information lie outside the effect estimate: the instrument controls pass, including a sham replay in which, on the 2,646 brief states of the w = 4 epochs, where production served that dose and the replay reproduces it, the real designation changed more served briefs than each of 20 matched sham designations would have (132 against 81–122, p = 1/21, the floor for 20 shams). Against these 20 shams, that is evidence that the dose acted specifically on what those epochs served. A second replay forces w = 4 in all 11,812 reconstructible states of the hash-verified trial window, control epochs included, so outside the w = 4 epochs it is a counterfactual and not a record of what was served; there too the real designation exceeds every sham (790 against 449–583, p = 1/21). The rank is not a calibrated randomization p-value, the second replay shares 2,016 states with the first (the w = 4 epochs except 09-01) and is not a replication, and neither says anything about outcomes (§4.0.1c). A hypothesis that was demoted from primary for requiring, at the pre-trial calibration share of altered briefs, a 955% effect nonetheless returns an interval excluding zero on all three legs, and its test rejects under the deposited reading, which we report as an unexplained rejection conditional on that reading: the denominator explains part of it and not the residual, and the observed reduction is many times what the few briefs the dose alters could carry in proportion to their share (a proportional-dilution calculation, not a causal bound) returned an interval excluding zero and, under the deposited reading, a rejection in versions up to rc9, and returns neither once every session is counted in the epoch of its start, as the registration requires: the earlier result did not survive the registered attribution of three long sessions that earlier versions split across epochs, a correction that moves both their exposure and the epoch and arm of their episodes, so it shows the result was not robust, not that the dose had no effect (corrected in rc10, qualified in rc11; §4.3); a second hypothesis inverts its conclusion on the removal of a single epoch whose exposure comes from one session with three episodes excludes zero only on the locked leg: its pre-committed sensitivity leg contains zero, and so does a post-hoc leg that removes a single epoch whose exposure comes from one session with three episodes (corrected 2026-10-04: the first version reported only the post-hoc leg; §4.2) excludes zero on the locked and the pre-committed legs under the registered BCa interval, contains zero on a post-hoc leg that removes a single epoch whose exposure comes from one session with three episodes, and is not rejected under the registered Holm rule (corrected in rc8: "only on the locked leg" held for the percentile interval of earlier versions, not for the registered BCa; §4.2) excludes zero on all three legs once the washout leaves the denominator, and is not rejected under the registered Holm rule (corrected in rc9: under the rc8 analysis, which kept the washout in the denominator, the post-hoc leg without 09-14 contained zero; §4.2) shares that history: its locked-leg interval excluded zero in every version that split sessions across epochs, all three legs did in rc9, and every leg contains zero once they are not, and the registered Holm rule never rejected it (corrected in rc10, qualified in rc15; §4.2); and the member of the hypothesis family that carried the substantive mechanism question collapses to 1.0 by construction, because two locks of our own pre-registration define its estimand incompatibly and neither of us noticed until the window had closed.

We take the position that these are the reportable results of a trial whose registered horizon outlived the intervention it ran and which is under-powered as a result, and that publishing them is the alternative to the two things a null of this shape is usually used for: silence, or a claim.

---