The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory

Abstract

Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use. The same reported test result can support confirmation when the outcome and stopping rule were fixed in advance, yet only exploration when the outcome was selected from twenty inspected results. We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. The SciStateForge compiler–reader framework turns distributed histories into explicit decision-governing state. Counterfactual pairs and targeted corruption–repair controls then test whether models use those dependencies and localize failures. Across four model families, compiled scientific state improves valid decision-making by 9.2–18.8 percentage points over a generic Event Graph. Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.

1 Introduction

An agent can recover the right fact from a laboratory history and still make the wrong scientific decision. Existing long-term-memory benchmarks probe long conversations, incremental interactions, state updates, selective forgetting, and on-policy personalization (Maharana et al., 2024; Wu et al., 2025; Hu et al., 2026; Jiayang et al., 2026). Yet their central question remains whether an agent can preserve, retrieve, or revise the relevant information. Scientific agents face a second question: is that information eligible to govern the current decision?

Consider two notebooks that both truthfully report p=0.03. In one, the researcher preregistered the outcome, fixed the sample size, and followed the stopping rule. In the other, the researcher inspected twenty outcomes and stopped after one crossed the significance threshold. The number is identical, but its evidential status differs: the former may support a confirmatory claim, whereas the latter remains exploratory. Preregistration fixes the analysis plan before observation, while selective-inference methods account for data-dependent selection (Nosek et al., 2018; Fithian et al., 2017). Better retrieval cannot resolve this distinction because the governing information is distributed across the analysis plan, outcome family, access history, and stopping rule. Returning the correct value without these dependencies can make an agent more confidently wrong.

We call this missing evaluation object scientific use validity (Figure 1): whether remembered evidence may influence an action under the scientific state governing its interpretation. This problem also arises when an outcome is viewed before a plan is locked, identically named measurements use different constructs, a blank denotes failure rather than zero, or sunk costs influence a prospective choice. Each case links a locally correct record to a decision through state elsewhere in the history. Scientific memory must therefore reconstruct the governing state, not merely retain the record.

This question is complementary to recent benchmarks for scientific agents. Scientific-agent benchmarks test broad capabilities, controlled tools, and open-ended or executable research workflows (Bragg et al., 2026; Panigrahi et al., 2026; Kon et al., 2026), while process evaluations show that final-answer accuracy can conceal unsupported reasoning (Shao et al., 2026). They do not isolate whether a memory reader uses the state that licenses a particular decision. Structured memory may score higher because of retrieval, added information, formatting, prompt length, or answer preference. One correct answer alone cannot distinguish these causes.

We introduce SciValidityBench, an executable benchmark factory built to make this distinction measurable. Each TaskSpec binds a generated history, scientific-state definition, legal actions, programmatic oracle, controlled interventions, and automatic scorer. The resulting 64 tasks cover five validity mechanisms across machine learning, A/B testing, clinical research, bioinformatics, systems, and causal inference. Domains change the scientific surface and event generator, while mechanisms declare governing state, interventions, and oracles through the same interface. At the center of the benchmark is SciStateForge, a compiler–reader pipeline that turns distributed experimental histories into explicit, decision-governing scientific state. We compare complete raw events, fixed BM25 retrieval, a generic Event Graph, a flat compiled-state view, and a typed scientific-state ledger. Counterfactual worlds hold all non-target information fixed while changing one decisive state value. Targeted corruption removes oracle-required state. Exact and edit-matched irrelevant repairs distinguish genuine state use from answer preference, generic text perturbation, and irrelevant-history interference.

Across four locked model families, compiled scientific state improves TaskSpec-balanced valid decision-making by 9.2–18.8 percentage points over a generic Event Graph. BM25 retrieval does not reproduce this cross-family gain, and content-identical flat and typed compiled views show that the central benefit comes from exposing governing state rather than from a particular graph serialization. Exact target repair exceeds matched irrelevant repair by 18.6 validity points on average. The resulting behavior identifies missing state use, representation-sensitive shortcuts, and irrelevant-memory interference rather than collapsing them into a single accuracy score. Our contributions are threefold:

1. Scientific use validity. We formalize the gap between recovering a correct record and deciding whether it may govern an action, then instantiate that distinction across five validity mechanisms with explicit state dependencies.

2. An executable benchmark factory. Sixty-four TaskSpecs across six domains share generators, state contracts, legal actions, programmatic oracles, interventions, and scorers, allowing new scientific settings to reuse the same evaluation interface.

3. Causal diagnosis and evidence. Paired worlds, targeted corruption, exact repair, and edit-matched irrelevant repair separate genuine state use from retrieval and answer preference. Compiled state beats Event Graph by 9.2–18.8 points across four model families.

2 Related Work

Agent memory: from recalling facts to using them in action. Persistent-memory agents consolidate and selectively retain experience in external stores (Park et al., 2023; Packer et al., 2024). Mem0 extends this pattern to scalable deployments (Chhikara et al., 2025), and surveys map the design space (Zhang et al., 2025). Benchmarks cover long-range and cross-session recall (Maharana et al., 2024; Wu et al., 2025; Tan et al., 2025; Tavakoli et al., 2026), continual adaptation and forgetting (Hu et al., 2026; Jiayang et al., 2026; Uddin et al., 2026), and memory-grounded action (He et al., 2026; Zhao et al., 2026; Wu et al., 2026). Recent designs span feedback-driven declarative and procedural adaptation, joint personal and system memory, episode-level knowledge versus execution demands, and hidden user-state recovery (Ai et al., 2026; Ma et al., 2026b; Wang et al., 2026b; Ma et al., 2026a). STALE, CAME-Bench, and Mem2ActBench target stale facts, contextual applicability, and historical tool constraints (Chao et al., 2026; Yang et al., 2026; Shen et al., 2026). These benchmarks measure recall and downstream utility. SciValidityBench holds the focal record fixed and varies the nonlocal conditions for its use, testing the content–state–action relation.

Scientific reasoning and scientific-agent benchmarks. Surveys chart scientific-agent stages, maturity, and reliability (Wang et al., 2026a). Benchmarks cover knowledge and problem solving (Wang et al., 2024; Laurent et al., 2024), data-driven discovery (Majumder et al., 2025; Gu et al., 2024; Chen et al., 2025), and controlled or end-to-end research environments (Bragg et al., 2026; Panigrahi et al., 2026; Kon et al., 2026). Scientific-memory work treats full-paper recall as budgeted restoration, while StatefulDiscovery externalizes investigation state for claim calibration (Sheverev et al., 2026; Chen et al., 2026a). These systems score completion, restoration, or discovery quality. We fix the history and action space, then test whether governing state permits evidence use.

Provenance and scientific validity. PROV traces entities, activities, and agents (Moreau and Missier, 2013), but traceability alone does not establish admissibility. Preregistration fixes the analysis plan before observation (Nosek et al., 2018), while selective inference accounts for data-dependent selection (Fithian et al., 2017). Data leakage can also invalidate claims when evaluation or artifact history is ignored (Kapoor and Narayanan, 2023). We turn this principle into five mechanisms and test whether readers reconstruct and obey the state governing valid use.

Behavioral and counterfactual diagnosis. MemFail isolates summarization, storage, and retrieval failures (Garg et al., 2026), while MemTrace follows memory evolution (Deng et al., 2026). CausalFlow and Causal Agent Replay intervene on execution traces (Bonagiri et al., 2026; Shah, 2026). CheckList probes behavioral failures, whereas contrast sets probe local decision boundaries (Ribeiro et al., 2020; Gardner et al., 2020). SeekBench separates answers from epistemic process (Shao et al., 2026), and explicit state need not be mechanistically faithful (Chen et al., 2026b). Our interventions target scientific state through one-value paired worlds, targeted corruption, and exact versus edit-matched irrelevant repair.

3 SciValidityBench

3.1 Why scientific use requires global state

A scientific notebook is an event trace, not yet a decision state. It may faithfully record plans, runs, observations, access operations, and previous choices while leaving their joint implication unstated. We call that cross-record implication global scientific state: the collection of constraints that determines which evidence may be combined, compared, interpreted as missing, disclosed, or allowed to influence a prospective action. “Global” refers to its scope across the history rather than to an unbounded summary of everything that occurred.

This object separates three questions that are often conflated. Factuality asks whether a recorded payload is correct. Provenance establishes where the item came from and how it was produced. Use validity concerns what that authentic record is licensed to support under the governing dependencies. A verified measurement can therefore remain real and traceable while being ineligible for pooling, confirmation, or resource allocation. The benchmark targets this last relation without weakening the first two. The same payload can therefore license different actions in otherwise paired histories.

Formally, let Epub​(w) denote the visible canonical-event history of world w. A TaskSpec τ declares a deterministic compiler Cτ, which produces the decision-governing state

It also provides a finite action space 𝒜τ and a programmatic oracle Oτ​(Gτ​(w))∈𝒜τ. For an agent decision a^, scientific use validity is

The measured object is consequently a state-conditioned action, rather than recovery of an isolated span.

3.2 What a TaskSpec specifies

The basic unit of SciValidityBench is an executable TaskSpec, not a fixed natural-language question. We write its functional contract as

where Dτ is a domain adapter, Wτ is a seeded world generator, Cτ defines scientific-state compilation, 𝒜τ lists legal decisions, Oτ returns the valid choice, Iτ constructs controlled interventions, and Sτ scores responses and diagnostic relations. The stored specification additionally fixes its version, episode schedule, visibility rules, budgets, corruption operators, and data-split policy. One template can thus instantiate many reproducible worlds while preserving the same scientific rule. Appendix A summarizes catalog coverage, contract fields, and one representative decision pattern from each module.

This contract turns an abstract validity concern into a complete test: the generator creates the history, the compiler identifies what governs its use, the model selects among declared actions, and code determines whether that selection is legitimate. No free-form judge is needed for the primary outcome. Stable identifiers and canonical serialization bind every score to the exact events and state from which it was derived.

Two orthogonal extension axes. The specification factorizes scientific surface from validity mechanism. To add a domain, a developer supplies new entities, units, artifacts, and event language through Dτ while reusing an existing state rule. To introduce a new module, one declares its governing variables, compiler relations, oracle, and interventions while retaining the common domain and scoring interfaces. This separation makes the suite a benchmark factory rather than a collection of unrelated datasets.

3.3 How SciStateForge compiles scientific state

SciStateForge realizes the TaskSpec contract as a compiler–reader pipeline (Figure 2). Its central operation is not summarization: it derives explicit, source-linked state from dependencies distributed across the trace.

1. Instantiate a scientific world. Given a TaskSpec and seed, the domain adapter generates a multi-episode history with a hidden world state and a reproducible public trajectory. Scientific surfaces vary by domain, but every trajectory uses the same event vocabulary: Plan, Run, Observe, Access, Write, Retrieve, and Decide.

2. Canonicalize the history. Each event receives a stable identifier, actor, sequence position, referenced objects, versioned payload, visibility, cost, and provenance. This normalized layer provides byte-stable replay and prevents domain-specific formatting from changing the underlying scientific world.

3. Compile governing relations. Module reducers map public events into a Scientific State Graph (SSG). Its nodes represent objects such as plans, observations, constructs, access records, absences, costs, and decisions. Typed edges materialize relations including Selected_From, Stopped_After, Sealed_Until, Comparable_With, Absent_Because, and Relevant_To. Derived fields retain links to the events that justify them. They add no new observations while making the scientific consequences of public evidence explicit.

This step is what distinguishes SciStateForge from a generic Event Graph. The latter preserves what happened and connects events through sequence or shared objects. The compiler additionally resolves what those occurrences mean for the pending decision. Its flat SSG and typed Scientific Validity Ledger renderings expose identical public nodes, attributes, and relations, allowing representation to change without silently changing the compiled content.

Figure 2 gives a concrete pass. The paired worlds contain the same event types, but A records lock@4 before access@6, whereas B records access@2 before lock@4. The Blind Memory reducer joins those timestamps, derives a post-lock or early-access status, and maps it to the corresponding confirmatory or exploratory action. Each derived value remains linked to the events that support it. The compiler terminates at this source-linked state. The reader still selects from the legal actions, and the TaskSpec oracle scores that selection. Compilation therefore resolves a cross-event scientific dependency rather than retrieving a different fact or executing the decision. Across the final catalog, 64 executable TaskSpecs crossed with ten unseen seeds instantiate 640 seeded worlds and expose 1,920 public event records.

4. Render, decide, and score. Oracle-only generator state remains with the validator. The reader receives a projection derived from the public history together with the current request and legal actions. Its choice is checked by the TaskSpec oracle, and the run emits both an item-level validity score and the identifiers needed for paired or repair-based diagnosis. Appendix B gives the canonical event schema, deterministic passes, and exact memory-view definitions.

3.4 Five modules across six scientific domains

The catalog contains 64 versioned TaskSpecs spanning five validity modules and six scientific domains. Table 1 shows that the modules are different mechanisms of one construct: whether the current scientific state licenses the proposed use of a locally correct record.

Domains supply scientific settings rather than duplicate module definitions. Machine learning contributes datasets, checkpoints, seeds, and metric implementations; A/B testing uses variants, traffic slices, endpoints, and scheduled looks; clinical research supplies protocols, cohorts, outcomes, and censoring records. Bioinformatics introduces assays, samples, replicate structure, and pipeline versions; systems research uses hardware, workloads, runtime logs, and benchmark configurations; causal inference instantiates exposures, estimands, adjustment sets, and identification choices. These scientific surfaces are summarized in Table 2.

This decomposition provides breadth without turning the suite into six unrelated collections. Inferential Ledger, Construct Drift, and Absence Algebra appear in every domain. Blind Memory and Sunk Cost are instantiated in selected settings where their operational stories are most direct. Because world generation is seeded, each TaskSpec yields controlled variation beneath the same decision contract rather than repeating one memorized answer. Cross-domain instances also resist surface shortcuts: one validity rule appears through different entities and terminology, while one scientific setting hosts multiple decision mechanisms.

3.5 Interventions and programmatic diagnosis

Correct actions alone cannot establish that a reader used the intended state. SciValidityBench therefore builds diagnostic interventions into the benchmark unit.

State-changing counterfactuals. For each TaskSpec-defined pair, Iτ changes one decisive public value while holding the TaskSpec, seed, surface form, legal actions, and non-target information fixed. The contract declares whether the oracle relation must flip or remain invariant. A model that repeats the same choice in a flip pair may have succeeded on one side through answer preference rather than state use.

Corruption and matched repair. Graph operators can omit, unlink, relabel, merge, duplicate, or inject a specific node or relation. Exact repair applies the inverse patch and restores the target graph, including its canonical hash. An edit-matched irrelevant repair leaves the decisive damage in place while modifying an unrelated location with the same operation signature. Recovery unique to the exact repair supports dependence on the intended scientific state. Equal recovery from both repairs instead indicates a generic text or structure effect.

Scoring beyond a leaderboard. The scorer records legal-action validity, the paired answer relation, and repair behavior from deterministic contracts. These outputs distinguish four useful failure patterns: failure to track the decisive value, a stable answer bias, sensitivity to irrelevant history, and inability to exploit a restored state. The benchmark consequently measures performance and supplies a causal behavioral account of where that performance comes from.

Together, canonical events, declarative TaskSpecs, domain adapters, explicit state compilation, and reversible interventions turn scientific use validity into a reusable evaluation interface. The resulting pipeline can expand to a new scientific setting without redefining the construct, or to a new validity mechanism without rebuilding the surrounding evaluation stack. Every extension retains the same diagnostic guarantees: executable scoring, state-changing pairs, and repair at a known causal location. Breadth therefore does not trade away failure interpretability.

4 Experiments

Our evaluation follows a three-step evidence chain. We first measure whether explicit scientific state improves valid decisions across independently executed readers. Retrieval, input length, generic graph structure, and typed serialization then test alternative explanations. Module effects, counterfactual pairs, and targeted repair respectively locate the benefit, verify state-dependent behavior, and diagnose failure.

4.1 Experimental setup

Primary matrix. The main evaluation contains all 64 program-valid TaskSpecs. Ten unseen seeds instantiate each specification for four open model families: Qwen3-14B, Gemma-3-12B-IT, Mistral-7B-Instruct-v0.3, and Phi-3.5-mini-instruct (Yang et al., 2025; Gemma Team et al., 2025; Jiang et al., 2023; Abdin et al., 2024). Crossing those factors with five memory views produces 64×10×4×5=12,800 original decisions. Flat SSG and typed SVL additionally receive one state-changing companion world per task–seed–reader, adding 5,120 counterfactual decisions. Completion is outcome-independent: invalid JSON, illegal actions, and unparsable answers count as invalid decisions. Across the six frozen evaluation matrices reported in Appendix C, the final suite contains 68,320 scored decisions, of which 17,920 belong to the primary matrix.

Representations and controls. Raw Events exposes the full public sequence. A fixed BM25 retriever selects at most five request-relevant event chunks. Event Graph preserves every public event and object but connects them only through generic sequence and mention edges. The two compiled conditions reveal the same public nodes, attributes, and governing relations: Flat SSG serializes them as records, whereas SVL uses a typed ledger. The prompt shell, current request, legal actions, seed, checkpoint, and temperature-zero decoding remain fixed within each comparison. Models are served with vLLM and return one action from the legal choice set (Kwon et al., 2023).

Outcomes and uncertainty. The primary measure is valid-action rate under the TaskSpec oracle. For every representation contrast, we first average paired seed differences within a TaskSpec and then weight all 64 specifications equally. Reported 95% intervals use 20,000 TaskSpec bootstrap resamples. Counterfactual relation accuracy tests whether two answers obey the required flip or invariance. Repair specificity compares restoring the damaged target with an edit-matched change at an irrelevant location. These diagnostics complement the valid-action rate by measuring how readers use the state behind each decision. Appendix C details models, matrices, and estimands.

4.2 Compiled state improves validity across settings

The primary matrix yields a consistent separation between scientific-state compilation and generic event organization. As shown in Table 3, typed SVL exceeds Event Graph by 18.8, 18.3, 11.3, and 9.2 validity points for Qwen, Gemma, Mistral, and Phi, respectively. All four 95% intervals remain above zero. Figure 3A summarizes the pattern. The result survives an exact-overlap audit. Removing 22 tasks leaves gains of 8.3–14.8 points for every reader across the remaining 42 tasks (Appendix D, Table 17).

Event Graph stays close to Raw Events despite containing all public events and their basic relations. Compilation materializes consequences such as Comparable_With, Absent_Because, or Stopped_After. The reader no longer has to reconstruct those cross-record dependencies from sequence alone. The 9.2–18.8 point gain comes from exposing the scientific relations that govern the current decision, not from organizing the same event history as a generic graph.

Flat SSG isolates compiled content from ledger syntax. It reaches 0.991/0.903/0.692/0.609 across the four readers, slightly above typed SVL in every row. Both renderings disclose identical nodes, attributes, and governing relations, so their shared improvement comes from the compiled scientific state. The small Flat–SVL differences show that readers vary in how they consume the same state.

Independent executions reproduce the contrast in Figure 3B. Hosted runs yield 47.0–48.6 point advantages across two readers and two seed blocks, with positive intervals throughout. An independent Qwen replication produces an 18.6-point gain [12.0,25.3], while Flat and typed SVL remain within five points. The direction persists across model identities and serving stacks. Appendices D and H give full results and replications.

4.3 What drives the gain?

Retrieval does not reproduce the cross-family pattern. BM25 raises Mistral by 3.0 points and Qwen by 0.3, but lowers the other two readers. Event Graph is the longest view for every tokenizer yet remains close to Raw Events. The shorter compiled projections lead instead, tying performance to explicit governing relations rather than retrieval depth or context length.

The source-level matrix changes one canonical public value before constructing any view. Matched Source exposes the complete derived-free fact closure and exactly matches Flat SSG’s full-chat token count. Across 12 balanced design families, Flat SSG exceeds Event Graph for every reader (Table 4). Scientific-relation compilation remains beneficial after the shared upstream intervention.

Exact token matching separates compilation from presentation. Flat outperforms Matched Source by 19.4 points for Qwen and 19.3 for Mistral, changes Gemma by 1.1, and trails it by 29.5 for Phi. Typed SVL is effectively tied with Flat for Qwen and Mistral but improves Gemma by 24.0 points and Phi by 48.8. The common advantage lies in compiled state. The easiest serialization depends on the reader.

4.4 Where compilation helps and whether readers use state

Benefits concentrate in particular module–reader combinations (Figure 4A). Blind Memory rises by 21.3–52.5 points across the four readers, and Inferential Ledger improves in every family. Absence Algebra primarily benefits Mistral. Sunk Cost is unchanged for Mistral and Phi but lower for Qwen and Gemma. The aggregate effect therefore reflects structured heterogeneity rather than a uniform catalog-wide lift.

Module effects locate where compilation helps. Paired worlds test whether the decisive state actually governs behavior. Counterfactual pairs hold task surface, graph structure, legal choices, and non-target information fixed while changing one public value. Relation accuracy scores the oracle-declared flip or invariance, so a fortunate answer on one side does not count as state tracking.

Panel B reveals a sharper hierarchy than single-world accuracy. For Flat SSG/typed SVL, relation accuracy reaches 1.000/0.998 for Qwen and 0.906/0.905 for Gemma, then falls to 0.561/0.550 for Mistral and 0.436/0.361 for Phi. Stronger readers convert explicit state into counterfactually consistent behavior. Weaker ones often retain stable answer preferences.

Panel C supplies an independent localization test. Exact restoration exceeds an edit-matched irrelevant repair by 26.9 points for Qwen, 32.8 for Gemma, and 15.0 for Mistral, but changes Phi by −0.3 points. The four-reader mean is 18.6. Cross-domain corruption fingerprints are stable for five of six operators in Qwen and Gemma, versus two for Mistral and one for Phi. High relation accuracy and stable fingerprints coincide in the stronger readers, whereas Mistral and Phi show weaker tracking and less specific recovery. The contrast distinguishes damage to a governing relation from a broader failure to use compiled state. Appendices E and F give complete diagnostic results.

5 Conclusion

Scientific memory must recover a true record and determine whether it is licensed for the current decision. SciValidityBench makes this scientific-use validity executable through 64 TaskSpecs spanning five mechanisms and six domains, while SciStateForge compiles distributed histories into source-linked decision state. Across four model families, compiled state improves valid-action selection over Event Graph by 9.2–18.8 points. The advantage persists after removing exact state–action overlaps and cannot be reproduced by retrieval, longer inputs, or generic graph structure. Counterfactual pairs and matched repairs distinguish genuine state dependence from answer preference and localize failures, shifting memory evaluation from retaining facts to reconstructing the conditions under which they may be compared, promoted, or acted upon. TaskSpecs separate scientific surfaces from validity rules for agents writing laboratory histories.

AI use statement

Generative AI tools were used only to polish the manuscript’s language, including grammar, clarity, and style. They did not determine the research ideas, methods, experimental design, implementation, data, analysis, or conclusions. The authors reviewed and verified the complete manuscript and take full responsibility for all of its content.

Ethics Statement

The benchmark uses procedurally generated histories and fictitious clinical scenarios. It involves no human participants, patient records, personal data, or confidential information. No human-subjects research was conducted, so IRB review was not required. The authors take responsibility for the ethical use and release of the benchmark, experiments, and associated materials.

Reproducibility statement

The appendices document the TaskSpec contracts, memory-view definitions, evaluation settings, programmatic scoring, and diagnostic interventions. The public GitHub repository provides the final 64-TaskSpec catalog, deterministic generators spanning six domains, the SciStateForge compiler, all five evaluated memory views, counterfactual and corruption/repair operators, executable examples, and release tests. Every benchmark world is generated deterministically from a TaskSpec and seed, and oracle-only state is excluded from model-visible inputs.

References

• M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, Q. Cai, V. Chaudhary, D. Chen, D. Chen, W. Chen, Y. Chen, Y. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V. Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. D. Giorno, A. Goswami, S. Gunasekar, E. Haider, J. Hao, R. J. Hewett, W. Hu, J. Huynh, D. Iter, S. A. Jacobs, M. Javaheripi, X. Jin, N. Karampatziakis, P. Kauffmann, M. Khademi, D. Kim, Y. J. Kim, L. Kurilenko, J. R. Lee, Y. T. Lee, Y. Li, Y. Li, C. Liang, L. Liden, X. Lin, Z. Lin, C. Liu, L. Liu, M. Liu, W. Liu, X. Liu, C. Luo, P. Madan, A. Mahmoudzadeh, D. Majercak, M. Mazzola, C. C. T. Mendes, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portet, R. Pryzant, H. Qin, M. Radmilac, L. Ren, G. de Rosa, C. Rosset, S. Roy, O. Ruwase, O. Saarikivi, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, Y. Shen, S. Shukla, X. Song, M. Tanaka, A. Tupini, P. Vaddamanu, C. Wang, G. Wang, L. Wang, S. Wang, X. Wang, Y. Wang, R. Ward, W. Wen, P. Witte, H. Wu, X. Wu, M. Wyatt, B. Xiao, C. Xu, J. Xu, W. Xu, J. Xue, S. Yadav, F. Yang, J. Yang, Y. Yang, Z. Yang, D. Yu, L. Yuan, C. Zhang, C. Zhang, J. Zhang, L. L. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, and X. Zhou Phi-3 technical report: a highly capable language model locally on your phone. External Links: 2404.14219, Link Cited by: §4.1.
• Ai et al. (2026) Q. Ai, Y. Tang, C. Wang, J. Long, W. Su, and Y. LIU MemoryBench: a benchmark for memory and continual learning in LLM systems. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §2.
• Bonagiri et al. (2026) A. Bonagiri, D. Borkar, G. J. Anderias, S. Rafatirad, and H. Homayoun CausalFlow: causal attribution and counterfactual repair for LLM agent failures. External Links: 2605.25338, Link Cited by: §2.
• Bragg et al. (2026) J. Bragg, M. D’Arcy, N. Balepur, D. Bareket, B. Dalvi, S. Feldman, D. Haddad, J. D. Hwang, P. Jansen, V. Kishore, B. P. Majumder, A. Naik, S. Rahamimov, K. Richardson, A. Singh, H. Surana, A. Tiktinsky, R. Vasu, G. Wiener, C. Anastasiades, S. Candra, J. Dunkelberger, D. Emery, R. Evans, M. Hamada, R. Huff, R. Kinney, M. Latzke, J. Lochner, R. Lozano-Aguilera, C. Nguyen, S. Rao, A. Tanaka, B. Vlahos, P. Clark, D. Downey, Y. Goldberg, A. Sabharwal, and D. S. Weld AstaBench: rigorous benchmarking of AI agents with a scientific research suite. External Links: 2510.21652, Link Cited by: §1, §2.
• Chao et al. (2026) H. Chao, Y. Bai, R. Sheng, T. Li, and Y. Sun STALE: can LLM agents know when their memories are no longer valid?. External Links: 2605.06527, Link Cited by: §2.
• Chen et al. (2026a) J. Chen, S. Liu, and L. Yang StatefulDiscovery: evidence-calibrated claim formation in open-ended scientific discovery. External Links: 2606.11851, Link Cited by: §2.
• Chen et al. (2026b) Y. Chen, P. Qian, S. Wang, C. Peng, H. Xu, S. Li, and Y. Sun Explicit state elicitation is not enough: a controlled audit of memory-policy classification. External Links: 2608.17247, Link Cited by: §2.
• Chen et al. (2025) Z. Chen, S. Chen, Y. Ning, Q. Zhang, B. Wang, B. Yu, Y. Li, Z. Liao, C. Wei, Z. Lu, V. Dey, M. Xue, F. N. Baker, B. Burns, D. Adu-Ampratwum, X. Huang, X. Ning, S. Gao, Y. Su, and H. Sun ScienceAgentBench: toward rigorous assessment of language agents for data-driven scientific discovery. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, pp. 96934–96990. External Links: Link Cited by: §2.
• Chhikara et al. (2025) P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav Mem0: building production-ready AI agents with scalable long-term memory. External Links: 2504.19413, Link Cited by: §2.
• Deng et al. (2026) X. Deng, R. Zhong, H. Peng, X. Lu, Y. Wu, G. Li, B. Xu, Y. Yao, J. Fang, H. Cao, J. Guo, Y. Yuan, Z. Ma, Y. Yu, R. Hu, B. Dong, H. Zhu, and N. Zhang MemTrace: tracing and attributing errors in large language model memory systems. External Links: 2605.28732, Link Cited by: §2.
• Fithian et al. (2017) W. Fithian, D. Sun, and J. Taylor Optimal inference after model selection. External Links: 1410.2597, Link Cited by: §1, §2.
• Gardner et al. (2020) M. Gardner, Y. Artzi, V. Basmov, J. Berant, B. Bogin, S. Chen, P. Dasigi, D. Dua, Y. Elazar, A. Gottumukkala, N. Gupta, H. Hajishirzi, G. Ilharco, D. Khashabi, K. Lin, J. Liu, N. F. Liu, P. Mulcaire, Q. Ning, S. Singh, N. A. Smith, S. Subramanian, R. Tsarfaty, E. Wallace, A. Zhang, and B. Zhou Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, T. Cohn, Y. He, and Y. Liu (Eds.), Online, pp. 1307–1323. External Links: Link, Document Cited by: §2.
• Garg et al. (2026) I. Garg, N. Kolhe, D. Song, and X. Zhao MemFail: stress-testing failure modes of LLM memory systems. External Links: 2605.26667, Link Cited by: §2.
• Gemma Team et al. (2025) Gemma Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, D. Tian, M. Eyal, C. Cherry, J. Peter, D. Sinopalnikov, S. Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma, A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Feng, A. Kolesnikov, A. Bendebury, A. Abdagic, A. Vadi, A. György, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang, A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, B. Wu, B. Shahriari, B. Petrini, C. Chen, C. L. Lan, C. A. Choquette-Choo, C. Carey, C. Brick, D. Deutsch, D. Eisenbud, D. Cattle, D. Cheng, D. Paparas, D. S. Sreepathihalli, D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-Plucińska, H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, I. Ballantyne, I. Szpektor, I. Nardini, J. Pouget-Abadie, J. Chan, J. Stanton, J. Wieting, J. Lai, J. Orbay, J. Fernandez, J. Newlan, J. Ji, J. Singh, K. Black, K. Yu, K. Hui, K. Vodrahalli, K. Greff, L. Qiu, M. Valentine, M. Coelho, M. Ritter, M. Hoffman, M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, N. Chauhan, N. Sachdeva, O. Bunyan, P. Botarda, P. Caron, P. K. Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, P. Xu, P. Stanczyk, P. Tafti, R. Shivanna, R. Wu, R. Pan, R. Rokni, R. Willoughby, R. Vallu, R. Mullins, S. Jerome, S. Smoot, S. Girgin, S. Iqbal, S. Reddy, S. Sheth, S. Põder, S. Bhatnagar, S. R. Panyam, S. Eiger, S. Zhang, T. Liu, T. Yacovone, T. Liechty, U. Kalra, U. Evci, V. Misra, V. Roseberry, V. Feinberg, V. Kolesnikov, W. Han, W. Kwon, X. Chen, Y. Chow, Y. Zhu, Z. Wei, Z. Egyed, V. Cotruta, M. Giang, P. Kirk, A. Rao, K. Black, N. Babar, J. Lo, E. Moreira, L. G. Martins, O. Sanseviero, L. Gonzalez, Z. Gleicher, T. Warkentin, V. Mirrokni, E. Senter, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, Y. Matias, D. Sculley, S. Petrov, N. Fiedel, N. Shazeer, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, J. Alayrac, R. Anil, Dmitry, Lepikhin, S. Borgeaud, O. Bachem, A. Joulin, A. Andreev, C. Hardin, R. Dadashi, and L. Hussenot Gemma 3 technical report. External Links: 2503.19786, Link Cited by: §4.1.
• Gu et al. (2024) K. Gu, R. Shang, R. Jiang, K. Kuang, R. Lin, D. Lyu, Y. Mao, Y. Pan, T. Wu, J. Yu, Y. Zhang, T. M. Zhang, L. Zhu, M. A. Merrill, J. Heer, and T. Althoff BLADE: benchmarking language model agents for data-driven science. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 13936–13971. External Links: Link, Document Cited by: §2.
• He et al. (2026) Z. He, Y. Wang, C. Zhi, Y. Hu, T. Chen, L. Yin, Z. Chen, T. A. Wu, S. Ouyang, Z. Wang, J. Pei, J. McAuley, Y. Choi, and A. Pentland MemoryArena: benchmarking agent memory in interdependent multi-session agentic tasks. External Links: 2602.16313, Link Cited by: §2.
• Hu et al. (2026) Y. Hu, Y. Wang, and J. McAuley Evaluating memory in LLM agents via incremental multi-turn interactions. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2.
• Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed Mistral 7B. External Links: 2310.06825, Link Cited by: §4.1.
• Jiayang et al. (2026) C. Jiayang, D. Ru, L. Qiu, Y. Li, X. Cao, Y. Song, and X. Cai AMemGym: interactive memory benchmarking for assistants in long-horizon conversations. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, pp. 4413–4446. External Links: Link Cited by: §1, §2.
• Kapoor and Narayanan (2023) S. Kapoor and A. Narayanan Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4 (9), pp. 100804. External Links: ISSN 2666-3899, Document, Link Cited by: §2.
• Kon et al. (2026) P. T. J. Kon, Q. Ding, J. Liu, X. Zhu, J. Peng, J. Xing, Y. Huang, Y. Qiu, J. Srinivasa, M. Lee, M. Chowdhury, M. Zaharia, and A. Chen EXP-Bench: can AI conduct AI research experiments?. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, pp. 120555–120604. External Links: Link Cited by: §1, §2.
• Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, New York, NY, USA, pp. 611–626. External Links: ISBN 9798400702297, Link, Document Cited by: §4.1.
• Laurent et al. (2024) J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Ponnapati, A. D. White, and S. G. Rodriques LAB-Bench: measuring capabilities of language models for biology research. External Links: 2407.10362, Link Cited by: §2.
• Ma et al. (2026a) E. Ma, Y. Zhou, W. Huang, J. Yang, H. Ma, Z. Wang, C. Li, C. Miao, P. S. Yu, and Z. Wang MemAudit: auditing long-term agent memory via hidden user-state recovery. External Links: 2606.24595, Link Cited by: §2.
• Ma et al. (2026b) Z. Ma, J. Liu, H. Tang, Y. Ma, R. Wang, Y. Yuan, Y. ZHENG, and J. HAO Benchmarking continual agent memory for online learning, transfer, and forgetting. In ICLR 2026 Workshop on Lifelong Agents: Learning, Aligning, Evolving, External Links: Link Cited by: §2.
• Maharana et al. (2024) A. Maharana, D. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y. Fang Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 13851–13870. External Links: Link, Document Cited by: §1, §2.
• Majumder et al. (2025) B. P. Majumder, H. Surana, D. Agarwal, B. Dalvi Mishra, A. Meena, A. Prakhar, T. Vora, T. Khot, A. Sabharwal, and P. Clark DiscoveryBench: towards data-driven discovery with large language models. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, pp. 4556–4579. External Links: Link Cited by: §2.
• Moreau and Missier (2013) L. Moreau and P. Missier PROV-DM: the PROV data model. Note: W3C Recommendation External Links: Link Cited by: §2.
• Nosek et al. (2018) B. A. Nosek, C. R. Ebersole, A. C. DeHaven, and D. T. Mellor The preregistration revolution. Proceedings of the National Academy of Sciences 115 (11), pp. 2600–2606. External Links: Document, Link, https://www.pnas.org/doi/pdf/10.1073/pnas.1708274114 Cited by: §1, §2.
• Packer et al. (2024) C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez MemGPT: towards LLMs as operating systems. External Links: 2310.08560, Link Cited by: §2.
• Panigrahi et al. (2026) S. S. Panigrahi, J. Videnović, and M. Brbic HeurekaBench: a benchmarking framework for AI co-scientist. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, pp. 11029–11061. External Links: Link Cited by: §1, §2.
• Park et al. (2023) J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein Generative agents: interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New York, NY, USA. External Links: ISBN 9798400701320, Link, Document Cited by: §2.
• Ribeiro et al. (2020) M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh Beyond accuracy: behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, pp. 4902–4912. External Links: Link, Document Cited by: §2.
• Shah (2026) J. Shah Causal agent replay: counterfactual attribution for LLM-agent failures. External Links: 2606.08275, Link Cited by: §2.
• Shao et al. (2026) J. Shao, Y. Lin, M. P. Lohani, Y. Miao, and B. Luo Do LLM agents know how to ground, recover, and assess? evaluating epistemic competence in information-seeking agents. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2.
• Shen et al. (2026) Y. Shen, K. Li, W. Zhou, and S. Hu Mem2ActBench: a benchmark for evaluating long-term memory utilization in task-oriented autonomous agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, pp. 8173–8190. External Links: Link, Document, ISBN 979-8-89176-390-6 Cited by: §2.
• Sheverev et al. (2026) M. Sheverev, D. Finkelstein, and S. Nikolenko Beyond memory leaderboards: evaluating scientific memory as budgeted context restoration. External Links: 2607.16848, Link Cited by: §2.
• Tan et al. (2025) H. Tan, Z. Zhang, C. Ma, X. Chen, Q. Dai, and Z. Dong MemBench: towards more comprehensive evaluation on the memory of LLM-based agents. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 19336–19352. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §2.
• Tavakoli et al. (2026) M. Tavakoli, A. Salemi, C. Ye, M. Abdalla, H. Zamani, and J. R. Mitchell Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs. In International Conference on Learning Representations, External Links: Link Cited by: §2.
• Uddin et al. (2026) M. N. Uddin, K. Shubham, E. Blanco, C. Baral, and G. Wang From recall to forgetting: benchmarking long-term memory for personalized agents. In Findings of the Association for Computational Linguistics: ACL 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, pp. 26814–26841. External Links: Link, Document, ISBN 979-8-89176-395-1 Cited by: §2.
• Wang et al. (2024) X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, S. Zhang, Y. Sun, and W. Wang SciBench: evaluating college-level scientific problem-solving abilities of large language models. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, pp. 50622–50649. External Links: Link Cited by: §2.
• Wang et al. (2026a) X. Wang, J. Xu, S. Lian, Y. Chen, H. Guo, F. Zhu, Y. Shao, M. Ren, H. Yi, A. H. Feng, H. Yang, T. Wu, H. Hu, S. Xiang, X. Zhang, and C. Liu The hitchhiker’s guide to autonomous research: a survey of scientific agents. IEEE Transactions on Pattern Analysis and Machine Intelligence (), pp. 1–20. External Links: Document, Link Cited by: §2.
• Wang et al. (2026b) Y. Wang, Z. Zhang, M. Chi, K. Yu, Y. Li, M. Peng, B. Tong, C. Zhang, Y. Zhou, and J. Li EvoMemBench: benchmarking agent memory from a self-evolving perspective. External Links: 2605.18421, Link Cited by: §2.
• Wu et al. (2026) D. Wu, Z. Ji, A. Kawatkar, B. Kwan, J. Gu, N. Peng, and K. Chang LongMemEval-V2: evaluating long-term agent memory toward experienced colleagues. External Links: 2605.12493, Link Cited by: §2.
• Wu et al. (2025) D. Wu, H. Wang, W. Yu, Y. Zhang, K. Chang, and D. Yu LongMemEval: benchmarking chat assistants on long-term interactive memory. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, pp. 86809–86836. External Links: Link Cited by: §1, §2.
• Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link Cited by: §4.1.
• Yang et al. (2026) R. Yang, Y. Jiang, Y. Jiang, P. Kargupta, Y. Zhang, and J. Han Grounding agent memory in contextual intent. In Findings of the Association for Computational Linguistics: ACL 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, pp. 12008–12042. External Links: Link, Document, ISBN 979-8-89176-395-1 Cited by: §2.
• Zhang et al. (2025) Z. Zhang, Q. Dai, X. Bo, C. Ma, R. Li, X. Chen, J. Zhu, Z. Dong, and J. Wen A survey on the memory mechanism of large language model-based agents. ACM Trans. Inf. Syst. 43 (6). External Links: ISSN 1046-8188, Link, Document Cited by: §2.
• Zhao et al. (2026) Y. Zhao, B. Yuan, J. Huang, H. Yuan, Z. Yu, H. Xu, L. Hu, A. Shankarampeta, Z. Huang, W. Ni, Y. Tian, and J. Zhao AMA-Bench: evaluating long-horizon memory for agentic applications. External Links: 2602.22769, Link Cited by: §2.

Appendix A Benchmark Design Details

A.1 Catalog coverage

The benchmark contains 64 executable TaskSpecs spanning five scientific-use validity modules and six domains. Modules define why a record may or may not govern a decision. Domains determine the scientific objects, units, and event language used to express that rule. This factorization allows either axis to expand without redesigning the evaluation interface. The resulting coverage is summarized in Table 5.

A.2 Executable TaskSpec contract

A TaskSpec is a program that produces a reproducible family of scientific worlds, not a fixed natural-language question. It contains every component needed to generate a history, compile the decision-governing state, intervene on that state, and score the resulting action. Table 6 specifies the shared executable interface.

The stored specification also fixes its schema version, episode schedule, visibility policy, legal choices, prompt surface, corruption operators, and data split. Before model execution, validators reject a counterfactual whose oracle relation is inconsistent and a repair that does not reconstruct the target state.

A.3 Representative module examples

Table 7 instantiates the common interface once per module, highlighting the state relation that changes or preserves the valid decision.

Appendix B SciStateForge Compilation and Memory Views

B.1 Canonical event layer

SciStateForge first converts domain-specific histories into a shared event layer. Domain payloads remain versioned data rather than being embedded in free-form notebook prose. Table 8 lists the normalized fields that every domain supplies.

B.2 Deterministic compilation

For a world w and TaskSpec τ, the compiler performs four passes:

• 1.

  validate and order the public canonical events Epub​(w);

• 2.

  instantiate typed objects for plans, observations, constructs, access records, absences, costs, and pending decisions;

• 3.

  derive governing relations such as Selected_From, Stopped_After, Sealed_Until, Comparable_With, Absent_Because, and Relevant_To;

• 4.

  project public nodes and relations into a model-facing view while retaining links from derived fields to their supporting events.

The resulting state Gτ​(w)=Cτ​(Epub​(w)) is a deterministic closure over the visible history. It introduces no new observation: the compiler makes cross-record consequences explicit while keeping oracle-only generator state outside the reader input.

B.3 Memory-view definitions

The five views in Table 9 progressively separate raw history, retrieval, generic event structure, and compiled scientific state.

Input length does not track representation quality: the median counts in Table 10 shows that Event Graph is the longest view for every reader.

Appendix C Experimental Details

C.1 Readers and evaluation matrices

The open-weight evaluation uses Qwen3-14B, Gemma-3-12B-IT, Mistral-7B-Instruct-v0.3, and Phi-3.5-mini-instruct. Each model is served alone through vLLM with deterministic decoding and the same choice-only output contract. Invalid JSON, illegal choices, refusals, and unparsable responses remain invalid decisions rather than being resampled.

Table 11 connects each evaluation matrix to its scientific question and reports the number of model decisions.

C.2 Estimands and uncertainty

For views a and b, the analysis averages paired seed differences within TaskSpec t,

and then weights TaskSpecs equally. The reported 95% intervals use 20,000 TaskSpec resamples. Family-balanced controls instead weight the 12 module–design families equally and resample those families.

For counterfactual pairs, relation accuracy tests whether the reader’s two actions obey the required flip or invariance. Both-world validity requires the oracle action on each side. Repair specificity subtracts recovery after an edit-matched irrelevant change from recovery after restoring the damaged target.

C.3 Scoring and completion

All program-valid tasks in a scheduled matrix enter its denominator. The scorer records action validity, parse status, pair relation, TaskSpec version, and the model-visible input. Infrastructure retries do not replace a terminal model response, preventing the evaluation from retaining only easy-to-parse or successful outputs.

Appendix D Complete Representation Results

D.1 Absolute rates and paired contrasts

Compiled views lead Event Graph for every open reader. Flat SSG is slightly higher than typed SVL in this matrix, indicating that their shared scientific state drives the cross-family improvement. Table 12 reports absolute validity. Table 13 gives the paired representation effects.

D.2 Module and domain structure

Blind Memory and Inferential Ledger consistently reward explicit governing relations. Sunk Cost instead tests whether a reader can suppress an irrelevant historical investment. Exposing more structured history does not guarantee an improvement when the reader remains attracted to that distractor. The complete reader–module matrix appears in Table 14.

The typed-SVL minus Event Graph contrast is positive in all 24 reader–domain cells. Clinical and causal cells produce the largest gains for Qwen and Gemma, while Mistral peaks in bioinformatics. These descriptive differences partly reflect the catalog’s unequal module mix and do not isolate a causal domain effect. Phi’s effects are smaller but remain positive across the six domains (Table 15).

D.3 Dependence and shortcut sensitivities

Equal family weighting and leave-one-family-out analysis retain positive effects for all readers. This prevents a large template family from dominating the catalog-wide estimate. Table 16 reports both estimates.

Some TaskSpecs contain a public state value whose literal string equals a legal choice. Removing all 22 such tasks leaves 42 tasks without this equality. Every reader retains a positive typed-SVL minus Event Graph effect, ruling out direct value copying as the explanation for the main pattern (Table 17).

Appendix E Counterfactual State-Use Evaluation

E.1 Pair construction

A companion world changes one decisive public value while preserving the TaskSpec, seed, request, legal choices, surface language, and non-target information. The oracle states whether the correct action must flip or remain invariant. Relation accuracy is one only when the reader’s two actions obey that requirement.

This test is stricter than scoring the worlds separately. Repeating exploratory, for example, may be correct on the contaminated side of an Inferential Ledger pair but cannot count as state tracking when the clean side requires confirm. Conversely, a Sunk Cost pair can change past investment while requiring an invariant action.

E.2 Aggregate and module-level results

Table 18 compares single-world validity, both-world validity, and relation accuracy for the two compiled views.

Qwen follows the required relation almost perfectly in both compiled views, and Gemma remains above 0.90. Mistral and Phi show a larger gap between single-world validity and state-dependent consistency. The module breakdown localizes these differences: Mistral benefits from the typed Blind representation but remains weak on Inferential pairs. Phi’s Sunk Cost relation accuracy is 1.000 in both views, but both-world validity is only 0.500, so this invariant-pair score alone does not establish correct state use. Phi also fails most Absence pairs (Table 19).

Appendix F Corruption and Repair Diagnostics

F.1 Reversible state interventions

Each corruption applies one TaskSpec-compatible edit to the compiled state. The operator has a deterministic inverse and an edit signature used to create a size-matched irrelevant repair. Table 20 defines the six edit types.

The 24-task matrix contains 60 compatible TaskSpec–operator pairs. Each task–seed–reader contributes one shared intact baseline, while each compatible pair adds three intervention worlds: target corruption, exact target repair, and a matched edit at an unrelated location. Thus the matrix contains 24×6×4+60×6×4×3=4,896 decisions. Exact repair restores the original canonical state. The matched condition leaves the decisive damage in place, separating recovery caused by the governing information from recovery caused by a generic edit.

F.2 Repair specificity and operator fingerprints

Table 21 contrasts exact and irrelevant repair for each reader and module.

Exact repair restores 26.9 points for Qwen, 32.8 for Gemma, and 15.0 for Mistral relative to a matched irrelevant edit. Phi changes by -0.3 points. Blind and Absence damage are recoverable in the first three readers, whereas Sunk Cost and Inferential repairs expose weaker or reader-specific dependence.

For each operator, a ten-dimensional vector records target-versus-corrupted and target-versus-matched effects across the five modules. Missing module–domain cells contribute zeros. For each operator, we compute the centroid over nonzero domain vectors and report the mean cosine to that centroid. All-zero vectors are omitted, and an operator with fewer than two nonzero vectors receives 0.0. Cross-domain cosine similarity measures whether the resulting failure pattern is stable (Table 22).

Qwen and Gemma retain stable fingerprints for five operators, Mistral for Omit and Relabel, and Phi for Omit. Value pairs and repairs thus answer different questions: the former tests whether an intact reader follows a changed state; the latter asks whether restoring a damaged component selectively recovers valid decisions.

Appendix G Source and Serialization Controls

G.1 Upstream intervention design

The source-control evaluation changes one decisive value in the canonical public record before constructing any model input. Both worlds are then rendered as Raw Events, BM25 Events, Event Graph, Matched Source, Flat SSG, typed SVL, Opaque Flat, and Opaque SVL. Every view therefore inherits the same source pair, oracle relation, seed, request, and legal choices.

Matched Source exposes the complete public source-fact closure while forbidding compiler-derived fields. Neutral padding equalizes its full-chat token count with Flat SSG separately for every reader, TaskSpec, seed, and world. Opaque views replace state values, relation names, and action labels through a TaskSpec-specific codebook shared across both worlds and serializations. The two controls isolate compiled relations from input length and familiar scientific terminology. Table 23 reports the five contrasts needed to separate these explanations.

Flat SSG remains above Event Graph for every reader after the shared upstream intervention. Under exact length matching, Flat also outperforms Matched Source for Qwen and Mistral and remains similar for Gemma. These controls separate the benefit of materialized scientific relations from retrieval depth and context length.

Appendix H Independent Replications and Broader Memory Policies

H.1 Hosted-reader replication

Two hosted readers were evaluated on two independently generated seed blocks. The compiled-state advantage is positive in all four reader–block combinations, ranging from 47.0 to 48.6 points (Table 24).

H.2 Independent Qwen replication

A 4,480-decision independent Qwen evaluation reproduces the open-weight result. Typed SVL exceeds Event Graph by 18.6 points, while Flat SSG and typed SVL remain within one point (Table 25).

H.3 Additional memory-policy evaluation

An independently constructed 24-TaskSpec set spans all five modules and six domains. Across four open readers and ten unseen seeds, it produces 6,720 decisions. SVL exceeds the mean of BM25, summary, selective, and provenance memory by 28.5, 29.9, 14.4, and 11.1 points for Qwen, Gemma, Mistral, and Phi. The TaskSpec-resampling intervals remain positive for every reader (Table 26).

Table 27 decomposes the same comparison by validity module.

Construct Drift and Absence Algebra yield the largest extension effects. Sunk Cost is tied because its correct operation is to suppress irrelevant history, not expose more of it. Mistral’s negative Blind effect alongside its large Construct gain further demonstrates an interaction between representation, reader capability, and validity rule.

The structural audit in Table 28 explains which scientific objects and governing relations remain available under each policy.

Provenance preserves every node but none of the required governing edges. Evidence-Carrying Memory (ECM) retains 88.6% of nodes and scores 55.6% on Endpoints, the fraction of missing required edges with both endpoints visible (1.0 when none are missing). The behavioral results measure whether each reader can exploit the structure that remains.

Appendix I Observed Boundaries and Extensions

The best serialization varies by reader even when compiled content is fixed. Flat and typed views are nearly equivalent for Qwen and Mistral in the source-control evaluation, whereas typed SVL substantially improves Gemma and Phi. Scientific-memory evaluation should therefore vary both state content and reader-facing representation. Single-world validity can overstate state use. Mistral and Phi retain moderate validity on compiled views while their counterfactual relation accuracy falls well below Qwen and Gemma. Paired interventions separate sensitivity to the decisive value from an action that is merely plausible in one world.

More retained history is not always beneficial. Sunk Cost tasks require the reader to exclude past effort when future payoffs are unchanged. Future compilers should represent relevance and exclusion constraints, not only add facts and relations. The six domains provide controlled surfaces for shared validity rules. Longer histories, multimodal records, laboratory handoffs, and field-specific simulators can broaden ecological coverage without changing the TaskSpec oracle or intervention interface. Learned ingestion and memory-writing systems can be evaluated by comparing their induced state and downstream decisions with these contracts.

注意转载授权 · 阅读原文