Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agents

   Yonathan Efroni Affiliation: Tel Aviv University    Paul Sajda Affiliation: Columbia University    Kaveh Hassani Affiliation: Meta AI Joint last authors    Anirudh Goyal Affiliation: Meta AI Joint last authors

Abstract

Long-horizon research agents must decide both how to investigate and what to investigate next as evidence accumulates. This is hard to learn because such decisions are sparse in long execution traces, and their consequences may emerge several investigations later. We introduce Meta-reasoning for Iterative Research Agents (MIRA), a hierarchical architecture separating research allocation from execution. An outer-loop meta-reasoner curates context from a persistent research record, then writes a work order for the next investigation or ends the episode. A fresh inner-loop executor carries out each work order, making execution part of the transition between meta-reasoning actions. Without policy training, MIRA improves long-horizon inference and allocates additional compute more effectively in theorem proving and open-ended neural-architecture research. Its decision boundaries also provide natural units for credit assignment. At each boundary, we train a generative critic to forecast expected remaining return from partial states, outperforming token-level alternatives. Cross-environment pretraining improves forecasting and adaptation, yielding a transferable prior for valuing partial progress. We use this prior to initialize MIRA-AC, a generative actor–critic jointly trained to forecast remaining return and choose the next investigation, without a separate critic model. MIRA-AC concentrates policy optimization on meta-reasoning decisions, enabling efficient long-horizon reinforcement learning without directly optimizing the longer execution traces they initiate. Training MIRA-AC on the model’s own proxy hill-climbing signals improves gold performance across four autoresearch environments; the actor transfers with cross-environment value initialization. Together, these results show that meta-reasoning can be learned as an explicit policy for directing long-horizon autonomous research.

^††correspondence: Ankur Samanta (as7416@columbia.edu), Anirudh Goyal (agi@meta.com)^††date: October 1, 2026

1 Introduction

Long-horizon research requires more than executing individual investigations well. As evidence accumulates, an agent must decide which hypothesis to test, which result to verify, whether to refine or abandon a direction, and when to stop. These decisions draw on evidence from many experiments (Gao et al., 2026; Zou et al., 2026), which can quickly exceed the model’s context window (Kausik et al., 2026). The decisions themselves are sparse within much longer execution traces, and their consequences may be delayed (Sutton and Barto, 2018; Samanta et al., 2026a): an unsuccessful experiment can eliminate an important hypothesis, while a locally promising direction can ultimately lead to a dead end. In this sense, the quality of a research trajectory often depends less on the outcome of any single investigation than on the choices that determine what to investigate next. Learning to conduct research therefore requires isolating the decisions that allocate future research effort from the executions they initiate and assigning them credit for their downstream consequences. This motivates the central question of this work:

Can a research agent learn what to investigate next from the downstream consequences of its previous research decisions?

We view this as a problem of meta-reasoning: deciding which computation a resource-bounded agent should perform next and when it should stop (Russell and Wefald, 1991). For research agents, we take the relevant computation to be an investigation. Sec. 2 formalizes research at the level of investigations. Sec. 3 introduces MIRA, a hierarchical architecture that separates deciding what to investigate from carrying out the investigation. At decision t, the outer loop curates evidence from a persistent research record into an observation ot. The meta-reasoning policy then samples a complete decision response at, comprising its reasoning followed by either a work order or a termination decision. For a work order, a fresh executor carries out the investigation and updates the persistent research record.

At the meta-reasoning level, the potentially long executor trace is therefore compressed into one decision-level transition ξt=(ot,at,rt,ot+1,dt), where dt denotes true episode termination. The complete decision response is the action; its implementation, experimentation, and tool use are part of the environment transition. This isolates the decisions that determine how future research effort is allocated from the longer computation used to execute them.

Sec. 4 develops a model of remaining research value. Making research decisions explicit also creates a natural boundary for value prediction and credit assignment. Because an investigation’s immediate outcome need not reflect its eventual contribution, we learn its expected remaining research return Vπ​(ot)=𝔼π​[Gt∣ot]. We train a decision-level generative critic to forecast this quantity once per research decision from the same observation available to the policy. It uses the language model’s native next-token distribution to represent beliefs over future returns (Zhang et al., 2025; Bellemare et al., 2017). The critic forecasts the future potential of the current curated research observation rather than ranking candidate work orders.

Sec. 5 uses this signal to assign delayed credit to the research policy. We introduce MIRA-AC, a shared generative actor–critic jointly trained to forecast remaining return and choose what to investigate next from each curated research observation. Because one model plays both roles, no separate critic model is required. MIRA-AC reconstructs complete research episodes to assign delayed credit to research decisions. A single decision-level advantage is propagated to every sampled token in the corresponding decision response, including its reasoning and resulting work order or termination decision. Context curation, executor generations, experiments, and tool calls receive no direct policy loss. MIRA-AC therefore concentrates policy optimization on what to investigate next without needing to directly optimize the much longer execution trajectories. We train it with a buffered single-rollout asynchronous optimization scheme (Hou et al., 2026) that accommodates variable-duration research episodes. Overall, we show that jointly training value forecasting and meta-reasoning from the same curated research observations—while assigning credit directly to the resulting decisions—can produce a stronger policy.

Sec. 6 evaluates each stage in turn. First, untrained MIRA improves long-horizon inference in theorem proving on IMOProofBench (Luong et al., 2025) and in open-ended neural-architecture research with Residual Matrix Transformers (Mak and Flanigan, 2025) and Loop Transformers (Geiping et al., 2025). In the latter settings, it selects per-cell residual-write gates for RMT and signed attention with a leaky recurrent update for the Loop Transformer. Second, a decision-level generative critic forecasts remaining return more accurately than token-level alternatives, and cross-environment pretraining improves value forecasting and adaptation. Third, under proxy supervision, the strongest decision-level policy-learning configuration improves the primary outcome over token-level regression across symbolic regression (Shojaee et al., 2025), physics discovery (Wiemann et al., 2026), Bayesian causal discovery on networks from the Bayesian Network Repository (Scutari, n.d.), and CPU-based neural-architecture research. Crucially, training on proxy hill-climbing feedback collected from the model’s own research trajectories improves gold-evaluation performance across all four environments. MIRA-AC provides additional gains in several environments, and a policy trained without BNLearn actor updates transfers to BNLearn when initialized from a critic pretrained across environments, including BNLearn. This separation between proxy training and gold evaluation mirrors autoresearch settings in which proxy evaluations support inexpensive iteration, while gold evaluation may, for example, require conducting in vivo experiments (Stokes et al., 2020) or running a large-scale production system (Novikov et al., 2025). Together, these results show that research allocation can be effectively governed by an explicit, learnable meta-reasoning policy rather than remaining an implicit byproduct of long-horizon generation.

2 Research as a Sequential Decision Process

Long-horizon research interleaves two kinds of computation: deciding what to investigate next and carrying out the chosen investigation. Our central modeling choice is to place the learning boundary between them. One research decision is one action; when that action selects an investigation, the potentially much longer computation used to execute it is part of the environment transition.

2.1 Decision Process

Let Ht denote the persistent research state immediately before research decision t. It contains the current artifact together with evidence accumulated from earlier investigations and the remaining research budget. Meta-reasoning includes both evidence curation and action selection. In this work, we hold curation fixed and train the core action-selection policy. A fixed context-curation procedure C constructs the observation ot=C⁡(Ht) presented to that policy. Retrieval and summarization used to construct ot belong to observation construction rather than the action.

Conditioned on ot, the policy selects at∼πθ(⋅∣ot), where at is the complete sampled meta-reasoning response: its emitted reasoning followed by either a work order or a termination decision. A work order specifies a research objective, a proposed approach, and criteria for determining whether the investigation succeeded.

For a work order, an executor ℰ carries out the investigation. Its execution may contain many model generations, tool calls, experiments, and revisions, but remains part of a single environment transition, Ht→at,ℰHt+1.

Let dt∈{0,1} indicate true termination after decision t. For a non-terminal transition, the next observation is ot+1=C⁡(Ht+1). For a terminal transition, we write ot+1=⊥ and use the convention V⁡(⊥)=0. A termination decision is itself a terminal research action.

At the meta-reasoning level, the stored transition is therefore ξt:=(ot,at,rt,ot+1,dt). Context resets, executor restarts, and learner-batch boundaries do not set dt=1.

Writing T=min⁡{t:dt=1}, a logical research episode is τ=(ξ1,…,ξT).

Thus, the unit of control is the research decision: the policy chooses what to investigate or whether to terminate, while the executor determines how a selected investigation is carried out.

2.2 Research Feedback and Return

Our environments expose proxy feedback during research and apply a separate gold evaluation only after final submission. The gold evaluator reports the task-specific final metrics and is never available to the agent during the research episode.

Proxy feedback.

Let qt∈[0,1] be the normalized proxy quality after decision t, with q0 the initial quality, and define q¯t=max0≤u≤t⁡qu. The proxy reward is the improvement in the incumbent, rtproxy=q¯t−q¯t−1. Hence the remaining proxy return telescopes: Gtproxy=∑u=tTruproxy=q¯T−q¯t−1. Each environment uses a monotone proxy normalization fixed before training. The proxy and gold evaluations need not rank artifacts identically.

Gold-binary feedback.

Gold evaluation may contain continuous metrics as well as an environment-specific binary success criterion. For the Gold binary training condition, let Y∈{0,1} denote this terminal success outcome. Intermediate rewards are zero and the terminal reward is Y, so Gtgold​-​bin=Y.

For either feedback signal, we write Gt=∑u=tTru for the undiscounted remaining return. Environment-specific definitions of qt and Y are given in App. B.

2.3 Remaining Research Value

For a fixed continuation policy and research budget, we define Vπ​(ot)=𝔼π​[Gt∣ot]. Because ot is a curated observation, this is an observation-conditioned value rather than a value of the complete research state. Under proxy feedback, Vπ​(ot) is the expected additional best-so-far improvement from continuing research. Under gold-binary feedback, it is the probability of success under the continuation policy. It is not the maximum improvement theoretically attainable from the underlying research state.

The remainder of the method instantiates this decision process: MIRA exposes the research actions, a critic forecasts remaining value from the resulting observations, and MIRA-AC assigns downstream credit to the policy that selects them.

3 MIRA: Exposing Meta-Reasoning Decisions

The decision-level formulation in Sec. 2 requires an agent architecture that exposes individual investigations as explicit actions while preserving evidence across a much longer research trajectory. In particular, the system must maintain research state across investigations, separate the choice of what to investigate from the computation used to carry it out, and produce a well-defined decision boundary at which outcomes can later be valued and credited. MIRA realizes these requirements with a persistent research substrate, an outer-loop meta-reasoner, and a fresh inner-loop executor.

3.1 MIRA Architecture

Separating research allocation from execution requires evidence to persist across investigations. We thus introduce MIRA, a hierarchical research harness in which an outer-loop meta-reasoner and an inner-loop executor are instantiated afresh at each iteration over a persistent research substrate.

Persistent research substrate.

Each task has a Git repository that persists throughout the research episode. It stores the current artifact, prior experiments and their outcomes, analyses, code or proof artifacts, and handovers from earlier executors. The repository implements the persistent research state Ht; model contexts are temporary.

Outer-loop meta-reasoner.

At each decision boundary, the context-curation procedure constructs ot from the persistent record. The meta-reasoning policy then samples the complete decision response at: its emitted reasoning followed by either a work order or a termination decision. A work order specifies the next objective, proposed approach, and success criteria. The outer loop therefore allocates the next unit of research effort.

Fresh inner-loop executor.

Each work order launches a fresh execution agent through the Codex harness (OpenAI, 2025), with the underlying model specified by the experiment. It receives the fixed task instructions, the work order, and access to the persistent workspace and task-specific tools. The executor performs the investigation, updates the artifact, and commits a handover recording its evidence and unresolved questions. Both outer- and inner-loop contexts reset between decisions; only the persistent research substrate survives. Fig. 1 illustrates this separation between research allocation and execution.

3.2 MIRA Research Procedure

Algorithm 1 formalizes this research procedure. Each outer-loop decision produces exactly one decision-level transition, regardless of the length of the corresponding executor trace. The shared procedure and executor interface remain fixed across tasks; App. C gives the context construction, evaluation, and submission details.

3.3 Learning Boundary

MIRA’s architectural separation also determines where MIRA-AC applies learning. It optimizes the policy that generates decision responses and the critic that forecasts remaining return, while holding curation, execution, and the task environment fixed.

Making research allocation explicit identifies where to assign credit. The remaining challenge is that a decision’s consequences may become clear only after several later investigations; Sec. 4 forecasts remaining value from the resulting observations, and Sec. 5 uses it for policy optimization.

4 Learning to Forecast the Future Value of Research Progress

An investigation’s immediate outcome need not reflect its eventual contribution. A failed experiment may eliminate an incorrect hypothesis and enable later progress, while an immediate improvement may lead to a dead end. We therefore learn a value function Vπ​(ot) at MIRA’s research-decision boundaries.

The environments report progress in different raw units, so we use the gold and proxy returns defined in Sec. 2.2. Each environment maps its proxy metric through a fixed environment-specific transform to a quality in [0,1]. These transforms are scaled to distinguish improvements attainable within the research budget and convert partial trajectories into marginal rewards. App. B.2 describes their anchors, relation to gold evaluation, and construction for new environments.

4.1 Decision-Level Generative Critic

The critic produces one value forecast immediately before each research action. This aligns value forecasting with the unit of control, rather than with intermediate token prefixes inside the language-model generation of a decision response.

We parameterize the critic using the language model’s next-token distribution. Following Samanta et al. (2026b), we elicit its beliefs by renormalizing the next-token log probabilities assigned to a set of single-token labels, each representing a possible future-return value. Let 𝒵={z1<⋯<zK} denote these values. For observation ot, pϕ​(zi∣ot)=exp⁡ℓϕ,i​(ot)/∑k=1Kexp⁡ℓϕ,k​(ot), and Vϕ​(ot)=∑i=1Kpϕ​(zi∣ot)​zi.

We use support {0.0,0.1,…,1.0} for proxy returns and {0,1} for gold-binary outcomes. The critic forecasts expected remaining return under the continuation policy; it does not estimate maximum attainable improvement or score candidate work orders.

4.2 Critic Objective

The training target is the Monte Carlo return Gt from the complete research episode. For zj<Gt<zj+1, we project the target onto its neighboring support points by setting yj​(Gt)=(zj+1−Gt)/(zj+1−zj) and yj+1​(Gt)=(Gt−zj)/(zj+1−zj), with all other components zero. If Gt=zj, then yj​(Gt)=1. The projection preserves the target expectation.

For a batch ℬ, the critic minimizes

Operationally, we select the model’s next-token log probabilities for the K support labels and renormalize them over that support with a log-softmax. We then evaluate the KL objective above; its parameter-dependent term is −∑iyi(Gt)logpϕ(zi∣ot), because the target entropy is constant with respect to ϕ.

4.3 Cross-Environment Value Pretraining

Research environments differ in artifacts, tools, and objectives, but partial research observations may share predictive structure. For a target environment d∈𝒟, leave-one-environment-out (LOO) pretraining uses trajectories from the source set 𝒟∖{d}, after which the critic is adapted on target-environment support episodes and evaluated on held-out target query episodes. The experiments below compare this initialization with target-only and single-source training.

Value forecasting alone does not change the research policy. The next section uses downstream outcomes to assign credit to the decisions that select what to investigate.

5 MIRA-AC: Learning What to Investigate Next

We now ask whether value estimates at MIRA’s decision boundaries can improve the policy that selects research objectives. We introduce MIRA-AC, a shared generative actor–critic that assigns delayed credit across complete research episodes and uses buffered asynchronous optimization to accommodate their variable durations.

5.1 Shared Actor and Critic

MIRA-AC uses one language model with shared parameters θ. In actor mode it represents πθ; in critic mode its restricted next-token distribution defines Vθ as in Sec. 4.1.

The shared model is initialized from a pretrained critic checkpoint. Actor and critic objectives use separate optimizer states while updating the shared model. Thus, unlike a separate actor–critic, value learning also updates representations used to generate subsequent research decisions.

5.2 Credit Assignment Across Research Decisions

MIRA-AC reconstructs each logical episode τ before computing learning targets. Context resets and learner-batch boundaries may partition an episode in storage, but only dt=1 stops value propagation.

The critic is trained on the exact return Gt. For actor training, we use generalized advantage estimation (Schulman et al., 2016) with γ=1 and λT=1−1/(α​T), with the fixed hyperparameter α given in Table 3.

Because asynchronous collection allows successive decisions in one episode to use different critic versions, let ct denote the critic version used at decision t. At rollout time we record the current baseline vt=Vθct​(ot) and same-version successor bootstrap v~t+1(ct)=Vθct​(ot+1). The temporal-difference residual is δt=rt+γ⁡(1−dt)​v~t+1(ct)−vt.

At a non-terminal boundary where the critic version changes, define Bt=vt+1−v~t+1(ct), where vt+1=Vθct+1​(ot+1); otherwise Bt=0. Advantages are computed backward as

The bridge aligns critic baselines across model versions; it does not modify the Monte Carlo target. App. I gives the complete episode-reconstruction details.

5.3 Decision-Level Actor Objective

Although each research action receives one decision-level advantage, the language model generates it autoregressively. Let at,j denote token j of action at, let Lt be its sampled length, and let bt denote the behavior-policy version that generated it. Define ρt,j=πθ​(at,j∣ot,at,<j)/πθbt​(at,j∣ot,at,<j). We mask policy-gradient terms outside the allowed importance-ratio range using mt,j=𝟏[1−ϵlow<ρt,j<1+ϵhigh].

For a batch of N research decisions, the actor objective is

Tokens outside the ratio interval are excluded from the policy-gradient term but remain in the policy-mismatch penalty. The trainable span is the complete sampled meta-reasoning decision response, including its emitted reasoning and resulting work order or termination decision. Prompts, curated observations, executor generations, tool calls, and experiments are masked.

5.4 Buffered Single-Rollout Asynchronous Policy Optimization

Investigations have heterogeneous wall-clock durations, so synchronized rollout cohorts can leave capacity idle. MIRA-AC instead collects completed episodes continuously into separate actor and critic buffers, which trigger updates independently on the shared model. If both are ready, the critic update is applied first, followed by the actor update within the same atomic update transaction. The resulting parameters are published once. A live episode adopts newly published parameters only at its next research-decision boundary and may therefore span multiple model versions.

Algorithm 2 summarizes training. Buffer thresholds, optimizer settings, rollout limits, and details of low-rank adaptation (LoRA) (Hu et al., 2022) are given in App. H.

\FloatBarrier

6 Experiments

We evaluate the approach in four stages. First, we ask whether MIRA improves long-horizon inference before policy training. Second, we test whether remaining research value can be forecast at MIRA’s decision boundaries and whether this forecasting capability transfers across research environments. Third, we evaluate whether decision-level credit improves the research policy and whether the learned policy transfers across environments. Finally, we analyze how training changes research behavior and how MIRA’s decision-level decomposition affects where policy learning is applied and the cost of collecting long-horizon trajectories.

These stages place different demands on the experimental setting. To evaluate MIRA before learning, we use full-scale long-horizon settings in theorem proving and open-ended neural-architecture research. Value pretraining and policy learning, however, require many complete research trajectories. We therefore evaluate the learning components in scalable autoresearch environments that support repeated training and held-out evaluation.

6.1 Does MIRA Improve Long-Horizon Inference Without Policy Training?

Experimental setup.

All comparisons in this subsection use frozen model weights. We compare GPT-5.5, Codex, and MIRA. GPT-5.5 uses the task interface directly; Codex operates as one persistent tool-using session; and MIRA assigns each investigation to a fresh Codex inner loop. The Codex and MIRA conditions use the same underlying GPT-5.5 model, Codex harness, and tools, while MIRA adds explicit meta-reasoning and curated context resets. We test the three conditions on IMOProofBench (Luong et al., 2025) and HillClimbBench-RMT, and compare MIRA with Codex on HillClimbBench-Loop. For theorem proving, we also include output-token-matched continuation controls.

Theorem proving.

On IMOProofBench Advanced, we count a problem as solved only when the grader awards full credit. Using GPT-5.5, MIRA raises the mean normalized grader score from 67.1% under direct inference to 100%, solving all 30 problems in each of three trials. With Qwen3.6-27B (Qwen Team, 2026), MIRA also more than doubles the model’s score (Fig. 4(a)). All conditions receive the same runtime budget, but MIRA makes better use of its inference budget to iterate on its solution, while the baselines submit much earlier. To determine whether its gains come from this allocation or simply from generating more tokens, we force the baselines to continue until they match MIRA’s output-token count on each problem. This recovers most of the performance gap, but the residual gap is concentrated on four problems. On these problems, GPT-5.5 and Codex primarily extend or repair existing proof directions, whereas MIRA more often revisits earlier evidence and explores alternatives. We characterize this behavior using strategic inertia, the fraction of consecutive semantic steps within the same research direction. Across natural-stopping runs using GPT-5.5, mean inertia is 0.958 for direct inference, 0.862 for Codex, and 0.772 for MIRA. MIRA’s lower inertia means that it switches between proof directions more often. Fig. 5 illustrates this behavior on Problem 23; App. D provides the metric definition and additional examples.

Neural-architecture research.

We introduce HillClimbBench to evaluate MIRA in open-ended experimental research. Each environment starts from a task’s paper-derived reference architecture and asks the agent to improve it through code changes and experiments. We study two architecture-research tasks; Sec. 5 uses a separate 50-task collection, HillClimbBench-CPU, for policy learning. Each task provides proxy evaluations that the agent can use to hill-climb during research. After submission, a held-out gold evaluation scores the final design.

Residual Matrix Transformer.

The Residual Matrix Transformer (RMT) (Mak and Flanigan, 2025) replaces the usual vector-valued residual stream of the Transformer architecture (Vaswani et al., 2017) with a matrix that each layer reads from and writes to. The research problem is to regulate where attention and MLP updates write into this state without discarding its matrix structure. HillClimbBench-RMT asks the agent to improve the architecture under that constraint. We run matched 24-hour GPT-5.5 campaigns under the three orchestration conditions. GPT-5.5 stays close to conventional training and scaling choices, while Codex refines an RMT-specific normalization family. After more elaborate routing mechanisms prove ineffective at scale, MIRA shifts from normalization and cross-axis mixing to local write control and selects learned per-cell residual-write gates. MIRA achieves the lowest proxy and gold loss among the three campaigns (Fig. 4(b)). App. E provides the matched controls, architecture, initialization, and ablations.

Loop Transformer.

A Loop Transformer (Geiping et al., 2025) repeatedly applies a shared block, increasing effective depth without introducing distinct parameters at every iteration. Its central challenge is to revise the recurrent state without erasing information needed by later applications of the block. HillClimbBench-Loop asks the agent to improve this information flow. We compare matched 48-hour MIRA and Codex campaigns using GPT-6 Astra. MIRA first hypothesizes that repeated core applications overwrite useful state. When an adaptive retention gate does not beat constant interpolation with the previous state, it retains the simpler leaky update. After memory and routing alternatives produce small or inconsistent gains, MIRA hypothesizes that repeated positive attention may reinforce information that later recurrences struggle to remove. It arrives at what it calls signed attention, which subtracts one attention read over shared values from another, and tests it against a matched positive-mixture control before scaling. This resembles a fixed-coefficient variant of Differential Attention (Ye et al., 2025) applied in a Loop Transformer. MIRA achieves lower NLL than Codex under both proxy and gold evaluation. Both campaigns also outperform the task reference architecture on the proxy metric. The selected RMT and Loop designs both regulate updates to persistent model state: the RMT design scales how strongly attention and MLP branches write to each matrix cell, while the Loop design allows recurrent attention to suppress as well as reinforce information and preserves part of the previous state. Fig. 6 shows the progression of both research campaigns; App. F details the matched controls, operator, initialization, and selected designs.

Context curation.

Across the Loop campaign, MIRA retains a stable task scaffold while selecting the notes, code, and measurements relevant to the active experiment. Recent artifacts and the latest handover preserve continuity across context resets, while the curated context contracts after final candidate selection (Fig. 7).

These studies show how MIRA reallocates inference as evidence accumulates. Making each meta-reasoning decision explicit also provides a natural boundary for forecasting remaining progress and assigning credit from later outcomes.

6.2 Can We Forecast Remaining Research Value?

Using proxy-marginal returns, we compare three decision-level critics—scalar, categorical-head, and generative—with scalar and generative token-level baselines. For each environment, we form five support/query splits, each with task-disjoint sets of 50 support and 50 query episodes. Under matched conditions, each critic starts from the same Qwen-27B initialization and trains on all 50 support episodes. We evaluate it on the query episodes using mean per-episode root mean squared error (RMSE). For query episodes 𝒬, this is

The decision-level generative critic achieves the lowest mean held-out RMSE among the tested designs, reducing RMSE by an absolute 0.110 relative to its token-level generative counterpart (Fig. 8(a)). We use this design in the subsequent transfer and policy-learning experiments.

We test whether critic pretraining improves forecasting in an unseen research environment. For each target environment, we compare a target-only model with models initialized by pretraining on one source environment or on all remaining environments. The latter is the LOO initialization defined above. We then adapt each critic on 0, 1, 5, 10, 25, or 50 target-environment support episodes and evaluate it on the same 50 held-out query episodes. This measures how quickly forecasting improves from each initialization as target-environment data increases. Across environments, LOO initialization enables better zero-shot forecasting and faster adaptation than either single-source pretraining or no pretraining. After adaptation on five target-environment episodes, it reduces RMSE by an absolute 0.053 compared with target-only training (Fig. 8(b)). We use pretrained critics to assign delayed credit and train MIRA’s research-decision policy.

6.3 Does Decision-Level Credit Improve the Research Policy?

Experimental protocol.

We evaluate on four environments: BNLearn causal discovery, constructed from the Bayesian Network Repository (Scutari, n.d.); Physics Discovery (Wiemann et al., 2026), SRBench symbolic regression (Shojaee et al., 2025), and HillClimbBench-CPU. The first three each use 100 tasks. HillClimbBench-CPU has 50 paper-derived tasks, each with two independently initialized rollouts. Qwen-27B is the underlying model for every MIRA agent role. In trained configurations, the learned checkpoint is used only for action selection and value forecasting; curation and execution remain fixed. We organize the comparison as an ablation ladder over supervision, credit granularity, critic parameterization, and actor–critic co-training. We begin with a separate actor–critic that uses token-level scalar regression and gold-binary outcomes. This Gold binary / Token regression condition is our analogue of single-rollout asynchronous methods such as SAO (Hou et al., 2026), which typically optimize all rollout tokens. Restricting the actor loss to the complete sampled MIRA decision response—its reasoning plus work order or termination decision—tests the same approach for learning a meta-reasoning policy. Holding the critic parameterization and granularity fixed, we replace terminal feedback with marginal improvements in the proxy metric. Under proxy supervision, i.e., training on proxy returns rather than the gold-binary outcome, we then move value forecasting to MIRA’s decision boundaries using decision-level scalar regression (Step regression) and replace scalar regression with the decision-level generative critic (Step generative). We evaluate the generative critic under both supervision regimes, first with separate policy and critic parameters and then with MIRA-AC training.

Token regression repeats the decision return target at each token prefix within the meta-reasoning action; the decision-level regression and generative critics forecast once per MIRA decision. In the separate-critic variants, the actor starts from the base policy and the critic from its pretrained checkpoint. MIRA-AC initializes the shared actor–critic from the pretrained critic checkpoint. Accordingly, this comparison evaluates the combined MIRA-AC design, not parameter sharing alone. All comparisons hold the MIRA harness, executor, tools, research horizon, inference budgets, and environment feedback fixed. Each environment-specific run makes one pass over a fixed, ordered set of 100 training episodes. The cross-environment run uses 300 episodes, comprising 100 from each of its three actor-training environments. We evaluate each run’s final policy checkpoint using gold evaluation.

Training with decision-level credit assignment improves long-horizon research.

MIRA-AC shows that long-horizon research can be improved by assigning credit directly to the meta-reasoning decisions that shape a trajectory. Trained from proxy hill-climbing feedback across multiple autoresearch environments, the policy achieves higher gold-evaluation point estimates than untrained MIRA in all four environments. Among the separate actor–critic ablations under proxy-marginal supervision, the strongest decision-level variant improves the primary outcome over Token regression in every environment. Step generative performs best on Symbolic Regression, Physics Discovery, and HillClimbBench-CPU, while Step regression performs best on BNLearn. MIRA exposes each meta-reasoning decision, and MIRA-AC applies policy loss to its complete sampled response while masking the much longer executor trace. MIRA-AC therefore improves research outcomes without directly optimizing the much longer executor traces, demonstrating the leverage of targeted credit assignment at meta-reasoning boundaries.

Jointly learning to forecast partial progress and act from the same curated research observations produces a stronger policy.

Ordinarily, the critic affects the policy only through the advantages used in the actor loss. Unlike Step generative, which uses separate actor and critic adapters, MIRA-AC applies both objectives to the same model with shared parameters. The critic loss therefore directly trains the policy’s representation to recognize evidence predictive of future progress. Under proxy-marginal supervision, the environment-specific MIRA-AC policy further improves the primary outcome on the three environments where Step generative is strongest. BNLearn instead favors Step regression. The critic loss provides an outcome-supervised auxiliary objective for learning what to investigate next.

MIRA-AC demonstrates the viability of training a general meta-reasoning policy.

Beyond the environment-specific ablations, we train a single MIRA-AC policy from proxy hill-climbing feedback across Symbolic Regression, Physics Discovery, and HillClimbBench-CPU. The cross-environment policy matches or exceeds the strongest separate-adapter primary outcome in every environment. On BNLearn, it matches Step regression without receiving BNLearn actor updates. This experiment builds on the transferable critic prior established in Sec. 6.2: MIRA-AC starts from a critic pretrained on all four environments, including BNLearn, to forecast progress from curated research observations. Its BNLearn result therefore provides evidence that the meta-reasoning policy learned from proxy feedback in the other environments transfers to BNLearn. Together, these results show that MIRA-AC can be scaled to a general meta-reasoning policy that learns reusable research strategies for deciding what to investigate across a variety of research domains.

6.4 What Changes After Meta-Reasoning Policy Training?

Beyond final task outcomes, we analyze how policy training changes research behavior, which generated tokens receive actor loss, and how context curation affects rollout input costs.

Analysis of trained meta-reasoning trajectories reveals notable shifts in research behavior, particularly a greater tendency to target unresolved uncertainty.

We use an LLM judge to compare 400 matched trajectory pairs while hiding policy identities, using a rubric covering hypothesis revision, experiment choice, evidence use, robustness testing, and stopping decisions. The judge first evaluates research behavior without seeing final outcomes. After outcomes are revealed, it assesses which policy’s outer-loop behavior contributed more strongly to the corresponding trajectory’s gold outcome. It favors MIRA-AC in 203 pairs and untrained MIRA in 131, with 66 tied or unclear; among clear attributions, 60.8% favor MIRA-AC.

Across environments, the judge more often identifies MIRA-AC trajectories that revise explanations, test competing ideas, check whether gains persist, and abandon stalled lines of inquiry. The largest shifts are an increase in trajectories containing a test that distinguishes competing ideas, from 16.8% to 28.0%, and a decrease in trajectories repeatedly trying minor variants of an unproductive idea, from 33.5% to 18.8% (Fig. 9). In Symbolic Regression, this appears as changing the functional form in response to structured errors rather than continuing to tune coefficients. In Physics Discovery, it appears as choosing experiments that vary one factor to distinguish competing physical laws. In HillClimbBench-CPU, it appears as leaving a plateaued sequence of local modifications for an architectural change tied to the diagnosed bottleneck. In BNLearn, which receives no actor updates, the transferred policy targets specific unresolved edges before expanding to broader graph search. App. K provides matched case analyses and trajectory excerpts, including cases where these changes help or hurt final performance.

Actor updates target only the research-decision responses.

Across the four environment-specific training runs, comprising 400 episodes, complete sampled meta-reasoning decision responses account for 25.8% of accounted decision and execution output tokens and are the only generated tokens that receive actor loss. Context-curation output is excluded because it was not fully logged. Fig. 10 shows this separation within one complete episode and across the full training set.

Curated context limits repeated-history input.

At each research decision, MIRA reads selected evidence from the persistent record instead of replaying the entire preceding trajectory. Under our accounting, MIRA-AC processes an average of 10.6 million input tokens per episode across the 400 training episodes. We compare this recorded cost with fixed-trajectory projections that replay the same trajectories while carrying prior context either in full or with auto-compaction at 200k tokens. Both replay alternatives require more input under this projection as research history accumulates (Fig. 11). App. H details the accounting and projection assumptions.

7 Limitations and Future Work

MIRA-AC learns only part of the meta-reasoning process. The policy that generates research decisions is trained, while evidence curation and execution remain fixed. This makes learning tractable and provides a clean credit-assignment boundary, but it also means that the policy can act only on evidence surfaced by the curator and can improve outcomes only through investigations the executor is capable of carrying out. Relevant evidence omitted during curation cannot be recovered by the actor or critic, and improvements in the executor may change which research decisions are valuable. An important next step is therefore to learn curation, allocation, and possibly execution jointly while preserving decision boundaries for credit assignment.

The behavioral results raise a broader question: what should a general meta-reasoner learn about how to conduct research? Policy training shapes not only task performance, but the model’s research temperament—how readily it abandons an incumbent, gathers more evidence, or commits to a larger change. In HillClimbBench-CPU, MIRA-AC’s willingness to pursue substantial architectural changes can escape local optima that trap untrained MIRA on some tasks, yet the same willingness can backfire when preserving and patiently refining the incumbent is better. The calibration problem recurs elsewhere: a new experiment is useful while physical laws remain ambiguous but wasteful once they are resolved; a richer symbolic expression can improve proxy fit while harming extrapolation; and broad graph search can be inferior to a one-edge repair. The objective is therefore not maximal exploration or aggressiveness, but context-sensitive judgment about when to persist, refine, test, pivot, or stop.

The environments used for policy training cover only a small fraction of the diversity required to learn this judgment. A general policy will require a deliberately constructed training distribution spanning distinct research regimes—exploration and exploitation, bold pivots and incremental refinement, continued investigation and stopping—so that the model learns not one research style, but which style and granularity of investigation a situation warrants. In this work, we demonstrate that decision-level training improves outcomes, changes research behavior, and transfers to an environment held out from actor training, providing a roadmap for scaling toward that goal.

We demonstrate that meta-reasoning updates transfer to an environment held out from actor training: MIRA-AC improves on BNLearn despite receiving no BNLearn actor updates. The model is, however, exposed to BNLearn during value pretraining. Future work should examine broader transfer to environments excluded from both value pretraining and actor training.

Our experiments use a single MIRA configuration with the same Codex-based inner-loop harness across research environments. This holds the research interface fixed while varying the task. Training across heterogeneous and evolving inner-loop scaffolds is a natural next step for testing whether learned meta-reasoning remains useful as execution capabilities change.

Our learning experiments also depend on proxy feedback. Dense proxy signals make repeated autoresearch training feasible, but a proxy need not rank research directions in the same way as the final gold evaluation. Our results show that proxy-trained policies can improve gold-evaluated outcomes in the environments studied here, but they do not establish when such transfer should be expected or how robust learning is to systematically misaligned proxies. Future work should characterize the degree of proxy–gold alignment required for stable policy improvement and develop methods that can learn from sparse gold feedback without giving up the sample efficiency of intermediate signals.

The behavioral analysis should likewise be interpreted as descriptive. Its conclusions rely on an LLM judge assigning process labels and comparing matched trajectories. These annotations provide evidence that training changes observable research behavior, but they do not establish that any individual behavior—such as testing competing hypotheses, abandoning stalled directions, or making larger pivots—causes the associated performance difference. More direct interventions on research strategy, together with human or task-specific behavioral evaluation, would be needed to identify which learned behaviors are causally useful and under which research regimes.

Finally, the current optimization results are limited by scale. We train LoRA adapters rather than all model parameters, and our asynchronous procedure is evaluated only at investigation durations supported by the present infrastructure. We have not tested episodes in which individual investigations last hours or days and a single trajectory spans many more actor and critic versions. At those timescales, delayed and heterogeneous feedback, policy staleness, and interference between actor and critic updates to shared parameters may become substantially more important. Scaling to full-parameter training and longer-lived research episodes will therefore require studying update scheduling, value calibration under changing policies, and stable joint optimization over much larger temporal gaps.

More broadly, MIRA-AC assumes that research can be decomposed into meaningful decision boundaries at which accumulated evidence is summarized, future value can be estimated, and credit can be assigned. The environments studied here support that decomposition. Whether the same abstraction remains effective for less structured research, where useful investigations overlap, branch, or evolve continuously, is an open question. Testing that boundary is important for determining how far decision-level meta-reasoning can scale toward more general autonomous research.

8 Conclusion

This work asks whether a research agent can learn what to investigate next from the downstream consequences of its previous research decisions. Across the settings studied here, our results support a positive answer. MIRA represents long-horizon research as a sequence of explicit meta-reasoning decisions over a persistent research record, treating the potentially much longer execution of each investigation as part of an environment transition. This decomposition improves long-horizon inference even before policy training and provides a natural boundary at which remaining research value can be forecast and delayed outcomes assigned to earlier decisions.

Building on this representation, MIRA-AC jointly trains one model to forecast remaining return from a curated research observation and choose how to allocate research effort next. Decision-level value forecasting outperforms token-level alternatives, and decision-level policy learning improves gold-evaluated research outcomes, including when trained from proxy feedback, without directly optimizing the longer executor trajectories. We also find initial evidence that actor updates learned across research environments can transfer to a held-out actor-training environment when initialized from a cross-environment value model.

The broader implication is that research strategy need not remain an implicit byproduct of long-horizon generation. Once decisions about what to investigate are represented explicitly and linked to their downstream consequences, they become objects that can be valued, credited, and improved through experience. This suggests a path toward research agents that learn not only to execute individual investigations more effectively, but to improve the strategy by which they decide what research is worth doing next.

References

• Agarwal et al. (2025) D. Agarwal, B. P. Majumder, R. Adamson, M. Chakravorty, S. R. Gavireddy, A. Parashar, H. Surana, B. D. Mishra, A. McCallum, A. Sabharwal, and P. Clark AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise. In Advances in Neural Information Processing Systems, Note: arXiv:2507.00310 External Links: Link, Document Cited by: Appendix A.
• Amos and Kolter (2017) B. Amos and J. Z. Kolter OptNet: differentiable optimization as a layer in neural networks. External Links: 1703.00443, Link Cited by: item 34.
• Arik and Pfister (2019) S. O. Arik and T. Pfister TabNet: attentive interpretable tabular learning. External Links: 1908.07442, Link Cited by: item 47.
• Arora et al. (2023) S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré Zoology: measuring and improving recall in efficient language models. External Links: 2312.04927, Link Cited by: item 50.
• Battaglia et al. (2016) P. W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu Interaction networks for learning about objects, relations and physics. External Links: 1612.00222, Link Cited by: item 19.
• Bellemare et al. (2017) M. G. Bellemare, W. Dabney, and R. Munos A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70, pp. 449–458. External Links: Link Cited by: §1.
• Chai et al. (2025) J. Chai, S. Tang, R. Ye, Y. Du, X. Zhu, M. Zhou, Y. Wang, W. E, Y. Zhang, L. Zhang, and S. Chen SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity’s Last Exam?. arXiv. Note: arXiv:2507.05241 External Links: Link, Document Cited by: Appendix A.
• Chan et al. (2025) J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. Mądry MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. In International Conference on Learning Representations, (en). Note: arXiv:2410.07095 External Links: Link Cited by: Appendix A.
• Chen et al. (2026) J. Chen, B. D. Mishra, J. Nam, R. Meng, T. Pfister, and J. Yoon MARS: Modular Agent with Reflective Search for Automated AI Research. In Proceedings of the 43rd International Conference on Machine Learning, Note: arXiv:2602.02660 External Links: Link, Document Cited by: Appendix A.
• Chen et al. (2018a) M. Chen, J. Pennington, and S. S. Schoenholz Dynamical isometry and a mean field theory of RNNs: gating enables signal propagation in recurrent neural networks. External Links: 1806.05394, Link Cited by: item 6.
• Chen (2017) M. Chen MinimalRNN: toward more interpretable and trainable recurrent neural networks. External Links: 1711.06788, Link Cited by: item 28.
• Chen et al. (2018b) R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud Neural ordinary differential equations. External Links: 1806.07366, Link Cited by: item 31.
• Chen et al. (2025) Z. Chen, S. Chen, Y. Ning, Q. Zhang, B. Wang, B. Yu, Y. Li, Z. Liao, C. Wei, Z. Lu, V. Dey, M. Xue, F. N. Baker, B. Burns, D. Adu-Ampratwum, X. Huang, X. Ning, S. Gao, Y. Su, and H. Sun ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. In International Conference on Learning Representations, Note: arXiv:2410.05080 External Links: Link, Document Cited by: Appendix A.
• Cheng et al. (2023) C. Cheng, A. Kolobov, D. Misra, A. Nie, and A. Swaminathan LLF-Bench: Benchmark for Interactive Learning from Language Feedback. arXiv. Note: arXiv:2312.06853 External Links: Link, Document Cited by: Appendix A.
• Chi et al. (2026) Y. Chi, D. Hong, D. Jiang, T. Luo, K. Yang, B. Zhang, Z. Cao, X. Fan, B. He, H. Hao, W. Jin, D. Lei, Q. Liu, H. Qian, B. Wang, S. Wang, Y. Zheng, Y. Zhou, C. Xiao, E. Cai, and Q. Na Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization. arXiv. Note: arXiv:2604.12290 External Links: Link, Document Cited by: Appendix A, Appendix A.
• Chilkuri and Eliasmith (2021) N. Chilkuri and C. Eliasmith Parallelizing legendre memory unit training. External Links: 2102.11417, Link Cited by: item 35.
• Choromanski et al. (2020) K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller Rethinking attention with performers. External Links: 2009.14794, Link Cited by: item 38.
• Cobbe et al. (2021) K. Cobbe, J. Hilton, O. Klimov, and J. Schulman Phasic policy gradient. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139, pp. 2020–2027. External Links: Link Cited by: Appendix A.
• Cohen and Welling (2016) T. S. Cohen and M. Welling Group equivariant convolutional networks. External Links: 1602.07576, Link Cited by: item 13.
• Dehghani et al. (2018) M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser Universal transformers. External Links: 1807.03819, Link Cited by: item 49.
• Dennis et al. (2026) S. Dennis, R. Patil, K. Shabahang, and H. Guo Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost. arXiv. Note: arXiv:2605.22502 External Links: Link, Document Cited by: Appendix A.
• Du et al. (2026) Y. Du, X. Yang, Z. Zhou, W. Liu, Z. Lei, Z. Chen, F. Liu, H. Wu, Y. Cai, Z. Liu, X. Zhu, W. Wang, L. Zhang, C. Qian, and S. Chen DataMaster: Data-Centric Autonomous AI Research. arXiv. Note: arXiv:2605.10906 External Links: Link, Document Cited by: Appendix A.
• Du et al. (2025) Y. Du, B. Yu, T. Liu, T. Shen, J. Chen, J. G. Rittig, K. Sun, Y. Zhang, A. Krishnan, Y. Zhang, D. Rosen, R. Pirone, Z. Song, B. Zhou, C. Masschelein, Y. Wang, H. Wang, H. Jia, C. Zhang, H. Zhao, M. Ester, N. Hacohen, T. Head-Gordon, C. P. Gomes, H. Sun, C. Duan, P. Schwaller, and W. Jin Accelerating Scientific Discovery with Autonomous Goal-evolving Agents. arXiv. Note: arXiv:2512.21782 External Links: Link, Document Cited by: Appendix A.
• Duan et al. (2025) S. Duan, Y. Tian, Q. Bing, and X. Shao Bayes-Entropy Collaborative Driven Agents for Research Hypotheses Generation and Optimization. arXiv. Note: arXiv:2508.01746 External Links: Link, Document Cited by: Appendix A.
• Gao et al. (2026) S. Gao, A. Fang, and M. Zitnik AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation. arXiv. Note: arXiv:2605.28655 External Links: Link, Document Cited by: Appendix A, §1.
• Garnelo et al. (2018) M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. M. A. Eslami, and Y. W. Teh Neural processes. External Links: 1807.01622, Link Cited by: item 32.
• Geiping et al. (2025) J. Geiping, S. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein Scaling up test-time compute with latent reasoning: a recurrent depth approach. arXiv preprint arXiv:2502.05171. External Links: Link Cited by: §1, §6.1.
• Goel et al. (2025) S. Goel, R. Hazra, D. Jayalath, T. Willi, P. Jain, W. F. Shen, I. Leontiadis, F. Barbieri, Y. Bachrach, J. Geiping, and C. Whitehouse Training AI Co-Scientists Using Rubric Rewards. arXiv. Note: arXiv:2512.23707 External Links: Link, Document Cited by: Appendix A.
• Goldie et al. (2026) A. D. Goldie, Z. Wang, A. Hayler, D. Nathani, E. Toledo, K. Thampiratwong, A. Kalisz, M. Beukman, H. Erlebach, A. Letcher, S. Reddy, C. Wibault, T. Wolf, C. O’Neill, U. Berdica, N. Roberts, S. Rahmani, R. Raileanu, S. Whiteson, and J. N. Foerster DiscoGen: Procedural Generation of Algorithm Discovery Tasks in Machine Learning. In Proceedings of the 43rd International Conference on Machine Learning, Note: arXiv:2603.17863 External Links: Link, Document Cited by: Appendix A.
• Gottweis et al. (2026) J. Gottweis, W. Weng, A. Daryin, T. Tu, P. Sirkovic, A. Myaskovsky, G. Glowaty, F. Weissenberger, A. Orlandi, D. Popovici, A. Palepu, K. Rong, R. Tanno, K. Saab, F. Zhang, J. Blum, A. Carroll, K. Kulkarni, N. Tomasev, D. Zverinski, I. Rendulic, E. Vedadi, F. Hasler, L. Rimanic, M. Boia, I. Budiselic, B. Feinstein, M. Bellaiche, T. Sheffer, J. Freyberg, J. Ratcliff, O. Bertolli, K. Chou, A. Hassidim, B. Gokturk, A. Vahdat, Y. Guan, V. Dhillon, E. D. Vaishnav, B. Lee, T. R. D. Costa, J. R. Penadés, G. Peltz, Y. Matias, J. Manyika, D. Hassabis, Y. Xu, P. Kohli, A. Pawlosky, A. Karthikesalingam, and V. Natarajan Accelerating scientific discovery with Co-Scientist. Nature. Note: arXiv:2502.18864 External Links: Link, Document Cited by: Appendix A.
• Graves et al. (2014) A. Graves, G. Wayne, and I. Danihelka Neural turing machines. External Links: 1410.5401, Link Cited by: item 33.
• Gu et al. (2020) A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré HiPPO: recurrent memory with optimal polynomial projections. External Links: 2008.07669, Link Cited by: item 15.
• Gu and Dao (2023) A. Gu and T. Dao Mamba: linear-time sequence modeling with selective state spaces. External Links: 2312.00752, Link Cited by: item 26.
• Gu et al. (2021) A. Gu, K. Goel, and C. Ré Efficiently modeling long sequences with structured state spaces. External Links: 2111.00396, Link Cited by: item 42.
• Ha et al. (2016) D. Ha, A. Dai, and Q. V. Le HyperNetworks. External Links: 1609.09106, Link Cited by: item 17.
• Helfrich et al. (2017) K. Helfrich, D. Willmott, and Q. Ye Orthogonal recurrent neural networks with scaled Cayley transform. External Links: 1707.09520, Link Cited by: item 43.
• Hochreiter and Schmidhuber (1997) S. Hochreiter and J. Schmidhuber Long short-term memory. Neural Computation 9 (8), pp. 1735–1780. External Links: Document, Link Cited by: item 25.
• Hou et al. (2026) Z. Hou, Y. Li, J. Tang, and Y. Dong Single-rollout asynchronous optimization for agentic reinforcement learning. Note: arXiv:2607.07508 External Links: Link Cited by: §1, §6.3.
• Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §5.4.
• Huang et al. (2024) Q. Huang, J. Vora, P. Liang, and J. Leskovec MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235. Note: arXiv:2310.03302 External Links: Link, Document Cited by: Appendix A.
• Jaderberg et al. (2017) M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A.
• Jaderberg et al. (2015) M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu Spatial transformer networks. External Links: 1506.02025, Link Cited by: item 46.
• Jaegle et al. (2021) A. Jaegle, S. Borgeaud, J. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, O. Hénaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira Perceiver IO: a general architecture for structured inputs & outputs. External Links: 2107.14795, Link Cited by: item 37.
• Ji et al. (2026) Y. Ji, Z. Ma, Y. Wang, G. Chen, X. Chu, and L. Wu Tree Search for LLM Agent Reinforcement Learning. In International Conference on Learning Representations, Note: arXiv:2509.21240 External Links: Link, Document Cited by: Appendix A.
• Jiang et al. (2025a) D. R. Jiang, A. Samanta, Y. Yang, J. Bhandari, R. Munos, and T. Lu Iterative GRPO: batch-online multi-turn RL via single-turn RLHF. Note: arXiv:2511.21638 External Links: Link Cited by: Appendix A.
• Jiang et al. (2025b) Z. Jiang, D. Schmidt, D. Srikanth, D. Xu, I. Kaplan, D. Jacenko, and Y. Wu AIDE: AI-Driven Exploration in the Space of Code. arXiv. Note: arXiv:2502.13138 External Links: Link, Document Cited by: Appendix A.
• Jin et al. (2026) J. Jin, Y. Hu, K. Qiu, Q. Dai, C. Luo, G. Dong, X. Li, T. Zhao, X. Ma, G. Zhang, Z. Wu, B. Liu, Z. Yang, L. Li, L. Wang, H. Qian, Y. Zhu, and Z. Dou Toward Generalist Autonomous Research via Hypothesis-Tree Refinement. arXiv. Note: arXiv:2606.11926 External Links: Link, Document Cited by: Appendix A.
• Jolicoeur-Martineau (2025) A. Jolicoeur-Martineau Less is more: recursive reasoning with tiny networks. External Links: 2510.04871, Link Cited by: item 48.
• Kabra et al. (2026) S. Kabra, N. Abhyankar, S. Desai, P. Iyer, and C. K. Reddy LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs. arXiv. Note: arXiv:2605.24043 version: 1 External Links: Link, Document Cited by: Appendix A.
• Kalchbrenner et al. (2015) N. Kalchbrenner, I. Danihelka, and A. Graves Grid long short-term memory. External Links: 1507.01526, Link Cited by: item 12.
• Kausik et al. (2026) C. Kausik, A. Swaminathan, and N. Kallus The context gathering decision process: a POMDP framework for agentic search. Note: arXiv:2605.07042 External Links: Link Cited by: §1.
• Li et al. (2018) S. Li, W. Li, C. Cook, C. Zhu, and Y. Gao Independently recurrent neural network (IndRNN): building a longer and deeper RNN. External Links: 1803.04831, Link Cited by: item 18.
• Li et al. (2026) Y. Li, C. Shao, X. Liu, R. Zhao, P. Liu, H. Su, Z. Chen, Q. Yang, A. Xu, Y. Fang, Q. Zeng, T. Li, J. Xu, F. Xu, Y. Li, and T. Liu AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery. arXiv. Note: arXiv:2604.05550 External Links: Link, Document Cited by: Appendix A.
• Lin et al. (2026) M. Lin, J. Wu, Z. Wang, Z. Shi, Y. Sang, B. He, Z. Liu, T. Wei, Z. Wu, Z. Zhang, D. Wang, X. Zhang, B. Dumoulin, C. Xie, Y. Zhou, S. Wang, and H. Lu Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents. arXiv. Note: arXiv:2605.30621 External Links: Link, Document Cited by: Appendix A.
• Liu et al. (2018) R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, and J. Yosinski An intriguing failing of convolutional neural networks and the CoordConv solution. External Links: 1807.03247, Link Cited by: item 2.
• Liu et al. (2024) Z. Liu, Y. Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Soljačić, T. Y. Hou, and M. Tegmark KAN: kolmogorov-arnold networks. External Links: 2404.19756, Link Cited by: item 22.
• Locatello et al. (2020) F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf Object-centric learning with slot attention. External Links: 2006.15055, Link Cited by: item 45.
• Long et al. (2017) Z. Long, Y. Lu, X. Ma, and B. Dong PDE-Net: learning PDEs from data. External Links: 1710.09668, Link Cited by: item 36.
• Lu et al. (2024) C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv. Note: arXiv:2408.06292 External Links: Link, Document Cited by: Appendix A.
• Lu et al. (2021) L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence 3, pp. 218–229. External Links: Document, Link Cited by: item 5.
• Luong et al. (2025) T. Luong, D. Hwang, H. H. Nguyen, G. Ghiasi, Y. Chervonyi, I. Seo, J. Kim, G. Bingham, J. Lee, S. Mishra, A. Zhai, C. H. Hu, H. Michalewski, J. Kim, J. Ahn, J. Bae, X. Song, T. H. Trinh, Q. V. Le, and J. Jung Towards robust mathematical reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, External Links: Document, Link Cited by: §1, §6.1.
• Madsen and Johansen (2020) A. Madsen and A. R. Johansen Neural arithmetic units. External Links: 2001.05016, Link Cited by: item 30.
• Mak and Flanigan (2025) B. Mak and J. Flanigan Residual matrix transformers: scaling the size of the residual stream. In Proceedings of the 42nd International Conference on Machine Learning, Note: arXiv:2506.22696 External Links: Link Cited by: §1, §6.1.
• Mena et al. (2018) G. Mena, D. Belanger, S. Linderman, and J. Snoek Learning latent permutations with Gumbel-Sinkhorn networks. External Links: 1802.08665, Link Cited by: item 14.
• Mishra et al. (2025) S. Mishra, G. Sahu, M. Pedersoli, L. Charlin, J. Dolz, and C. Pal AInstein: Can LLMs Solve Research Problems From Parametric Memory Alone?. arXiv. Note: arXiv:2510.05432 External Links: Link, Document Cited by: Appendix A.
• Mnih et al. (2016) V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, Vol. 48, pp. 1928–1937. External Links: Link Cited by: Appendix A.
• Murphy et al. (2018) R. L. Murphy, B. Srinivasan, V. Rao, and B. Ribeiro Janossy pooling: learning deep permutation-invariant functions for variable-size inputs. External Links: 1811.01900, Link Cited by: item 21.
• Neil et al. (2016) D. Neil, M. Pfeiffer, and S. Liu Phased LSTM: accelerating recurrent network training for long or event-based sequences. External Links: 1610.09513, Link Cited by: item 39.
• Novikov et al. (2025) A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv. Note: arXiv:2506.13131 External Links: Link, Document Cited by: Appendix A, §1.
• OpenAI (2025) OpenAI Introducing Codex. Note: https://openai.com/index/introducing-codex/ Cited by: §3.1.
• Orvieto et al. (2023) A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De Resurrecting recurrent neural networks for long sequences. External Links: 2303.06349, Link Cited by: item 24.
• Pepe et al. (2026) A. Pepe, C. Lin, D. Magka, B. Acun, Y. N. Wu, A. Protopopov, C. Wu, and Y. Bachrach Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design. arXiv. Note: arXiv:2605.15871 External Links: Link, Document Cited by: Appendix A.
• Poli et al. (2023) M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré Hyena hierarchy: towards larger convolutional language models. External Links: 2302.10866, Link Cited by: item 16.
• Press et al. (2025) O. Press, B. Amos, H. Zhao, Y. Wu, S. K. Ainsworth, D. Krupke, P. Kidger, T. Sajed, B. Stellato, J. Park, N. Bosch, E. Meril, A. Steppi, A. Zharmagambetov, F. Zhang, D. Perez-Pineiro, A. Mercurio, N. Zhan, T. Abramovich, K. Lieret, H. Zhang, S. Huang, M. Bethge, and O. Press AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?. arXiv. Note: arXiv:2507.15887 External Links: Link, Document Cited by: Appendix A.
• Qiang et al. (2025) R. Qiang, Y. Zhuang, Y. Li, D. S. V. K, R. Zhang, C. Li, I. S. Wong, S. Yang, P. Liang, C. Zhang, and B. Dai MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering. arXiv. Note: arXiv:2505.07782 External Links: Link, Document Cited by: Appendix A.
• Qwen Team (2026) Qwen Team Qwen3.6-27B. Note: Hugging Face model repositoryRevision 6a9e13bd6fc8f0983b9b99948120bc37f49c13e9 External Links: Link Cited by: Appendix H, §6.1.
• Ramsauer et al. (2020) H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, V. Greiff, D. Kreil, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter Hopfield networks is all you need. External Links: 2008.02217, Link Cited by: item 29.
• Rank et al. (2026) B. Rank, H. Bhatnagar, A. Prabhu, S. Eisenberg, K. Nguyen, M. Bethge, and M. Andriushchenko PostTrainBench: Can LLM Agents Automate LLM Post-Training?. arXiv. Note: arXiv:2603.08640 External Links: Link, Document Cited by: Appendix A.
• Rein et al. (2025) D. Rein, J. Becker, A. Deng, S. Nix, C. Canal, D. O’Connel, P. Arnott, R. Bloom, T. Broadley, K. Garcia, B. Goodrich, M. Hasin, S. Jawhar, M. Kinniment, T. Kwa, A. Lajko, N. Rush, L. J. K. Sato, S. V. Arx, B. West, L. Chan, and E. Barnes HCAST: Human-Calibrated Autonomy Software Tasks. arXiv. Note: arXiv:2503.17354 External Links: Link, Document Cited by: Appendix A.
• Romera-Paredes et al. (2024) B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi Mathematical discoveries from program search with large language models. Nature 625 (7995), pp. 468–475. External Links: Document, Link Cited by: Appendix A.
• Russell and Wefald (1991) S. J. Russell and E. H. Wefald Principles of metareasoning. Artificial Intelligence 49 (1–3), pp. 361–395. External Links: Document Cited by: Appendix A, §1.
• Samanta et al. (2026a) A. Samanta, A. Magesh, A. Jain, Y. Yu, D. Jiang, K. Asadi, K. Hassani, P. Sajda, J. Bhandari, and Y. Efroni Credit assignment with resets in language model reasoning. arXiv preprint arXiv:2605.25507. Note: arXiv:2605.25507v2 External Links: Link Cited by: Appendix A, §1.
• Samanta et al. (2026b) A. Samanta, A. Magesh, T. Lancewicki, A. Jain, Y. Yu, P. Sajda, K. Hassani, A. Modi, D. R. Jiang, and Y. Efroni BayesBench: evaluating LLM belief trajectories under multi-turn evidence accumulation. External Links: 2606.30850, Link Cited by: §4.1.
• Satorras et al. (2021) V. G. Satorras, E. Hoogeboom, and M. Welling E⁡(n) equivariant graph neural networks. External Links: 2102.09844, Link Cited by: item 7.
• Schmidgall et al. (2025) S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, M. Moor, Z. Liu, and E. Barsoum Agent Laboratory: Using LLM Agents as Research Assistants. arXiv. Note: arXiv:2501.04227 External Links: Link, Document Cited by: Appendix A.
• Schulman et al. (2016) J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, External Links: Link Cited by: §5.2.
• Scutari (n.d.) M. Scutari Bayesian network repository. External Links: Link Cited by: §B.3, §1, §6.3.
• Shan et al. (2026) Z. Shan, H. Zhong, L. Wang, and L. Zhao Bringing value models back: generative critics for value modeling in LLM reinforcement learning. External Links: 2604.10701, Link Cited by: Appendix A.
• Shojaee et al. (2025) P. Shojaee, N. Nguyen, K. Meidani, A. B. Farimani, K. D. Doan, and C. K. Reddy LLM-SRBench: a new benchmark for scientific equation discovery with large language models. In Proceedings of the 42nd International Conference on Machine Learning, Note: Oral presentation; arXiv:2504.10415 External Links: Link Cited by: §B.3, §1, §6.3.
• Sitzmann et al. (2020) V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein Implicit neural representations with periodic activation functions. External Links: 2006.09661, Link Cited by: item 44.
• Song et al. (2025) H. Song, J. Jiang, Y. Min, J. Chen, Z. Chen, W. X. Zhao, L. Fang, and J. Wen R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. arXiv. Note: arXiv:2503.05592 External Links: Link, Document Cited by: Appendix A.
• Starace et al. (2025) G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese, and T. Patwardhan PaperBench: Evaluating AI’s Ability to Replicate AI Research. arXiv. Note: arXiv:2504.01848 External Links: Link, Document Cited by: Appendix A.
• Stokes et al. (2020) J. M. Stokes, K. Yang, K. Swanson, W. Jin, A. Cubillos-Ruiz, N. M. Donghia, C. R. MacNair, S. French, L. A. Carfrae, Z. Bloom-Ackermann, V. M. Tran, A. Chiappino-Pepe, A. H. Badran, I. W. Andrews, E. J. Chory, G. M. Church, E. D. Brown, T. S. Jaakkola, R. Barzilay, and J. J. Collins A deep learning approach to antibiotic discovery. Cell 180 (4), pp. 688–702.e13. External Links: Document, Link Cited by: §1.
• Sutton and Barto (2018) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. 2 edition, MIT Press. External Links: Link Cited by: §1.
• Szot et al. (2026) A. Szot, M. Kirchhof, O. Attia, and A. Toshev Expanding LLM Agent Boundaries with Strategy-Guided Exploration. arXiv. Note: arXiv:2603.02045 External Links: Link, Document Cited by: Appendix A.
• Tallec and Ollivier (2018) C. Tallec and Y. Ollivier Can recurrent neural networks warp time?. External Links: 1804.11188, Link Cited by: item 1.
• Tancik et al. (2020) M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, and R. Ng Fourier features let networks learn high frequency functions in low dimensional domains. External Links: 2006.10739, Link Cited by: item 8.
• van der Westhuizen and Lasenby (2018) J. van der Westhuizen and J. Lasenby The unreasonable effectiveness of the forget gate. External Links: 1804.04849, Link Cited by: item 20.
• Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin Attention is all you need. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §6.1.
• Veličković et al. (2017) P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio Graph attention networks. External Links: 1710.10903, Link Cited by: item 11.
• Vinyals et al. (2015) O. Vinyals, M. Fortunato, and N. Jaitly Pointer networks. External Links: 1506.03134, Link Cited by: item 40.
• Voelker et al. (2019) A. Voelker, I. Kajić, and C. Eliasmith Legendre memory units: continuous-time representation in recurrent neural networks. In Advances in Neural Information Processing Systems, Vol. 32, pp. 15544–15553. External Links: Link Cited by: item 23.
• Wan et al. (2025a) C. Wan, X. Dai, Z. Wang, M. Li, Y. Wang, Y. Mao, Y. Lan, and Z. Xiao LoongFlow: Directed Evolutionary Search via a Cognitive Plan-Execute-Summarize Paradigm. arXiv. Note: arXiv:2512.24077 External Links: Link, Document Cited by: Appendix A.
• Wan et al. (2025b) Y. Wan, J. Wang, L. Li, J. Liu, R. Zhu, and Z. Zhu Rethinking the Design of Reinforcement Learning-Based Deep Research Agents. arXiv. Note: arXiv:2510.15862 version: 3 External Links: Link, Document Cited by: Appendix A.
• Wang et al. (2026a) H. Wang, Y. Lan, B. Cao, L. Lin, and J. Chen SkillGrad: Optimizing Agent Skills Like Gradient Descent. arXiv. Note: arXiv:2605.27760 External Links: Link, Document Cited by: Appendix A.
• Wang et al. (2026b) M. Wang, R. Lin, K. Hu, J. Jiao, N. Chowdhury, E. Chang, and T. Patwardhan FrontierScience: Evaluating AI’s Ability to Perform Expert-Level Scientific Tasks. arXiv. Note: arXiv:2601.21165 External Links: Link, Document Cited by: Appendix A.
• Wang et al. (2025) Z. Wang, X. Zheng, K. An, C. Ouyang, J. Cai, Y. Wang, and Y. Wu StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization. arXiv. Note: arXiv:2505.15107 External Links: Link, Document Cited by: Appendix A.
• Weston et al. (2014) J. Weston, S. Chopra, and A. Bordes Memory networks. External Links: 1410.3916, Link Cited by: item 27.
• Wiemann et al. (2026) M. L. Wiemann, L. M. Smith, P. Melchior, S. Mishra-Sharma, A. G. Wilson, P. Izmailov, and C. Cuesta-Lázaro DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking. arXiv. Note: arXiv:2605.26087 External Links: Link, Document Cited by: Appendix A, §B.3, §1, §6.3.
• Wijk et al. (2025) H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, H. Karnofsky, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267. Note: arXiv:2411.15114 External Links: Link, Document Cited by: Appendix A.
• Wisdom et al. (2016) S. Wisdom, T. Powers, J. R. Hershey, J. Le Roux, and L. Atlas Full-capacity unitary recurrent neural networks. External Links: 1611.00035, Link Cited by: item 9.
• Xu et al. (2018) K. Xu, W. Hu, J. Leskovec, and S. Jegelka How powerful are graph neural networks?. External Links: 1810.00826, Link Cited by: item 10.
• Xu et al. (2026) W. Xu, A. Nie, R. Zheng, A. Modi, A. Swaminathan, and C. Cheng Formalizing Learning from Language Feedback with Provable Guarantees. In Proceedings of the 43rd International Conference on Machine Learning, Note: arXiv:2506.10341 External Links: Link, Document Cited by: Appendix A.
• Yamada et al. (2025) Y. Yamada, R. T. Lange, C. Lu, S. Hu, C. Lu, J. Foerster, J. Clune, and D. Ha The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. arXiv. Note: arXiv:2504.08066 External Links: Link, Document Cited by: Appendix A.
• Yang et al. (2025) X. Yang, X. Yang, S. Fang, Y. Zhang, J. Wang, B. Xian, Q. Li, J. Li, M. Xu, Y. Li, H. Pan, Y. Zhang, W. Liu, Y. Shen, W. Chen, and J. Bian R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science. arXiv. Note: arXiv:2505.14738 External Links: Link, Document Cited by: Appendix A.
• Yang et al. (2026) Y. Yang, Z. Gong, W. Huang, Q. Yang, Z. Zhou, Z. Huang, Y. Li, X. Gao, Q. Dai, B. Liu, K. Qiu, Y. Yang, D. Chen, X. Yang, and C. Luo SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv. Note: arXiv:2605.23904 External Links: Link, Document Cited by: Appendix A.
• Ye et al. (2025) T. Ye, L. Dong, Y. Xia, Y. Sun, Y. Zhu, G. Huang, and F. Wei Differential transformer. In International Conference on Learning Representations, Note: Oral presentation External Links: Link Cited by: Appendix F, §6.1.
• You et al. (2017) S. You, D. Ding, K. Canini, J. Pfeifer, and M. Gupta Deep lattice networks and partial monotonic functions. External Links: 1709.06680, Link Cited by: item 3.
• Zaheer et al. (2017) M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. Salakhutdinov, and A. Smola Deep sets. External Links: 1703.06114, Link Cited by: item 4.
• Zhang et al. (2025) L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal Generative verifiers: reward modeling as next-token prediction. In International Conference on Learning Representations, External Links: 2408.15240, Link Cited by: Appendix A, §1.
• Zhao et al. (2025) B. Zhao, D. Magka, M. Jiang, X. Li, R. Raileanu, T. Shavrina, J. Gagnon-Audet, K. Niu, S. Sodhani, M. Shvartsman, A. Lupu, A. Lupidi, E. Toledo, K. Hambardzumyan, M. Josifoski, T. Foster, L. Cipolina-Kun, A. Charnalia, D. Dunfield, A. H. Miller, O. M. Aodha, J. Foerster, and Y. Bachrach The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements. arXiv. Note: arXiv:2506.22419 External Links: Link, Document Cited by: Appendix A.
• Zhou et al. (2024) A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y. Wang Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models. In Proceedings of the 41st International Conference on Machine Learning, Note: arXiv:2310.04406 External Links: Link, Document Cited by: Appendix A.
• Zilly et al. (2016) J. G. Zilly, R. K. Srivastava, J. Koutník, and J. Schmidhuber Recurrent highway networks. External Links: 1607.03474, Link Cited by: item 41.
• Zou et al. (2026) Q. Zou, H. H. Lam, W. Zhao, T. Chen, Y. Tang, S. Yu, Y. Zhu, S. Anumasa, Z. Zhang, T. Zhang, C. Liu, Z. Jiang, A. Goyal, and D. Liu FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics. arXiv. Note: arXiv:2605.17373 External Links: Link, Document Cited by: Appendix A, Appendix A, §1.

Appendix Contents

Background A  Appendix A Related Work.A Experimental setup B  Appendix B Experimental Setup.B C  Appendix C MIRA Implementation.C Additional MIRA analyses D  Appendix D Semantic Exploration Graphs and Strategic Inertia.D E  Appendix E RMT Architecture Identified by MIRA.E F  Appendix F Loop Transformer Search and Selected Designs.F Critic and policy-learning details G  Appendix G Critic Training and Validation.G H  Appendix H MIRA-AC Training Details.H I  Appendix I Trajectory Stitching Across Artificial Boundaries.I J  Appendix J Additional MIRA-AC Results.J K  Appendix K Behavioral Analysis of MIRA-AC.K

Appendix A Related Work

Rational meta-reasoning.

MIRA builds on the classical account in Principles of Metareasoning (Russell and Wefald, 1991), which asks which computation a resource-bounded agent should perform next and when it should stop. In long-horizon research, the relevant computations are investigations: testing a hypothesis, revising an implementation, gathering evidence, or terminating the search. MIRA makes these allocation decisions explicit as work orders, thereby turning a classical meta-reasoning problem into a trainable sequential decision process.

Benchmarks for autonomous research.

Recent benchmarks expose complementary aspects of long-horizon research. MLAgentBench (Huang et al., 2024) and MLE-bench (Chan et al., 2025) evaluate iterative machine-learning engineering over executable tasks, while RE-Bench (Wijk et al., 2025) compares agents with human experts over increasing time budgets. PaperBench (Starace et al., 2025) evaluates end-to-end replication using author-developed hierarchical rubrics, and HCAST (Rein et al., 2025) grounds task difficulty in the time required by skilled humans. ScienceAgentBench (Chen et al., 2025) similarly derives data-driven discovery tasks from published scientific workflows.

Other environments broaden either the training interface or the space of research problems. MLE-Dojo (Qiang et al., 2025) provides interactive Kaggle environments with normalized rewards suitable for collecting learning trajectories; PostTrainBench (Rank et al., 2026) extends autoresearch to coupled post-training decisions; and FrontierScience (Wang et al., 2026b) targets expert-level scientific tasks. DiscoGen (Goldie et al., 2026) supplies procedurally generated algorithm-discovery tasks with meta-train/meta-test splits, whereas AlgoTune (Press et al., 2025) evaluates measurable speedups to established numerical programs. Finally, Frontier-Eng (Chi et al., 2026) characterizes how improvements diminish over extended engineering runs, and FML-bench (Zou et al., 2026) compares search strategies across controlled landscapes. In addition to introducing HillClimbBench, we cast several existing environments as instances of a common research decision process and ask whether research strategy can be learned across trajectories.

Scientific and algorithmic discovery agents.

Several systems automate substantial portions of the research lifecycle. The AI Scientist (Lu et al., 2024) and AI Scientist-v2 (Yamada et al., 2025) combine experiment execution, evaluation, and paper generation, with the latter adding tree search and structured memory. FunSearch (Romera-Paredes et al., 2024) established language-model-guided evolution as a mechanism for discovering programs, and AlphaEvolve (Novikov et al., 2025) scales this approach through model ensembles, evaluation cascades, and quality-diversity search. SAGA (Du et al., 2025) extends automation to the evolution of scientific objectives, while the AI co-scientist (Gottweis et al., 2026) and AutoDiscovery (Agarwal et al., 2025) organize hypothesis development through evolutionary selection and Bayesian surprise, respectively.

Research agents also differ in how they divide and retain work. R&D-Agent (Yang et al., 2025) separates researcher and developer roles, AutoScientists (Gao et al., 2026) coordinates decentralized teams through shared experimental memory, and MARS (Chen et al., 2026) performs budget-aware tree search over repository states. AutoSOTA (Li et al., 2026) targets reproduction and improvement of existing models, LLM-AutoSciLab (Kabra et al., 2026) casts discovery as active experimental design, and Arbor (Jin et al., 2026) refines explicit hypothesis trees. DiscoverPhysics (Wiemann et al., 2026) evaluates open-ended scientific reasoning in physics, whereas AInstein (Mishra et al., 2025) studies iterative critique using parametric knowledge alone. These systems primarily improve a fixed model through inference-time organization. MIRA is compatible with such execution mechanisms, but our central concern is learning the policy that chooses the next research objective from prior experience.

Search, memory, and evolving agent structure.

Existing agents encode a wide range of search biases. AIDE (Jiang et al., 2025b) uses explicit drafting, improvement, and debugging operators; LATS (Zhou et al., 2024) combines tree search with acting, planning, reflection, and model-generated values; and LoongFlow (Wan et al., 2025a) couples plan–execute–summarize cycles with evolutionary archives. DataMaster (Du et al., 2026) schedules exploration over the available budget, AIRA (Pepe et al., 2026) applies UCT-style search to neural-architecture discovery, and SciMaster (Chai et al., 2025) distributes exploration across multiple agents. Automated AI-research Speedrunning (Zhao et al., 2025) provides a controlled autoresearch setting, while Agent Laboratory (Schmidgall et al., 2025) organizes research into fixed phases. The diversity of successful scaffolds, together with the strategy-dependent results of Frontier-Eng (Chi et al., 2026) and FML-bench (Zou et al., 2026), motivates selecting research actions conditionally rather than committing to one search heuristic for an entire investigation.

A complementary line of work changes the agent’s persistent artifacts. SkillGrad (Wang et al., 2026a) treats trajectory diagnoses as textual gradients, and SkillOpt (Yang et al., 2026) uses bounded edits and validation to improve reusable skill documents. Harness self-evolution analysis (Lin et al., 2026) distinguishes the ability to modify a harness from the ability to benefit from those modifications, while Compiling Agentic Workflows into LLM Weights (Dennis et al., 2026) shows that procedural workflows can be distilled into model weights. MIRA keeps the execution harness fixed in our learning experiments and instead updates the model that allocates research effort; evolving the harness and learning the meta-reasoning policy are therefore complementary directions.

Feedback and value learning.

Language feedback can convey substantially more structure than a scalar reward, as illustrated empirically by LLF-Bench (Cheng et al., 2023) and formalized by Formalizing Learning from Language Feedback with Provable Guarantees (Xu et al., 2026). Training AI Co-Scientists Using Rubric Rewards (Goel et al., 2025) uses rubric rewards to train research-plan generation, while HypoAgents (Duan et al., 2025) uses Bayesian belief and entropy updates to guide which hypotheses to refine. Our experiments instead use scalar, outcome-grounded targets: either a gold-binary outcome or changes in a proxy experimental metric. This choice isolates whether delayed research outcomes alone can train value estimates at MIRA’s decision boundaries; versioned artifacts could support richer feedback in future work.

MIRA-AC also relates to shared actor–critic representation learning. A3C (Mnih et al., 2016) uses a shared backbone with separate policy and scalar-value outputs, while Phasic Policy Gradient (Cobbe et al., 2021) studies the balance between useful representation transfer and interference between policy and value objectives. UNREAL (Jaderberg et al., 2017) demonstrates how auxiliary prediction objectives can shape a shared policy representation. More recent generative approaches express verification or value estimation through language modeling itself: Generative Verifiers (Zhang et al., 2025) formulates verification as next-token prediction, and GenAC (Shan et al., 2026) uses a generative critic for value modeling. MIRA-AC similarly uses one language model and one next-token interface for both research actions and return prediction, without adding a scalar head or maintaining a separately resident critic.

Credit assignment in multi-turn agents.

Several methods localize outcome supervision within long interactions. Credit assignment can also proceed through a learned step-level critic: Iterative GRPO (Jiang et al., 2025a) fits a turn-level Qπ from logged Monte Carlo returns and uses it as the reward model for single-turn policy improvement in batch-online multi-turn settings. Credit Assignment with Resets (Samanta et al., 2026a) uses counterfactual continuations from intermediate states, and Tree-GRPO (Ji et al., 2026) obtains step-level relative signals from sibling branches that share a prefix. R1-Searcher (Song et al., 2025) learns retrieval behavior from outcome-only reinforcement learning, StepSearch (Wang et al., 2025) applies step-wise policy optimization to search-augmented reasoning, and strategy-guided exploration (Szot et al., 2026) operates over high-level language strategies. PokeeResearch (Wan et al., 2025b) studies reinforcement-learning design choices for deep-research agents. MIRA-AC differs in where it places the learning boundary: it reconstructs complete research episodes but applies policy loss only to the complete sampled decision responses that allocate future computation. Its decision-level generative critic assigns delayed credit at those boundaries, and buffered asynchronous optimization accommodates investigations with heterogeneous durations.

Appendix B Experimental Setup

We study four autoresearch environments: BNLearn causal discovery, Physics Discovery, SRBench symbolic regression, and HillClimbBench-CPU architecture research. In the reported policy evaluation, BNLearn, Physics, and SRBench each contain 100 tasks with one rollout per task. HillClimbBench-CPU contains 50 distinct paper-derived tasks with two independently initialized rollouts per task. Each rollout permits at most 20 MIRA iterations.

B.1 Shared Interaction Protocol

Each task is an isolated, persistent workspace. At every iteration, a fresh executor receives the MIRA work order, the fixed task prompt, and stable environment-specific instructions. The prompt identifies the available data and tools, interaction budget, editable files, submission artifact, and hidden information. The executor may write analyses and programs, run proxy evaluators, update its artifact, and leave evidence for later iterations. The shared procedure governing outer-loop and executor interactions is described in App. C; below we specify the environment-facing part of the prompt.

For the three scientific environments, policy training and reported evaluation use independently initialized trajectories on the same fixed task identities and seeds. They therefore test new research trajectories and held-out evaluations within the same tasks, not generalization to unseen task identities.

B.2 Defining Proxy Hill-Climbing Signals

Each environment exposes a different proxy measure of progress: graph-edit distance, trajectory error, expression error, or architecture performance. We map that measure to a higher-is-better quality qt∈[0,1] after iteration t, with q0 denoting the fixed initial quality. We construct each map from constants in the environment specification and freeze it before training.

The four maps use natural task-level anchors. BNLearn measures how much of the empty graph’s structural Hamming distance (SHD) has been removed. Physics scales trajectory mean squared error (MSE) by position variance and applies a fixed reciprocal map capped at 0.95. SRBench measures log-normalized mean squared error (log-NMSE) improvement from the initial expression toward the float64 numerical floor. HillClimbBench-CPU measures improvement from the supplied reference architecture toward the task’s ideal score of one. The exact equations appear below. The endpoints need not be attainable; they are placed to distinguish plausible within-budget improvements.

These proxies provide a task-local ordering of partial progress. They turn changes across successive research iterations into marginal rewards at a resolution suited to the current model class and fixed research budget. The mapping remains imperfect: it can miss real progress, reward a brittle shortcut, or saturate too early. Gold evaluation therefore remains the final measure of the selected artifact. It stays hidden during research, while reward depends on increases in best-so-far proxy quality.

For a new environment, the construction begins with a repeatable proxy metric that usually moves with the desired outcome. Natural anchors include a supplied baseline, an empty prediction, or a task-defined target. A monotone transform can spread raw errors that span orders of magnitude, while its scale should keep plausible improvements away from a narrow or saturated range. Invalid artifacts receive no improvement. The completed map is frozen before policy training. Development tasks then establish its usable range and its association with gold outcomes; poor alignment calls for redesign before final evaluation.

The qt values define best-so-far progress and marginal reward in Sec. 2.2; other visible diagnostics do not affect reward. After selection, a hidden gold evaluation produces the score reported in the results table. It uses the same hidden target as the proxy for BNLearn and separate held-out data or conditions for the other environments. A task is solved when this gold evaluation meets the environment-specific success criterion.

B.3 Environment Interfaces

BNLearn causal discovery.

We construct 100 tasks from the Alarm (32 tasks), Child (17), Insurance (24), and Water (27) networks in the Bayesian Network Repository (Scutari, n.d.). Each task is a randomly permuted, anonymized 14–34-node induced subgraph. A linear-Gaussian structural equation model provides 1,800 training and 600 visible validation observations.

The agent may analyze the visible data, query validation Bayesian information criterion (BIC) diagnostics, and, on an n-node task, request n interventions that report estimated total effects. These diagnostics do not determine reward, and the executor cannot query graph accuracy while carrying out an investigation. The final artifact is a directed acyclic graph (DAG).

After each completed iteration, the harness reports directed structural Hamming distance (SHD), including missing, extra, and reversed edges. For the empty-graph SHD S0 and submitted-graph SHD St, proxy quality is qt=clip⁡((S0−St)/S0,0,1). The best model-authored graph is retained and restored for final scoring under the same hidden-DAG objective. Its gold score is the normalized SHD gap closure above, evaluated on the submitted graph; the task is solved only by exact directed recovery.

Physics discovery.

We adapt DiscoverPhysics (Wiemann et al., 2026) into 100 anonymous worlds spanning 12 mechanisms and three difficulty levels. The agent receives a mission and topology-specific laboratory interface, but not the governing law. Across 20 MIRA iterations, it may allocate 15 experiment rounds among the first 19; the last iteration is reserved for finalization. It may validate experiment plans and repeatedly fit executable candidate laws to all acquired trajectories.

The agent selects an executable law and a short explanation. For acquired-trajectory MSE mt and position variance vt, proxy quality is qt=0.95/(1+mt/vt). After the fixed horizon, the law is evaluated once on held-out trajectories and the explanation is graded once by a hosted GPT-5.5 judge using a hidden world-specific reference and rubric. Let n be trajectory MSE normalized by the held-out world’s ground-truth trajectory variance, and let e be the explanation score. For an unsolved task, the gold score is the product of the threshold-scaled components 0.1/max⁡(n,0.1) and min⁡(1,e/0.75), capped strictly below one. A task is solved, and receives a gold score of one, when n<0.1 and e≥0.75.

SRBench symbolic regression.

We use 100 LLM-SRBench problems (Shojaee et al., 2025) from chemical reactions (30 tasks), physical oscillators (27), materials science (23), and population growth (20). Within each family, tasks span a frozen reference-expression complexity ranking. The official training data are split deterministically 80/20 into visible fit and validation sets; the in-distribution (ID) test set, optional out-of-distribution (OOD) set, and reference expression remain hidden.

The agent may analyze the available data freely and evaluate expressions within a constrained SymPy-compatible grammar. It must select one final expression. The environment reports proxy metrics but does not restore the candidate with the highest proxy metric, so final selection remains with the agent.

For validation NMSE Nt, fixed initial-expression NMSE B, and float64 floor ϵ, proxy quality is

The final expression is evaluated once on sealed ID-test data, with OOD metrics reported when available. Its gold score is R2 on the sealed ID test. The task is solved when pointwise relative error is at most 0.1 everywhere (and the prediction is exact when the target is zero).

HillClimbBench-CPU architecture research.

We select 50 research papers whose central architectural contribution can be implemented, trained, evaluated, and iteratively improved within a CPU-only budget, and construct one task from each paper. We run two independent trajectories per task. Each task prompt combines a technical paper summary, research brief, description of how gold evaluation differs from proxy evaluation, evaluation contract, and reference implementation. The agent tests architectural hypotheses while the host fixes the data, targets, loss, optimizer, training budget, evaluation seeds, and resource limits. Every candidate is trained from scratch on one CPU thread.

The 50 source papers are:

1. 1.

   Can Recurrent Neural Networks Warp Time? (Tallec and Ollivier, 2018);

2. 2.

   An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution (Liu et al., 2018);

3. 3.

   Deep Lattice Networks and Partial Monotonic Functions (You et al., 2017);

4. 4.

   Deep Sets (Zaheer et al., 2017);

5. 5.

   Learning Nonlinear Operators via DeepONet Based on the Universal Approximation Theorem of Operators (Lu et al., 2021);

6. 6.

   Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (Chen et al., 2018a);

7. 7.

   E(n) Equivariant Graph Neural Networks (Satorras et al., 2021);

8. 8.

   Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains (Tancik et al., 2020);

9. 9.

   Full-Capacity Unitary Recurrent Neural Networks (Wisdom et al., 2016);

10. 10.

    How Powerful are Graph Neural Networks? (Xu et al., 2018);

11. 11.

    Graph Attention Networks (Veličković et al., 2017);

12. 12.

    Grid Long Short-Term Memory (Kalchbrenner et al., 2015);

13. 13.

    Group Equivariant Convolutional Networks (Cohen and Welling, 2016);

14. 14.

    Learning Latent Permutations with Gumbel-Sinkhorn Networks (Mena et al., 2018);

15. 15.

    HiPPO: Recurrent Memory with Optimal Polynomial Projections (Gu et al., 2020);

16. 16.

    Hyena Hierarchy: Towards Larger Convolutional Language Models (Poli et al., 2023);

17. 17.

    HyperNetworks (Ha et al., 2016);

18. 18.

    Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN (Li et al., 2018);

19. 19.

    Interaction Networks for Learning about Objects, Relations and Physics (Battaglia et al., 2016);

20. 20.

    The Unreasonable Effectiveness of the Forget Gate (van der Westhuizen and Lasenby, 2018);

21. 21.

    Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs (Murphy et al., 2018);

22. 22.

    KAN: Kolmogorov-Arnold Networks (Liu et al., 2024);

23. 23.

    Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks (Voelker et al., 2019);

24. 24.

    Resurrecting Recurrent Neural Networks for Long Sequences (Orvieto et al., 2023);

25. 25.

    Long Short-Term Memory (Hochreiter and Schmidhuber, 1997);

26. 26.

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Gu and Dao, 2023);

27. 27.

    Memory Networks (Weston et al., 2014);

28. 28.

    MinimalRNN: Toward More Interpretable and Trainable Recurrent Neural Networks (Chen, 2017);

29. 29.

    Hopfield Networks is All You Need (Ramsauer et al., 2020);

30. 30.

    Neural Arithmetic Units (Madsen and Johansen, 2020);

31. 31.

    Neural Ordinary Differential Equations (Chen et al., 2018b);

32. 32.

    Neural Processes (Garnelo et al., 2018);

33. 33.

    Neural Turing Machines (Graves et al., 2014);

34. 34.

    OptNet: Differentiable Optimization as a Layer in Neural Networks (Amos and Kolter, 2017);

35. 35.

    Parallelizing Legendre Memory Unit Training (Chilkuri and Eliasmith, 2021);

36. 36.

    PDE-Net: Learning PDEs from Data (Long et al., 2017);

37. 37.

    Perceiver IO: A General Architecture for Structured Inputs & Outputs (Jaegle et al., 2021);

38. 38.

    Rethinking Attention with Performers (Choromanski et al., 2020);

39. 39.

    Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences (Neil et al., 2016);

40. 40.

    Pointer Networks (Vinyals et al., 2015);

41. 41.

    Recurrent Highway Networks (Zilly et al., 2016);

42. 42.

    Efficiently Modeling Long Sequences with Structured State Spaces (Gu et al., 2021);

43. 43.

    Orthogonal Recurrent Neural Networks with Scaled Cayley Transform (Helfrich et al., 2017);

44. 44.

    Implicit Neural Representations with Periodic Activation Functions (Sitzmann et al., 2020);

45. 45.

    Object-Centric Learning with Slot Attention (Locatello et al., 2020);

46. 46.

    Spatial Transformer Networks (Jaderberg et al., 2015);

47. 47.

    TabNet: Attentive Interpretable Tabular Learning (Arik and Pfister, 2019);

48. 48.

    Less is More: Recursive Reasoning with Tiny Networks (Jolicoeur-Martineau, 2025);

49. 49.

    Universal Transformers (Dehghani et al., 2018); and

50. 50.

    Zoology: Measuring and Improving Recall in Efficient Language Models (Arora et al., 2023).

The proxy evaluator measures both in-distribution performance and a task-specific structural extrapolation split. Each task fixes in advance a scoring function that combines these measurements into a scalar proxy metric ut∈[0,1], with one denoting the task’s declared ideal. Only valid candidates whose proxy metric exceeds that of the retained incumbent replace it. For the frozen reference metric u0, proxy quality is qt=clip⁡((ut−u0)/(1−u0),0,1). After selection, the retained architecture and task reference are retrained on the same five gold-evaluation seeds under harder structural conditions. The task’s gold evaluator maps each run to a score in [0,1], and the submission’s gold score is the mean across its five runs. A submission meets the benchmark’s success criterion when the lower endpoint of the two-sided 95% Student-t confidence interval over the five paired candidate-minus-reference scores is positive. In the policy comparison, a win is a paired evaluation in which the trained-policy submission has a higher gold score than the matched untrained-MIRA submission.

B.4 Evaluation

We report the proxy best-so-far quality q¯t and the gold score separately. Comparisons use matched task instances and interaction budgets; confidence intervals use a task-clustered bootstrap.

Appendix C MIRA Implementation

This appendix describes how MIRA constructs each decision context, executes the resulting work order, and delegates evaluation and submission to the task environment.

C.1 Context Construction

The outer-loop context is rebuilt at every iteration. Its fixed portion contains the task statement, a compact workspace summary, the latest proxy evaluation, a structured summary of the immediately preceding work episode and a pointer to its handover when available, and the remaining budget. The full episode record and handover text are not copied automatically. Instead, the fixed curator has read-only access to repository files, diffs, commit history, and prior work records, from which it selects a bounded set of decision-relevant evidence. The fixed portion and this selected evidence form the new decision context.

The policy is instructed to select a bounded objective or coherent experiment set while considering its information value, cost, remaining time, and risk to the current solution. Its work order states the objective, relevant evidence, resource guidance, acceptance criteria, and stopping condition. The same structure is used for theorem proving, with experiments replaced by proof construction and verification.

C.2 Decision and Execution

A fresh executor in the Codex harness receives the work order, its selected repository references, and the stable environment instructions. It may inspect and modify the workspace, run tests and authorized tools, and evaluate candidates. It validates substantive work, commits useful artifacts and evidence, and leaves a concise handover. Recorded failures and contradictory results remain available to later decisions through the repository.

The repository—including the current candidate, experiments, results, analyses, and handovers—persists across iterations; the model contexts do not. Any proxy evaluation produced after execution becomes evidence for the next decision. When an environment maintains an incumbent, it protects the strongest validated artifact against later regressions.

C.3 Environment-Controlled Evaluation and Submission

The environment adapter supplies the task-specific prompt, tools, proxy feedback, reward mapping, submission artifact, and terminal evaluator. It also determines who authorizes submission: Codex may submit under inner-loop control, the policy chooses between further investigation and submission under outer-loop control, and the environment evaluates only after research ends under external control.

A run ends when a model-visible success condition is met, submission is authorized, or the iteration or wall-clock budget is exhausted. Gold evaluation remains outside both model contexts and is applied only to the selected repository artifact. Environments that retain an incumbent may restore it before final evaluation.

Appendix D Semantic Exploration Graphs and Strategic Inertia

We construct the exploration graphs after each run has terminated. The graphs are used only for analysis and are never shown to the evaluated system.

Comparable evidence projection.

We project every run into the same chronological evidence format: the common task statement, text authored by the evaluated models, and the submitted proof. For MIRA, model-authored text includes the outer-loop policy’s work orders and the executors’ reports; for Codex, it includes assistant messages. We exclude commands, tool outputs, diffs, commits, repository metadata, handovers, submission reviews, and external evaluations from every condition. The same projection is applied at each forced-continuation checkpoint, and the same downstream extraction procedure is applied to every inference system.

Semantic graph extraction.

A GPT-5.5 pass at temperature zero converts the projected trace into a graph. A node represents a semantically meaningful proof or research step, such as an approach, claim, lemma, counterexample, experiment, candidate, result, or submission. Several raw turns may therefore map to one node, while one long response may contain several nodes. Nodes record their status, such as supported, proved, refuted, stalled, abandoned, or submitted. Typed edges record evidence-supported relationships including refinement, dependence, support, refutation, pivoting, supersession, and production of a result. Chronology alone does not create an edge.

The extractor receives the trajectory content but not the method name, model name, grader feedback, or final grader score. Every node and edge must cite a source segment in the projected evidence; outputs without valid provenance are rejected. The extracted nodes are ordered by first appearance in the trace. For visualization, the submission ancestry is placed on a central lane and other connected components are assigned separate lanes. Shortened labels and display coordinates affect only presentation, not the analysis.

Research-direction annotation.

A separate GPT-5.5 pass at temperature zero groups the fixed semantic nodes into research directions. A direction is a high-level hypothesis, proof plan, construction, mechanism, or algorithmic approach. Lemmas, tests, repairs, and refinements remain in the same direction when they serve the same high-level plan; a materially different plan receives a new identifier, and a later return to a previous plan reuses its identifier. This pass may correct the order of fixed nodes when their provenance establishes a different first-introduction order, but it may not add, remove, split, or merge nodes. It observes node content, provenance episode identifiers, and graph relations, but not the method or model name, final grader score, grader feedback, node status, terminal submission, graph title, or graph summary.

Strategic inertia.

Two consecutive non-submission semantic steps form a continuation when they belong to the same annotated research direction and a pivot otherwise. For graph 𝒢, we define I⁡(𝒢)=Ncontinue/(Ncontinue+Npivot). The root and terminal submission are excluded. Graphs with fewer than two non-submission nodes have no transitions and receive no inertia value. The statistic is descriptive: high inertia can sustain a promising direction or prolong a local minimum, while low inertia can expose alternatives or reflect premature switching.

For the natural GPT-5.5 comparison, the analysis contains 30 problems, three trials, and three inference systems. We compute inertia for each run, average trials within each problem, and obtain 95% confidence intervals by resampling problems with 10,000 bootstrap draws. Pairwise contrasts match runs by problem and trial, average the paired differences within each problem, and again bootstrap problems. Natural and forced-continuation runs are analyzed separately.

The four visualizations below are the problems on which a gap remains after output-token-matched continuation. They show the first natural-stopping trial and use the same extraction and layout procedure for all three inference systems.

Appendix E RMT Architecture Identified by MIRA

Matched search protocol.

We run one 24-hour GPT-5.5 campaign for each orchestration condition. All three campaigns use the same task reference architecture, proxy evaluator, compute budget, and parameter limit. Proxy evaluations guide research, and the final submitted architecture is scored separately by the gold evaluator.

Selected architecture.

An ordinary RMT stores each token’s residual state as a matrix Xℓ∈ℝDk×Dv. Attention and MLP branches read this state through a lower-rank interface, produce a matrix update Δ​Xℓ(b), and add it uniformly as Xℓ,out(b)=Xℓ,in(b)+Δ​Xℓ(b) for b∈{attn,mlp}. MIRA’s final model retains this matrix state and low-rank read/write interface, uses a GELU-gated linear unit (GEGLU) feed-forward block, and introduces a separate learned gate for each branch and layer: Xℓ,out(b)=Xℓ,in(b)+Gℓ(b)⊙Δ​Xℓ(b), where Gℓ(b)∈ℝDk×Dv. Every coefficient is initialized to one, so the new update is exactly the ordinary additive update at initialization. The two gates add only 0.056% to the model’s parameter count. Unlike a generic GEGLU block, this mechanism is specific to the matrix residual stream: it learns where each branch can amplify or suppress a proposed matrix-state write.

MIRA first tested the gate at small scale, then reran a final-scale comparison matched in geometry, seed, optimizer, and training tokens. Relative to the ungated GEGLU control, the gated model improved last-token cross-entropy by 0.009 at small scale and by 0.021 at the final scale. Both comparisons use one seed, and the ungated final-geometry control was not submitted to the gold evaluator. These ablations support the gate as a promising mechanism, but they do not isolate it as the cause of the full gold-loss improvement in Fig. 4(b).

The learned gates did not remain near their identity initialization. Attention gates generally amplify writes, whereas MLP gates generally suppress them, and both vary with depth. A two-way additive decomposition into row and column effects explains roughly three quarters of the variation in the depth-averaged attention map and nearly nine tenths in the MLP map, with stronger variation along the residual-key axis. This supports a structured, branch-, depth-, and address-dependent write-allocation interpretation. It does not establish dynamic routing: the gates are fixed across tokens and examples, and their low exact-map correlation across layers does not support a clean permanent partition into “durable” and “temporary” cells.

Appendix F Loop Transformer Search and Selected Designs

Matched search protocol.

We compare matched 48-hour MIRA and Codex campaigns using GPT-6 Astra. Both campaigns use the same model-scale and training-token rungs. At each rung, candidates and the task reference architecture are evaluated on the same 512-context proxy set with eight recurrent iterations. Intermediate search decisions use these results; after submission, the final incumbent from each campaign and the task reference are evaluated on a separate 4,873-context gold set. The proxy and gold results are reported separately in Fig. 6.

MIRA search trajectory.

MIRA begins by hypothesizing that repeated applications of the shared core overwrite useful state. It compares a learned adaptive retention gate with a constant interpolation between the previous and proposed states. The learned gate does not outperform this control, so subsequent candidates retain the simpler leaky update. After explicit fast and slow states, iteration-history mechanisms, and other routing alternatives produce small or inconsistent gains, MIRA hypothesizes that repeated positive attention may reinforce information that later recurrences struggle to remove. It tests the resulting subtractive attention mechanism against a matched positive-mixture control, then promotes it only after the advantage persists across seeds and model scales. The selected design combines what the campaign calls “signed attention” with a leaky recurrent update (Fig. 16). If the two branches produce attention reads A1 and A2 over shared values, the attention module returns A1+(A1−A2). Each branch separately uses ordinary softmax attention; the subtraction creates effective positive and negative coefficients over the shared values. The second branch copies the first branch’s query and key parameters at initialization, making the additional contrastive term exactly zero before training. Training can then break this symmetry, allowing the effective coefficients to suppress as well as reinforce value components. After the block proposes F⁡(h(k)) from recurrent state h(k), the leaky update mixes the previous and proposed states, h(k+1)=ρ​h(k)+(1−ρ)​F​(h(k)), with fixed ρ=0.25. The attention variant and leaky update therefore play complementary roles: one supports subtractive routing, while the other limits how completely a recurrent step can overwrite its input. Applied in a Loop Transformer, this mechanism resembles a fixed-coefficient variant of Differential Attention (Ye et al., 2025). It uses two full query–key branches initialized identically and the fixed mixture 2​A1−A2.

Comparison.

Codex selects a tangent-state update with local/global attention routing and learned write gates on both the attention and MLP branches, a complementary strategy that regulates how aggressively each recurrent pass changes the state. MIRA’s incumbent improves proxy loss at every completed scale. Its final proxy NLL is 3.0331, compared with 3.2419 for Codex and 3.2940 for the task reference. On the separate gold evaluation, its NLL is 3.0464, compared with 3.2275 for Codex and 3.2653 for the task reference.

Appendix G Critic Training and Validation

Targets and value contracts.

All critic variants use the stitched Monte Carlo target Gt. Under proxy-marginal supervision, Gt is the future increase in best-so-far proxy quality, where qt is defined for each environment in App. B.2. The corresponding proxy qualities are directed-SHD gap closure for BNLearn, acquired-trajectory normalized fit for Physics Discovery, validation log-NMSE gap closure for SRBench, and initial-to-ideal proxy-score gap closure for HillClimbBench-CPU.

Under gold-binary supervision, the label Y∈{0,1} records whether the final submission meets the environment’s gold success criterion. This means exact directed recovery for BNLearn; the joint hidden-trajectory and explanation criterion for Physics Discovery; the ID-test pointwise-accuracy criterion for SRBench; and a positive lower endpoint of the paired 95% confidence interval for improvement over the reference architecture for HillClimbBench-CPU. These are the same success criteria specified in App. B; the policy-comparison win rate is not a critic target.

Critic parameterizations and objectives.

For proxy-marginal targets, a scalar critic minimizes ℒscalar​(ϕ)=M−1​∑m=1M(Vϕ​(om)−Gm)2. For the categorical critics, let z1,…,zK be the ordered support. If zj<Gt<zj+1, we set yj​(Gt)=(zj+1−Gt)/(zj+1−zj) and yj+1​(Gt)=(Gt−zj)/(zj+1−zj), with all other entries zero; if Gt=zj, then yj​(Gt)=1. A categorical head produces qϕ​(zi∣o)=softmax⁡(hϕ​(o))i, while the generative critic uses the restricted next-token distribution pϕ​(zi∣o) from Sec. 4.1. They minimize ℒcat(ϕ)=M−1∑m=1MDKL(y(Gm)∥pϕ(⋅∣om)), replacing pϕ with qϕ for the categorical head. For the generative critic, the implementation gathers the next-token log probabilities of the support labels, renormalizes them across that support, and evaluates the KL objective in Sec. 4.2; no probability mass outside the support enters the critic loss.

Under gold-binary supervision, the scalar critic outputs a single logit ℓϕ​(o) and represents value as Vϕ​(o)=σ⁡(ℓϕ​(o)). It is trained with binary cross-entropy with logits,

The gold-binary generative critic instead restricts the next-token distribution to the support {0,1} and uses the corresponding one-hot target. Token-level variants repeat the decision return at each causal token prefix within the action; decision-level variants forecast once from the curated observation.

Target and forecast validation.

Proxy-marginal returns are validated and clipped to Gt∈[0,1−q¯t−1]. Before bootstrapping, temporal-difference targets, or advantage estimation, proxy-marginal forecasts must be finite and are clipped to the same interval. For a generative critic, Vϕ​(ot)=∑ipϕ​(zi∣ot)​zi. Gold-binary targets lie in {0,1}, and both the sigmoid head value and the expected value of the two-point generative distribution lie in [0,1].

Pretraining and target-environment adaptation.

We pretrain Qwen3.6-27B for one epoch with batch size 8, microbatch size 1, maximum sequence length 262,144, and seed 0. The optimizer is AdamW with learning rate 10−5, weight decay 0.01, gradient clipping at 1.0, 5% linear warmup, and cosine decay to 10% of the peak learning rate. Training uses bfloat16 and the LoRA configuration in Table 3. The value-forecasting experiments use four H200s; the four-environment critic used to initialize the general MIRA-AC policy uses eight H200s. For target-environment adaptation, we evaluate support sizes {0,1,5,10,25,50} over five paired support/query splits with 50 query episodes per split. Adaptation makes one episode-ordered pass using AdamW at learning rate 10−5 and LoRA dropout 0.05 on one H200.

For each target environment and paired support/query split, we first average the single-source pretraining condition across source environments. We then report averages over target environments and splits with 95% confidence intervals from a paired bootstrap over support/query splits.

Generative critic prompts.

The generative critic receives the same curated decision-scoped conversation as the actor, followed by the value query in Fig. 17. When forecasting at termination, the query marks the normal research budget as exhausted.

Value contract: <ENVIRONMENT-SPECIFIC VALUE-CONTRACT NAME>
<ENVIRONMENT-SPECIFIC VALUE-CONTRACT DESCRIPTION>

Observable value state:
Current best normalized quality before this decision: <CURRENT BEST NORMALIZED QUALITY>
Total decision horizon: <TOTAL DECISION HORIZON>
Decisions remaining including the current decision: <DECISIONS REMAINING>
<OPTIONAL TERMINAL-PHASE STATUS>

You are estimating the value of the research state in the conversation above.

Predict the future increase in the final best solution quality, using the normalized 0-to-1 quality scale.

0.0 means no further improvement over the best solution already found.
Higher values mean larger expected improvement in final best solution quality.

Use the raw metrics, errors, tests, benchmark scores, and experiment results as evidence, but do not answer in raw environment units.

On this normalized 0-to-1 scale, how much additional improvement over the current best solution do you expect from here?
A. 0.0
B. 0.1
C. 0.2
D. 0.3
E. 0.4
F. 0.5
G. 0.6
H. 0.7
I. 0.8
J. 0.9
K. 1.0

Answer with only the single letter for your estimate.

For gold-binary supervision, the environment-specific contract states the gold success event above, and the proxy-marginal question and eleven-point support are replaced by:

Predict the binary terminal outcome of the complete research episode.

0.0 means the environment’s authoritative terminal success condition will not be met.
1.0 means the environment’s authoritative terminal success condition will be met.

Use the raw metrics, errors, tests, benchmark scores, experiment results,
current progress, and remaining budget as evidence.

Will this research episode ultimately succeed?
A. 0.0
B. 1.0

Answer with only the single letter for your estimate.

Appendix H MIRA-AC Training Details

All policy-training runs use Qwen3.6-27B (Qwen Team, 2026) (Qwen-27B) with LoRA adapters. In separate actor–critic configurations, actor adapters target the standard attention and MLP projections, mixture-of-experts (MoE) experts, and latent fully connected projections. MIRA-AC targets the standard attention and MLP projections together with the model’s recurrent input and output projections. Separate-critic variants use distinct policy and critic adapters. In MIRA-AC, one shared adapter serves both roles, with separate optimizer states for actor and critic updates; online critic updates modify only the shared adapter’s MLP LoRA parameters. Each critic is initialized from a pretrained checkpoint matched to its critic parameterization and target environment. Separate actors start from the untrained MIRA policy. For each single-environment experiment, MIRA-AC initializes its shared adapter from a generative critic pretrained on trajectories from that environment under the matching supervision target. The cross-environment policy instead starts from a proxy-marginal generative critic pretrained jointly across all four environments, then receives online actor–critic updates on SRBench, Physics Discovery, and HillClimbBench-CPU.

Table 3 reports the configuration of the general cross-environment policy used for the main result. The reported checkpoint is the epoch-one shared adapter at version 94, after 49 actor updates and 90 critic updates. When both learners update in one transaction, their updates share a single published version.

We train one policy per configuration. Each environment-specific run makes one pass over a fixed ordered set of 100 episodes. The cross-environment run uses 300 episodes: 100 from each of its three actor-training environments. The scientific environments use 100 tasks with one rollout each; HillClimbBench-CPU uses 50 tasks with two rollouts each. Within each environment, the task order is deterministically shuffled from run seed 42; environments are interleaved by equal-weight deficit round robin, and generation seeds are tied to logical decision identifiers.

The actor packs each sealed learner batch dynamically into microbatches of at most 262,144 tokens and takes one optimizer step after accumulating their gradients. The online critic uses microbatch size 1 across four GPUs and takes one optimizer step after a complete pass without replacement through each sealed batch. The proxy generative critic uses support {0.0,0.1,…,1.0}; the gold-binary critic forecasts the gold-binary outcome on support {0,1}. We use γ=1 and exact Monte Carlo critic targets. Actor advantages use the episode-length-adaptive GAE configuration in Table 3.

The implementation uses the token-level importance ratio and range mask from Sec. 5.3, together with the decision-normalized actor objective in Eq. 3. Tokens outside the ratio interval are excluded from the policy-gradient term but remain in the squared-log-ratio penalty. The trainable span contains only the complete sampled decision response—including its emitted reasoning and end-of-turn token—whether it ends in a work order or a termination decision. Prompts, curated observations, tool calls, and executor outputs are masked.

The buffered schedule maintains at most 32 active episodes and immediately refills completed slots while budget remains. Every completed episode enters both buffers, which seal independently according to Table 3; remaining examples form final underfilled batches. In MIRA-AC, all ready role-specific updates are applied within one atomic transaction. The critic update precedes the actor update, recorded values remain detached during actor training, and the resulting shared adapter is published once. Live episodes adopt it only at the next meta-reasoning boundary.

Value-forecasting critic pretraining uses four H200 GPUs. The four-environment critic used to initialize the general policy instead uses eight H200 GPUs. The reported general-policy run uses two eight-H200 nodes for actor–critic inference and optimization and a separate two-node, 16-H200 pool for frozen executors.

Context-cost accounting.

The context-cost analysis in Fig. 11 compares MIRA-AC with fixed-trajectory replays that preserve the recorded work orders, executor actions, and tool interactions. The MIRA-AC baseline counts policy input together with exact or imputed executor input. The no-compaction replay carries all prior cross-iteration history into each later model call, while the auto-compaction projection caps that history at 200k tokens. The displayed range captures how often the carried history is reread. Its lower endpoint adds the history once during each active stage—curation, work-order selection, and execution. Its upper endpoint follows a sequential tool loop in which the updated context is read after each recorded curation request and executor turn. Frozen-curator base prompts and outputs, critic inputs, compaction-summary costs, and behavioral changes are not modeled. The comparison therefore concerns rollout-generation cost, not optimization cost or sample efficiency.

Appendix I Trajectory Stitching Across Artificial Boundaries

MIRA reconstructs a fresh curated context at every research decision. Context-window limits, compaction, sequence packing, and learner batching may further split a single logical episode into multiple samples. None of these boundaries is an environment termination. Treating them as terminal either removes future TD residuals from earlier advantages or places later rewards at the wrong temporal distance.

Let a logical episode τ=(ξ1,…,ξT) be partitioned into stored samples 𝒮1,…,𝒮K. The partition does not alter the termination indicator: dt=1 only when the environment truly terminates. Each stored transition contains ot, at, rt, ot+1, actor behavior log probabilities and version θbt, and the critic version ct assigned to the transition. For that critic version, the system records both the current baseline vt=Vθct​(ot) and its successor bootstrap v~t+1(ct)=Vθct​(ot+1).

An asynchronous episode may adopt a different critic before decision t+1. At a nonterminal boundary where this occurs, we use the bridge Bt from Sec. 5.2, and set Bt=0 otherwise. Terminal successor values and bridges are zero. The implemented residual and backward recurrence follow Sec. 5.2 and Eq. 2. The bridge aligns the outgoing critic’s successor baseline with the baseline recorded by the critic version assigned to the next decision.

After this episode-level pass, A^t is scattered back to its original sample and token span for actor training. The critic target is the Monte Carlo return Gt, computed separately over the same reconstructed episode. The recorded value forecasts are treated as constants during the actor update, so the policy loss does not differentiate through the shared critic baseline.

Partition invariance.

For any partition of a fixed logical episode into stored samples, stitching produces the same actor advantages and Monte Carlo critic targets as processing the unpartitioned episode, provided the logged transitions, critic versions, and successor bootstraps are preserved. Stitching restores the same ordered rewards, terminal indicators, baselines, and cross-version bridges. A backward induction on the recurrence therefore yields the same A^t at every decision, while every reward remains in the same Monte Carlo return.

In our implementation, γ=1 and the critic is trained on the exact Monte Carlo return Gt=∑u=tTru. Actor advantages use episode-length-adaptive λT=1−1/(α​T), giving A^t=δt+λT​(1−dt)​(A^t+1+Bt). The full Monte Carlo return receives weight λTT−1≥1−1/α at the earliest decision and greater weight at later decisions. The bridges correct baseline-coordinate changes but do not alter the Monte Carlo critic target. Stitching remains necessary: Gt must include rewards from all later MIRA iterations, even when those iterations occupy different contexts, samples, or learner batches.

The governing invariant is that only true environment termination may stop value bootstrapping; infrastructure boundaries never do.

Appendix J Additional MIRA-AC Results

Appendix K Behavioral Analysis of MIRA-AC

This appendix examines the changes we observe in the model’s meta-reasoning and scientific behavior as a result of training. We compare the cross-environment MIRA-AC policy with untrained MIRA on 400 matched evaluation pairs (800 trajectories): 100 pairs each from symbolic regression, Physics Discovery, HillClimbBench-CPU, and BNLearn. Each pair contains the two policies’ trajectories on the same evaluation instance. We call the held-out evaluation result the gold outcome. In both systems, an outer policy selects the next research direction, an inner executor carries it out, and the harness manages execution and can force continuation after an incomplete iteration.

K.1 Analysis Protocol

We use GPT-5.5 as an LLM judge in two passes. In the first pass, the judge sees the sequence of work orders, executor reports, and proxy feedback under anonymous labels A and B, but not the policy identities or gold outcomes. It records what each trajectory’s policy and executor did, how they responded to evidence, and whether the policy’s next action could resolve an uncertainty visible at the time. This assessment is fixed before the second pass.

In the second pass, the judge receives this fixed assessment and each side’s gold outcome; policy identity remains hidden. It separately assesses whether a decision was reasonable when made and whether it ultimately helped or harmed the gold result. The judge also attributes each claimed contribution among the outer policy, inner executor, harness, and unobserved factors. The pair-level verdict used below asks which side’s outer-policy decisions contributed more to its own final outcome. The verdict excludes executor-only and harness-only contributions. Policy identity is revealed only after both passes are complete. All 400 pairs produced valid judgments in both passes.

For behavior rates, the unit is a trajectory, not an action. A trajectory is counted once for a behavior if the second-pass judge identifies at least one instance attributable at least partly to the outer policy; repeated instances do not increase the count. Behaviors attributed only to the executor, harness, or unobserved factors are excluded. The exception is productive risk, a whole-trajectory label indicating that a reasonable exploratory bet ultimately improved the gold result. Every rate therefore has a denominator of 400 trajectories per policy.

The matched comparisons describe observed differences between the policies. They do not estimate the effect of independently inducing a single behavior: executor stochasticity, interactions among decisions, and the evaluation environment can also affect the final result. The aggregate rates establish how common each behavior was; the excerpts below show what those behaviors looked like in individual trajectories.

K.2 Pooled Behavioral Changes

The outcome-aware judge credited MIRA-AC’s outer-policy decisions with the larger contribution in 203 pairs and untrained MIRA’s in 131; 66 pairs were tied or unclear. MIRA-AC therefore received 60.8% of the 334 decisive attributions. It was also more often classified as taking a defensible risk that improved the gold result (210/400 trajectories, 52.5%, versus 139/400, 34.8%).

The remaining rows of Fig. 9 describe what changed in the research process. Relative to untrained MIRA, MIRA-AC more often changed its explanation of the underlying mechanism or the form of its solution (185/400 vs. 148/400), ran a test that distinguished competing ideas (112/400 vs. 67/400), checked whether an improvement held across conditions or edge cases (28/400 vs. 10/400), and made an evidence-supported decision about whether to continue (53/400 vs. 29/400). It less often repeated minor variants that yielded little new information (75/400 vs. 134/400).

K.3 Selecting and Attributing Trajectory Examples

For each environment, the examples include at least one case in which MIRA-AC produced the better gold result and one in which it performed worse than untrained MIRA. We select cases only after computing the aggregate statistics across all trajectories, favoring cases with a clear chain from the available evidence to an outer decision, an executor result, and the gold outcome. Additional cases are included only when they clarify a distinct behavior or the boundary between policy and executor credit. The examples are thus illustrative rather than estimates of how common each behavior was, and every quotation is linked to its original decision or result record.

A case in which training hurt performance is included only when it reveals an interpretable decision tendency consistent with the training feedback or data distribution, not merely because MIRA-AC lost the pair. Its discussion reconstructs why the choice was reasonable given the information available at the time, explains why the proxy evidence favored a decision that later performed poorly on the gold evaluation, and states any suggested remedy as a testable hypothesis rather than an established correction.

A useful idea reported by the executor can still have been elicited by the outer loop. We distinguish five relationships between the outer decision and the eventual idea:

1. 1.

   Direct outer specification. The outer work order names the useful hypothesis, test, or planned response before execution.

2. 2.

   Outer-seeded executor discovery. The outer policy isolates the relevant uncertainty or asks for a test likely to resolve it. The executor then supplies the concrete mechanism or implementation.

3. 3.

   State scaffolding. Earlier outer decisions create or preserve a useful best candidate or body of evidence, but do not point the executor toward the eventual idea.

4. 4.

   Executor-independent pivot. The executor introduces a useful direction outside the requested experiment and its stated contingencies.

5. 5.

   Harness-enabled opportunity. Forced continuation or retry makes later work possible but contributes no research hypothesis. Any later outer decision or executor discovery is credited separately.

These categories describe an idea’s provenance, not whether it was good. We require a specific link in content between a work order and a later discovery. An executor idea is not outer-seeded merely because the work order came first, requested a progress update, or was followed by an iteration forced by the harness.

K.4 Symbolic Regression

In symbolic regression, productive research means identifying which variables and mathematical form explain the data, testing that explanation where it is most likely to fail, and resisting improvements that depend on a feature specific to the proxy validation set. Across the 100 matched tasks, MIRA-AC obtained the higher gold score in 58 cases, compared with 32 for untrained MIRA (10 ties). The LLM trajectory judge more often found that MIRA-AC ran a test designed to distinguish two explanations (14% vs. 6%), checked robustness or an edge case (12% vs. 4%), and made a well-supported decision to stop or continue (25% vs. 11%). It also found fewer premature stops (13% vs. 29%) and fewer stretches of repeated minor variants that added little information (3% vs. 13%). These rates count whether a behavior appeared at least once in a trajectory; the matched cases below illustrate how the same tendency to act on specific evidence could help or hurt.

Testing a specific error reveals a better equation.

In MatSci25 (pair symbolic:047), the task asks for a compact equation for stress as a function of strain ϵ and temperature T. The proxy fit suggested that the coefficient relating stress to strain varied with temperature, but its worst relative errors remained too large. MIRA-AC made that error the target of the next experiment:

Outer policy (B-D002): “Scan alternative parametric forms for the temperature-dependent modulus E⁡(T) in Stress ≈ϵ​E​(T) to reduce max relative error and potentially achieve acc_0_1.”

Executor (B-R002): “epsilon and T are perfectly correlated (r≈1.0) in the data, so fitting ϵp​E​(T) with p≠1 reduces MRE substantially.”

The resulting expression reduced the maximum relative error from 0.140 to 0.052. MIRA-AC scored 0.9949 on the gold evaluation; untrained MIRA scored 0. This was an outer-seeded executor discovery: the outer policy selected the relationship to inspect and the error criterion, which exposed the ϵ–T correlation, while the executor introduced the decisive generalization from ϵ to ϵp. The result therefore supports credit for the policy’s choice of question, not sole credit for the formula.

Optimizing proxy error preserves a model that does not extrapolate.

In the CRK25 reaction-kinetics task (pair symbolic:020), the target depends on time t and concentration A. Untrained MIRA tested whether time added information after conditioning on concentration; its executor reported “Pearson(residual,t)=0.0145, confirming t carries no independent predictive power once A is conditioned on” (A-R001), then refined equations using concentration alone until it passed the proxy relative-error threshold. MIRA-AC instead found a more elaborate candidate with the best proxy score, but that expression used a boundary value estimated from the observed data. Its own work order identified the danger:

Outer policy (B-D005): “The incumbent F34 expression […] achieves acc_0_1=True with max_re=0.052 and NMSE=0.000689, but has 5+ free parameters plus a hardcoded A_max from data maximum — raising OOD generalization risk.”

Executor (B-R005): “Discovered Family 38 (F38) — combining rational correction with A-correction in exponent — achieving a 1133× NMSE improvement.”

The proxy gain was substantial: validation NMSE fell from 6.89×10−4 to 6.08×10−7 while satisfying the relative-error criterion. Retaining the model was therefore reasonable given the available metrics, despite the known extrapolation risk. The gold result reversed the proxy ordering: untrained MIRA scored 0.9998 and MIRA-AC scored 0. The executor fitted and compared the candidate equations, but the outer policy directly set the criterion for choosing the final equation and selected the formula after naming its weakness. Forced continuation preserved later opportunities to work, but supplied neither the candidate equation nor the selection criterion.

Interpretation. Training did not simply make the model change the mathematical form of its answer more often: the judge attributed such a change to the outer policy, or jointly to the outer policy and executor, in 27% of MIRA-AC trajectories and 36% of untrained MIRA trajectories, but those trajectories produced the higher gold score 66.7% and 47.2% of the time, respectively. The two cases show when this tendency helps and when it hurts. A test tied to a systematic pattern in the errors can reveal a better equation; the same pressure to improve a proxy error can favor an expression that uses a constant fitted at the edge of the observed data and may not extrapolate. One testable remedy is to evaluate expressions containing such constants on held-out examples near and beyond the range seen during fitting.

Forced continuation appeared in 57 MIRA-AC trajectories and 17 untrained MIRA trajectories in this environment. The attribution analysis excludes harness-only contributions, but forced continuation gave the policies unequal opportunities to make later decisions. This weakens a causal interpretation of the aggregate difference.

K.5 Physics Discovery

Physics Discovery gives the agent a limited budget for simulator experiments and then evaluates its submitted law on hidden trajectories. Productive research therefore uses an experiment to distinguish physical explanations or test conditions in which they disagree; repeatedly fitting the same acquired trajectories need not produce new evidence. Across 100 matched tasks, MIRA-AC obtained the higher gold score in 65 cases and untrained MIRA in 26 (nine ties). The LLM judge more often observed MIRA-AC revising its mechanism (51% vs. 22%), running a discriminating experiment (28% vs. 15%), and testing an edge case (13% vs. 4%). Repeated minor variations of an unproductive idea fell from 51% to 23%, and premature stopping from 27% to 6%. Perfect gold scores rose only from eight to ten; much of the outcome gain was better partial recovery of the hidden law.

Varying one parameter separates two candidate laws.

In pair physics:012, both trajectories inferred a plausible distance-dependent interaction, but the dependence on the second particle’s parameter remained unresolved. Untrained MIRA stopped after five research iterations even though its executor reported that “14 experiment rounds and unlimited MSE evaluations remain for further improvement” (A-R002). MIRA-AC eventually isolated the ambiguity:

Outer policy (B-D013): “The cheapest discriminating action is one experiment round varying p2 independently.”

Executor (B-R014): “Discovered the correct charge model: p1/p2 (not p1/p2 as previously believed).”

MIRA-AC then preserved its best prior candidate when later changes to the numerical integration failed to improve the fit. It received a gold score of 1.000, compared with 0.005 for untrained MIRA. The outer policy selected the controlled parameter variation and, in the following work order, made the competing coupling laws explicit. The executor ran the experiment and identified the exact relation. This is not an independent executor pivot. A forced continuation preserved the opportunity to run the following iteration, but the preceding policy decision had already selected the informative parameter variation.

After finding the law, further exploration chases a proxy metric.

Pair physics:001 shows the same willingness to act on uncertainty after the model had largely resolved which physical law applied. MIRA-AC’s executor first reported, “Discovered the force law: F = -p1*p2 * r_hat / r²” (A-R002). The outer policy nevertheless redirected the remaining experiment budget:

Outer policy (A-D007): “pivot directly to acquiring diverse trajectories to improve trajectory_shaping_quality.”

Executor (A-R007): “0.006 gain over 6 rounds.”

Untrained MIRA instead instructed its executor to “probe distance scaling, sign reversal, and magnitude coupling” (B-D002), validated the inverse-square law, and scored 1.000 on the gold evaluation. MIRA-AC exhausted all 15 experiment rounds and scored 0.00085. Acquiring broader trajectories could reasonably have exposed a hidden failure and did improve the visible trajectory_shaping_quality score, but the new trajectories no longer distinguished between competing physical laws. The costly pivot was directly specified by the outer policy; the executor implemented it. Forced continuation made later work possible but supplied no scientific direction.

Interpretation. The principal change in Physics Discovery was not simply that MIRA-AC did more work. It was more willing to run a new experiment when uncertainty remained. That tendency helped when the uncertainty distinguished physical laws and hurt when the policy continued optimizing a proxy metric after competing physical laws were no longer in question. A policy trained to seek incremental improvements in proxy score could learn this preference, although the matched trajectories alone cannot establish that explanation. A direct follow-up is to require each costly experiment to state which competing physical laws it is meant to distinguish, or stop collecting data solely to improve the proxy metric after several rounds with negligible gains.

K.6 HillClimbBench-CPU: Larger Architectural Changes

HillClimbBench-CPU asks the agent to improve a neural architecture under a fixed CPU training budget. Productive research must therefore do more than change hyperparameters: it must identify a computational bottleneck, propose an architecture that addresses it, and distinguish a genuine improvement from a proxy-evaluation shortcut. Across 100 matched pairs, MIRA-AC achieved the higher gold score in 55, compared with 45 for untrained MIRA. Its trajectories were more often judged to change the model’s computation rather than only its settings (73% vs. 60%), or to take a productive risk (66% vs. 55%), and less often remained in an unproductive sequence of minor variants after they stopped improving (23% vs. 41%). The shift was not uniformly beneficial: continuing despite contrary evidence was also more common (26% vs. 10%). When the judge examined the research process without seeing the final scores, it preferred untrained MIRA in 53 pairs and MIRA-AC in 45 (two ties). MIRA-AC’s gold-score advantage therefore came from decisions with larger effects on the final outcomes, not from a uniformly better research process.

An architecture change that matched the task.

In the scaled-Cayley forgetting task, the model had to retain a Cayley-rotation structure while extrapolating to longer horizons and larger amplitudes. After the current architecture reached an extrapolation ceiling, MIRA-AC’s outer policy identified the restrictive component: “Allowing scale > 1 enables growth modes while preserving the Cayley rotation structure” (B-D006). The executor implemented a clamped exponential scale and reported, “Score: 0.5710 (delta +0.4008 vs incumbent 0.1703),” with extrapolation rising from 0.072 to 0.615 (B-R006). The final gold score was 0.683, compared with 0.195 for the matched untrained trajectory, which continued to explore nearby direct-mapping variants.

The ability to model growth was specified directly by the outer policy. A later executor report introduced a diagonal preconditioner, which scales different internal modes by different amounts, but only after the outer policy had focused the search on scaling those modes; we therefore treat that detail as an outer-seeded executor discovery rather than an independent pivot. The executor supplied part of the implementation, while the outer loop had already identified the model’s inability to scale different internal modes as the bottleneck.

The same willingness to change the architecture can follow the wrong signal.

The Universal Transformer task evaluates a model that repeatedly applies the same block and learns when to stop while solving cellular automata. MIRA-AC instead ordered the executor to “Switch to a direct local-neighbor transition” (B-D001). This was plausible given the available evidence: cellular-automaton updates are local, and successive refinements raised the proxy score to 0.985. The gold evaluator nevertheless assigned the final candidate a score of 0 after marking it invalid. Untrained MIRA preserved the benchmark’s intended design—“Single weight-tied attention+MLP block reused across steps” (A-D001)—and scored 0.809 on the gold evaluation.

Both architectures were specified directly by their respective outer policies; the contrast is not explained by an independent executor discovery or a harness-provided hypothesis. The trained policy made a meaningful architectural bet, but the proxy signal rewarded a solution tailored to the proxy evaluator that did not survive the gold evaluation. The zero alone does not show whether the candidate violated the intended architecture or merely failed because of an implementation error. Because the actor was trained to seek incremental improvements in proxy score, selecting a rapidly improving proxy candidate was aligned with its training signal even though that candidate failed gold evaluation. One useful safeguard would require architecture-changing work orders to state which intended architectural constraint they preserve and to pass basic validity tests across input shapes before being selected as the best candidate. Whether this retains the gains from architecture search while filtering proxy shortcuts remains to be tested.

Together, the two cases clarify the aggregate shift. MIRA-AC more often pivoted from repeated minor variants to changes in the computation itself. That behavior created value when the proposed architecture addressed the true bottleneck, but produced a larger downside when the proxy evaluator rewarded an easier shortcut.

K.7 BNLearn: Resolving Remaining Graph Uncertainty

BNLearn asks the agent to recover a causal directed acyclic graph from observational data and a limited intervention budget. An intervention fixes one variable to a chosen value and reveals how other variables respond. Useful exploration is usually conditional: choose an intervention that separates a few remaining graph hypotheses, or search a small, explicitly enumerated set of edge edits after localizing the remaining errors. We summarize those errors using structural Hamming distance (SHD), the number of missing, extra, or reversed edges. Because many trajectories reach the score ceiling, different research paths often have the same final score. The outcome-aware judge credited MIRA-AC with decisions that contributed more to its own final outcome in 25 of 100 pairs and untrained MIRA in 22, with 53 ties. MIRA-AC trajectories more often contained a test aimed at resolving uncertainty about a specific edge (55% vs. 34%) and a productive risk (61% vs. 40%), but less often preserved the strongest current graph while deciding what to try next (19% vs. 32%). BNLearn was held out from actor training, although it was included in critic pretraining; these differences therefore reflect behavior transferred from other training environments.

Targeted interventions fix the remaining graph errors.

On bnlearn:004, MIRA-AC reduced its graph to one missing, one extra, and one reversed edge. Its outer policy then selected two interventions: “Intervene on X0” and “Intervene on X16” (A-D003). The executor reported, “Fixed all 3 SHD errors using 2 strategic interventions (X0, X16)” (A-R003), identifying the precise reversal, deletion, and addition implied by the new evidence about which variables were affected. MIRA-AC achieved a gold score of 1.0; untrained MIRA repeatedly launched related combinatorial searches after visible wall-time failures and finished at 0.833.

The outer policy receives direct credit for selecting the two informative interventions and the narrowly defined correction problem. The executor receives credit for identifying the exact three edge edits; because those edits followed from the requested interventions, this is an outer-seeded executor discovery rather than an executor-independent pivot. The useful behavior was not exploration in the abstract, but spending scarce interventions on observations that could change a small set of decisions.

Broad search loses to a one-edge repair.

The reverse pattern appears on bnlearn:057. Untrained MIRA used two interventions to raise its score for recovering which variables affect one another from 0.935 to 0.982, then issued the narrow work order “close the SHD=1 gap by identifying and adding the single missing edge” (B-D009). The executor compared the effects implied by the current graph with the variables affected by each intervention and reported, “Identified and added the single missing edge X15-¿X14 to close the SHD=1 gap” (B-R009). Untrained MIRA therefore reached a gold score of 1.0. MIRA-AC’s reconstruction based on its interventions made substantial progress, but its early searches incurred wall-time failures and later graph-learning methods and exhaustive edge-swap searches concluded that the current SHD=7 graph was “the best achievable” (A-R011). It finished at 0.844.

Untrained MIRA’s outer policy directly asked the executor to find the one missing edge; the executor supplied the exact edge, making the solution an outer-seeded executor discovery. MIRA-AC’s broader search was reasonable after several methods agreed on the same best graph, but it failed to turn the available evidence into a sufficiently narrow correction. This is consistent with a preference learned in other environments for gathering new evidence, even in an environment where many near-ceiling solutions require only a small repair to the remaining graph errors. A late-stage work order could instead test the smallest set of edge edits consistent with the intervention evidence before rebuilding the graph or expanding to a broad search.

BNLearn therefore shows both sides of the same tendency. Targeted experiments can close the last graph errors efficiently, but exploration that begins before the remaining errors are identified can spend the remaining budget without improving the graph. The large number of tied outcomes also means that many differences in behavior do not appear in final-score comparisons.

K.8 Cross-Environment Interpretation

Across the four environments, training did not simply increase the amount of exploration. The trained outer policy more often acted on unresolved uncertainty: run a test that could separate competing explanations, change the form of the solution when the current class of solutions stopped improving, or abandon a line of minor variants that had stopped yielding new information. In Physics Discovery and HillClimbBench-CPU this often produced larger changes to the proposed physical law or neural architecture. In symbolic regression and BNLearn it more often appeared as a controlled test or a focused attempt to resolve the remaining errors.

The same shift explains the observed degradations. Acting on uncertainty was valuable when resolving that uncertainty improved the gold evaluation, but costly when a proxy metric rewarded a shortcut, a constant fitted to the edge of the observed data, or additional data collection after the central question was already resolved. The trained policy therefore made the model’s research decisions more likely to change the final result, for better or worse, rather than uniformly more rigorous. The paired examples suggest three concrete directions for future evaluation: require costly experiments to name the hypotheses they separate, test architecture and expression changes on deliberately shifted or out-of-distribution cases before selecting them as the final answer, and prefer the smallest set of possible edits once only a few uncertainties remain.

注意转载授权 · 阅读原文