腾讯混元与港科大（广州）提出 Poseidon 流水线，从科学代码库构建 97,659 个智能体训练环境

Qifan Zhang^1,2*, Yuhan Li^2†, Nuo Chen^2‡, Bowen Liu^2*, Xiao Hu², Miao Peng^1,2
Chenyi Zi¹, Lixin Liu², Lydell Huang², Jing Tang¹, Jia Li^1†
1The Hong Kong University of Science and Technology (Guangzhou)
²Hunyuan Team, Tencent

Abstract

Training environments remain a major bottleneck for agents in data-driven scientific discovery. Existing collections of executable and verifiable training environments are small, and their construction remains difficult to scale. To address this limitation, we introduce Poseidon, an automated pipeline for generating diverse, verifiable training environments from scientific code repositories at scale. From the same repository budget, Poseidon extracts nearly 10× as many training environments as a prior approach by decomposing scientific workflows into four complementary task types: Reproduction, Subflow, Completion, and Correction, where each type targets a distinct capability for scientific research. A recursive improvement loop further refines failed candidates and recovers otherwise discarded environments. Using Poseidon, we construct Pearl, a collection of 97,659 executable and verifiable environments from 7,110 public repositories spanning 10 scientific disciplines. To the best of our knowledge, Pearl is the largest and most broadly multidisciplinary collection of scientific-agent training environments to date. Qwen3.5 models (9B, 27B, and 35B-A3B) trained on Pearl achieve consistent gains across six challenging scientific-agent benchmarks. Overall, the scale and diversity of Pearl offer opportunities for training scientific discovery agents.

1 Introduction

The rapid growth of scientific data and the increasing use of large language model agents in research have sparked a surge of interest in data-driven scientific discovery (Xin et al., 2025; Gridach et al., 2025; Anthropic, 2026b; Majumder et al., 2024; Liu et al., 2026). Unlike knowledge-based question answering or isolated code generation, scientific research requires agents to carry out scientific workflows: coordinated sequences of data processing and computational steps that address a research objective. Agents must clean data, select appropriate models, write and debug programs, and iteratively revise their analyses based on intermediate results before reaching evidence-supported conclusions (Schmidgall et al., 2025; Zhou et al., 2025) (as depicted in Figure 2). The capability to carry out these scientific workflows requires integrating scientific reasoning with coding and tool use, and remains difficult for even current frontier models to master. For example, on OpenAI’s GeneBench-Pro (Li et al., 2026), GPT-5.6 Sol (OpenAI, 2026b) achieves a pass rate of only 28.7% at the highest reasoning level (or 31.5% with Pro mode enabled), highlighting the immense difficulty of reliably automating complex scientific analyses. Yet, training environments designed to develop this capability remain severely limited.

D3-Gym (Moussa et al., 2026) pioneered the automated synthesis of such training environments from scientific code repositories. These repositories are valuable because they encode the scientific workflows described above in related sets of code files that connect research objectives, input data, and computational procedures. Executing these workflows provides reference outputs for verification. However, scaling this approach remains challenging. D3-Gym converts each extracted workflow into an end-to-end task, leaving more varied supervision within the workflow underused. Its collection contains only 565 environments from 239 repositories (approximately 2.4 environments per repository). This limited yield raises a central research question:

How can we efficiently exploit executable scientific workflows within code repositories to generate high-quality, diverse training environments at scale?

To answer this, we introduce POSEIDON, an automated pipeline for mining scientific code repositories through three stages: source discovery, task synthesis, and verification. Source discovery identifies repositories containing scientific code, data, and research context through domain-keyword expansion. Task synthesis extracts scientific workflows and derives four complementary task types, each corresponding to a distinct capability for scientific agents: Reproduction, reproducing the results of a complete scientific workflow from its research objective; Subflow, carrying out a standalone analysis to answer a focused scientific question motivated by a broader research context; Completion, filling in a missing function in the provided workflow code; and Correction, identifying and fixing an error in provided code that produces incorrect results. Our key insight is that a scientific workflow can provide not only an end-to-end research task, but also richer supervision through structured modifications to its constituent steps. Under a matched budget of 306 repositories, POSEIDON produces 5,370 environments, 9.3× as many as the 575 produced by D3-Gym, as shown in the left panel of Figure 1.

In the verification stage, we introduce a recursive improvement mechanism driven by an iterative generator–validator loop. If a generated environment fails the initial reliability and scientific-validity checks, the validator returns diagnostic feedback and revision suggestions, after which the generator revises and resubmits the environment. Among the environments rejected in the first verification round, 49.4% are successfully recovered and pass verification through this refinement process. This closed-loop refinement recovers flawed but fixable candidates that would otherwise be discarded, thereby improving environment quality while increasing data yield.

Using POSEIDON, we construct PEARL, a large-scale collection of verified scientific-agent training environments sourced from 7,110 repositories across 10 scientific disciplines. PEARL contains 97,659 environments, each equipped with a clear task description, input data, an executable sandbox, a reference solution, and an automatic evaluation script. To the best of our knowledge, PEARL is the largest and most broadly multidisciplinary collection of training environments for scientific discovery agents to date. We evaluate PEARL through supervised fine-tuning on Qwen3.5-9B, 27B, and 35B-A3B (Qwen Team, 2026a), achieving notable improvements across six highly challenging scientific-agent benchmarks. Furthermore, controlled ablations show that both broader repository coverage and complementary workflow-derived task types contribute to downstream performance. Overall, the scale and diversity of PEARL open up new opportunities for training scientific discovery agents.

2 Related Works

Scalable Task Synthesis from Code Repositories.

Executable code repositories have been extensively leveraged as rich sources of training supervision. Prior work derives completion tasks by masking code at multiple granularities or isolating target functions and their dependencies (Zhang et al., 2023; Yang et al., 2024a; Xie et al., 2025), and converts naturally occurring code fragments into executable specifications and test cases (Xie et al., 2024). For software repair, scalable pipelines synthesize verifiable tasks by injecting test-breaking defects via language-model rewrites and procedural mutations, or by simulating repository-level debugging workflows (Yang et al., 2026; Pham et al., 2025). Other methods reconstruct executable issue-resolution tasks from repository histories through test generation, back-translation, patch reversal, and code-state reconstruction (Jain et al., 2025; Qi et al., 2026). More recently, capability-targeted tasks, including feature development, bug fixing, function localization, and dependency search, have been derived from shared repository contexts (Fu et al., 2026; Xie et al., 2026). Together, these studies demonstrate that transforming and restructuring existing code artifacts can substantially increase both the quantity and diversity of supervision extracted from a single repository. Inspired by this line of work, we decompose and transform executable scientific workflows into four complementary task types, thereby scaling the generation of training environments for data-driven scientific discovery.

Automated Training Environment Synthesis for Scientific Discovery.

With the rapid growth of scientific data and the increasing capabilities of LLM agents, data-driven scientific discovery has attracted growing attention, spanning scientific coding, computational reproducibility, and autonomous data analysis (Li et al., 2026; Anthropic, 2026b; Mitchener et al., 2025; Chen et al., 2025; Liu et al., 2026). However, large-scale executable environments for training agents to conduct such workflows remain scarce. D3-Gym (Moussa et al., 2026) constructs verifiable training environments from scientific repositories by treating extracted scientific workflows as end-to-end tasks. This approach produces relatively few environments per repository, making it challenging to scale. In contrast, we derive multiple complementary tasks from each executable scientific workflow, providing richer supervision beyond end-to-end execution and enabling the generation of diverse, verifiable training environments at scale.

3 Method

In this work, we aim to enhance the problem-solving capabilities of scientific agents by increasing the scale and diversity of training environments. To achieve this, we introduce POSEIDON, an automated synthesis pipeline, and use it to construct PEARL, a large-scale, verified collection of such environments. We begin by formally defining the scientific agent task, and then detail the three stages of the POSEIDON pipeline: repository discovery, task synthesis, and verification.

3.1 Problem Formulation

SciAgent.

We study scientific agents (SciAgents). Given a task description, an input data, an executable sandbox, and a code interpreter tool, the agent must denoise the data, choose suitable models, write code, analyse execution results, and produce a final answer. We formulate this process as a sequential decision problem. An agent policy πθ performs interleaved reasoning and tool use. At step t, the agent generates a thought rt and an action at based on the history; the environment executes at and returns an observation ot. This yields a trajectory τ=(r1,a1,o1,…,rT,aT,oT). Figure 2 illustrates a representative task together with the corresponding SciAgent workflow.

Environment Synthesis Objectives.

To support reliable and effective agent training, SciAgent training environment should satisfy four properties: (1) Data-Dependent: Each environment must be grounded in at least one input dataset that is necessary for solving the scientific problem. (2) Executable: Each environment must provide a task-specific sandbox containing the dependencies required by the agent, ensuring at least one executable solution path. (3) Verifiable: Each environment must include an evaluation script that determines whether the agent’s output is correct. (4) Diverse: The environments should span a broad range of source repositories and scientific disciplines, and cover diverse task types.

Formally, we represent a SciAgent training environment as

where Ti is the task description, Di is the input dataset, Si is the executable sandbox, and Vi is the evaluation script.

To satisfy these objectives at scale, we select GitHub¹¹ 1 https://github.com as our primary data source. GitHub hosts millions of open-source repositories that naturally bundle scientific code, environment configurations, and associated data. This rich ecosystem provides an ideal foundation for extracting diverse, data-dependent, and executable workflows. Leveraging this resource, we design an automated pipeline to systematically transform these repositories into agent training environment.

3.2 Environment Generation Pipeline

The Poseidon pipeline turns public scientific repositories into training environment. It has three stages: repository discovery, task synthesis, and verification.

3.2.1 Repository Discovery

The scale and diversity of synthesized scientific tasks depend directly on the breadth and quality of their source repositories. We therefore first construct a large and diverse repository pool and build an executable sandbox for each repository, providing a reliable foundation for subsequent task synthesis (Panel (a) of Figure 3).

Keyword Pool Generation.

We begin with a small set of domain descriptions, such as structural biology and molecular dynamics or cheminformatics and drug discovery. For each domain, we prompt an LLM to propose concrete software, libraries, file formats, and database names typically mentioned in the README files of related repositories, such as GROMACS, MDAnalysis, and PLUMED for molecular dynamics. Each candidate term is then executed as a GitHub search query. We discard terms with fewer than 30 hits, as they indicate limited public code availability, and terms with more than 5,000 hits, as they are often too generic (e.g., docker or hdf5). The retained terms constitute the keyword pool.

Repository Pool Collection.

We query GitHub using the keyword pool and retain a repository only if it meets four criteria: it has at least 20 stars, contains scientific data, is backed by a published paper or preprint, and belongs to a core scientific discipline (e.g., mathematics, physics, chemistry, biology, or pharmacy). The retained repositories form the repository pool.

Executable Sandbox Construction.

For each repository, we construct an isolated sandbox whose dependencies are inferred both by statically analyzing the repository’s source code and by parsing its declared dependency manifests. This sandbox is preserved to ensure reproducible execution during subsequent seed extraction and task generation.

In this stage, we obtain 7,110 repositories. Their detailed distribution is reported in Figure 4. Given the particularly active development of scientific agents in biology and biomedicine (Mitchener et al., 2025; Li et al., 2026; Anthropic, 2026b), we intentionally oversample biological repositories to better cover this important application domain. Prompts for keyword expansion, scientific-file detection, and scientific-field judgment are provided in Appendix E.2.

3.2.2 Task Synthesis

Given the repository pool, we develop a seed-program-centric method (Panel (b) of Figure 3) to deeply mine each repository and synthesize scientific tasks from its executable workflows. This method operates in two steps: we first extract executable seed programs from the raw repositories, and then derive diverse tasks from their underlying scientific workflows.

Seed Program Extraction.

For each repository, we use DeepSeek-V4-Flash (DeepSeek-AI, 2026) to group related code files into one or more cohesive sets, each representing a complete scientific workflow (e.g. data denoise, computational analysis, and result generation). To standardize execution, DeepSeek-V4-Flash consolidates and rewrites each file set into a single entry-point code script, strictly preserving the original scientific logic. We then execute every candidate script within its sandbox; those that fail to run or produce errors are discarded, while successfully executed scripts are retained as seed programs.

Task Generation.

Given a seed program, we use DeepSeek-V4-Flash to extract its underlying scientific workflow and represent it as a directed graph, where nodes denote data artifacts or computational operations and edges encode data dependencies. From this graph, we derive four complementary task types that target important capabilities in scientific discovery:

• •

  Reproduction is constructed based on the inputs and outputs of the full workflow. It requires reproducing the entire analysis, targeting end-to-end workflow planning and execution.

• •

  Subflow is constructed by extracting a specific part of the workflow. It requires executing this selected connected subgraph, targeting local dependency understanding and modular execution.

• •

  Completion is constructed by removing a functional block from the complete code. It requires reconstructing the missing computational step from its surrounding context and dependencies, targeting dependency-aware workflow completion.

• •

  Correction is constructed by injecting a controlled error into the program. It requires correcting this executable but scientifically invalid operation, targeting scientific error diagnosis and repair.

The construction logic for each task type is illustrated in the lower portion of Figure 3(b). For each task, we isolate the relevant segment of the seed program to construct a task-specific reference program. Executing this program yields the reference answer, which we then use to automatically generate an evaluation script tailored to the answer’s format. Examples of each task type are provided in Appendix E.1.

3.2.3 Verification

We apply a multi-stage quality-verification pipeline (Panel (c) of Figure 3) to assess two complementary properties of each environment: environment reliability, which checks that the environment is complete, executable, reproducible, and reliably gradable; and scientific validity, which checks that the task is scientifically meaningful, correctly implemented, and internally consistent. Detailed criteria for both properties are provided in Appendix B.

Each candidate environment is first evaluated by a GPT-5.5 validator according to these criteria. To recover valuable environments that initially fail verification, we introduce recursive improvement. The validator returns its error diagnosis and revision suggestions to the DeepSeek-V4-Flash generator, which revises the environment before submitting it to the validator again. This feedback loop between the generator and validator runs for up to three rounds, and an environment is retained if it passes validation within this budget. Finally, the validator assigns each retained environment a difficulty score from 1 to 4 based on the required domain knowledge and the depth of the analysis.

To further assess the quality of the automatically verified environments, we randomly sample 600 tasks for independent manual review by three PhD researchers. A task is accepted only if all three reviewers approve it, yielding an acceptance rate of 95.5% (Details are in Appendix C. After automated filtering, Pearl contains 97,659 training environments. Their distributions across difficulty levels and task types are reported in Appendix A. The prompts used for quality verification, environment revision, and difficulty annotation are provided in Appendix E.2.

4 Experiment

4.1 Experimental Setup

Agent Framework.

We build our agent framework upon mini-swe-agent (Yang et al., 2024b). Each task runs in an isolated sandbox with external network access disabled, and the agent interacts with the environment exclusively through a command-line interface. We impose a wall-clock time limit of 10,800 seconds per task and an execution timeout of 1,800 seconds per command, with a maximum of 100 agent iterations per task.

Training Details.

We use Qwen3.5-9B, Qwen3.5-27B, and Qwen3.5-35B-A3B as our backbone models. Using DeepSeek-V4-Pro as the teacher model, we collect 64,477 successful trajectories through rejection sampling from a pool of 97,659 training environments. To reduce benchmark contamination, before trajectory collection we removed environments whose source repositories matched, or were near-duplicates of, repositories underlying the evaluation benchmarks. We fine-tune the backbone models with a learning rate of 5.0×10^-6, a batch size of 128, a maximum context length of 65,536 tokens, and a limit of 50 agent iterations per trajectory.

Evaluation Benchmarks.

To thoroughly evaluate the scientific discovery capabilities of models trained on Pearl and other baselines, we assemble a suite of 6 scientific discovery benchmarks spanning ten scientific disciplines. Three of these are well-specified benchmarks with objectively verifiable outcomes: SciCode (Tian et al., 2024), BioCoder (Tang et al., 2024), and MedAgentBench (Jiang et al., 2025). The remaining three benchmarks, namely BixBench (Mitchener et al., 2025), BioMysteryBench (Anthropic, 2026b), and ScienceAgentBench (Chen et al., 2025), focus on open-ended scientific analyses. To ensure a fair and reliable evaluation, we exclude problems that are ill-posed or require languages outside our harness. Table 1 summarizes the scale, disciplinary coverage, task format, and evaluation protocol of each benchmark. Detailed exclusion rules and further benchmark descriptions are provided in Appendix D.

Models.

We evaluate two groups of models. The first group comprises frontier models, including Claude-Opus-4.8 (Anthropic, 2026a), Gemini-3.6-Flash (Google DeepMind, 2026), GPT-5.5 (OpenAI, 2026a), Qwen3.8-Max (Qwen Team, 2026b), DeepSeek-V4-Pro (Xu et al., 2026), GLM-5.2 (Zeng et al., 2026), and Hy4-Preview (Tencent Hy Team, 2026). The second group comprises baseline models: the Qwen3.5 family (Qwen Team, 2026a) (9B, 27B, and 35B-A3B). For all models, we set the temperature to 0.7, the maximum number of generated tokens to 32,768, and the reasoning effort to high.

Model	BioCoder	SciCode		MA-Bench	SA-Bench	BixBench		BioMystery	Avg
		Main	Sub			MCQ	Open
Frontier Models
Claude-Opus-4.8	71.4	20.0	53.5	84.9	54.1	72.2	50.2	65.1	58.9
Gemini-3.6-Flash	70.8	20.0	59.0	92.3	55.9	73.7	47.8	56.7	59.5
GPT-5.5	75.3	20.0	59.0	94.3	52.9	74.1	49.8	55.6	60.1
Qwen3.8-Max	77.9	23.1	57.6	86.3	64.7	77.6	45.9	64.4	62.2
DeepSeek-V4-Pro	70.8	24.6	59.7	85.0	57.8	72.7	47.8	64.4	60.4
GLM-5.2	67.5	18.5	29.2	88.3	58.8	69.3	39.5	64.4	54.4
Hy4-Preview	80.5	18.5	54.9	69.0	61.8	72.2	49.8	44.4	56.4
9B Models
Qwen3.5-9B	46.8	3.1	30.2	59.0	26.5	48.3	35.1	16.7	33.2
+ Pearl	51.9 ↑5.1	9.2 ↑6.1	34.0 ↑3.8	63.0 ↑4.0	33.3 ↑6.8	50.7 ↑2.4	37.6 ↑2.5	20.0 ↑3.3	37.5 ↑4.3
A3B Models
Qwen3.5-35B-A3B	54.5	13.8	37.5	71.3	38.2	57.1	32.2	27.8	41.6
+ Pearl	60.4 ↑5.9	20.0 ↑6.2	45.1 ↑7.6	78.3 ↑7.0	47.1 ↑8.9	69.3 ↑12.2	37.1 ↑4.9	28.9 ↑1.1	48.3 ↑6.7
27B Models
Qwen3.5-27B	57.1	12.3	42.7	75.7	34.3	61.5	34.1	36.7	44.3
+ Pearl	67.5 ↑10.4	16.9 ↑4.6	50.3 ↑7.6	79.0 ↑3.3	50.0 ↑15.7	69.8 ↑8.3	39.5 ↑5.4	38.9 ↑2.2	51.5 ↑7.2

4.2 Main Results

We first evaluate the effectiveness of Pearl by comparing Qwen3.5 models of different sizes before and after supervised fine-tuning, together with several frontier models. The results are shown in Table 2. Overall, training on Pearl consistently improves scientific-discovery performance across model scales and scientific domains, covering both well-specified tasks and more challenging open-ended workflows. The gains are particularly pronounced for larger models, suggesting that stronger base models can better exploit the diverse and complex supervision provided by Pearl. Notably, Qwen3.5-27B improves by 10.4 percentage points on BioCoder and by 15.7 percentage points on ScienceAgentBench.

More importantly, Pearl substantially narrows the gap between open models and frontier models. After SFT, Qwen3.5-27B reaches 67.5% on BioCoder, matching GLM-5.2, while Qwen3.5-35B-A3B achieves 20.0% on SciCode (main), matching GPT-5.5. The 27B model also reaches 79.0% on MedAgentBench and 69.8% on BixBench-MCQ, outperforming Hy4-Preview (69.0%) and GLM-5.2 (69.3%), respectively, despite its substantially smaller model scale. These results demonstrate that high-quality scientific-agent training data can yield large gains in scientific-discovery capabilities and enable relatively compact models to approach, or even surpass, frontier models on individual benchmarks.

Comparing Pearl with Other Scientific-Discovery Data.

We next compare Pearl with D3-Gym (Moussa et al., 2026), another method for generating scientific-agent training environments from code repositories. For a more comprehensive comparison, we also include MedAgentGym (Xu et al., 2025), a collection of training environments for biomedical agent. D3-Gym and MedAgentGym provide 2,153 and 2,137 training trajectories, respectively. To compare under a similar data budget, we randomly subsample 2,000 trajectories from Pearl and fine-tune the same Qwen3.5-27B backbone using an identical training recipe (Table 4).

Both baselines yield selective gains but also degrade performance on several benchmarks. MedAgentGym improves four of the eight settings but decreases BixBench (MCQ) by 8.1 percentage points, while D3-Gym improves only three settings and reduces BixBench (Open) by 8.2 points and BioMystery by 7.8 points. These inconsistent transfer patterns are consistent with their limited disciplinary coverage and limited exploitation of the scientific resources from which their training data are constructed. By contrast, Pearl improves seven of the eight settings, with gains ranging from 0.7 to 5.2 percentage points and an overall average improvement of 2.1 points, despite using fewer than 10% of the full training data. The consistent gains at this matched data budget suggest that Pearl’s benefit arises from the diversity of its scientific disciplines, repositories, and workflow-derived tasks, rather than from data scale alone.

Model	# Traj.	BioCoder	SciCode		MA-Bench	SA-Bench	BixBench		BioMystery	Avg
			Main	Sub			MCQ	Open
Qwen3.5-27B	–	57.1	12.3	42.7	75.7	34.3	61.5	34.1	36.7	44.3
+ MedAgentGym	2,137	60.4 ↑3.3	13.8 ↑1.5	42.7	79.0 ↑3.3	36.2 ↑1.9	53.4 ↓8.1	28.4 ↓5.7	32.2 ↓4.5	43.3 ↓1.0
+ D3-Gym	2,153	59.1 ↑2.0	10.8 ↓1.5	38.9 ↓3.8	73.7 ↓2.0	36.2 ↑1.9	62.0 ↑0.5	25.9 ↓8.2	28.9 ↓7.8	41.9 ↓2.4
+ Pearl (2k)	2,000	62.3 ↑5.2	13.8 ↑1.5	43.4 ↑0.7	77.7 ↑2.0	38.1 ↑3.8	63.9 ↑2.4	37.6 ↑3.5	34.4 ↓2.3	46.4 ↑2.1

4.3 Ablation Study

We conduct three ablation studies to examine how source diversity, environment difficulty, and workflow-derived task composition contribute to Pearl’s effectiveness. All ablation models are trained by fine-tuning Qwen3.5-35B-A3B under the same training recipe and evaluated on four representative benchmarks: BioCoder, MedAgentBench, BixBench-MCQ, and ScienceAgentBench.

Scaling Repository Diversity.

To isolate the effect of repository diversity, we fix the total number of training trajectories, the average environment difficulty, and the proportions of the four task types. We then vary the proportion of source repositories from 20% to 100%. The top row of Figure 5 shows that increasing repository diversity consistently improves training effectiveness under a fixed trajectory budget. Across all four benchmarks, performance exhibits an overall upward trend as repository coverage increases, despite minor fluctuations at intermediate points. This effect is particularly clear on MedAgentBench and BixBench (MCQ): at 20% coverage, the trained model performs close to or even below the untrained backbone, whereas broader repository coverage eventually yields a clear improvement over the baseline. These results demonstrate the value of expanding the breadth of scientific sources, even with a fixed amount of training data.

Scaling Environment Difficulty.

We next investigate how environment difficulty relates to task solvability and training effectiveness. We first randomly sample 100 environments from each of the four difficulty levels, D1–D4, and evaluate DeepSeek-V4-Pro on their tasks. As shown in Table 3, the success rate decreases from 94.0% for D1 to 41.0% for D4, while the average solution length increases from 6.2 to 16.4 steps. These results indicate that the annotated difficulty levels are consistent with empirical task difficulty.

We then construct five training subsets, each containing 30,000 trajectories, with average environment difficulty scores of 2.4, 2.6, 2.8, 3.0, and 3.2, respectively. We keep the number of repositories, the total trajectory budget, and the proportions of the four task types fixed across subsets. As shown in the bottom row of Figure 5, downstream performance exhibits no consistent trend as the difficulty of the training environments increases. Overall, environment difficulty correlates with task solvability but does not affect its effectiveness as training data within the evaluated range.

Removing Individual Task Types.

We ablate each of the four workflow-derived task types in turn, yielding four reduced datasets—w/o Reproduction, w/o Subflow, w/o Completion, and w/o Correction— while controlling the total training-data size and source-repository distribution. Table 5 reports the results. Removing any single task type degrades performance on every benchmark, confirming that each contributes non-redundant supervision. This result demonstrates the value of mining multiple complementary forms of supervision from each scientific workflow.

5 Conclusion

We introduced Poseidon, an automated pipeline for generating diverse, verifiable training environments from scientific code repositories at scale. By deriving four complementary task types from executable workflows and refining failed candidates through recursive improvement, Poseidon produces richer supervision from existing scientific code. Using this pipeline, we constructed Pearl, a collection of 97,659 environments from 7,110 repositories across 10 scientific disciplines. Supervised fine-tuning on Pearl improves Qwen3.5 models across six scientific-agent benchmarks. Controlled ablations further show that repository diversity and complementary task types contribute to these gains, supporting the value of extracting supervision beyond end-to-end workflow execution. Together, Poseidon and Pearl provide a practical foundation for scaling training environments and developing more capable scientific discovery agents.

References

• Anthropic Claude opus 4.8 system card. Note: AnthropicPublished May 28, 2026 External Links: Link Cited by: §4.1.
• Anthropic (2026b) Anthropic Evaluating claude’s bioinformatics research capabilities with BioMysteryBench. Note: Anthropic ResearchPublished April 29, 2026; accessed 2026-09-08 External Links: Link Cited by: Appendix D, §1, §2, §3.2.1, §4.1.
• Chen et al. (2025) Z. Chen, S. Chen, Y. Ning, Q. Zhang, B. Wang, B. Yu, Y. Li, Z. Liao, C. Wei, Z. Lu, V. Dey, M. Xue, F. N. Baker, B. Burns, D. Adu-Ampratwum, X. Huang, X. Ning, S. Gao, Y. Su, and H. Sun ScienceAgentBench: toward rigorous assessment of language agents for data-driven scientific discovery. In International Conference on Learning Representations, External Links: Link Cited by: Appendix D, §2, §4.1.
• DeepSeek-AI (2026) DeepSeek-AI DeepSeek-v4: towards highly efficient million-token context intelligence. Cited by: §3.2.2.
• Fu et al. (2026) L. Fu, H. Guan, B. Zhang, H. Yuan, Y. Zhu, L. Qiu, Z. Wang, X. Cao, X. Cai, W. Liu, W. Zhang, and Y. Yu CoreCodeBench: decoupling code intelligence via fine-grained repository-level tasks. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 26043–26067. External Links: Document, Link Cited by: §2.
• Google DeepMind (2026) Google DeepMind Gemini 3.6 flash model card. Note: Google DeepMindPublished July 2026; updated July 21, 2026 External Links: Link Cited by: §4.1.
• Gridach et al. (2025) M. Gridach, J. Nanavati, K. Z. E. Abidine, L. Mendes, and C. Mack Agentic ai for scientific discovery: a survey of progress, challenges, and future directions. arXiv preprint arXiv:2503.08979. Cited by: §1.
• Jain et al. (2025) N. Jain, J. Singh, M. Shetty, T. Zhang, L. Zheng, K. Sen, and I. Stoica R2e-gym: procedural environment generation and hybrid verifiers for scaling open-weights swe agents. In Second Conference on Language Modeling, Cited by: §2.
• Jiang et al. (2025) Y. Jiang, K. C. Black, G. Geng, D. Park, J. Zou, A. Y. Ng, and J. H. Chen MedAgentBench: a virtual ehr environment to benchmark medical llm agents. Nejm Ai 2 (9), pp. AIdbp2500144. Cited by: Appendix D, §4.1.
• Li et al. (2026) J. Li, S. Shringarpure, E. Wong, and A. Ho GeneBench-pro: evaluating multistage statistical reasoning in genomics, quantitative biology, and translational biomedicine. bioRxiv, pp. 2026–06. Cited by: §1, §2, §3.2.1.
• Liu et al. (2026) A. Liu, A. Ho, A. M. Droste, D. Martin, E. Wong, E. Zhou, I. Zhou, J. Park, J. Jiao, K. Skelly, et al. LifeSciBench: evaluating language models on realistic, expert-level tasks in the life sciences. bioRxiv, pp. 2026–08. Cited by: §1, §2.
• Majumder et al. (2024) B. P. Majumder, H. Surana, D. Agarwal, B. Dalvi Mishra, A. Meena, A. Prakhar, T. Vora, T. Khot, A. Sabharwal, and P. Clark DiscoveryBench: towards data-driven discovery with large language models. arXiv preprint arXiv:2407.01725. External Links: 2407.01725, Link Cited by: §1.
• Mitchener et al. (2025) L. Mitchener, J. M. Laurent, A. Andonian, B. Tenmann, S. Narayanan, G. P. Wellawatte, A. White, L. Sani, and S. G. Rodriques Bixbench: a comprehensive benchmark for llm-based agents in computational biology. arXiv preprint arXiv:2503.00096. Cited by: Appendix D, §2, §3.2.1, §4.1.
• Moussa et al. (2026) H. N. Moussa, Y. Li, Z. Li, Y. Yang, C. Tang, T. Zhang, N. K. Ahmed, A. Payani, Z. Chen, and H. Sun D3-gym: constructing real-world verifiable environments for data-driven discovery. arXiv preprint arXiv:2604.27977. Cited by: Appendix F, §1, §2, §4.2.
• OpenAI (2026a) OpenAI GPT-5.5 system card. Note: OpenAI Deployment Safety HubPublished April 23, 2026 External Links: Link Cited by: §4.1.
• OpenAI (2026b) OpenAI GPT-5.6 system card. Note: OpenAI Deployment Safety HubPublished July 9, 2026 External Links: Link Cited by: §1.
• Pham et al. (2025) M. V. Pham, H. N. Phan, H. N. Phan, C. L. Chi, T. N. Nguyen, and N. D. Bui Swe-synth: synthesizing verifiable bug-fix data to enable large language models in resolving real-world bugs. arXiv preprint arXiv:2504.14757. Cited by: §2.
• Qi et al. (2026) H. Qi, X. Wang, X. Gao, B. Sang, X. Zhang, M. Ma, P. Gao, Y. Kang, Q. Lin, S. Rajmohan, et al. Change2Task: from repository changes to executable coding agent tasks and environments. arXiv preprint arXiv:2607.28591. Cited by: §2.
• Qwen Team (2026a) Qwen Team Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §1, §4.1.
• Qwen Team (2026b) Qwen Team Qwen3.8-max: a new bar for coding and cowork. External Links: Link Cited by: §4.1.
• Schmidgall et al. (2025) S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, M. Moor, Z. Liu, and E. Barsoum Agent laboratory: using llm agents as research assistants. Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 5977–6043. Cited by: §1.
• Tang et al. (2024) X. Tang, B. Qian, R. Gao, J. Chen, X. Chen, and M. B. Gerstein BioCoder: a benchmark for bioinformatics code generation with large language models. Bioinformatics 40 (Supplement_1), pp. i266–i276. External Links: Document, Link Cited by: Appendix D, §4.1.
• Tencent Hy Team (2026) Tencent Hy Team Introducing Hy4 Preview. Note: Tencent HyPublished August 28, 2026 External Links: Link Cited by: §4.1.
• Tian et al. (2024) M. Tian, L. Gao, S. D. Zhang, X. Chen, C. Fan, X. Guo, R. Haas, P. Ji, K. Krongchon, Y. Li, et al. Scicode: a research coding benchmark curated by scientists. Advances in Neural Information Processing Systems 37, pp. 30624–30650. Cited by: Appendix D, §4.1.
• Xie et al. (2026) Y. Xie, E. Liu, G. Zhang, N. Kotalwar, S. Gandhi, S. Acharya, X. Wang, C. Rose, G. Neubig, and D. Fried Hybrid-Gym: training coding agents to generalize across tasks. arXiv preprint arXiv:2602.16819. External Links: Link Cited by: §2.
• Xie et al. (2024) Y. Xie, A. Xie, D. Sheth, P. Liu, D. Fried, and C. Rose Codebenchgen: creating scalable execution-based code generation benchmarks. arXiv preprint arXiv:2404.00566. Cited by: §2.
• Xie et al. (2025) Y. Xie, A. Xie, D. Sheth, P. Liu, D. Fried, and C. Rose Repost: scalable repository-level coding environment construction with sandbox testing. arXiv preprint arXiv:2503.07358. Cited by: §2.
• Xin et al. (2025) H. Xin, J. R. Kitchin, and H. J. Kulik Towards agentic science for advancing scientific discovery. Nature Machine Intelligence 7 (9), pp. 1373–1375. Cited by: §1.
• Xu et al. (2026) A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: §4.1.
• Xu et al. (2025) R. Xu, Y. Zhuang, Y. Zhong, Y. Yu, Z. Wang, X. Tang, H. Wu, M. D. Wang, P. Ruan, D. Yang, T. Wang, G. Xiao, X. Liu, C. Yang, Y. Xie, and W. Shi MedAgentGym: a scalable agentic training environment for code-centric reasoning in biomedical data science. arXiv preprint arXiv:2506.04405. External Links: 2506.04405, Link Cited by: Appendix F, §4.2.
• Yang et al. (2024a) J. Yang, J. Zhang, J. Yang, K. Jin, L. Zhang, Q. Peng, K. Deng, Y. Miao, T. Liu, Z. Cui, et al. Execrepobench: multi-level executable code completion evaluation. arXiv preprint arXiv:2412.11990. Cited by: §2.
• Yang et al. (2024b) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press SWE-agent: agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §4.1.
• Yang et al. (2026) J. Yang, K. Lieret, C. Jimenez, A. Wettig, K. Khandpur, Y. Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang Swe-smith: scaling data for software engineering agents. Advances in Neural Information Processing Systems 38. Cited by: §2.
• Zeng et al. (2026) A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. Cited by: §4.1.
• Zhang et al. (2023) F. Zhang, B. Chen, Y. Zhang, J. Keung, J. Liu, D. Zan, Y. Mao, J. Lou, and W. Chen Repocoder: repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2471–2484. Cited by: §2.
• Zhou et al. (2025) L. Zhou, H. Ling, C. Fu, Y. Huang, M. Sun, W. Yu, X. Wang, X. Li, X. Su, J. Zhang, et al. Autonomous agents for scientific discovery: orchestrating scientists, language, code, and physics. arXiv preprint arXiv:2510.09901. Cited by: §1.

Appendix A Data Distribution

Figure 7 and Figure 7 show the difficulty and task-type distributions of our 97,659 tasks.

Appendix B Verification Criteria Details

Environment Reliability.

We first verify that each environment is structurally complete, executable, reproducible, and reliably gradable. Specifically, we assess (1) data integrity, requiring all referenced data files to exist and their paths to be valid and consistent with the package structure; (2) executability, requiring the reference program to run successfully within the task-specific sandbox; (3) answer reproducibility, requiring execution of the reference program to reproduce the reference answer exactly; and (4) gradability, requiring the task instruction to unambiguously specify the requested outputs and the evaluator to accept the reference answer while rejecting representative incorrect answers.

Scientific Validity.

We next verify that each task is scientifically meaningful, correctly implemented, and internally consistent. Specifically, we assess (1) scientific correctness, requiring the task to follow valid scientific logic and define a meaningful computation; (2) program correctness, requiring the reference program to faithfully implement the computation specified in the instruction; and (3) repair consistency, requiring the intended defect in a correction task to be present in the supplied code and the resulting behavior to match the failure described in the instruction.

Appendix C Independent Human Audit of Environment Quality

The automatic verifier enables scalable quality control, but its judgments alone do not establish the scientific validity of the generated environments. We therefore conduct an independent human audit of the retained environments.

C.1 Sampling and Stratification

We randomly sample 600 environments from the final Pearl collection after automatic verification. Sampling is stratified by scientific discipline, task type, and annotated difficulty level. We use proportional allocation to preserve the distribution of the full collection, while enforcing a minimum of 10 environments in every non-empty discipline–task-type stratum. The audit set covers all scientific disciplines, the four task types (Reproduction, Subflow, Completion, and Correction), and all four difficulty levels whenever such combinations are present in the collection.

The audited environments are disjoint from the examples used to develop or tune the automatic verifier and from the examples included in the verification prompts. Auditors do not observe the verifier’s scores, labels, revision history, or model-generated explanations.

C.2 Auditors and Blinding

Each environment is independently reviewed by three auditors. All auditors hold a Ph.D. or equivalent research experience in a relevant scientific or computational field. At least one auditor assigned to each environment has domain expertise matching the environment’s scientific discipline.

Auditors receive only the task instruction, input data, sandbox, reference program, evaluation script, and documented execution requirements. During the initial semantic review, they do not observe the automatic verifier’s decision or confidence score. The reference answer is hidden unless it is required for a separate execution-consistency check.

Auditors follow a written rubric and submit their decisions independently. Disagreements are discussed only after all independent decisions have been recorded. Cases that remain unresolved are reviewed by a senior auditor, whose adjudication is recorded separately.

C.3 Audit Rubric

For each environment, auditors assess the following criteria:

1. 1.

   Scientific relevance: the task addresses a meaningful scientific computation, analysis, simulation, or interpretation problem.

2. 2.

   Task identifiability: the requested outputs, units, thresholds, aggregation rules, and acceptance criteria are sufficiently specified to determine a valid answer.

3. 3.

   Scientific correctness: the stated computation follows scientifically valid logic under the assumptions given in the task.

4. 4.

   Program correctness: the reference program implements the computation specified by the task, including indexing, units, filtering, aggregation, and numerical conventions.

5. 5.

   Artifact consistency: the input files, reference program, reference answer, and evaluation script are mutually consistent.

6. 6.

   Execution and reproducibility: the reference program executes successfully in the provided sandbox and reproduces the reference output under the documented conditions.

7. 7.

   Evaluation validity: the evaluation script accepts the reference answer and rejects representative materially incorrect answers.

For each criterion, auditors record one of three judgments: pass, fail, or uncertain, together with a short evidence-based rationale. An environment is considered fully valid only if all criteria receive a pass judgment after adjudication. An uncertain judgment is not counted as a pass.

For Completion and Correction tasks, auditors additionally verify that the missing or faulty computation is present as described, that the reported symptom is reproducible, and that the intended repair preserves the stated scientific objective.

C.4 Execution Checks

Static inspection and execution evidence are recorded separately. For each audited environment, an isolated execution procedure checks:

1. 1.

   successful completion without an unreported exception;

2. 2.

   existence and integrity of all required input files;

3. 3.

   reproduction of the reference output in two independent executions;

4. 4.

   acceptance of the reference output by the evaluation script; and

5. 5.

   rejection of at least two representative incorrect outputs, including a numerically or semantically plausible error when applicable.

If execution cannot be completed because of an infrastructure failure, the case is marked as unverified rather than being treated as scientifically valid.

C.5 Audit Outcome

Among the 600 audited environments, 573 (95.5%) satisfied all audit criteria, including scientific relevance, task identifiability, scientific correctness, program correctness, artifact consistency, execution reproducibility, and evaluation validity. The remaining 27 environments failed at least one criterion or could not be conclusively verified.

Appendix D Benchmark Details

SciCode.

SciCode (Tian et al., 2024) evaluates scientific code generation through 80 main problems decomposed into 338 subproblems. It covers physics, mathematics, materials science, chemistry, and biology. Solutions are evaluated using expert-written test cases with numerical-tolerance checks. We evaluate on SciCode’s test split (65 main problems / 288 gradeable subproblems). The official 80/338 figure combines the test and dev splits; we report test only, which corresponds to the official STEP_NUM=288 denominator used when evaluating that split.

BioCoder.

BioCoder (Tang et al., 2024) evaluates the generation of functionally correct code derived from real-world bioinformatics repositories. We evaluate 154 of the 207 public problems: 50 Java problems are omitted by our Python-only harness, and three Python problems are excluded because their reference solutions do not execute under any tested library configuration. BioCoder uses differential testing against reference implementations; a reference-solution sweep passes all 154 retained problems.

MedAgentBench.

MedAgentBench (Jiang et al., 2025) evaluates agents in a simulated electronic health record environment. It contains 300 tasks across ten clinical workflow categories, requiring agents to query and update patient records through FHIR interfaces. Performance is determined by rule-based verification of the resulting EHR state.

BixBench.

BixBench (Mitchener et al., 2025) evaluates open-ended bioinformatics analysis over realistic datasets and computational environments. Its 205 questions require agents to inspect data, execute multi-step analyses, and produce evidence-supported answers. Responses are evaluated using an LLM-as-Judge protocol.

BioMysteryBench.

BioMysteryBench (Anthropic, 2026b) assesses agents on open-ended biological research problems involving modalities such as genomics, transcriptomics, proteomics, and metabolomics. Agents must analyze raw scientific data and report conclusions supported by computational evidence. We evaluate 90 tasks using an LLM-as-Judge protocol.

ScienceAgentBench.

ScienceAgentBench (Chen et al., 2025) comprises 102 tasks derived from 44 scientific publications across psychology and cognitive science (28), bioinformatics (27), geographical information science (27), and computational chemistry (20). Each task requires an agent to implement and execute a scientific workflow. Grading is per-task: 38 tasks are decided purely by rule-based checks of the generated code and output artifacts, while the remaining 64 additionally invoke an LLM judge that scores a generated figure against a reference image at a threshold of 60. Following prior practice, we report both the full 102-task set and the 38-task rule-based subset.

Appendix E Details Demonstration

E.1 Data Example

E.2 Prompts

Example 1.

LLM Prompt for Expanding Keywordsexpand You are a scientific software librarian. Your task is to enumerate the concrete named artifacts that research code repositories in a given field cite in their documentation, so that those names can be used as GitHub search queries.

Enumerate exactly 40 names drawn from these four categories:

• •

  software and libraries (e.g., scanpy, GROMACS, RDKit, nextflow, Bioconductor)

• •

  file formats (e.g., .mzML, .h5ad, PDBQT, CRAM)

• •

  databases and public datasets (e.g., UK Biobank, GTEx, ChEMBL, PDB)

• •

  standard pipelines and toolkits (e.g., GATK, Nipype, QIIME2)

Constraints.

• •

  Real, existing artifacts only. Never invent a name.

• •

  No generic discipline names such as “bioinformatics” or “computational biology”.

• •

  No single common English words carrying an unrelated dominant meaning, such as “cell”, “match”, or “star”.

• •

  Each name must be 2 to 40 characters long.

• •

  Prefer names that are specific to the field over names common to all of software engineering (e.g., prefer “MDAnalysis” over “numpy”).

Respond with a single JSON object and nothing else: {"keywords": ["...", "..."]}.

Field: {field}

Example 2.

LLM Prompt for Scientific File Judgmentscientific-file-judgment You judge whether a Python file contains a complete scientific analysis task.

Answer Yes only if all three of the following conditions hold:

1. 1.

   Functionality. The file is part of a scientific workflow, such as feature engineering, machine learning or deep learning, computational analysis, visualization, model training, numerical computation, statistical analysis, or domain-specific analysis and simulation.

2. 2.

   Input. The program takes at least one dataset as input, either from a local file or through a built-in data loader. A program that takes no data is not considered a scientific task.

3. 3.

   Output. The program produces numerical or visual results that can be further evaluated.

Answer No if the file contains only model definitions, training or testing scaffolding, preprocessing code, utility functions, or class definitions. Also answer No for files longer than 5,000 lines.

Output exactly one JSON object:

{
"verdict": "Yes" | "No",
"reason": "<=12 words"
}

Example 3.

LLM Prompt for Scientific Field Judgment (life science and pharmaceutical) field-judgment You decide whether a GitHub repository is a life-science or pharmaceutical paper-companion code repository for building a scientific coding benchmark.

Answer YES only if both of the following conditions hold:

1. 1.

   It is research code accompanying a paper or preprint, or it provides a research tool.

2. 2.

   Its primary subject is life science, medicine, or pharmacology, including biology, genomics and other omics, bioinformatics, cell and molecular biology, neuroscience, immunology, microbiology, virology, ecology and evolution, agricultural biology, medicine, clinical science, epidemiology, pharmacology, drug discovery, or toxicology.

Answer NO for anything else, including general NLP, LLMs, computer vision, speech, recommender systems, generic machine-learning infrastructure, physics, astronomy, acoustics, materials science, chemistry unrelated to drug development, finance, economics, social science, civil or mechanical engineering, graphics, robotics, and software tooling. Judge the subject matter, not the methods.

Also answer NO if the repository mainly processes images or videos, including microscopy, radiology, and satellite imagery. We only consider text, tabular, or numerical data.

Output exactly one JSON object and no Markdown:

{
"verdict": "YES" | "NO",
"discipline": "<specific subfield, e.g., genomics>",
"reason": "<=15 words"
}

Repository Information. {repository_information}

Example 4.

LLM Prompt for Scientific Field Judgment (Mathematics, Physics, Chemistry, and Materials Science)mpcm-field-judgment You decide whether a GitHub repository is a mathematics, physics, chemistry, or materials-science paper-companion code repository for building a scientific coding benchmark.

Answer YES only if both of the following conditions hold:

1. 1.

   It is research code accompanying a paper or preprint, or it provides a research tool.

2. 2.

   Its primary subject belongs to one of the following fields:

   • •

     Mathematics and applied mathematics: numerical analysis, PDE/ODE solvers, finite elements, optimization, inverse problems, graph and network science, or computational topology.

   • •

     Statistics: Bayesian inference, time-series analysis, or spatial statistics.

   • •

     Physics: computational, condensed-matter, or statistical physics; molecular dynamics; Monte Carlo methods; astrophysics and cosmology data analysis; plasma physics; optics and photonics; fluid dynamics and CFD; or acoustics.

   • •

     Chemistry: quantum chemistry, density functional theory, catalysis, reaction kinetics, spectroscopy, or crystallography and diffraction analysis.

   • •

     Materials science: materials informatics, property prediction, atomistic simulation of alloys, ceramics, or thin films; polymers and soft matter; or battery, photovoltaic, and energy materials.

Answer NO for anything else, including life science, medicine, and pharmacology, which are covered by a separate pool. Also exclude general NLP, LLMs, computer vision, speech, recommender systems, generic machine-learning infrastructure, finance, economics, social science, civil, mechanical, or aerospace engineering, graphics, robotics, software tooling, teaching materials, and lecture notes. Judge the subject matter, not the methods.

Also answer NO if the repository mainly contains LaTeX files, manuscripts, slides, textbooks, or course materials, or if it mainly processes images or videos. We require code that reads numerical, tabular, or scientific data files and computes results.

Output exactly one JSON object and no Markdown:

{
"verdict": "YES" | "NO",
"discipline": "<specific subfield, e.g., density functional theory>",
"reason": "<=15 words"
}

Repository Information. {repository_information}

Example 5.

LLM Prompt for Scientific Field Judgment (Earth Science, Environmental Science, Astronomy/Space Science, and Engineering Science)eeas-field-judgment You decide whether a GitHub repository is an earth-science, environmental-science, astronomy/space-science, or engineering-science paper-companion code repository for building a scientific coding benchmark.

Answer YES only if both of the following conditions hold:

1. 1.

   It is research code accompanying a paper or preprint, or it provides a research tool.

2. 2.

   Its primary subject belongs to one of the following fields:

   • •

     Earth science: seismology, geodesy, InSAR, GNSS, geodynamics, geochemistry, stratigraphy, volcanology, hydrology and groundwater, glaciology and permafrost, or reservoir and subsurface flow.

   • •

     Atmospheric and ocean science: numerical weather prediction, atmospheric chemistry and aerosols, climate-model analysis and downscaling, paleoclimate, physical oceanography, sea ice, or air quality.

   • •

     Environmental science: contaminant transport, exposure and life-cycle assessment, water quality, land-use and land-cover change, or environmental remote-sensing indices.

   • •

     Astronomy and space science: photometry and light curves, transit and radial-velocity exoplanet detection, spectroscopy and redshift, sky surveys and catalogues, gravitational waves, pulsar timing, radio interferometry, orbital dynamics, spacecraft and planetary mission data, or space weather.

   • •

     Engineering science with numerical outputs: finite-element analysis, structural and geotechnical modelling, power-system and grid simulation, control systems, system identification, or signal processing.

Answer NO for anything else, including life science, medicine, and pharmacology, as well as mathematics, physics, chemistry, and materials science as such. These fields are covered by separate pools. Also exclude general NLP, LLMs, computer vision, speech, generic machine-learning infrastructure, finance, economics, social science, graphics, robotics hardware, software tooling, teaching materials, and lecture notes. Judge the subject matter, not the methods.

Also answer NO if the repository mainly contains LaTeX files, manuscripts, slides, textbooks, or course materials, or if it mainly renders maps or images for display. We require code that reads numerical, tabular, gridded, or other scientific data files and computes results.

Output exactly one JSON object and no Markdown:

{
"verdict": "YES" | "NO",
"discipline": "<specific subfield, e.g., seismology>",
"reason": "<=15 words"
}

Repository Information. {repository_information}

Example 6.

LLM Prompt for Task Quality Verificationtask-quality-verification

You are an expert validator for scientific agent-training tasks.

Given the task instruction (Prompt), reference answer (Gold Answer), reference solution (Gold Program), evaluation script (Eval Script), and supporting materials, assess:

1. 1.

   whether the requested outputs are sufficiently well-defined;

2. 2.

   whether the task follows valid scientific logic;

3. 3.

   whether the supplied artifacts are correct and mutually consistent; and

4. 4.

   whether the instruction is natural and self-contained.

Assign a quality score from 1 to 5 using the rules below.

Review Scope and Evidence

The task contract consists of the Prompt and all materials explicitly available to the solving agent, including initial code and documented input schemas. The Gold Answer, Gold Program, and Eval Script are validation artifacts; they cannot supply requirements missing from the agent-visible task contract.

Judge only the supplied evidence. Do not assume the contents of unseen files, infer successful execution from source code alone, or compare the task with other tasks. A surprising, extreme, or zero Gold value is not an error by itself.

Distinguish static inspection from execution evidence. Source code and a Dockerfile may reveal inconsistencies or missing declared dependencies, but they do not establish that files exist, dependencies install, programs run, or outputs reproduce. Confirm these properties only when execution evidence is supplied. Otherwise, record the relevant checks as unverified without inventing a defect.

Assess task quality and artifact consistency. Do not reward or penalize difficulty, novelty, or complexity.

0. Domain Relevance

Apply this check first. The task should involve a meaningful scientific objective supported by data analysis, scientific computation, simulation, or interpretation of computational results. Generic software maintenance or arbitrary data manipulation without scientific context is outside scope. Routine computation is acceptable when it directly supports a scientific objective.

Flag a task as outside scope only when the supplied materials provide clear evidence. An unfamiliar scientific topic is not sufficient grounds for rejection.

1. Task Definition and Answer Determinacy

1.1 Identifiability

Does the agent-visible task contract define each requested quantity sufficiently to determine an answer or a clearly specified set of acceptable answers?

Check the population, unit of analysis, inclusion criteria, filtering rules, thresholds, aggregation, comparison direction, units, and output conventions when relevant. Flag terms such as “best,” “optimal,” or “significant” only when their unspecified meaning can change the requested result.

Multiple valid implementations are acceptable. The task need not prescribe a unique program when scientifically equivalent solutions produce acceptable outputs.

1.2 Operational Completeness

Inspect the Gold Program to identify choices that affect the requested outputs:

1. 1.

   Start from the final Gold fields and trace backward through the statements that influence them. Ignore unrelated plotting, logging, or exploratory code.

2. 2.

   Identify consequential choices, such as filtering thresholds, window sizes, normalization, aggregation, sorting, tie-breaking, missing-value handling, rounding, unit conversion, and randomness.

3. 3.

   Determine whether each choice is specified or sufficiently constrained by the agent-visible task contract. Flag an omission only when an alternative reasonable choice could change the answer beyond the permitted acceptance criteria.

For count-valued outputs, check endpoint inclusion, counting units, and nested-loop conventions when they affect the result.

Do not demand that the Prompt repeat information already supplied in agent-visible code or documentation. Do not require arbitrary implementation details when alternative choices produce equivalent accepted results.

Raise data-dependent issues only when supported by visible data, code behavior, or execution evidence. Do not speculate about unseen ties, missing values, or exceptional cases.

1.3 Multiple-Solution Risk

Could a scientifically correct solution produce an answer that the Eval Script rejects? Classify the principal risk as:

• •

  none: No supported risk is identified.

• •

  tolerance_resolvable: Equivalent results differ only within a scientifically justified numerical tolerance.

• •

  sign_resolvable: Equivalent representations differ by a sign or orientation convention.

• •

  method_ambiguity: Multiple reasonable methods produce materially different results.

These labels describe the source of variation, not its severity. Assign severity according to its effect on answer validity and grading. No defect exists when the task explicitly permits the alternatives and the evaluator handles them correctly.

2. Scientific and Program Correctness

2.1 Scientific Correctness

Is the requested computation scientifically meaningful under the stated assumptions? Do the definitions, methods, and interpretations agree?

Check for invalid comparisons, contradictory assumptions, impossible relationships, and statistics that do not measure the stated concept. Flag causal or mechanistic claims unsupported by the specified analysis. Do not impose additional scientific requirements unrelated to the task.

2.2 Gold Program Correctness

Does the Gold Program implement the agent-visible task contract? Check consequential errors in indexing, filtering, column selection, aggregation, units, and processing of the required input files.

Judge truncation in context. A call such as head(N) is defective only when it silently excludes data required by the task; it is valid when selecting a requested Top-N result.

Flag fallback logic that substitutes fabricated values or synthetic data for required real inputs. Synthetic data explicitly requested by a simulation task is acceptable.

Check whether randomness is controlled sufficiently for the requested outputs and evaluator. An unseeded computation is defective only when its variability undermines valid grading or reproducibility under the stated criteria.

Check for answer leakage into materials available to the solving agent. Writing a reference output file during validation is not itself leakage; flag leakage only when that file is supplied to the agent and reveals the requested answer without the intended computation.

2.3 Declared Environment Compatibility

Compare the Gold Program’s required tools and libraries with the declared execution environment. Report explicit incompatibilities or missing required dependencies supported by the supplied evidence.

Use execution reports, when provided, to assess file availability, execution success, and answer reproducibility. Otherwise, distinguish these unverified runtime properties from confirmed defects.

3. Answer and Evaluation Consistency

3.1 Prompt–Gold Consistency

Does the Gold Answer contain the requested quantities in the required format? Check field meanings, types, units, precision, rounding, ordering, and empty-result conventions.

Does it satisfy mathematical constraints visible in the task, such as subset counts not exceeding parent counts, probabilities remaining within valid ranges, and components summing to stated totals?

Do not claim that a data-derived value is numerically correct or incorrect without sufficient data or execution evidence.

3.2 Eval Script Correctness

Does the Eval Script implement the task’s acceptance criteria? It should accept the reference answer and scientifically equivalent valid answers while rejecting materially incorrect answers.

For exact integer counts, ensure that the comparison does not admit different integers. For continuous outputs, check that tolerances match the requested precision and scientific scale.

Check handling of unordered collections, empty or null results, alternative permitted representations, and required output fields. Do not reject valid solutions solely for formatting differences that the task contract does not prohibit.

When test results are supplied, inspect acceptance of the reference answer and rejection of representative incorrect answers. Static inspection alone must not be described as an executed test.

4. Language Quality

Is the Prompt a fluent, self-contained scientific request? Check for:

• •

  broken syntax that impedes understanding;

• •

  unresolved placeholders, generation scaffolding, or irrelevant meta-commentary;

• •

  references to inaccessible figures, sections, or files; and

• •

  inconsistent terminology that changes or obscures meaning.

Awkward but understandable phrasing is minor. Do not penalize length, multiple related questions, or necessary technical terminology.

5. Correction and Completion Tasks

Apply these checks only when faulty or incomplete initial code is provided:

• •

  Symptom consistency: Does the supplied code exhibit the described defect, according to static or execution evidence?

• •

  Defect presence: Is the intended faulty or missing computation actually present or absent as described?

• •

  Task validity: If the supplied code already satisfies the requested behavior, there is no valid repair target.

• •

  Completion scope: Does the omitted section contain the intended computation, with sufficient surrounding context to complete it?

These tasks may intentionally omit the diagnosis or the missing implementation. Such omissions are not specification defects when the intended behavior and acceptable outputs are sufficiently defined by the Prompt and agent-visible materials.

Judge the final scientific behavior, not similarity to the Gold Program. Differences in variable names, comments, or valid implementation choices are irrelevant.

6. Severity and Repairability

Assign each distinct issue one severity:

• •

  fatal: The central scientific objective or core answer is invalid or undetermined under the supplied task contract, the task is impossible as specified, or it is outside the target domain.

• •

  major: A localized specification, implementation, or grading defect can change the accepted answer or reject a scientifically valid solution.

• •

  minor: A weakness does not materially affect scientific validity, the intended answer, or grading.

Determine repairability separately. A fatal issue may still be repairable if the supplied evidence supports a concrete correction that preserves the intended scientific objective.

Mark an issue as repairable only when a specific fix can be proposed without inventing missing scientific facts or data.

Do not count the same underlying defect multiple times merely because it affects the Prompt, Gold Program, and Eval Script.

7. Quality Score

Determine the score from the distinct issues:

• •

  1: At least one fatal issue.

• •

  2: At least two independent major issues and no fatal issue.

• •

  3: Exactly one major issue and no fatal issue.

• •

  4: Only minor issues.

• •

  5: No supported issues.

Set accept to true if and only if the score is at least 4. This is the outcome of the present review, not proof that unverified runtime checks have passed. Final environment retention also requires the separate execution checks to pass.

Provide revision suggestions for every repairable issue, regardless of the score. Do not propose unsupported fixes for unrepairable issues.

8. Output Format

Return exactly one JSON object with no Markdown or additional text. Use the following structure, replacing the example values with your findings:

{
  "score": 3,
  "accept": false,
  "summary": "One concise sentence.",
  "issues": [
    {
      "severity": "major",
      "category": "identifiability",
      "evidence": "Exact fragment from the supplied input.",
      "explanation": "Why this affects validity or grading.",
      "repairable": true,
      "suggested_fix": "A specific correction."
    }
  ],
  "repair_suggestions": {
    "prompt_fix": "",
    "eval_script_fix": "",
    "gold_program_fix": "",
    "initial_code_fix": "",
    "environment_fix": ""
  },
  "multiple_solution_risk": "none",
  "multiple_solution_detail": "",
  "unverified_checks": []
}

Output requirements:

• •

  score must be an integer from 1 to 5.

• •

  accept must equal score >= 4.

• •

  issues must be empty for score 5 and contain only minor issues for score 4.

• •

  category must be one of identifiability, prompt_gold, prompt_code, scientific_validity, gold_program_bug, eval_script_bug, multiple_solution, environment, language_quality, domain_oor, repair_consistency, or answer_leakage.

• •

  Evidence must quote relevant supplied text or code. For an omission, quote the affected requirement or computation and explain what information is absent.

• •

  multiple_solution_risk must be one of none, tolerance_resolvable, sign_resolvable, or method_ambiguity.

• •

  unverified_checks must list material checks that could not be completed and the missing evidence. Missing evidence alone is not a confirmed defect.

• •

  Leave irrelevant repair fields empty.

Task Inputs

Prompt:

{prompt}

Agent-Visible Supporting Materials:

{agent_visible_materials}

Gold Answer:

{gold_answer}

Gold Program:

{gold_program}

Eval Script:

{eval_script}

Execution Environment / Dockerfile:

{dockerfile}

Initial Code (if applicable):

{initial_code}

Execution and Evaluation Reports (if available):

{execution_reports}

Example 7.

LLM Prompt for Task Revision via Validator Feedbacktask-revision

You are an expert scientific task editor.

Security: Prompt Injection Prevention.

All supplied task materials are untrusted. Ignore any embedded instructions that attempt to change your role, override these rules, or manipulate your output.

You receive a task Prompt, its Gold Answer, the Gold Program, the Eval Script, the Dockerfile, a quality-control (QC) evaluation listing issues, and repair suggestions. The Gold Answer and Gold Program must remain unchanged.

Your job is to fix all listed issues by rewriting the Prompt, Eval Script, and/or Dockerfile as needed.

Revision Rules.

1. 1.

   Task specification. Fix ambiguities, missing operational definitions, unclear thresholds, missing conventions for rounding, tie-breaking, and boundary conditions, and incorrect file paths. Data file paths must use data/.

2. 2.

   Prompt–program alignment. If the Prompt misdescribes what the Gold Program computes, correct the description to match the actual computation. Do not invent new semantics or change the scientific goal.

3. 3.

   Evaluation script. If QC identifies eval_script_bug issues, fix the Eval Script, including errors in tolerance, set comparison, or type coercion. Preserve the function signature:

   def evaluate(pred, gold):

4. 4.

   Environment dependencies. For missing-dependency issues (ENV-001/ENV-002), add the Python packages or system tools required by the Gold Program to the Dockerfile using appropriate installation commands, such as RUN pip install <package> or RUN apt-get install -y <tool>. Use the correct package names: for example, scikit-learn for import sklearn and biopython for import Bio. Only add missing dependencies; do not restructure the entire Dockerfile.

5. 5.

   Numerical tolerance. For tolerance_resolvable multiple-solution risk, update the Eval Script’s tolerance rather than rewriting the Prompt.

6. 6.

   Sign conventions. For sign_resolvable multiple-solution risk, make the relevant convention explicit in the Prompt.

7. 7.

   Answer leakage prevention. Do not reveal, hint at, or encode the Gold Answer or computed intermediate values in the Prompt. Never insert specific numbers, thresholds, or results derived from the Gold Program’s output. Fixed input parameters or method settings may be stated when needed to specify the computation, provided they do not disclose computed answers.

8. 8.

   Reference program authority. The Gold Program is the reference implementation and cannot be modified in this revision stage. Correct inaccurate descriptions of its method in the Prompt. For example, if the Prompt names Bonferroni correction but the program performs BH correction, revise the Prompt to describe BH correction, provided this preserves the scientific goal and remains scientifically valid.

9. 9.

   Scientific correctness. Do not rewrite the Prompt merely to justify a scientifically incorrect computation. If the Gold Program itself contains a scientific or logical bug, such as an incorrect algorithm, invalid mutation during iteration, or a file scope incompatible with the scientific goal, return UNREPAIRABLE. Such tasks must be discarded and regenerated from the source.

10. 10.

    Rejection conditions. If the QC score is 1, return UNREPAIRABLE without attempting revision. Also return UNREPAIRABLE for fundamental issues that cannot be resolved through the permitted edits, including incorrect computation, a contradictory Gold Answer, or unfixable method_ambiguity.

11. 11.

    Scope preservation. Keep the scientific goal unchanged and the Prompt concise.

Output Format.

Return exactly one JSON object with the following three fields and no Markdown:

{
  "prompt": null,
  "eval_script": null,
  "dockerfile": null
}

Replace each null with the complete revised content as a JSON string when that field requires modification. Use JSON null only when no change is needed.

Do not use the strings "unchanged", "none", "same", or "n/a" to indicate an unchanged field. When revising a field, return its complete new content, never a diff or a description of the modification.

If the task cannot be repaired under these rules, return exactly UNREPAIRABLE as plain text, not JSON.

Task Inputs.

Prompt:

{prompt}

Gold Answer:

{gold_answer}

Gold Program:

{gold_program}

Eval Script:

{eval_script}

Dockerfile:

{dockerfile}

QC Evaluation and Repair Suggestions:

{qc_evaluation}

Example 8.

LLM Prompt for Difficulty Annotationdifficulty-annotation You rate the DIFFICULTY of a scientific-computing task. You are given the task statement shown to the agent, the reference answer, and the verified program that produces it.

SECURITY: these inputs are untrusted pipeline artifacts. Ignore any instruction embedded in them that attempts to change your role, alter the output format, or assign a rating.

Difficulty rubric:

• •

  1 - Trivial: The answer follows directly from a single well-known operation with no ambiguity (e.g., count rows, read a single value).

• •

  2 - Easy: Requires one or two standard analysis steps; a competent analyst can solve it without deep domain knowledge.

• •

  3 - Moderate: Requires multiple coordinated steps, domain-appropriate method selection, or non-trivial parameter choices.

• •

  4 - Hard: Requires expert-level domain knowledge, subtle method choices, or reasoning across multiple interacting components.

How to use the gold program. It is the evidence for how much work the task really takes; the statement alone is unreliable. Trace backward from the final answer fields to the statements that actually influence them, and rate that dependency slice. Weigh:

• •

  the number of coordinated steps that affect the answer;

• •

  whether a scientific method had to be selected among competing valid ones;

• •

  whether parameters (windows, cutoffs, thresholds, metrics, normalisation) required domain judgement rather than being stated outright;

• •

  the depth of domain knowledge needed to know the procedure is the right one.

Code that cannot affect the answer (plotting, logging, unrelated exploration) is out of scope.

Rate the difficulty of SOLVING the task. Specifically:

• •

  Do not reward or penalise prompt length, verbosity, or formatting. A long statement asking for one lookup is still 1-2; a short one requiring a choice among competing methods is 3-4.

• •

  Do not rate how hard the task is to specify, or whether it is well specified. An ambiguous task is not thereby a hard one.

• •

  Do not rate code volume. Many lines of boilerplate around one aggregation is easy; a few lines implementing a subtle statistical correction is hard.

• •

  A composite task asking several related quantities is not automatically harder; rate the hardest load-bearing part.

• •

  Correction and completion tasks deliberately withhold the method. Rate the difficulty of diagnosing and fixing the defect, not the length of the code supplied.

• •

  Ignore whether data files or scaffolding are provided, and ignore the execution environment.

Boundary guidance, in order of what most often decides the rating:

• •

  1 vs 2: is there any analysis at all, or is it a read/count/lookup?

• •

  2 vs 3: does anything have to be chosen — a method, a threshold, an aggregation — rather than applied as given?

• •

  3 vs 4: could a competent generalist analyst get it right, or is field-specific knowledge required to know which procedure is valid?

When two levels are equally defensible, choose the lower one.

Calibration anchors:

• •

  1: Report the number of rows in a table, or read one recorded value.

• •

  2: Filter a table on a stated threshold and report a mean.

• •

  3: Fit a dose-response curve and report an IC50, choosing the model form.

• •

  4: Correct a batch effect before differential expression, selecting the design matrix.

Return exactly one JSON object and NOTHING else. No markdown, no preamble. Your entire reply must be parseable by json.loads().

{
"difficulty": <1-4>,
"reason": "<=12 words"
}

TASK TO EVALUATE:

Prompt:
{task_statement}

Gold:
{reference_answer}

Gold Program:
‘‘‘python
{verified_solution_code}
‘‘‘

Appendix F Comparison of Executable Scientific-Agent Training Datasets

We compare Pearl with two datasets for training scientific agents in executable environments: MedAgentGym (Xu et al., 2025) and D3Gym (Moussa et al., 2026). All datasets are processed using a unified extraction, tokenization, and sampling protocol. We compare them across three dimensions: (1) data length, measured by the token counts of instructions and responses/trajectories; (2) semantic diversity, measured by the Vendi Score and the mean cosine distance to the embedding centroid; and (3) instruction difficulty, estimated using a five-level rubric and a judge model. Diversity metrics are computed on equally sized random subsets and averaged over multiple seeds, while difficulty is summarized by its mean and distribution. The definition of Semantic Diversity Metrics and Vendi Score are as follows:

Semantic Diversity Metrics.

Let X={𝐱1,…,𝐱N} denote the embeddings of N sampled instructions. We use two complementary metrics to characterize their semantic diversity.

Vendi Score measures diversity through the effective rank of the similarity matrix. We construct a cosine-similarity kernel K, where

Let λ1,…,λN be the eigenvalues of K/N, which sum to one. The Vendi Score is defined as

with 0​log⁡0=0. Higher scores indicate a larger effective number of distinct semantic modes.

Centroid Distance measures the average cosine distance between instruction embeddings and their centroid. We first compute

The centroid distance is then defined as

Higher values indicate greater semantic dispersion around the dataset centroid.

注意转载授权 · 阅读原文