Auditing Pairwise Equivalence Judgments: Self-Critique Effects and Diversity Measurement in Multi-Agent Hypothesis Generation

Ji Young Byun Affiliation: Johns Hopkins University Affiliation: Baltimore, MD 21218, USA Email: jbyun13@jhu.edu    Anthony Hu    Jesse Rogers Affiliation: Merck & Co., Inc. Affiliation: West Point, PA 19486, USA Email: jesse.rogers@merck.com    Roujia Wang Affiliation: Merck & Co., Inc. Affiliation: West Point, PA 19486, USA Email: roujia.wang@merck.com    Manasa Kesapragada Affiliation: Merck & Co., Inc. Affiliation: West Point, PA 19486, USA Email: manasa.kesapragada@merck.com    Falgun Shah Affiliation: Merck & Co., Inc. Affiliation: West Point, PA 19486, USA Email: falgun.shah@merck.com

Abstract

Multi-agent systems built on large language models (LLMs) are increasingly applied to scientific discovery and hypothesis generation. Both the effect of refinement and the diversity of the delivered set are hard to interpret before experimental ground truth exists, and both are typically reported by deciding, for pairs of generated hypotheses, whether they describe the same underlying mechanism. We study two evaluation questions that rest on this pairwise equivalence judgment: (1) how much self-critique changes delivered hypotheses beyond run-to-run variability, and (2) how the equivalence rule used to group hypotheses affects measured diversity. Our study workflow generates mechanistic explanations represented as causal chains from a molecular initiating event through intermediate biological steps to an observed outcome, revised through self-critique rounds before a final consolidation step. Across four independent proprietary instances, we hold opening hypotheses fixed, rerun the downstream workflow with 0, 1, and 5 critique rounds, and score matched hypothesis pairs with an LLM-as-a-judge. Relative to matched same-depth reruns, moving from 0 to 1 round produces 34.5 percentage points (pp) of additional mechanism-level divergence, whereas 1 to 5 rounds adds 1.3 pp. We then compare three equivalence rules: term frequency–inverse document frequency (TF–IDF) similarity, dense embeddings, and the same LLM-as-a-judge. To characterize how these rules respond to specific edits, we construct controlled hypothesis pairs that either alter only wording or equivalent biological terminology while preserving the causal explanation, or replace one component of the causal chain while holding the remaining structure fixed. All three rules are invariant to meaning-preserving edits, but when the initiating event is replaced, the LLM-as-a-judge identifies 83% of valid pairs as different mechanisms, versus 0% for TF–IDF and 8% for embeddings; varying only the rubric that defines same mechanism moves this figure from 38% to 96%. Together, these results show that pairwise equivalence judgments are a measurement choice: how mechanism equivalence is defined affects both the estimated effect of self-critique and the measured diversity of generated hypotheses.

⁰⁰footnotetext: Accepted to the Agentic AI for Biological Discovery Workshop (NeurIPS 2026).

1 Introduction

Large language models (LLMs) are increasingly being incorporated into scientific workflows, where agentic systems propose and refine hypotheses (Gottweis et al., 2025; Ghareeb et al., 2026), interpret experimental data (Huang and others, 2025; Chen et al., 2025), design candidate molecules (Bran et al., 2024; Swanson et al., 2025), and plan follow-up experiments (Huang and others, 2025). Evaluating such systems departs from the standard benchmark setting. Scientific hypotheses often concern mechanisms whose truth is not yet established when they are proposed, and the experiments required to adjudicate them can be expensive and time-consuming. Correctness therefore becomes available, if at all, only after the system must already be evaluated, necessitating evaluation criteria that can be applied directly to the generated hypotheses.

In the absence of such an external signal, recent systems evaluate generated hypotheses by comparing them with one another, both while refining candidates and when reporting the delivered set. The AI co-scientist ranks hypotheses through an Elo tournament of pairwise scientific debates, and separately consolidates its output set by clustering hypotheses in an embedding-based proximity graph (Gottweis et al., 2025). Robin ranks candidates by asking an LLM-as-a-judge which of two hypotheses is better across all pairs and fitting a Bradley–Terry–Luce model to the outcomes (Ghareeb et al., 2026). Si et al. (2025) use both: generated ideas are deduplicated by thresholding pairwise embedding similarity before an LLM-as-a-judge orders the survivors. These systems apply different comparison rules to different purposes, with LLM-based preference judgments prioritizing candidates and embedding similarity supporting deduplication.

Pairwise equivalence judgments provide a common measurement basis for two evaluation questions that are otherwise treated separately. First, assessing the effect of self-critique requires determining how much the delivered hypotheses change as critique depth varies, relative to run-to-run variability across repeated executions at the same critique depth. Second, measuring the diversity of a generated set requires determining when two hypotheses should be treated as distinct explanations rather than alternative formulations of the same explanation. Although these questions concern different properties of the workflow, both involve the same hypothesis comparison task of deciding whether a pair should be treated as the same or different.

We study this problem in a multi-agent hypothesis generation workflow. Given an unexpected experimental finding, the workflow proposes mechanistic hypotheses that explain it, each represented as a causal chain from an initiating event through intermediate biological steps to that finding. For the first question, we hold the opening hypotheses fixed and vary only critique depth, then read cross-depth divergence against matched same-depth reruns, since repeating an identical configuration also changes the delivered hypotheses (Atil et al., 2024). For the second, no labels exist for whether two generated hypotheses share a mechanism, so we compare equivalence rules on constructed pairs whose intended relation is known: some edits preserve the mechanism and should not change a rule’s verdict, while others replace a component of the causal chain and should.

• •

  We identify the pairwise equivalence judgment—deciding whether two hypotheses are the same or different—as a shared measurement basis for two evaluation questions that are usually treated separately: what changes beyond same-condition run-to-run variability, and what counts as one mechanism when measuring diversity.

• •

  Replaying the downstream workflow from identical opening hypotheses, we show that the largest change in delivered mechanisms occurs when critique is first introduced: moving from 0 to 1 round produces 34.5 pp of divergence beyond matched rerun variability, whereas moving from 1 to 5 rounds adds 1.3 pp.

• •

  We audit three equivalence rules with controlled perturbations. TF–IDF, embeddings, and an LLM-as-a-judge behave alike on meaning-preserving edits, but diverge when the initiating event is replaced: the LLM-as-a-judge reads 83% of these pairs as different mechanisms, compared with 8% for embeddings and 0% for TF–IDF, and varying only the judge’s rubric moves this rate from 38% to 96%.

2 Related Work

LLM agents for scientific hypothesis generation.

LLM agents are now applied across the stages of a scientific workflow, from interpreting experimental data and designing candidate molecules to planning and executing experiments, and benchmarks have been built to measure these capabilities (Boiko et al., 2023; Bran et al., 2024; Laurent et al., 2024; Huang and others, 2025; Swanson et al., 2025; Chen et al., 2025). Single-agent designs pursue it through iteration, alternating between generation and self-evaluation modules and reporting that self-evaluation improves the novelty and relevance of proposals (Yang et al., 2025). Multi-agent designs instead distribute the same work across specialized roles: the AI co-scientist separates generation, reflection, ranking, evolution, and proximity-based consolidation (Gottweis et al., 2025), Robin couples literature-grounded generation with experimental analysis (Ghareeb et al., 2026), and VirSci organizes a simulated research team that generates, discusses, and assesses ideas before selecting one (Su et al., 2025).

Self-critique and iterative refinement.

Refinement is a standard way to spend additional computation on a candidate output before delivering it. A single agent can revise its own output from model-generated feedback or environmental signals (Madaan et al., 2023; Shinn et al., 2023), and multi-agent designs assign the critique to a separate instance that challenges the proposal before a final response is produced (Du et al., 2024). Test-time scaling studies place this sequential revision alongside parallel sampling, aggregation, and verifier-based merging as alternative ways to allocate the same inference budget, both for LLMs and for the agents (Byun et al., 2026; Zhu et al., 2025). How much the added computation helps varies with the setting: intrinsic self-correction without external feedback can leave reasoning unimproved (Huang et al., 2024; Kamoi et al., 2024) and debate does not always outperform simpler ensembling (Smit et al., 2024). In each case the effect is read against a gold answer or a task-specific metric, a reference that open-ended hypothesis generation makes available only after the fact.

Comparative evaluation without a reference.

When an external reference is available, a generation can be scored against it. FActScore decomposes text into atomic claims and assesses their evidential support, while ALCE evaluates the correctness and completeness of generated citations (Min et al., 2023; Gao et al., 2023). Without such a reference, evaluation compares generations with one another, and the common instrument is a pairwise LLM-as-a-judge that decides which of two outputs is better (Zheng et al., 2023; Liusie et al., 2024). Because no reference is available for evaluating the LLM-as-a-judge itself, responsible use requires understanding its behavior (Shi et al., 2024; Eiras et al., 2025).

3 Problem Formulation: Evaluation without Experimental Ground Truth

Given an unexpected experimental finding, a generation run delivers a set of mechanistic hypotheses ℋ={h1,…,hn} that could explain it, where each hi is a causal chain from an initiating event through intermediate biological steps to that finding, and n varies across runs. Candidate hypotheses may be iteratively revised through self-critique before the final set is produced. Because the underlying biological mechanism is unknown at generation time, no experimental ground truth is available for deciding whether a proposed mechanism is correct. Such comparison requires an equivalence rule ρm​(hi,hj)∈{0,1}, indexed by the comparison method m, that returns 1 when two hypotheses express the same mechanism.

What changes beyond run-to-run variability?

Write ℛK for the set of runs performed with up to K critique rounds. For a run r∈ℛK, let ℋr denote the set of hypotheses produced after self-critique. To compare two runs r and r′, we form a set of matched hypothesis pairs M⁡(r,r′), where each pair (h,h′) contains corresponding hypotheses from ℋr and ℋr′. A blinded LLM-as-a-judge supplies the equivalence rule here, with ρLLM​(h,h′)∈{0,1} assigning 1 when the pair describes the same mechanism and 0 otherwise. We define the divergence between the two runs as the fraction of matched hypothesis pairs judged to describe different mechanisms,

Here, |M⁡(r,r′)| is the number of matched hypothesis pairs used to compare the two runs. Thus, D⁡(r,r′)=0 when all matched pairs are judged to describe the same mechanism, while larger values indicate greater mechanism-level divergence between the two runs.

A cross-depth difference alone does not indicate how much of the observed change is specific to critique depth, because rerunning the same configuration can also produce different hypotheses. Let D¯K,K′ denote the mean divergence over run pairs with critique depths K and K′, and let D¯K,K′within denote the mean divergence over all same-depth run pairs from the two configurations. We define the extra divergence as

with the opening hypotheses held fixed so that critique depth is the only experimentally varied configuration. A positive Δ⁡(K,K′) indicates that runs at different critique depths diverge more than expected from rerunning the corresponding same-depth configurations, whereas Δ⁡(K,K′)=0 indicates no additional divergence beyond this run-to-run variability.

What counts as one mechanism when measuring diversity?

Once hypotheses have been generated, reporting their diversity requires grouping ℋ into distinct mechanisms and counting the resulting groups. Here m ranges over the candidate comparison methods—TF–IDF similarity, dense embeddings, and an LLM-as-a-judge—and the same grouping procedure is applied to the pairwise decisions of each,

where 𝒫 clusters ℋ under ρm and returns the number of resulting clusters. Because no reference is available for scoring ρm either, we audit the candidate rules with targeted perturbations: some preserve the underlying mechanism, whereas others replace a specified component of the causal chain.

4 Experimental Setup

We study our multi-agent workflow on four independent proprietary instances, each defined by an unexpected experimental finding and its retrieved evidence (Figure 1). The instances are proprietary and are anonymized as A–D throughout. For each instance, we perform five independent hypothesis-generation and refinement processes, using retrieved evidence and up to K rounds of self-critique before consolidation.

Generation and evaluation are kept separate: all evaluation is performed post hoc and does not influence hypothesis generation or refinement. Generation uses Claude Sonnet 4.6 for the proposer and critic. Two judges are used post hoc, both Claude Opus 4.6 at temperature 0: one labels how the proposer responds to each critique, and one applies the equivalence rule that decides whether two hypotheses describe the same mechanism. Dense representations use Amazon Titan Text Embeddings V2. The equivalence judgments are additionally repeated across three model families (Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.5) to assess sensitivity. These models are used exclusively for post-hoc evaluation and provide no feedback to the generation process. Appendix B provides the details.

5 Results

We first examine how changing critique depth alters the delivered mechanisms relative to the run-to-run variability observed under the same critique setting. We then examine how different equivalence rules affect measured diversity by changing which generated hypotheses are treated as distinct mechanisms.

5.1 Critique-depth effects relative to run-to-run variability

Critique is actively incorporated during refinement.

Across the critique exchanges from the four proprietary instances, a blinded post-hoc LLM-as-a-judge decomposes the critiques into 332 individual demands and finds that 61% are conceded, 27% are answered, 9% are partially addressed, and 3% are dropped. These results show that the refinement process engages with the critiques, but do not establish how much critique changes the final hypotheses or whether those changes are scientifically beneficial.

Most critique-depth divergence appears after the first critique round.

Holding the opening hypotheses fixed, we replay the downstream process with 0, 1, and 5 critique rounds and compare the resulting hypotheses against matched same-depth reruns. All comparisons are conducted within instance to account for differences in the number and composition of generated hypotheses. The blinded LLM-as-a-judge is not given the critique depth or provenance of either hypothesis.

Under the primary evaluator shown in Figure 2, same-depth reruns with no critique show 0% divergence for all four instances, whereas reruns with 1 critique round show 58%–76% divergence across instances. Although downstream generation and consolidation remain stochastic, the zero-round condition is substantially reproducible at the mechanism level. Relative to the corresponding same-depth baselines, moving from 0 to 1 critique round produces 34.5 percentage points (pp) of extra divergence, whereas moving from 1 to 5 rounds adds 1.3 pp. Moving from 0 to 5 rounds produces 39.8 pp of extra divergence. Thus, most of the cross-depth separation appears when critique is first introduced, with little additional separation from later rounds. This pattern is robust to evaluator choice: across three evaluator configurations, the 0-to-1 extra divergence ranges from 34.5 to 37.3 pp, while the corresponding same-depth rerun ranges are 0%–5% at zero critique and 58%–76% at one critique round (Appendix A).

The preceding divergence estimates were obtained under a single equivalence rule, implemented by an LLM-as-a-judge that determined whether each matched pair described the same mechanism. Pairwise equivalence judgments also underlie the measurement of output diversity. When hypotheses are generated in parallel, estimating diversity requires grouping hypotheses judged to express the same explanation, and the resulting number of distinct groups may vary with the equivalence rule used. We therefore compare three equivalence rules applied to the same set of hypotheses.

5.2 Measured diversity depends on the equivalence rule

We compare three rules for deciding whether two generated hypotheses describe the same mechanism. TF–IDF similarity scores a pair by weighted lexical overlap, so that shared rare terms count for more than shared common ones (Jones, 1972). Dense embedding similarity replaces this surface match with cosine distance between learned sentence representations, which can align paraphrases that share little vocabulary (Reimers and Gurevych, 2019). An LLM-as-a-judge is shown both hypotheses and returns a same-or-different verdict under a written rubric (Zheng et al., 2023). We examine the pairwise decisions directly, using controlled perturbations that either preserve the underlying mechanism or alter a specified component of the causal chain.

Targeted perturbations probe what each rule treats as a distinct mechanism.

Each hypothesis contains a natural language mechanistic explanation and a causal chain from an initiating event through intermediate biological steps to the observed findings. We construct four controlled variants, each changing a specific aspect of the original hypothesis. Wording Only rewrites the mechanistic explanation while leaving the causal chain unchanged. Wording + Chain additionally rewrites the chain using audited biological aliases while preserving the same entities and causal structure. Both are designed to preserve the underlying mechanism. First Step Swapped replaces the initiating event while keeping the downstream structure fixed and is designed to represent a different mechanism under our study definition. Middle Step Swapped replaces one intermediate causal step while preserving the initiating event and terminal finding. Details are included in Appendix C.

Targeted perturbations reveal different rule sensitivities.

All three rules classify every valid Wording Only and Wording + Chain pair as the same mechanism, indicating robustness to paraphrasing and audited alias substitutions (Figure 3). Differences emerge when the causal structure is altered. For First Step Swapped, the LLM-as-a-judge classifies 83% of pairs as different, compared with 0% for TF–IDF and 8% for embeddings, suggesting that global similarity measures may be less sensitive when a single causally important component changes while most of the hypothesis remains unchanged. For Middle Step Swapped, the LLM-as-a-judge classifies only 13% of pairs as different, while TF–IDF and embeddings classify none as different, indicating that even the LLM-as-a-judge often treats changes to intermediate causal steps as variation within the same mechanism. This motivates examining how the LLM-as-a-judge’s decisions change when the evaluation rubric is specified differently.

LLM-as-a-judge behavior depends on prompt specification.

We test whether the LLM-as-a-judge’s same or different decisions depend on how the evaluation criteria are specified. We compare three prompt variants: the full rubric used throughout the paper, a step-by-step rubric that separately evaluates the initiating event, causal route, and terminal outcome before the overall verdict, and a no-rubric version that provides only the comparison task and verdict field. All prompts are provided in Appendix E.

Explicit criteria improve sensitivity to causal differences.

In Figure 4, all three prompts remain invariant to Wording Only and Wording + Chain edits. The largest difference appears for First Step Swapped: the no-rubric prompt classifies 38% of pairs as different, compared with 83% under the full rubric and 96% under the step-by-step rubric. The step-by-step rubric also increases sensitivity to Middle Step Swapped changes, from 13% under the full rubric to 21%, while explicitly separating judgments of the initiating event, causal route, and terminal outcome. Reversing the presentation order changes decisions for 5.6% of controlled perturbation pairs under each prompt variant, although the affected controlled pairs differ across rubrics (Appendix D). Together, these results suggest that making the comparison criteria more explicit helps the LLM-as-a-judge distinguish causally meaningful differences while preserving invariance to meaning-preserving edits.

6 Conclusion

We studied two questions for evaluating scientific hypothesis generation before experimental ground truth is available: how much self-critique changes delivered mechanisms beyond run-to-run variability, and how the equivalence rule affects measured diversity. Moving from 0 to 1 critique round produced 34.5 pp of extra divergence, whereas 1 to 5 rounds added 1.3 pp. Under controlled perturbations, all three rules were robust to meaning-preserving edits but differed on causal changes: for First Step Swapped, the LLM-as-a-judge read 83% of pairs as different, compared with 8% for embeddings and 0% for TF–IDF, and this rate varied from 38% to 96% with the rubric alone. These results suggest that evaluation should account for variability and carefully consider both the equivalence method and the rubric used to interpret measured diversity.

Limitations.

Our study is limited to four proprietary instances and five repetitions per critique depth, and no biological ground truth is available for the generated hypotheses. The results therefore characterize how the workflow outputs and evaluation methods behave, rather than whether critique improves scientific correctness or whether a particular grouping corresponds to the biologically correct mechanism.

Our post-hoc evaluation also relies on repeated LLM-based comparisons, which may become computationally expensive as the number of generated hypotheses increases. Pairwise mechanism comparisons grow rapidly with the size of the hypothesis set, while robustness analyses across judge models and prompt variants introduce additional inference cost.

Finally, measured diversity remains sensitive to evaluator design. TF–IDF and embedding results depend on similarity thresholds, while LLM same or different judgments vary with judge model, prompt specification, and presentation order (Appendix D). Because the judges and generators belong to the same broad class of LLMs, agreement across judge families does not rule out shared biases such as self-preference or sensitivity to superficial properties. Our controlled perturbations also probe only a limited set of semantic and causal changes. Broader, systematically constructed perturbation suites and independently specified evaluation rubrics would provide stronger evidence about which causal distinctions an evaluator captures and how robustly it captures them.

Future work.

A first direction is to improve the efficiency of post-hoc evaluation, for example through selective pairwise comparison, cascaded evaluators that invoke an LLM-as-a-judge only for ambiguous pairs, or distillation of LLM judgments into smaller models. A second direction is to strengthen the evaluation reference itself. Blinded domain experts could help validate controlled perturbations, refine the rubric for distinguishing mechanisms, and adjudicate same/different decisions. Measuring expert–expert and expert–model agreement across a broader range of mechanistic differences would help identify where automated evaluators are reliable and where their judgments remain ambiguous. Ultimately, prospective experimental evidence will be needed to determine whether critique-induced changes and evaluator-defined distinctions correspond to scientifically meaningful biological mechanisms.

References

• Atil et al. (2024) B. Atil, A. Chittams, L. Fu, F. Ture, L. Xu, and B. Baldwin Non-determinism of “deterministic” LLM settings. arXiv preprint arXiv:2408.04667. Cited by: §1.
• Boiko et al. (2023) D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes Autonomous chemical research with large language models. Nature 624 (7992), pp. 570–578. External Links: Document Cited by: §2.
• Bran et al. (2024) A. M. Bran, S. Cox, O. Schilter, C. Baldassari, A. D. White, and P. Schwaller Augmenting large language models with chemistry tools. Nature Machine Intelligence 6, pp. 525–535. External Links: Document Cited by: §1, §2.
• Byun et al. (2026) J. Y. Byun, Y. Park, N. Azizan, and R. Chellappa Test-time scaling in clinical decision making. In Proceedings of the 9th International Conference on Medical Imaging with Deep Learning, Proceedings of Machine Learning Research, Vol. 315, pp. 2118–2142. External Links: Link Cited by: §2.
• Chen et al. (2025) Z. Chen, S. Chen, Y. Ning, Q. Zhang, B. Wang, B. Yu, Y. Li, Z. Liao, C. Wei, Z. Lu, V. Dey, M. Xue, F. N. Baker, B. Burns, D. Adu-Ampratwum, X. Huang, X. Ning, S. Gao, Y. Su, and H. Sun ScienceAgentBench: toward rigorous assessment of language agents for data-driven scientific discovery. In International Conference on Learning Representations, Cited by: §1, §2.
• Du et al. (2024) Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, Cited by: §2.
• Eiras et al. (2025) F. Eiras, E. Zemour, E. Lin, and V. Mugunthan Know thy judge: on the robustness meta-evaluation of LLM safety judges. arXiv preprint arXiv:2503.04474. Cited by: §2.
• Gao et al. (2023) T. Gao, H. Yen, J. Yu, and D. Chen Enabling large language models to generate text with citations. In Proceedings of EMNLP, Cited by: §2.
• Ghareeb et al. (2026) A. E. Ghareeb, B. Chang, L. Mitchener, et al. A multi-agent system for automating scientific discovery. Nature 655, pp. 497–505. Cited by: §1, §1, §2.
• Gottweis et al. (2025) J. Gottweis, W.-H. Weng, A. Daryin, et al. Towards an AI co-scientist. External Links: 2502.18864 Cited by: §1, §1, §2.
• Huang et al. (2024) J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations, Cited by: §2.
• Huang et al. (2025) K. Huang et al. Biomni: a general-purpose biomedical AI agent. bioRxiv. External Links: Document Cited by: §1, §2.
• Jones (1972) K. S. Jones A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation 28 (1), pp. 11–21. Cited by: §5.2.
• Kamoi et al. (2024) R. Kamoi, Y. Zhang, N. Zhang, J. Han, and R. Zhang When can LLMs actually correct their own mistakes? a critical survey of self-correction of LLMs. Transactions of the Association for Computational Linguistics 12, pp. 1417–1440. Cited by: §2.
• Laurent et al. (2024) J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Ponnapati, A. D. White, and S. G. Rodriques LAB-Bench: measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10362. Cited by: §2.
• Liusie et al. (2024) A. Liusie, P. Manakul, and M. Gales LLM comparative assessment: zero-shot NLG evaluation through pairwise comparisons using large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pp. 139–151. Cited by: §2.
• Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, et al. Self-Refine: iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, Cited by: §2.
• Min et al. (2023) S. Min, K. Krishna, X. Lyu, et al. FActScore: fine-grained atomic evaluation of factual precision in long-form text generation. In Proceedings of EMNLP, Cited by: §2.
• Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-BERT: sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pp. 3982–3992. Cited by: §5.2.
• Shi et al. (2024) L. Shi, C. Ma, W. Liang, W. Ma, and S. Vosoughi Judging the judges: a systematic study of position bias in LLM-as-a-judge. arXiv preprint arXiv:2406.07791. Cited by: §2.
• Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, Cited by: §2.
• Si et al. (2025) C. Si, D. Yang, and T. Hashimoto Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers. In International Conference on Learning Representations, Cited by: §1.
• Smit et al. (2024) A. P. Smit, N. Grinsztajn, P. Duckworth, T. D. Barrett, and A. Pretorius Should we be going MAD? a look at multi-agent debate strategies for LLMs. In Proceedings of the 41st International Conference on Machine Learning, Cited by: §2.
• Su et al. (2025) H. Su, R. Chen, S. Tang, Z. Yin, X. Zheng, J. Li, B. Qi, Q. Wu, H. Li, W. Ouyang, P. Torr, B. Zhou, and N. Dong Many heads are better than one: improved scientific idea generation by a LLM-based multi-agent system. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pp. 28201–28240. External Links: Document Cited by: §2.
• Swanson et al. (2025) K. Swanson, W. Wu, N. L. Bulaong, J. E. Pak, and J. Zou The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies. Nature. External Links: Document Cited by: §1, §2.
• Yang et al. (2025) F. Yang, C. Ye, M. D. Ma, Y. Xiao, M. Yang, and W. Wang BioVerge: a comprehensive benchmark and study of self-evaluating agents for biomedical hypothesis generation. In NeurIPS Workshop on AI for Science, Cited by: §2.
• Zheng et al. (2023) L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, Cited by: §2, §5.2.
• Zhu et al. (2025) K. Zhu, H. Li, S. Wu, T. Xing, D. Ma, X. Tang, M. Liu, J. Yang, J. Liu, Y. E. Jiang, C. Zhang, C. Lin, J. Wang, G. Zhang, and W. Zhou Scaling test-time compute for LLM agents. arXiv preprint arXiv:2506.12928. Cited by: §2.

Appendix A Critique-Depth Effects Relative to Run-to-Run Variability

This analysis quantifies how much changing critique depth alters the delivered mechanisms beyond the run-to-run variability observed when the same critique setting is repeated. Holding the opening hypotheses fixed, we replay the downstream workflow with 0, 1, and 5 critique rounds and use a blinded LLM-as-a-judge to compare matched delivered hypotheses. For each run pair, divergence D⁡(r,r′) is the proportion of matched hypothesis pairs classified as different mechanisms. Extra divergence Δ⁡(K,K′) is the mean cross-depth divergence minus the mean divergence between same-depth reruns at the corresponding depths. Formal definitions are provided in Section 3.

For each of instances A–D, one donor run provides five opening hypotheses, which are held fixed across all replays while every downstream stage is rerun. Each critique depth is repeated five times per instance, yielding 15 runs per instance and 60 runs in total. The 15 runs produce 105 run-pair comparisons per instance and 420 across all four instances, comprising 4,863 matched hypothesis pairs.

For each instance and each pairwise depth comparison, the five runs from each depth are pooled, and all 126 unique partitions of the ten runs into two groups of five are considered. We recompute Δ for each partition, and the two-sided p value is the proportion of partitions that produce an absolute Δ at least as large as the observed value. For the aggregate result, this permutation is performed separately within each instance, after which the four instance-specific Δ values are averaged.

Evaluator sensitivity.

Opus 4.6 serves as the primary LLM-as-a-judge throughout this study. To assess whether the observed critique-depth pattern depends on the choice of judge model, we repeat the comparison using Opus 4.6, Sonnet 4.6, and GPT-5.5. For each judge, we recompute extra divergence, defined as cross-depth divergence relative to the corresponding same-depth rerun variability.

All three judges produce the same qualitative pattern (Figure 5). The 0-to-1 comparison ranges from +34.5 to +37.3 pp of extra divergence, and the 0-to-5 comparison ranges from +39.8 to +43.3 pp; both comparisons are significant under every judge (p<0.0001). In contrast, the 1-to-5 comparison ranges from +0.5 to +2.3 pp. Thus, across these judges, most cross-depth separation appears when critique is first introduced, with limited additional separation from later rounds.

Appendix B Pairwise Mechanism Comparison Methods

We compare three methods for determining whether a pair of hypotheses describes the same mechanism. Each method receives the same hypothesis statement, mechanistic explanation, and causal chain. In our TF–IDF implementation, vectors are constructed using sublinear term-frequency scaling and English stopword removal; pairs with cosine similarity of at least 0.25 are classified as the same mechanism. Dense embeddings are generated using amazon.titan-embed-text-v2, with pairs classified as the same mechanism when their cosine similarity is at least 0.77. This threshold is selected by matching the marginal same-pair rate to the range produced by the other methods, without using individual pair decisions as labels. The LLM-as-a-judge evaluator uses claude-opus-4-6-v1 at temperature 0 and directly returns a binary same/different decision. The LLM-as-a-judge is queried three times per pair, including one presentation-order reversal, and the majority decision is used. The full presentation-order sensitivity is examined in Appendix D, and evaluation rubric and its two prompt variants are in Appendix E.

Appendix C Controlled Perturbations for Grouping Decisions

Perturbation definitions.

Each perturbation pair consists of one generated hypothesis and one edited variant designed to test how the comparison methods respond to a specific change. Wording Only rewrites the mechanistic explanation while leaving the causal chain unchanged, whereas Wording + Chain also rewrites the chain using audited biological aliases while preserving the ordered entities and causal relations. Both are intended to preserve the same mechanism and test invariance to meaning-preserving changes. First Step Swapped replaces the initiating event while holding the downstream chain fixed and is treated as a different mechanism under the study rubric. Middle Step Swapped replaces one intermediate step while preserving the initiating event and terminal finding.

Generation and structural validation.

Variants are generated using claude-sonnet-4-6 at temperature 0.3 and retained only if they pass deterministic validation against the original causal chain. Wording Only must preserve all chain steps and regulatory directions, Wording + Chain must preserve the ordered entities after alias normalization, and the step-swapped variants must change exactly one step at the designated position. Constructions that fail these checks are excluded.

Appendix D Sensitivity Analyses for Pairwise Comparison Methods

Threshold sensitivity and cross-method agreement.

At the selected thresholds of 0.25 for TF–IDF and 0.77 for embeddings, the three comparison methods achieve 85%–89% raw pairwise agreement and chance-corrected agreement of κ∈[0.59,0.74]. Sweeping the TF–IDF and embedding thresholds changes their same/different decisions, but no evaluated threshold produces complete agreement among the methods; the highest chance-corrected agreement with another method is κ=0.74. Thus, the observed differences among the methods are not attributable to a single threshold setting.

Judge-model and presentation-order sensitivity.

On the 180 pairs drawn from the generated hypothesis set, repeating the full-rubric LLM judgment at temperature 0 in the same presentation order changes none of the decisions, whereas reversing the order of the two hypotheses changes 5 decisions (3%). Repeating the comparison with three judge models yields 96.1%–98.3% pairwise agreement and κ=0.86–0.94. These results indicate high consistency across repeated evaluations and judge models, with a small residual dependence on presentation order.

Rubric and order interactions.

We compare the full, step-by-step, and no-rubric prompts on the same controlled perturbation pairs while holding the judge model, temperature, sampling procedure, and rendered inputs fixed (Figure 4). Exact repetition in the same order changes no decisions under any prompt. Presentation-order reversal changes 7 decisions under each prompt, but the affected pairs differ: two are shared between the full and step-by-step rubrics, and none are shared with the no-rubric prompt.

Appendix E LLM-as-a-judge Prompts

E.1 Critique-engagement judge

The critique-engagement judge decomposes each critique into individual demands and classifies the proposer’s response to each demand as conceded, answered, partially addressed, or dropped.

E.2 LLM-as-a-judge and rubric variants

The LLM-as-a-judge determines whether two hypotheses describe the same or different mechanism. We present the full rubric used throughout the pairwise analyses, a step-by-step variant that records component-level judgments before the overall verdict, and a no-rubric variant that provides only the comparison task.

注意转载授权 · 阅读原文