Can We Believe AI Peer Review?

   Haoyuan Li    Zhongsheng Wang Affiliation: Gillian Dobbie    Jiamou Liu Affiliation: University of Auckland Affiliation: Auckland, New Zealand

Abstract

As large language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer’s judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.

1 Introduction

As large language models become integrated into research workflows, AI-assisted peer review is moving from experimental studies into practical use. Submission volumes at major AI conferences have continued to grow, increasing the pressure on peer review. Conferences including ICLR and AAAI have begun exploring AI assistance in the review process (Zou and Thakkar, 2025; AAAI, 2025). Meanwhile, researchers are using AI review tools to assess paper quality, obtain revision suggestions, and judge the effects of revisions through changes in scores. Both applications rest on a shared premise: AI judgments are reliable enough to provide meaningful evidence for research decisions.

This premise does not necessarily hold. First, AI scores are stochastic: even when the paper, model, and review protocol remain fixed, repeated reviews can yield different scores, making an individual score or an observed gain an unreliable representation of the model’s own assessment. Second, AI judgments depend on the model: reviewers may have different evaluation preferences and, even when they assess original papers similarly, may not endorse the same revisions. Ignoring these issues can lead users to mistake random fluctuations for differences in paper quality and score gains under a particular model for broadly endorsed improvements. How much, then, can we trust AI scores and the gains observed after revision?

Prior work has examined AI reviewers’ evaluation capabilities, scoring biases, and sensitivity to paper presentation. ReviewEval (Garg et al., 2025), TitleTrap (Du, 2025), and LLM-as-a-Reviewer (Li et al., 2026a) study different aspects of AI reviewing, while Rhetoric (Li et al., 2026b) further reveals differences in reviewers’ responses to revision gains. Practical use of AI scores nevertheless calls for a connected set of answers: How reliable is a single score? How many reviews are needed to estimate a reviewer’s expected score accurately? Can score estimation be improved under a limited reasoning budget? Does an observed gain reflect random fluctuation, or an improvement endorsed by a particular reviewer? These questions determine when AI scores can serve as research feedback.

To address these questions, we conduct approximately 14,000 AI reviews of 200 papers from ICLR 2025 and ICLR 2026, systematically studying scoring randomness, estimation accuracy, and cross-model agreement on revision gains. Our main contributions are:

Quantifying the reliability of a single AI score. We use repeated sampling to analyze score fluctuations and their effects on relative paper rankings, then derive sufficient review budgets for specified mean-score error and success-probability targets.

Exploring methods to improve scoring reliability. We examine how score granularity affects variability and systematic shifts, and test whether auxiliary score probabilities improve distribution estimation under a limited reasoning budget.

Assessing the credibility of revision gains. Through repeated scoring and cross-model evaluation, we separate scoring fluctuations from reviewer dependence and test whether different reviewers consistently endorse the same revisions through score gains.

2 Related Work

AI peer review. Existing studies primarily assess LLM reviewing ability through the quality of review comments. ReviewEval (Garg et al., 2025) examines factuality, analytical depth, and the actionability of suggestions; LLM-as-a-Reviewer (Li et al., 2026a) compares model and human reviews and tests resistance to prompt injection. These studies help establish whether LLMs can generate high-quality reviews, but devote less attention to how review outcomes inform subsequent decisions. In particular, the extent of score variation when the same paper is reviewed repeatedly, and whether these scores reliably distinguish paper quality before and after revision, remain understudied.

Score stability and estimation. Prior work finds inconsistencies across repeated LLM scoring runs and systematic biases (Stureborg et al., 2024); output formats and explanation requirements also affect evaluation results (Chiang and Lee, 2023). To address scoring uncertainty, G-Eval (Liu et al., 2023) uses score-token probabilities, while Homan et al. (2026) examine how many repeated ratings statistical tests require. However, existing work rarely directly answers how much a single score differs from a mean over repeated scores when the reviewer and reviewing conditions are fixed. We target the expected score and score distribution under fixed reviewing conditions, analyze how repeated sampling affects estimation precision, and evaluate the gains from probability-assisted estimation against independently sampled target scores.

AI-assisted revision. Beyond evaluating papers, LLM scores can be used to select and guide revisions. TitleTrap (Du, 2025) and Rhetoric (Li et al., 2026b) show that changes to titles and rhetorical presentation can affect AI scores. A score increase may therefore reflect either an improvement in substantive paper quality or a selector’s adaptation to particular scoring preferences. Existing work has not sufficiently separated these sources. We hold the candidate edit set fixed and vary the selector and scorer separately, comparing score gains under shared revision content. This tests whether LLM-score-guided revision strategies remain stable across reviewers and how much of the observed gains can be attributed to sampling uncertainty.

3 Problem Formulation and Study Design

3.1 What Does a Review Score Estimate?

Score distributions and repeated sampling. We treat one LLM paper-review procedure as one sample. Given a paper Pi and review configuration r (model, prompt, reasoning settings, and execution environment), an individual score is a random variable Yr​(Pi) with expectation μr,i and variance σr,i2. For the same paper, we perform k independent reviews and use their mean Y¯r,i,k to estimate the expected score under the fixed configuration. Under IID scores within each paper, this mean has variance σr,i2/k.

Evaluation validity. We use the mean human review score Hi as an external reference for paper quality. Correlation and error relative to this reference, together with AUC for acceptance status, serve as indicators of evaluation validity.

Scoring reliability. For a fixed paper and review configuration, scores should exhibit low randomness, and their expectation should be reliably estimable from a finite number of repeated reviews. We report both score stability and relative-ranking stability as indicators of scoring reliability.

Evaluation validity for revisions. For a revision Pi→Pi′, the reviewer’s expected gain is Δr,i=μr​(Pi′)−μr​(Pi), estimated by the difference between independent mean scores assigned by the same Scorer to the original and revised manuscripts. Gains from different Scorers are compared on identical revised versions. For fixed original and revised manuscripts, independent scores on the two sides give the gain estimate a variance of (σr,i2+σr,i′2)/k. An observed score increase contains sampling error from both versions. In the crossover experiment, we score each version repeatedly and keep the Scorer of the original-paper baseline consistent, establishing separate controls for within-model fluctuations and between-model response differences.

3.2 Papers and Review Configurations

We draw experimental samples from public ICLR 2025 and ICLR 2026 submissions by stratifying jointly on official subject area and mean human score interval, selecting 100 papers per year. The paper lists and configurations are provided in the supplementary material.

For LLM paper review, PDFs are converted into ordered text and images, retaining the main text, figures, tables, and appendices; review inputs are anonymized. A common task asks models to assess novelty, technical correctness, experimental evidence, and limitations, then produce an overall score from 1 to 10 with review comments.

Original integer scoring uses GPT-5.6-Sol, GPT-6-Luna, and GPT-6-Sol, with ten reviews per paper per configuration, totaling 6,000. The granularity experiment compares integer and forced-noninteger protocols for the latter two models. Distribution estimation targets GPT-5.6-Sol reasoning scores. Revision experiments use Luna and Claude Haiku 5.5, with three scores each for original and revised manuscripts; one paper has 224 images, exceeding Claude’s input limit, so crossover comparisons use the shared 199 papers. Additional configuration details are provided in the supplementary material.

4 How Reliable Is a Single AI Review?

We first characterize score variation and validity for individual papers, then test whether independent reviews reproduce the same paper rankings. We further estimate ranking reliability under finite sampling and recommend review budgets.

4.1 Repeated Scores and Human References

Table 1 summarizes ten-score repeatability and external associations. Across configurations, 30.0%–56.5% of papers receive ten identical scores, while 91.5%–100% have a score range of at most one point. Fluctuations mainly occur between adjacent categories. GPT-6-Sol has a within-paper SD of 0.272, compared with 0.374 and 0.386 for the other configurations, and a Spearman correlation of 0.409 with human scores. Final-acceptance AUC is 0.7134 for GPT-5.6-Sol, 0.6484 for GPT-6-Luna, and 0.7138 for GPT-6-Sol.

Within-paper SD averages unbiased paper-level variances before taking the square root. Each set of ten scores forms 45 unordered pairs; exact agreement is 72.86%, 72.36%, and 85.19%, respectively. GPT-6-Sol stays within a one-point range for all 200 papers, including 113 with ten identical scores.

4.2 Do Independent Reviews Reproduce Rankings?

We form disjoint equal-sized blocks from each paper’s ten scores, comparing means of one through five reviews. Enumerating all unordered block partitions gives 45, 630, 2,100, 1,575, and 126 partitions, respectively. Each configuration covers 9,900 within-year paper pairs, averaged over all partitions. Rankings are classified as the same strict order, a tie in at least one block, or reversed strict orders.

More reviews mainly improve strict ranking reproducibility by reducing ties. From one to five reviews per block, strict agreement rises from 62.14% to 84.47% for GPT-5.6-Sol and from 63.00% to 85.71% for Luna. GPT-6-Sol rises from 57.52% to 74.13%, with 25.08% of pairs still tied in at least one five-score block. Its reversal rate is 0.79%, below 2.15% and 2.44% for the other configurations. Finer means establish more strict comparisons, including reversals of pairs previously tied. We next specify a sampling target through absolute mean-score estimation error.

4.3 How Many Reviews Are Enough?

Weighting the sampled papers equally, we define the overall success rate of k new reviews as

We assume IID scores within each paper, independence across papers, and future scores independent of the pilot. We require A⁡(k,ϵ)≥0.95. This means that, for a randomly selected paper, the mean of repeated reviews falls within ϵ points of the model’s expected score with probability at least 95%. Let V=N−1​∑iσi2. Under independent and identically distributed reviews, Chebyshev’s inequality gives

This budget depends only on the average variance and requires neither normally distributed scores nor a known score distribution. Substituting the average sample variance V^ from ten pilot scores gives planning budgets of 45, 48, and 24 reviews at a 0.25-point tolerance, and 12, 12, and 6 reviews at a 0.5-point tolerance.

Confidence control with finite pilot data. We also use the known score range [1,10] to construct an upper confidence bound U for V. The sample variance of ten scores for each paper equals the average half-squared difference over all score pairs. Treating it as a U-statistic over unordered score pairs, we obtain a one-sided 95% upper bound depending only on V^ through random disjoint pairing and a KL-Chernoff bound (Hoeffding, 1963). Each model has an effective pairing count of 200​⌊10/2⌋=1000; the 9,000 score pairs that share observations are not treated as independent. Specifically, let B=(10−1)2/2=40.5 and M=1000, and solve for the root above V^:

where kl⁡(p,q)=p​log⁡(p/q)+(1−p)​log⁡((1−p)/(1−q)). This inversion satisfies Pr⁡(V>U)≤0.05. The key step expresses all overlapping score pairs as the conditional expectation of random disjoint matchings, then uses Jensen’s inequality to obtain the moment-generating-function bound for 1000 independent bounded kernels. Substituting U into the same budget formula gives a sufficient budget for an overall success rate of 95% on future new reviews, with 95% pilot confidence for each model.

The two budgets correspond to different information requirements: planning budgets reflect the scale of observed variance, while confidence-controlled budgets also cover score tails that have not been sufficiently observed. Halving the tolerance approximately quadruples both budgets. In practice, the budget can be selected according to the error tolerance and confidence requirement. The bound’s construction, proof, and simultaneous-control results for all three models appear in the supplementary material.

Figure 3 plots the budgets for the three configurations at a tolerance of 0.5 points as lower bounds on the success rate. Dashed curves substitute observed variances, and solid curves use their upper confidence bounds, corresponding to the planning scale and confidence-controlled sufficient budgets, respectively. The curves are calculated from the variance relationship without resampling the observed frequencies of the ten scores.

5 Can We Improve Scoring Reliability?

Repeated reviews improve the reliability of scores and paper rankings, but reaching reliable judgments about closely scored papers may require many independent calls. Can AI-score reliability be improved without substantially increasing review costs?

5.1 Finer Scores Change Both Variance and Location

If high-quality peer review is viewed as a ranking of paper quality, that ranking need not be inherently discrete. Scores, however, are discrete integers. This motivates testing whether finer output granularity improves model stability. We fix paper inputs and all other review requirements, replacing only the integer protocol with a 1–10 score required to have a nonzero fractional part. Luna and Sol each produce five repeated scores for all 200 papers.

Score variability decreases for both models, while their means rise by 0.693 and 0.266, respectively. Luna’s MAE relative to mean human scores also decreases, whereas Sol’s MAE remains largely unchanged. The decimal protocol thus changes both location and variability on the scoring scale. To estimate the original integer reviewer, we next retain its protocol and use additional information to improve estimation.

5.2 Estimating the Same Reviewer with Fewer Reasoning Calls

Beyond comparing integer and decimal scoring, we evaluate two simple strategies for estimating the integer score distribution pi of a fixed GPT-5.6-Sol reasoning configuration with fewer reasoning calls. Quick extracts token probabilities for candidate integer scores from one non-reasoning call, normalizes them, adjusts their concentration using temperature T, and calibrates score locations through an affine mapping s↦a+b​s. Mapped locations are clipped to [1,10], and probability mass at non-integer positions is allocated linearly to the two adjacent score categories, yielding qQ. Standard combines this estimate with the empirical distribution of two target reasoning scores: qS=w​p^2+(1−w)​qQ.

We use the 198 papers with available auxiliary probabilities and evaluate both strategies over 30 random paper-level splits, each containing 99 training and 99 test papers. Quick uses the first valid non-reasoning output for each paper. Its parameters T,a,b are selected by minimizing Brier loss against the training papers’ reasoning scores, over a grid of six temperatures, 13 offsets, and three slopes. With these parameters fixed, Standard selects w from eight candidate weights by enumerating all ten ways to divide each training paper’s five reasoning scores into two inputs and three independent targets. Evaluation on test papers follows the same input–target separation, with all parameters fixed from the training set.

Risk evaluation with independent targets. For a predicted distribution q and independent target scores Z1,…,Zm, we compute

Conditional on q, its expectation is ‖q−pi‖22. The collision rate between target scores estimates ‖pi‖22, so variability from the finite target sample is not directly added to prediction risk. For n direct reasoning samples, risk is (1−‖pi‖22)/n, also estimated using the collision rate of independent scores.

Standard reduces risk by 0.0388 relative to two direct reasoning samples, with a paired 95% interval of [-0.0524, -0.0255]; its risk difference from three reasoning samples is +0.0018 [-0.0057, 0.0095]. Auxiliary probabilities provide useful information alongside two reasoning samples, reaching a risk level close to three direct reasoning samples. Quick’s risk is close to that of a single reasoning sample, and fusion is more effective than replacing the scorer alone. The table reports the call composition separately; actual dollar costs depend on input length and caching.

The effect of fusion on risk can also be expressed directly. If the auxiliary call and the two reasoning scores are conditionally independent, and the empirical reasoning distribution is unbiased for pi, then 𝔼​‖qS−pi‖22=w2​(1−‖pi‖22)/2+(1−w)2​𝔼​‖qQ−pi‖22. The reasoning term captures direct-sampling error, while the auxiliary term captures the discrepancy between the calibrated distribution and the target. Selecting the weight on training papers allocates risk between the two terms; debiased test-set evaluation assesses whether that choice remains effective on papers excluded from fitting.

5.3 Independent-Target Validation

A separate existing experiment collects ten additional independent reasoning scores for five papers from each year, using subsets of the original five scores for prediction. Quick and Standard are fitted on the other 188 papers. Standard obtains a Brier score of 0.353 and Wasserstein-1 distance of 0.19; three direct reasoning samples obtain 0.379 and 0.18, respectively. The independent targets give a similar estimation pattern to the main experiment. These losses include finite-target variability and are reported separately from the debiased risk in Table 5; Table 6 retains all five baselines. This ten-paper evaluation uses new target scores and separates evaluation papers from parameter training.

6 Can We Trust Revision Gains?

Accurately estimating one reviewer’s expected score does not establish whether other reviewers endorse its judgments. This matters particularly for AI-assisted manuscript revision: when a revised paper receives a higher score, can we trust that increase to reflect an effective improvement rather than a particular reviewer’s preferences? Through controlled manuscript revision and cross-model reviewing, we study whether different reviewers respond consistently to the same edits and how much of the observed disagreement can be explained by scoring randomness.

6.1 Holding Revisions Fixed

With more precise score estimation, we examine another use case: AI scores as revision feedback. The workflow has three roles: the Writer generates candidate text edits from the original manuscript and existing reviews; the Selector decides whether to accept each edit; and the Scorer evaluates the complete manuscript after edits are applied. The Scorer does not receive selection rationales. Revisions are required to preserve facts, experimental data, and non-text materials.

Candidate edits generated by Luna are selected by Luna or Haiku and then scored by both, forming a complete 2×2 crossover. Haiku-generated edits add three role combinations, for seven configurations in total. Each Scorer independently scores the original and revised manuscripts three times each, and all comparisons use the same 199 papers. Of Luna’s 9,327 candidate edits, 8,459 are accepted by both Selectors, with a mean per-paper acceptance-set Jaccard similarity of 0.919; the final versions still differ for 182 papers. This lets us compare selection effects with fixed candidate edits and scoring effects with fixed final versions.

6.2 Opposite Gains for the Same Revisions

For identical revisions generated and selected by Luna, Luna assigns a mean gain of +0.474 and Haiku -0.348, a difference of 0.822 [0.717, 0.928]. Replacing the Selector with Haiku still yields +0.472 and -0.296, respectively; the cross-scorer gap changes by only -0.054 [-0.116, 0.010]. Figure 4 shows the separation in scorer responses under the same selection condition.

Luna also assigns positive average gains to Haiku-generated revisions, while Haiku assigns its own selected revisions a gain of +0.050. The result is asymmetric: Luna endorses revisions from both Writers; Haiku assigns negative gains to Luna’s revisions and near-zero gains to its own. Changing the Selector has a smaller effect on the aggregate results for Luna’s candidate edits. Across these two sets of Luna revisions, the final Scorer changes the direction of the gain.

These controls serve different purposes. Holding the final manuscript fixed and changing the Scorer directly measures differing responses to the same textual edits; holding the Writer’s candidate edits fixed and changing the Selector tests whether selection changes this gap. The two Selectors accept many of the same edits. Even when final versions differ, Luna and Haiku assign opposite aggregate gains under both selection conditions. Cross-reviewer disagreement can thus be studied without comparing two revisions with different content.

6.3 Agreement on Originals Does Not Predict Agreement on Gains

The two Scorers have a Pearson correlation of 0.804 on original-paper means, but their paper-level gain correlations for identical revisions fall to 0.415, 0.407, and 0.085. Gain directions agree exactly for 28.1%, 30.2%, and 41.2% of papers, respectively, including cases in which both assign zero gain.

We estimate the noise contribution from repeated scores: the noise variance of original-paper means is so2/3, and that of gains is (so2+sv2)/3. We subtract these components from the corresponding between-paper variances while retaining the cross-model covariance. Adjusted correlations for Luna revisions are approximately 0.52, still below 0.843 for original papers. Scoring fluctuations explain part of the correlation decline, while opposite aggregate gains for fixed revised versions directly demonstrate differing reviewer responses.

7 Conclusion

Using AI scores as research feedback involves two inferences: from individual outputs to expected judgments, and from those judgments to revision gains. Repeated reviews improve mean-score precision, with budgets determined by average variance and error tolerance and further controlled for uncertainty in finite pilot data. Under a fixed scoring protocol, auxiliary probabilities combined with a small number of reasoning scores improve distribution estimation; changing output granularity alters both variability and the scoring center. The revision crossover further shows that even highly correlated original-paper judgments can lead to gains in opposite directions for identical revisions. Score precision clarifies one reviewer’s judgment, while cross-reviewer evaluation of shared revisions tests whether this feedback receives consistent endorsement.

Limitations

Sampling budgets target new independent scores under a fixed configuration; changes to model versions, prompts, or service environments change the target. Confidence bounds use the full 1–10 range and are conservative when tail observations are limited. Human scores come from current public records, and PDFs may be updated versions; the two years use different rating categories, and correlations do not directly observe latent quality. Calibration is validated on one API configuration, while the revision crossover includes only two models and omits one role combination. Factual preservation in revisions is required by the editing protocol but has not been validated through blinded human assessment of compliance and quality gains; using existing reviews to generate edits may also affect the independence assumption in noise adjustment. Experiments use public anonymized manuscripts. The findings analyze scoring behavior; automated editing and scoring do not replace authors’ responsibility for paper content.

References

• AAAI (2025) AAAI AAAI launches AI-powered peer review assessment system. Note: Association for the Advancement of Artificial Intelligence External Links: Link Cited by: §1.
• Chiang and Lee (2023) C. Chiang and H. Lee A closer look into using large language models for automatic evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 8928–8942. External Links: Link, Document Cited by: §2.
• Du (2025) S. Du TitleTrap: probing presentation bias in LLM-based scientific reviewing. In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems, M. Akter, T. Chowdhury, S. Eger, C. Leiter, J. Opitz, and E. Çano (Eds.), Mumbai, India, pp. 119–125. External Links: Link, Document, ISBN 979-8-89176-305-0 Cited by: §1, §2.
• Garg et al. (2025) M. K. Garg, T. Prasad, T. Singhal, C. Kirtani, M. Mandal, and D. Kumar ReviewEval: an evaluation framework for AI-generated reviews. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, pp. 20542–20564. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §1, §2.
• Hoeffding (1963) W. Hoeffding Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association 58 (301), pp. 13–30. External Links: Document, Link Cited by: §D.3, §4.3.
• Homan et al. (2026) C. M. Homan, F. Korn, D. Pandita, and C. Welty How many ratings per item are necessary for reliable significance testing?. In Findings of the Association for Computational Linguistics: EACL 2026, V. Demberg, K. Inui, and L. Marquez (Eds.), Rabat, Morocco, pp. 4258–4273. External Links: Link, Document, ISBN 979-8-89176-386-9 Cited by: §2.
• Li et al. (2026a) L. Li, J. Xiong, C. Zhu, R. Yu, C. Chen, J. Wang, R. Ma, and Z. Lu LLM-as-a-reviewer: benchmarking their ability, divergence, and prompt injection resistance as paper reviewers. External Links: 2605.25415, Document, Link Cited by: §1, §2.
• Li et al. (2026b) M. Li, C. Wang, X. Li, X. Zeng, D. Li, P. Shi, D. Zhou, and T. Zhou How can rhetoric reward-hack AI reviewers? dissecting rhetorical sensitivity in AI-based peer review. External Links: 2608.08975, Document, Link Cited by: §1, §2.
• Liu et al. (2023) Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu G-eval: NLG evaluation using Gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 2511–2522. External Links: Link, Document Cited by: §2.
• Stureborg et al. (2024) R. Stureborg, D. Alikaniotis, and Y. Suhara Large language models are inconsistent and biased evaluators. External Links: 2405.01724, Document, Link Cited by: §2.
• Zou and Thakkar (2025) J. Zou and N. Thakkar Leveraging LLM feedback to enhance review quality. Note: ICLR Blog External Links: Link Cited by: §1.

Appendix A Data and Sampling

A.1 Submission Populations and Sample Composition

The study includes 100 papers each from ICLR 2025 and ICLR 2026. The sampling populations contain 11,520 and 19,474 submissions with publicly available official reviews, respectively, including accepted, rejected, withdrawn, and desk-rejected papers. The paper lists were fixed before downloading PDFs or running AI reviews. AI scores, PDF availability, and overlap with earlier samples played no role in selection.

The 2025 sample was stratified jointly by official primary subject area and mean human-score bin. The 2026 sample uses the second version of the sampling design, which adds submission status to these dimensions.

A.2 Stratified Quotas and Within-Stratum Sampling

For 2025, integer quotas for three nested sample sizes of 100, 300, and 1,000 were jointly solved using SciPy milp and HiGHS. For each subject area, score bin, and nonempty area-by-score-bin stratum, the sample count was constrained to the floor or ceiling of its population share multiplied by the sample size, keeping the count discrepancy strictly below one paper. Additional constraints ensured coverage of subject areas and nonempty score bins. Within each stratum, papers were sorted by their exact mean human score, with random ordering of ties, and selected by systematic sampling with a random start. The sample of 1,000 was drawn first, followed by 300 from that sample and then 100 from the 300.

The second 2026 design uses area-by-score-bin-by-status strata. Status categories are Poster, Oral, rejected, withdrawn, and desk rejected. Status quotas were assigned by the largest-remainder method, with the total number of accepted papers fixed at 27. Area, score-bin, and cross-stratum quotas retained the floor/ceiling constraints, and within-stratum selection again used systematic sampling after sorting by exact mean score. Section A.3 reports the final status quotas and counts.

The human mean for each paper is the unweighted average of its valid official review scores. Bins are assigned using the exact, unrounded mean. Nonempty bins range from [1,2) to [9,10] in 2025. The 2026 population includes means below 1, with nonempty bins from [0,1) to [8,9); its [9,10] bin is empty. The complete distributions follow.

A.3 Subject-Area and Submission-Status Distributions

All 21 official primary subject areas are represented. The table retains the original area names without merging small areas.

The 2025 sample contains 36 accepted, 41 rejected, and 23 withdrawn papers. The 2026 sample contains 27 accepted, 43 rejected, 26 withdrawn, and 4 desk-rejected papers. Together, these comprise 63 accepted, 84 rejected, 49 withdrawn, and 4 desk-rejected papers.

Quota rounding allows rare categories to enter a small sample. For example, only nine papers in the 2025 population have a mean human score of at least 9, and one is retained in the sample of 100: a sample share of 1% versus a population share of approximately 0.078%. Coverage of subject areas and score bins does not imply coverage of every small cross-stratum; some receive a rounded quota of zero. Results give equal weight to the selected papers, without reweighting to population shares.

A.4 Human Scores, PDF Versions, and Acceptance Labels

Human scores are taken from the public records in the frozen data and may include post-rebuttal updates. Paper PDFs may also be updated versions. Human score options are 1, 3, 5, 6, 8, and 10 for 2025, and 0, 2, 4, 6, 8, and 10 for 2026. Section C therefore also reports associations and errors separately by year.

Acceptance labels are derived from final public status: accepted_* is labeled 1, and rejected, withdrawn, and desk-rejected papers are labeled 0. The primary analysis uses all 200 papers, with 63 positives and 137 negatives. Excluding withdrawals leaves 151 papers: 63 positives and 88 negatives. The full and withdrawal-excluded samples contain 100 and 77 papers in 2025, and 100 and 74 in 2026. Positive-negative pairs with equal predicted scores receive a contribution of 0.5 to AUC.

A.5 Shared Samples and Paper Lists Across Experiments

The two papers missing auxiliary probabilities differ from the paper affected by Claude’s image limit. The full calibration sample contains 198 papers, with 63 accepted and 135 nonaccepted. Excluding its 48 withdrawn papers leaves 150 papers, with 63 accepted and 87 nonaccepted. The revision experiment uses a fixed shared set of 199 papers: 100 from 2025 and 99 from 2026.

Scoring models receive anonymized manuscript content. Human scores, acceptance status, and other models’ results remain in the analysis metadata.

Appendix B Review and Revision Protocols

B.1 Models and Review Settings

Each experiment fixes its review configuration. Sections C and D use ten original-manuscript scores per paper, whereas distribution estimation in Section E uses a separately stored set of five target reasoning scores per paper.

Auxiliary probability calls use reasoning.effort="none", top_logprobs=20, and max_output_tokens=4096, and request output-token log probabilities. Target reasoning scores use the same manuscript input and scoring schema, with reasoning.effort="medium" and an output budget of 16,000 tokens. Temperature is not explicitly set in the requests.

Each review runs in an independent session. Models can access only the preloaded manuscript text and images; tools, web search, memory, and prior context are disabled. Analyses use valid results that pass format and execution checks.

B.2 Manuscript Inputs and Image Processing

Each original manuscript is represented by text.json, index.json, and figures/. Text records contain a stable id, PDF page number, record type, and text. The sequence in index.json specifies the interleaving of text and images. The input loader follows this sequence to assemble the full text, table text, references, and appendices. Images are loaded after relative-path and SHA-256 checks. Adjacent text records may be merged without changing their order.

All scorers receive preloaded text and images. Haiku processes images in the same order, resizing the longest side to 1,568 pixels and re-encoding them before submission, with a limit of 100 images per call. Paper 2026-paper_067 contains 224 images and exceeds this limit, so the revision crossover experiment uses the remaining 199 papers.

Across the 200 input packages, PDF page markers total 4,923 pages and image-index entries total 2,584. The median, 90th percentile, and maximum page counts are 23, 38, and 69; the corresponding image counts are 9, 23, and 224. There are 133,967 text records containing 15,324,311 characters. Page counts come from PDF page markers in the packages, and image counts from index entries rather than individual figure or table panels.

B.3 Integer-Scoring Task

The scorer acts as an ICLR reviewer, reads the complete anonymized manuscript with interleaved images, evaluates novelty, technical correctness, experimental evidence, and limitations, and returns an integer overall score from 1 to 10. The anchors are 1 for strong reject, 3 for reject, 5 for marginally below the acceptance threshold, 6 for marginally above it, 8 for accept, and 10 for strong accept. Other intermediate integers are permitted.

The output contains an overall score, confidence from 1 to 5, a brief summary, up to three strengths, up to three concerns, and a scoring rationale. Key judgments cite PDF page numbers. The scorer does not receive human scores, final decisions, or other reviews. Confidence is not used to weight scores.

B.4 Forced-Decimal Protocol

The decimal experiment keeps the inputs, models, and other requirements unchanged, replacing the integer requirement with a mandatory nonzero fractional part. The score field changes from integer to number, while its range remains 1–10. Local validation additionally checks that the score is noninteger. All 1,000 valid scores from each model satisfy this requirement.

An earlier pilot that merely allowed decimals was discarded and is excluded from all analyses. The full forced-decimal experiment reused the 20-paper pilot run under the same protocol, then incorporated the remaining papers to obtain five scores each for 200 papers per model. The integer controls were verified paper by paper and round by round against the first five scores in the ten-review original-manuscript experiment, matching the number of repetitions.

B.5 Inputs to the Writer, Selector, and Scorer

The Luna Writer uses the first existing GPT-5.6-Sol review as feedback; source records identify this configuration for all 200 papers. The Haiku Writer uses Haiku’s first review of the original manuscript. The two Writer conditions thus follow their respective feedback pipelines. With the Writer fixed, both Selectors read the same candidate diffs. With the final manuscript fixed, both Scorers read the same text version. Scorers receive no candidate diffs, selection rationales, other scorers’ scores, or comparative prompt identifying the manuscript as revised.

The Writer returns text records to replace. The Selector gives an acceptance decision and a reason for each candidate edit, with exactly one decision per candidate. The program validates record IDs, decision counts, and output format before generating the final manuscript.

The Writer is instructed to preserve facts, numerical values, data, evidential scope, record order, page numbers, and nontext fields. Leading instructions and page markers must remain unchanged. Returned text is merged into a candidate manuscript, from which block-level diffs are generated. Each Selector comment is limited to 32 characters; truncations of longer Claude comments are recorded. The final manuscript is copied from the original records, replacing only the text of accepted edits with candidate text. Section F.5 reports structural checks.

Appendix C Extended Score Reliability Results

C.1 Metrics and Aggregation Units

Let the ten scores for paper i be Yi​1,…,Yi,10, with mean Y¯i. Its unbiased within-paper variance is

The mean within-paper variance is V^=N−1​∑iSi2, and the within-paper SD reported here is V^. The SD of all scores additionally includes differences in mean scores between papers. The identical-10/10 rate is the fraction of papers with ten identical observations, and the range is the maximum minus the minimum score. Pairwise exact agreement is first averaged over the (102)=45 unordered score pairs within each paper, then across papers.

All GPT-6-Sol papers have a score range of at most one point, and 113 have ten identical scores. The corresponding identical-score counts are 60 for GPT-5.6-Sol and 65 for Luna. Year-specific results show the same configuration differences: Sol’s within-paper SD is 0.2614 in 2025 and 0.2824 in 2026.

C.2 Association with Human Scores and Final Acceptance

Pearson correlation, Spearman correlation, MAE, and RMSE compare each paper’s ten-review mean with its public mean human score. AUC uses the same ten-review mean to predict the acceptance labels defined in Section A.4.

Spearman correlations range from 0.4017 to 0.5457 in 2025 and from 0.1572 to 0.2345 in 2026. Full-sample AUCs range from 0.6756 to 0.7852 in 2025 and from 0.6010 to 0.6185 in 2026. These year-specific results use the scores and labels for the corresponding year.

C.3 Score Concentration and Paper-Level Variance Distributions

Luna’s scores are concentrated at 2–4. Sol assigns a score of 5 in 1,042 cases, or 52.1% of its 2,000 scores. Scores from 5 to 7 account for 65.20% of Sol’s outputs, most of them at 5 or 6. The range of repeated scores and numerical separation between papers therefore describe different aspects of scoring.

C.4 First Five Versus Last Five Reviews

The ten reviews combine historical scores with later runs under the same configuration. The table separates rounds 1–5 from rounds 6–10, computes the difference between batch means within each paper, and bootstraps papers.

For all three models, the interval for the overall difference between batches includes zero. This analysis records shifts between batch positions; the precision budgets in Section D concern the expectation of future independent scores under a fixed configuration.

C.5 Ranking Reproducibility Across All Ten Reviews

For each paper, we construct two nonoverlapping blocks of k scores from its ten reviews. Treating the pair of blocks as unordered gives

The resulting numbers of partitions are 45, 630, 2,100, 1,575, and 126. For each partition, block means are compared between papers from the same year, then averaged over partitions and paper pairs. The two years yield 2​(1002)=9,900 distinct paper pairs.

Same denotes the same strict ordering in both blocks; Tie denotes a tie in at least one block; and Reverse denotes opposite strict orderings. These categories are mutually exclusive and exhaustive.

C.6 Close-Pair Analysis with Independent Grouping Scores

To examine close paper pairs, we group pairs by the absolute difference between their first-five means, reserving the last five scores for ranking comparisons. Among the last five, single-review blocks enumerate all 10 partitions, and two-review blocks enumerate all 15 partitions. Grouping and ranking comparisons thus use separate scores.

The first-five mean differences are grouped as 0, (0,0.4], (0.4,1], (1,2], and greater than 2 points. These differences are finite-sample estimates. Same, Tie, and Reverse below measure reproducibility between two subsequent independent review blocks, without treating the grouping estimate as the correct ordering.

For pairs whose first-five mean difference lies in (0,0.4], subsequent single-review comparisons produce ties in 74.84%–85.01% of cases; with two-review means, this remains 55.73%–68.45%. For the (1,2] group, strict same-order rates with two-review means reach 97.95%–99.58%. Improvements averaged across all pairs therefore combine contributions from different score-gap regions.

Each paper participates in multiple pairs, creating dependence across pairs. We enumerate all partitions exactly and report descriptive proportions, without treating the 9,900 paper pairs as independent observations for binomial intervals.

C.7 Semantic Consistency in the Existing Review Corpus

The existing text-analysis corpus contains five original-manuscript reviews each from Luna and Sol and three from Haiku, for a shared set of 199 papers. It excludes the later rounds 6–10. We use every available round in this fixed corpus: five reviews yield 10 within-model pairs, three yield 3, and cross-model comparisons yield 25 or 15 pairs.

Each strength and concern is encoded with sentence-transformers/all-mpnet-base-v2. For two lists of the same polarity, we compute the pairwise cosine-similarity matrix and use the Hungarian algorithm for one-to-one matching. The sum of matched cosines is divided by the longer list’s length, with unmatched items contributing zero:

All strength and concern lists in the corpus are nonempty. Review-pair values are first averaged within each paper and condition, then across papers. The table covers 15,522 review pairs. Confidence intervals retain the original analysis’s 5,000 paper-level bootstrap resamples.

Repeated reviews from the same model have closer opinion lists on average, and concern lists are closer than strength lists. An existing assistant inspection of 40 matched opinion pairs found shared cores, partial overlaps, and complementary observations; it does not constitute a human blind-evaluation gold standard. General-purpose NLI also misclassified some compound technical statements. We therefore retain the reproducible continuous semantic measure rather than converting automatic entailment thresholds into rates of substantive agreement.

Appendix D Review Budgets and Mean-Score Precision

D.1 Average Success Probability Across Papers

We give equal weight to a fixed set of N=200 papers. For paper i, let μi and σi2 denote the mean and variance of scores under a fixed configuration. Let Y¯i,k be the mean of k future independent scores, and define

The target is A⁡(k,ϵ)≥1−β. This is the overall probability that the mean of k scores estimates the model’s expected score to within ϵ for a uniformly selected sample paper. The main paper sets β=0.05 and considers ϵ=0.25 and 0.5.

Under independent, identically distributed scoring within each paper, Var⁡(Y¯i,k)=σi2/k. Applying Chebyshev’s inequality to the failure event |Y¯i,k−μi|≥ϵ gives

This expression gives a sufficient budget when the score scale is known and the mean and variance exist. It requires neither a between-paper score gap nor a normality assumption or an empirical histogram as ground truth.

D.2 Planning Budgets from Observed Variance

With n pilot scores per paper, estimate within-paper variance by V^=N−1​∑iSi2. Substituting V^ into the preceding expression gives the Planning budget. With ten pilot scores, the three configurations have V^ values of 0.140056, 0.148889, and 0.074056. Planning budgets are 45, 48, and 24 at a tolerance of 0.25 points, and 12, 12, and 6 at 0.5 points.

Planning expresses the budget implied by the observed variance. The next section also accounts for uncertainty in variance estimation from a finite pilot sample.

D.3 An Upper Bound on Mean Variance from Nonoverlapping Pairs

Scores lie in [a,b]=[1,10]. Define the symmetric kernel

For two independent, identically distributed scores, 𝔼​h​(Y,Y′)=σi2. The unbiased sample variance for each paper also equals the average kernel value over all score pairs:

Let K=⌊n/2⌋. Randomly permute each paper’s n scores and form K nonoverlapping pairs from the first 2​K scores. Let Ti be their mean kernel value and T=N−1​∑iTi its average across papers. Conditional on the observed scores, every unordered pair has the same probability of selection, so

For fixed permutations, there are M=N​K nonoverlapping kernel values. Assuming calls are independent across papers, these values are independent, and their expectations average to V. For any real λ, Jensen’s inequality gives

For X∈[0,1], the chord bound for the exponential function gives

The kernel values may have different expectations. Multiplying the moment-generating functions of the independent variables and applying concavity of the logarithm yields a bound depending only on the overall mean V/B:

For v<V, taking λ<0 and optimizing the Chernoff parameter gives

where

Given the observed V^, let Uα be the root above V^ of

Inverting the lower-tail bound yields

The score range also implies V≤(b−a)2/4=20.25, so the final upper bound can be capped at 20.25. The derivation uses a Chernoff argument for bounded variables and random grouping of a U-statistic; see Hoeffding (1963).

With ten pilot reviews, all 45 pairs per paper, or 9,000 pairs overall, contribute to the unbiased estimate V^, while the concentration bound uses M=200×5=1,000. These quantities serve variance estimation and confidence control, respectively; the overlapping 9,000 pairs are not treated as independent kernel values.

D.4 The Two 95% Levels

Choose k⁡(Uα)=⌈Uα/(β​ϵ2)⌉ from the pilot data. For separately collected, independent future scores,

The main paper uses α=β=0.05. The first 95% is the confidence level for the variance upper bound constructed from the pilot data; the second is the overall success probability of future mean-score estimation. The budget k counts new independent reviews and excludes the existing pilot reviews.

Averaging over both pilot-data and future-score randomness gives a joint success lower bound of (1−α)​(1−β). This equals 0.9025 when α=β=0.05. To require a joint lower bound of at least 0.95 in advance, one can choose α=0.01,β=0.04, giving 0.99×0.96=0.9504. Section D.5 reports the corresponding budgets.

D.5 Simultaneous Control Across Models and Different Tolerances

Assigning failure probability α/3 to each model’s variance bound gives simultaneous coverage of at least 1−α for all three bounds by Bonferroni’s inequality. Each model’s overall future-score success probability remains 1−β.

Halving the tolerance approximately quadruples the variance-based budget, and a higher success target also increases the budget. All tables round budgets upward to integer review counts.

D.6 Five Versus Ten Pilot Reviews

Ten pilot reviews increase the number of independent kernel values available to the bound from 400 to 1,000. Observed variances change little, but the upper bounds tighten: Certified budgets at a half-point tolerance decrease from 53, 55, and 43 to 33, 35, and 24.

D.7 A Reference Bound Using Only the Score Range

Without pilot variance estimates, the paper-specific Hoeffding bound gives

Thus k≥⌈81​log⁡(2/β)/(2​ϵ2)⌉ suffices to control each paper’s failure probability and hence the average failure probability. With β=.05, tolerances of 0.25 and 0.5 points require 2,391 and 598 reviews, respectively. Our primary analysis uses pilot variances and their upper bounds to exploit the observed dispersion of scores.

Appendix E Score Granularity and Distribution Estimation

E.1 Integer Versus Forced-Decimal Scoring

We compare the two protocols within papers, with five scores per protocol: 1,000 observations per model and protocol. Integer scores are exactly the first five from the ten-review original-manuscript experiment. The table also reports year-specific results and both acceptance-label conventions.

The forced-decimal protocol raises Luna’s mean score by 0.693 points and Sol’s by 0.266. Their within-paper SDs decrease from 0.390 to 0.329 and from 0.286 to 0.249, respectively. Luna’s MAE decreases in both years. Sol’s MAE changes in opposite directions across years, with both intervals including zero.

Output granularity affects exact agreement and score range under the integer protocol. If scores switch only between adjacent integers s and s+1, with probabilities 1−π and π, the variance is π⁡(1−π) and the probability of disagreement between two reviews is 2​π​(1−π). This two-point model shows how switching between adjacent levels can produce frequent disagreement within a small range. The decimal experiment directly measures the variance and score-location changes caused by a protocol change, without assuming that lower variance implies a more accurate estimate of an unchanged scoring target.

E.2 Construction of Quick and Standard

The target is the score distribution pi​(s) under the fixed GPT-5.6-Sol medium integer configuration, for s∈{1,…,10}. An auxiliary call uses a nonreasoning configuration and extracts the visible probabilities of valid candidate score tokens. Restricting candidates to 1–10 and renormalizing yields ri​(s). The analysis does not assign probability mass to candidates absent from the response.

Quick first adjusts probability concentration:

Here 𝒮i is the valid candidate set returned by the call. It then maps score locations to xs=clip⁡(a+b​s,1,10) and linearly distributes the mass at each location between adjacent integers. Integer locations receive all their mass; otherwise, the weights at ⌊xs⌋ and ⌈xs⌉ are ⌈xs⌉−xs and xs−⌊xs⌋, respectively. Denote the resulting distribution by qQ,i.

Standard combines Quick with two reasoning scores from the target configuration:

A Quick prediction uses one auxiliary call. Standard uses that same auxiliary call and two reasoning reviews.

E.3 Parameter Search and Train–Test Splits

The 198 papers are sorted by anonymous ID, and 30 random 99/99 train–test splits are generated without additional stratification by year. The search grid is

Quick has 234 parameter combinations. First, (T,a,b) is selected using Brier loss against five target scores per training paper. Next, all ten partitions of five scores into two inputs and three targets are enumerated, and the mean training loss selects one of eight candidate weights. Test papers do not participate in parameter selection, and Standard’s input and target scores do not overlap.

The parameter combinations selected across the 30 splits are as follows.

At test time, Quick consistently uses the first valid auxiliary output and is evaluated against five independent target scores. Standard enumerates ten partitions per paper, first averaging over partitions, then over that paper’s test-set appearances, and finally equally across the 198 papers. Each paper appears in the test set 8–23 times.

E.4 Squared Distribution Risk with Independent Targets

For a predicted distribution q and independent target scores Z1,…,Zm, define

Conditional on q, the expectation of the second term is −2∑sq(s)pi(s) and that of the final term is ∑spi​(s)2. Therefore,

The final term estimates the squared probability sum through collisions between independent target scores, removing variation due to a finite target sample. Individual paper-level risk estimates may be negative; these values are retained when averaging.

For a frequency estimator based on n direct scores, expected risk is

Direct-sampling baselines estimate the numerator from the collision statistic of the five existing scores and divide by n. Risks for 1, 2, 3, and 5 reviews are estimates of expected risk under this sampling law, without treating the five-score histogram as a known target distribution.

For an auxiliary prediction conditionally independent of two reasoning scores, and a frequency estimator unbiased for pi, Standard’s risk decomposes as

The cross term vanishes because the reasoning-frequency error has mean zero and is conditionally independent of the auxiliary prediction. Weights are selected on training papers, and the combined predictor’s risk is evaluated on test papers.

E.5 Pooled and Year-Specific Risks

Overall Standard risk is 0.0829. Year-specific risks are 0.0932 in 2025 and 0.0727 in 2026, with 99 papers per year; two direct reasoning reviews yield 0.1520 and 0.0914. All intervals hold the original parameters and splits fixed while resampling papers, preserving the same within-paper aggregation order.

E.6 Bootstrap with Parameter Refitting

For each paper-level refit bootstrap, we draw 198 papers with replacement from the 198-paper set, record each paper’s multiplicity, and use it as a weight within the original 30 train–test splits. The training side searches all 234 Quick parameter combinations and eight Standard weights again. The test side continues to exclude training papers for that split. Refit results retain the original within-paper averaging over test appearances and are then aggregated using the bootstrap paper weights.

This procedure preserves the original split design while propagating uncertainty from paper resampling and parameter selection. We run 1,000 resamples using frozen scores and probabilities, without new reviews. The original, unresampled fit and paper-level risks match the saved main-paper results to a tolerance of 10−12.

With refitting, the risk difference between Standard and two reasoning reviews is −0.0388, with a 95% interval of [−0.0525, −0.0259]. Relative to three reasoning reviews, it is +0.0018, with an interval of [−0.0062, +0.0091]. The finding that Standard reduces risk relative to two reasoning reviews and approaches the risk of three persists after parameter reselection.

E.7 Acceptance AUC Under the Same Held-Out Design

All methods use the same 198 papers, parameters, and test splits as the risk analysis. Predictions are distributional expected scores; parameters are not fitted to acceptance labels. Direct-reasoning baselines enumerate all equal-sized subsets of the five scores, while Standard enumerates the ten two-input combinations. AUC is computed on test papers for each combination, then averaged over combinations and the 30 splits.

On all 198 papers, Quick exceeds one reasoning review in AUC by +0.0197, and Standard exceeds two reasoning reviews by +0.0131. Excluding withdrawals gives differences of +0.0141 and +0.0115. This table reports point estimates; intervals for squared risk concern a different statistic and are not AUC intervals.

E.8 Independent-Target Validation on Ten Papers

The existing independent-target validation sampled five papers per year, excluding the long-input paper 2026-paper_067 before sampling. Each paper received ten additional GPT-5.6-Sol medium target scores. Predictions use the original five scores. Quick and Standard are fitted on the other 188 papers, yielding T=8,a=0.2,b=0.9,w=0.7.

For Quick, every available auxiliary output for a paper forms a separate one-call prediction, and losses are averaged without first combining the outputs into a single prediction. Standard uses the first auxiliary output and all two-score combinations from the original five scores. Direct-reasoning methods enumerate all subsets of the corresponding size. All methods first average over prediction combinations within each paper, then equally over the ten papers.

Brier loss is computed against each of the ten one-hot targets and then averaged, including randomness in the target scores themselves. W1 compares the predicted distribution with the empirical CDF of the ten target scores, in score points. Modal accuracy compares the modes of the prediction and the ten target scores, choosing the smallest score in a tie. Mean-score error and P⁡(Y≥6) error compare the respective means and tail probabilities.

In this independent-target set, Standard has Brier loss 0.353, W1 of 0.194 points, and modal accuracy of 69%; three reasoning reviews yield 0.379, 0.178 points, and 77%. These metrics measure the full distribution, cumulative distribution, and most likely score, respectively, and are reported together.

E.9 Historical Call Costs

The existing cost audit uses the original script’s historical pretax USD rates per million tokens: 4 for ordinary input, 5 for cache writes, 0.4 for cached input, and 20 for output. Call cost is computed from usage fields as

Here I denotes input tokens, C cached-input tokens, W cache-write tokens, and O output tokens.

First and subsequent calls differ in cache state, so the table reports both historical averages. The main paper’s R/Q columns describe call composition; this table gives observed costs for those calls without combining different cache states into a single deployment price. The frozen summaries do not contain consistent end-to-end latency measurements for comparing methods under matched conditions, so no latency estimate is reported.

Appendix F Revision Crossover Experiment

F.1 Seven-Cell Design and Paired Scoring

Three letters denote Writer, Selector, and Scorer, in that order, with L for Luna and H for Haiku. Luna candidates are processed by two Selectors and evaluated by two Scorers, yielding LLL, LHL, LLH, and LHH. Haiku candidates yield HLL, HHL, and HHH; HLH was not collected. All cells use the same 199 papers.

Section B.5 describes the Writers’ feedback sources. Each Writer’s candidate set is held fixed across Selectors, and both Scorers read the same final manuscript for a given Writer/Selector combination. Each Scorer independently evaluates the original and revised manuscripts three times each. A Scorer’s original-manuscript baseline is reused across revision conditions. Paper-level gain is defined as

Positive, zero, and negative gains are determined by the difference between the two sums of three scores, avoiding floating-point changes to direction. Overall gains and cross-Scorer differences use paired paper-level resampling.

Both years retain opposite average gains for Luna revisions under the two Scorers. For example, year-specific gains for LLL are +0.280 and +0.670, whereas those for LLH are −0.417 and −0.279. The year-specific results for LHL and LHH likewise remain positive and negative, respectively.

F.2 Candidate Edits and Selector Agreement

The two Selectors independently accept or reject each candidate edit. For accepted sets Ai,Bi, Jaccard similarity is Ji=|Ai∩Bi|/|Ai∪Bi|, defined as 1 when both sets are empty. The table reports candidate counts and acceptance intersections for each Writer.

Of Luna’s 9,327 candidates, 8,459 are accepted by both Selectors and 661 receive different decisions. Decisions are identical for 17 papers; the other 182 have different final texts. Candidate acceptance and final scoring occur at different stages: a large overlap in accepted candidates coexists with opposite gains from the two Scorers on fixed final manuscripts.

Haiku generates 2,587 candidates. Luna and Haiku accept 2,442 and 2,450, respectively, with 2,354 accepted by both. Mean paper-level Jaccard similarity is 0.9198. The Writers produce different numbers of edits, while selection agreement within their respective candidate sets is similar.

F.3 Paper-Level Gain Directions and Cross-Model Differences

Beyond overall gains, we classify each paper’s difference between three-review means as positive, zero, or negative. The table gives complete direction contingency tables for the three shared-manuscript conditions.

For Luna-written, Luna-selected manuscripts, 46 papers have positive Luna gains and negative Haiku gains, and none show the reverse combination. Both gains are positive for 27 papers, negative for 19, and zero for 10. The two strict reversal combinations total 61 papers for Luna-written, Haiku-selected manuscripts and 23 for Haiku-written, Haiku-selected manuscripts.

At least one Scorer assigns zero gain in 53.8%, 45.7%, and 70.4% of papers across the three conditions. Direction disagreement can therefore be separated into strict reversals and cases involving zero gain; the full contingency tables preserve this distinction.

F.4 Scoring Noise and Correlation Adjustment

Under a model with fixed manuscripts and independent, zero-mean scoring errors, the noise variance of the three-review original mean is eo,i=So,i2/3, and the noise variance of the gain estimate is

Let vx,vy be the observed across-paper variances for two Scorers, ex,ey their mean noise variances, and c their observed covariance. Keeping the covariance unchanged and subtracting diagonal noise terms gives

The original-manuscript correlation is 0.804 before adjustment and 0.843 after it. Observed gain correlations for the two Luna revision conditions are 0.415 and 0.407, rising to 0.521 and 0.518 after adjustment. For Haiku revisions, the observed correlation is 0.085 and the adjusted value is approximately 0.230, with a wide interval. Some bootstrap samples have nonpositive noise-adjusted variances or correlations outside the usual range. The table retains valid-resample counts and out-of-range cases.

The adjustment assumes independent errors across models and between original and revised manuscripts. Writers use existing first reviews to generate revisions; Section B.5 specifies their actual relationship to the original-score baselines. We use noise subtraction to diagnose the contribution of scoring error. Mean cross-Scorer differences for fixed final manuscripts are computed directly from paired scores and do not depend on this adjustment.

F.5 Structural Checks of Manuscript Packages

We compare original and final packages paper by paper for the four Writer/Selector conditions, checking record counts and order, fields other than text, leading instructions and page-marker text, image indices and stored hashes, and the existence of referenced image files.

All four conditions pass these checks. The Luna-written conditions accept 8,857 and 8,722 text blocks; the Haiku-written conditions accept 2,442 and 2,450. Recomputing SHA-256 for all 9,440 image files across the four final-package conditions gives exact matches to the original image-index hashes. This audit covers input-package structure, text-replacement rules, and image files. The study did not separately collect blinded expert labels for factual preservation or quality changes in the revisions.

F.6 Real Revision Examples Selected by a Fixed Rule

Cases are selected from five direction strata in LL: Luna positive/Haiku negative, both positive, both negative, Luna negative/Haiku positive, and at least one zero. Papers in each stratum are sorted by a fixed hash rule, and the first is selected. The Luna-negative/Haiku-positive stratum is empty. One additional case is selected by the same rule from Luna-positive/Haiku-zero papers in the Haiku-written, Haiku-selected condition.

Each case shows the first accepted block of suitable display length after sorting edits by page number and text-block ID. Both original and revised text must contain 70–750 characters after whitespace normalization; if no block qualifies, the first accepted block is used. Scores always refer to the complete revised manuscript. The displayed block illustrates the edit and is not assigned responsibility for the whole-paper gain.

F.6.1 Case 1: Luna Positive / Haiku Negative

Paper 2025-paper_092; Luna Writer/Luna Selector; 46 papers in this direction stratum. The displayed edit is on PDF page 3, text block 27, diff d0003, and was accepted by this condition’s Selector.

Original:

3. We demonstrate theoretically how a) SGFT can learn highly performant policies, despite the bias introduced by the SGFT objective and b) how this enables SGFT to overcome compounding errors which plague standard MBRL approaches.

Revised:

3. We analyze how value-transfer bias and model error affect SGFT policies. The resulting guarantee is conditional on uniform closeness of Vsim to the optimal real-world value and improvability under real-world dynamics; neither condition is empirically verified in our experiments.

F.6.2 Case 2: Both Positive

Paper 2026-paper_005; Luna Writer/Luna Selector; 27 papers in this direction stratum. The displayed edit is on PDF page 1, text block 10, diff d0002, and was accepted by this condition’s Selector.

Original:

Despite rapid progress, two gaps remain. First, no research has scaled EFMs beyond the levels nec- essary to fully exploit this potential—previous efforts have been limited by computational resources and dataset availability. Second, no systematic, quantitative analysis of EFM scaling behavior has been conducted, leaving fundamental questions unanswered about optimal resource allocation and scaling strategies for neural data—in particular, how to allocate a fixed compute budget across model size and number of training epochs.

Revised:

Despite rapid progress, two gaps remain. First, previous EFM efforts have explored a narrower range of model and data scales than those considered here, in part because of computational resources and dataset availability. Second, systematic quantitative analysis of EFM scaling behavior remains limited, leaving open questions about resource allocation and how to divide a fixed compute budget between model size and training duration.

F.6.3 Case 3: Both Negative

Paper 2025-paper_020; Luna Writer/Luna Selector; 19 papers in this direction stratum. The displayed edit is on PDF page 2, text block 15, diff d0004, and was accepted by this condition’s Selector.

Original:

We show that each Koopman operator is represented by the exponential of a matrix constructed with Toeplitz matrices and diagonal matrices. This allows us to apply Krylov subspace methods (Gallopou- los & Saad, 1992; Güttel, 2013; Hashimoto & Nodera, 2016) to compute the estimation of Koopman operators with low computational costs. By virtue of the universal property of Toeplitz matrices (Ye & Lim, 2016), we can show the universality of the proposed model with a linear algebraic approach. We also show a generalization bound of the proposed model using a reproducing kernel Hilbert space (RKHS) associated with the Fourier functions. We can analyze both the universality and generalization error with the same framework.

Revised:

We represent each layer by the exponential of a matrix constructed from Toeplitz and diagonal matrices. This structure allows Krylov subspace methods to compute matrix-exponential actions. Using the universal property of Toeplitz matrices (Ye & Lim, 2016), we establish a finite-dimensional function-approximation result. We also analyze generalization using an RKHS associated with Fourier functions, with the bound depending on operator norms and requiring appropriate controls on the model class.

F.6.4 Case 4: At Least One Zero

Paper 2026-paper_066; Luna Writer/Luna Selector; 107 papers in this direction stratum. The displayed edit is on PDF page 1, text block 9, diff d0003, and was accepted by this condition’s Selector.

Original:

Large Language models (LLMs) such as Transformer (Vaswani et al., 2017), BERT (Devlin et al., 2019a), PaLM (Chowdhery et al., 2023), and GPT-4o (Hurst et al., 2024) have demonstrated re- markable capabilities in natural language understanding and generation, which helps to achieve a wide range of tasks such as language translation, sentiment analysis, and question answering. Simi- larly, video Transformers such as TimeSformer (Bertasius et al., 2021), ViViT (Arnab et al., 2021), and Video Swin Transformer (Liu et al., 2022) have illustrated remarkable progress in capturing spatio-temporal dynamics, greatly advancing applications such as video understanding and editing.

Revised:

Video Transformers, including TimeSformer (Bertasius et al., 2021), ViViT (Arnab et al., 2021), and Video Swin Transformer (Liu et al., 2022), model spatio-temporal dependencies. This manuscript examines the computational cost of exact block attention and a proposed dynamic approach. The review identified fundamental correctness and dimensionality errors in that approach; the results below therefore define the target computation, not a validated efficient algorithm.

F.6.5 Case 5: Haiku Revision, Luna Positive / Haiku Zero

Paper 2025-paper_093; Haiku Writer/Haiku Selector; 47 papers in this direction stratum. The displayed edit is on PDF page 1, text block 20, diff d0002, and was accepted by this condition’s Selector.

Original:

To address this gap, in this paper, we curate a comprehensive ex- trapolation benchmark consisting of seven datasets sourced from Matminer (Ward et al., 2018), accompanied by well-defined train/validation/test splits based on real-

Revised:

To address this gap, in this paper, we curate a comprehensive extrapolation benchmark consisting of seven tasks derived from four datasets sourced from Matminer (Ward et al., 2018), accompanied by well-defined train/validation/test splits based on real-

Appendix G Reproduction Resources

The code package includes all data, code, configurations, and external files needed to reproduce the analyses and figures.

Appendix H Ethical Considerations and Responsible Use

Data Sources and Licensing. This study uses publicly available ICLR 2025 and 2026 submissions and associated review metadata obtained through OpenReview. We selected 100 papers from each year for academic analysis of AI-based peer review. The original manuscripts and their contents remain subject to the copyrights and applicable licenses of their respective authors. Our reported findings primarily consist of independently computed statistics and analyses. Any redistribution of source materials or derived artifacts is subject to the applicable licenses and terms of use.

Use of AI Models. Large language models are the primary research objects in this study. They are used to evaluate manuscripts, generate and select candidate revisions, and assess scoring consistency. These model-generated reviews and revisions constitute experimental outputs rather than independent human assessments. The experimental procedures, statistical analyses, and interpretation of the results remain the responsibility of the authors.

Ethical Risks. The study analyzes publicly available research manuscripts and review information without recruiting human participants. Manuscript inputs are anonymized for AI evaluation. Our findings should not be interpreted as evidence that AI-generated reviews can replace human peer reviewers or reliably determine publication decisions. Automated manuscript revisions may also introduce inaccuracies, and their scientific validity requires independent verification.

注意转载授权 · 阅读原文