我们能否相信AI同行评审？1.4万次AI评审揭示评分随机性与修订增益分歧

   Haoyuan Li    Zhongsheng Wang 单位： Gillian Dobbie    Jiamou Liu 单位： University of Auckland 单位： Auckland, New Zealand

   Haoyuan Li    Zhongsheng Wang Affiliation: Gillian Dobbie    Jiamou Liu Affiliation: University of Auckland Affiliation: Auckland, New Zealand

摘要

Abstract

随着大语言模型被广泛用于科学审稿和稿件修改，AI 评分正日益作为评估论文质量和指导修改的反馈依据。然而，单一 AI 评分能否可靠反映审稿人的判断，以及基于该评分获得的修改增益能否得到其他审稿人的认可，仍缺乏充分研究。我们使用 ICLR 2025 和 2026 的 200 篇论文及约 14,000 条 AI 审稿记录，研究评分可靠性、评分分布估计以及不同模型对修改增益的一致性。即使在相同条件下，重复审稿的评分也会波动。更细粒度的评分协议能降低评分方差，但也会系统性地改变平均分，而分配给稿件修改的增益在很大程度上取决于审稿人。

As large language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer’s judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.

1 引言

1 Introduction

随着大语言模型融入研究工作流，AI 辅助同行评审正从实验研究走向实际应用。主要 AI 会议的投稿量持续增长，加大了同行评审的压力。包括 ICLR 和 AAAI 在内的会议已开始探索在审稿流程中引入 AI 辅助（Zou and Thakkar, 2025；AAAI, 2025）。与此同时，研究人员正使用 AI 审稿工具评估论文质量、获取修改建议，并通过评分变化判断修改效果。这两类应用都基于一个共同前提：AI 判断足够可靠，能为研究决策提供有意义的证据。

As large language models become integrated into research workflows, AI-assisted peer review is moving from experimental studies into practical use. Submission volumes at major AI conferences have continued to grow, increasing the pressure on peer review. Conferences including ICLR and AAAI have begun exploring AI assistance in the review process (Zou and Thakkar, 2025; AAAI, 2025). Meanwhile, researchers are using AI review tools to assess paper quality, obtain revision suggestions, and judge the effects of revisions through changes in scores. Both applications rest on a shared premise: AI judgments are reliable enough to provide meaningful evidence for research decisions.

这一前提未必成立。首先，AI 评分具有随机性：即使论文、模型和审稿协议保持不变，重复审稿也可能给出不同评分，使得单个评分或观测到的增益无法可靠代表模型自身的评估。其次，AI 判断依赖于模型：不同审稿人可能有不同的评估偏好，即使对原始论文的评估相似，也未必认可相同的修改。忽视这些问题可能使用户将随机波动误认为论文质量差异，并将特定模型下的评分增益误认为得到广泛认可的改进。那么，我们能在多大程度上信任 AI 评分以及修改后观测到的增益？

This premise does not necessarily hold. First, AI scores are stochastic: even when the paper, model, and review protocol remain fixed, repeated reviews can yield different scores, making an individual score or an observed gain an unreliable representation of the model’s own assessment. Second, AI judgments depend on the model: reviewers may have different evaluation preferences and, even when they assess original papers similarly, may not endorse the same revisions. Ignoring these issues can lead users to mistake random fluctuations for differences in paper quality and score gains under a particular model for broadly endorsed improvements. How much, then, can we trust AI scores and the gains observed after revision?

已有工作考察了 AI 审稿人的评估能力、评分偏差以及对论文呈现方式的敏感性。ReviewEval (Garg et al., 2025)、TitleTrap (Du, 2025) 和 LLM-as-a-Reviewer (Li et al., 2026a) 研究了 AI 审稿的不同方面，而 Rhetoric (Li et al., 2026b) 进一步揭示了审稿人对修改增益反应的差异。然而，实际使用 AI 评分需要一组相互关联的答案：单个评分有多可靠？需要多少次审稿才能准确估计某位审稿人的期望评分？在有限推理预算下能否改进评分估计？观察到的增益反映的是随机波动，还是某位特定审稿人所认可的改进？这些问题决定了 AI 评分何时能作为研究反馈。

Prior work has examined AI reviewers’ evaluation capabilities, scoring biases, and sensitivity to paper presentation. ReviewEval (Garg et al., 2025), TitleTrap (Du, 2025), and LLM-as-a-Reviewer (Li et al., 2026a) study different aspects of AI reviewing, while Rhetoric (Li et al., 2026b) further reveals differences in reviewers’ responses to revision gains. Practical use of AI scores nevertheless calls for a connected set of answers: How reliable is a single score? How many reviews are needed to estimate a reviewer’s expected score accurately? Can score estimation be improved under a limited reasoning budget? Does an observed gain reflect random fluctuation, or an improvement endorsed by a particular reviewer? These questions determine when AI scores can serve as research feedback.

为回答这些问题，我们对来自 ICLR 2025 和 ICLR 2026 的 200 篇论文进行了约 14,000 次 AI 审稿，系统研究了评分随机性、估计准确性以及跨模型在修改增益上的一致性。我们的主要贡献是：

To address these questions, we conduct approximately 14,000 AI reviews of 200 papers from ICLR 2025 and ICLR 2026, systematically studying scoring randomness, estimation accuracy, and cross-model agreement on revision gains. Our main contributions are:

量化单个 AI 评分的可靠性。 我们使用重复采样分析评分波动及其对论文相对排名的影响，进而推导出在指定平均分误差和成功概率目标下所需的审稿预算。

Quantifying the reliability of a single AI score. We use repeated sampling to analyze score fluctuations and their effects on relative paper rankings, then derive sufficient review budgets for specified mean-score error and success-probability targets.

探索提高评分可靠性的方法。 我们考察评分粒度如何影响变异性和系统性偏移，并测试辅助评分概率能否在有限推理预算下改进分布估计。

Exploring methods to improve scoring reliability. We examine how score granularity affects variability and systematic shifts, and test whether auxiliary score probabilities improve distribution estimation under a limited reasoning budget.

评估修改增益的可信度。 通过重复评分和跨模型评估，我们将评分波动与审稿人依赖性区分开来，并检验不同审稿人是否通过评分增益一致地认可相同的修改。

Assessing the credibility of revision gains. Through repeated scoring and cross-model evaluation, we separate scoring fluctuations from reviewer dependence and test whether different reviewers consistently endorse the same revisions through score gains.

2 相关工作

2 Related Work

AI 同行评审。 现有研究主要通过评审意见的质量来评估 LLM 的审稿能力。ReviewEval (Garg et al., 2025) 考察事实性、分析深度和建议的可操作性；LLM-as-a-Reviewer (Li et al., 2026a) 比较模型与人类审稿并测试对提示注入的抵抗能力。这些研究有助于确定 LLM 能否生成高质量评审，但较少关注评审结果如何影响后续决策。特别是，同一论文被反复审稿时评分变化的程度，以及这些评分能否可靠地区分修改前后的论文质量，仍研究不足。

AI peer review. Existing studies primarily assess LLM reviewing ability through the quality of review comments. ReviewEval (Garg et al., 2025) examines factuality, analytical depth, and the actionability of suggestions; LLM-as-a-Reviewer (Li et al., 2026a) compares model and human reviews and tests resistance to prompt injection. These studies help establish whether LLMs can generate high-quality reviews, but devote less attention to how review outcomes inform subsequent decisions. In particular, the extent of score variation when the same paper is reviewed repeatedly, and whether these scores reliably distinguish paper quality before and after revision, remain understudied.

评分稳定性与估计。已有研究发现，重复的 LLM 评分运行之间存在不一致，并存在系统性偏差（Stureborg et al., 2024）；输出格式和解释要求也会影响评估结果（Chiang and Lee, 2023）。为应对评分不确定性，G-Eval（Liu et al., 2023）使用评分 token 概率，而 Homan et al. (2026) 考察了统计检验需要多少次重复评分。然而，现有工作很少直接回答：在评审者和评审条件固定的情况下，单个评分与重复评分的均值相差多少。我们针对固定评审条件下的期望评分和评分分布，分析重复采样如何影响估计精度，并评估概率辅助估计相对于独立采样的目标评分所带来的增益。

Score stability and estimation. Prior work finds inconsistencies across repeated LLM scoring runs and systematic biases (Stureborg et al., 2024); output formats and explanation requirements also affect evaluation results (Chiang and Lee, 2023). To address scoring uncertainty, G-Eval (Liu et al., 2023) uses score-token probabilities, while Homan et al. (2026) examine how many repeated ratings statistical tests require. However, existing work rarely directly answers how much a single score differs from a mean over repeated scores when the reviewer and reviewing conditions are fixed. We target the expected score and score distribution under fixed reviewing conditions, analyze how repeated sampling affects estimation precision, and evaluate the gains from probability-assisted estimation against independently sampled target scores.

AI 辅助修改。除评估论文外，LLM 评分还可用于选择和指导修改。TitleTrap（Du, 2025）和 Rhetoric（Li et al., 2026b）表明，标题和修辞呈现的变化会影响 AI 评分。因此，评分提高可能反映论文实质质量的提升，也可能反映选择器对特定评分偏好的适应。现有工作尚未充分区分这些来源。我们固定候选修改集合，分别改变选择器和评分器，在共享修改内容下比较评分增益。这检验了 LLM 评分引导的修改策略在不同评审者之间是否保持稳定，以及观察到的增益中有多少可归因于采样不确定性。

AI-assisted revision. Beyond evaluating papers, LLM scores can be used to select and guide revisions. TitleTrap (Du, 2025) and Rhetoric (Li et al., 2026b) show that changes to titles and rhetorical presentation can affect AI scores. A score increase may therefore reflect either an improvement in substantive paper quality or a selector’s adaptation to particular scoring preferences. Existing work has not sufficiently separated these sources. We hold the candidate edit set fixed and vary the selector and scorer separately, comparing score gains under shared revision content. This tests whether LLM-score-guided revision strategies remain stable across reviewers and how much of the observed gains can be attributed to sampling uncertainty.

3 问题形式化与研究设计

3 Problem Formulation and Study Design

3.1 评审评分估计的是什么？

3.1 What Does a Review Score Estimate?

评分分布与重复采样。我们将一次 LLM 论文评审过程视为一个样本。给定论文 Pi 和评审配置 r（模型、提示词、推理设置和执行环境），单个评分是一个随机变量 Yr​(Pi)，其期望为 μr,i，方差为 σr,i2。对同一篇论文，我们进行 k 次独立评审，并用其均值 Y¯r,i,k 估计固定配置下的期望评分。在每篇论文内评分满足 IID 的假设下，该均值的方差为 σr,i2/k。

Score distributions and repeated sampling. We treat one LLM paper-review procedure as one sample. Given a paper Pi and review configuration r (model, prompt, reasoning settings, and execution environment), an individual score is a random variable Yr​(Pi) with expectation μr,i and variance σr,i2. For the same paper, we perform k independent reviews and use their mean Y¯r,i,k to estimate the expected score under the fixed configuration. Under IID scores within each paper, this mean has variance σr,i2/k.

评估有效性。我们使用人类评审评分的均值 Hi 作为论文质量的外部参照。相对于该参照的相关性和误差，以及录用状态的 AUC，作为评估有效性的指标。

Evaluation validity. We use the mean human review score Hi as an external reference for paper quality. Correlation and error relative to this reference, together with AUC for acceptance status, serve as indicators of evaluation validity.

评分可靠性。对于固定的论文和评审配置，评分应表现出较低的随机性，并且其期望应能从有限次重复评审中可靠地估计。我们同时报告评分稳定性和相对排序稳定性，作为评分可靠性的指标。

Scoring reliability. For a fixed paper and review configuration, scores should exhibit low randomness, and their expectation should be reliably estimable from a finite number of repeated reviews. We report both score stability and relative-ranking stability as indicators of scoring reliability.

修订稿的评估效度。对于修订 Pi→Pi′，审稿人的预期增益为 Δr,i=μr​(Pi′)−μr​(Pi)，由同一 Scorer 对原稿和修订稿分别给出的独立平均分之差估计。不同 Scorer 的增益在相同的修订版本上进行比较。对于固定的原稿和修订稿，两侧的独立评分使增益估计的方差为 (σr,i2+σr,i′2)/k。观测到的分数提升包含两个版本的抽样误差。在交叉实验中，我们对每个版本重复评分，并保持原稿基线的 Scorer 一致，从而分别建立模型内波动和模型间响应差异的对照。

Evaluation validity for revisions. For a revision Pi→Pi′, the reviewer’s expected gain is Δr,i=μr​(Pi′)−μr​(Pi), estimated by the difference between independent mean scores assigned by the same Scorer to the original and revised manuscripts. Gains from different Scorers are compared on identical revised versions. For fixed original and revised manuscripts, independent scores on the two sides give the gain estimate a variance of (σr,i2+σr,i′2)/k. An observed score increase contains sampling error from both versions. In the crossover experiment, we score each version repeatedly and keep the Scorer of the original-paper baseline consistent, establishing separate controls for within-model fluctuations and between-model response differences.

3.2 论文与评审配置

3.2 Papers and Review Configurations

我们从公开的 ICLR 2025 和 ICLR 2026 投稿中抽取实验样本，按官方学科领域和人类平均分区间联合分层，每年选取 100 篇论文。论文列表和配置见补充材料。

We draw experimental samples from public ICLR 2025 and ICLR 2026 submissions by stratifying jointly on official subject area and mean human score interval, selecting 100 papers per year. The paper lists and configurations are provided in the supplementary material.

对于 LLM 论文评审，PDF 被转换为有序的文本和图像，保留正文、图表、表格和附录；评审输入经过匿名化处理。一项通用任务要求模型评估新颖性、技术正确性、实验证据和局限性，然后给出 1 到 10 的总体评分及评审意见。

For LLM paper review, PDFs are converted into ordered text and images, retaining the main text, figures, tables, and appendices; review inputs are anonymized. A common task asks models to assess novelty, technical correctness, experimental evidence, and limitations, then produce an overall score from 1 to 10 with review comments.

原始整数评分使用 GPT-5.6-Sol、GPT-6-Luna 和 GPT-6-Sol，每种配置下每篇论文十条评审，共 6,000 条。粒度实验比较后两个模型的整数协议和强制非整数协议。分布估计针对 GPT-5.6-Sol 的推理评分。修订实验使用 Luna 和 Claude Haiku 5.5，原稿和修订稿各评三次；有一篇论文含 224 张图像，超出 Claude 的输入限制，因此交叉比较使用共同的 199 篇论文。更多配置细节见补充材料。

Original integer scoring uses GPT-5.6-Sol, GPT-6-Luna, and GPT-6-Sol, with ten reviews per paper per configuration, totaling 6,000. The granularity experiment compares integer and forced-noninteger protocols for the latter two models. Distribution estimation targets GPT-5.6-Sol reasoning scores. Revision experiments use Luna and Claude Haiku 5.5, with three scores each for original and revised manuscripts; one paper has 224 images, exceeding Claude’s input limit, so crossover comparisons use the shared 199 papers. Additional configuration details are provided in the supplementary material.

4 单条 AI 评审有多可靠？

4 How Reliable Is a Single AI Review?

我们首先刻画单篇论文的分数变异和效度，然后检验独立评审能否复现相同的论文排名。我们进一步估计有限采样下的排名可靠性，并推荐评审预算。

We first characterize score variation and validity for individual papers, then test whether independent reviews reproduce the same paper rankings. We further estimate ranking reliability under finite sampling and recommend review budgets.

4.1 重复评分与人类参照

4.1 Repeated Scores and Human References

表 1 汇总了十次评分的可重复性和外部关联。各配置中，30.0%–56.5% 的论文获得十个完全相同的分数，而 91.5%–100% 的论文分数极差不超过一分。波动主要发生在相邻类别之间。GPT-6-Sol 的论文内标准差为 0.272，其他配置分别为 0.374 和 0.386，其与人类评分的 Spearman 相关系数为 0.409。最终录用 AUC 为：GPT-5.6-Sol 0.7134，GPT-6-Luna 0.6484，GPT-6-Sol 0.7138。

Table 1 summarizes ten-score repeatability and external associations. Across configurations, 30.0%–56.5% of papers receive ten identical scores, while 91.5%–100% have a score range of at most one point. Fluctuations mainly occur between adjacent categories. GPT-6-Sol has a within-paper SD of 0.272, compared with 0.374 and 0.386 for the other configurations, and a Spearman correlation of 0.409 with human scores. Final-acceptance AUC is 0.7134 for GPT-5.6-Sol, 0.6484 for GPT-6-Luna, and 0.7138 for GPT-6-Sol.

论文内 SD 先对无偏的论文级方差取平均，再取平方根。每组十条评分构成 45 个无序对；完全一致率分别为 72.86%、72.36% 和 85.19%。GPT-6-Sol 在全部 200 篇论文上极差均不超过一分，其中 113 篇获得十个完全相同的分数。

Within-paper SD averages unbiased paper-level variances before taking the square root. Each set of ten scores forms 45 unordered pairs; exact agreement is 72.86%, 72.36%, and 85.19%, respectively. GPT-6-Sol stays within a one-point range for all 200 papers, including 113 with ten identical scores.

4.2 独立评审能复现排名吗？

4.2 Do Independent Reviews Reproduce Rankings?

我们将每篇论文的十个评分划分为互不重叠、大小相等的块，比较一到五条评审的均值。枚举所有无序块划分，分别得到 45、630、2,100、1,575 和 126 种划分。每种配置覆盖 9,900 个同年论文对，并在所有划分上取平均。排名被分类为相同严格顺序、至少一个块中出现并列，或严格顺序相反。

We form disjoint equal-sized blocks from each paper’s ten scores, comparing means of one through five reviews. Enumerating all unordered block partitions gives 45, 630, 2,100, 1,575, and 126 partitions, respectively. Each configuration covers 9,900 within-year paper pairs, averaged over all partitions. Rankings are classified as the same strict order, a tie in at least one block, or reversed strict orders.

更多评审主要通过减少并列来提高严格排名的可复现性。从每块一条评审增加到五条评审，GPT-5.6-Sol 的严格一致率从 62.14% 升至 84.47%，Luna 从 63.00% 升至 85.71%。GPT-6-Sol 从 57.52% 升至 74.13%，仍有 25.08% 的论文对在至少一个五评分块中并列。其反转率为 0.79%，低于其他配置的 2.15% 和 2.44%。更精细的均值可建立更多严格比较，包括对先前并列论文对的反转。接下来，我们通过绝对均值估计误差来指定采样目标。

More reviews mainly improve strict ranking reproducibility by reducing ties. From one to five reviews per block, strict agreement rises from 62.14% to 84.47% for GPT-5.6-Sol and from 63.00% to 85.71% for Luna. GPT-6-Sol rises from 57.52% to 74.13%, with 25.08% of pairs still tied in at least one five-score block. Its reversal rate is 0.79%, below 2.15% and 2.44% for the other configurations. Finer means establish more strict comparisons, including reversals of pairs previously tied. We next specify a sampling target through absolute mean-score estimation error.

4.3 多少条评审才足够？

4.3 How Many Reviews Are Enough?

对抽样论文等权，我们将 k 条新评审的总体成功率定义为

Weighting the sampled papers equally, we define the overall success rate of k new reviews as

我们假设每篇论文内评分独立同分布，论文之间相互独立，且未来评分与试点评分独立。我们要求 A⁡(k,ϵ)≥0.95。这意味着，对于随机选取的论文，重复评审的均值落在模型期望评分 ϵ 分以内的概率至少为 95%。令 V=N−1​∑iσi2。在独立同分布评审下，切比雪夫不等式给出

We assume IID scores within each paper, independence across papers, and future scores independent of the pilot. We require A⁡(k,ϵ)≥0.95. This means that, for a randomly selected paper, the mean of repeated reviews falls within ϵ points of the model’s expected score with probability at least 95%. Let V=N−1​∑iσi2. Under independent and identically distributed reviews, Chebyshev’s inequality gives

该预算仅取决于平均方差，既不要求评分服从正态分布，也不要求已知评分分布。代入十个试点评分的平均样本方差 V^，在 0.25 分容差下得到规划预算为 45、48 和 24 条评审，在 0.5 分容差下为 12、12 和 6 条评审。

This budget depends only on the average variance and requires neither normally distributed scores nor a known score distribution. Substituting the average sample variance V^ from ten pilot scores gives planning budgets of 45, 48, and 24 reviews at a 0.25-point tolerance, and 12, 12, and 6 reviews at a 0.5-point tolerance.

有限试点数据下的置信控制。我们还利用已知评分范围 [1,10] 为 V 构造上置信界 U。每篇论文十个评分的样本方差等于所有评分对上半平方差的平均值。将其视为无序评分对上的 U 统计量，我们通过随机不相交配对和 KL-Chernoff 界（Hoeffding, 1963）得到一个仅通过 V^ 计算的单侧 95% 上界。每个模型的有效配对数为 200​⌊10/2⌋=1000；共享观测的 9,000 个评分对不被视为独立。具体而言，令 B=(10−1)2/2=40.5 且 M=1000，在 V^ 之上求解根：

Confidence control with finite pilot data. We also use the known score range [1,10] to construct an upper confidence bound U for V. The sample variance of ten scores for each paper equals the average half-squared difference over all score pairs. Treating it as a U-statistic over unordered score pairs, we obtain a one-sided 95% upper bound depending only on V^ through random disjoint pairing and a KL-Chernoff bound (Hoeffding, 1963). Each model has an effective pairing count of 200​⌊10/2⌋=1000; the 9,000 score pairs that share observations are not treated as independent. Specifically, let B=(10−1)2/2=40.5 and M=1000, and solve for the root above V^:

其中 kl⁡(p,q)=p​log⁡(p/q)+(1−p)​log⁡((1−p)/(1−q))。该反演满足 Pr⁡(V>U)≤0.05。关键步骤是将所有重叠评分对表示为随机不相交匹配的条件期望，然后利用 Jensen 不等式得到 1000 个独立有界核的矩生成函数界。将 U 代入同一预算公式，可得到在未来新评审上总体成功率达到 95% 的充分预算，且对每个模型具有 95% 的试点置信度。

where kl⁡(p,q)=p​log⁡(p/q)+(1−p)​log⁡((1−p)/(1−q)). This inversion satisfies Pr⁡(V>U)≤0.05. The key step expresses all overlapping score pairs as the conditional expectation of random disjoint matchings, then uses Jensen’s inequality to obtain the moment-generating-function bound for 1000 independent bounded kernels. Substituting U into the same budget formula gives a sufficient budget for an overall success rate of 95% on future new reviews, with 95% pilot confidence for each model.

两个预算对应不同的信息需求：规划预算反映观测到的方差规模，而置信度控制预算还覆盖尚未被充分观测到的分数尾部。将容限减半会使两个预算大约变为四倍。实践中，可根据误差容限和置信度要求选择预算。该界的构造、证明以及三个模型的同时控制结果见补充材料。

The two budgets correspond to different information requirements: planning budgets reflect the scale of observed variance, while confidence-controlled budgets also cover score tails that have not been sufficiently observed. Halving the tolerance approximately quadruples both budgets. In practice, the budget can be selected according to the error tolerance and confidence requirement. The bound’s construction, proof, and simultaneous-control results for all three models appear in the supplementary material.

图 3 以成功率下界的形式绘制了三种配置在 0.5 分容限下的预算。虚线曲线代入观测方差，实线曲线使用其置信上界，分别对应规划规模和置信度控制的充分预算。曲线由方差关系计算得出，未对十个分数的观测频率进行重采样。

Figure 3 plots the budgets for the three configurations at a tolerance of 0.5 points as lower bounds on the success rate. Dashed curves substitute observed variances, and solid curves use their upper confidence bounds, corresponding to the planning scale and confidence-controlled sufficient budgets, respectively. The curves are calculated from the variance relationship without resampling the observed frequencies of the ten scores.

5 我们能否提高评分可靠性？

5 Can We Improve Scoring Reliability?

重复评审提高了分数和论文排名的可靠性，但对分数接近的论文做出可靠判断可能需要许多独立调用。能否在不显著增加评审成本的情况下提高 AI 评分可靠性？

Repeated reviews improve the reliability of scores and paper rankings, but reaching reliable judgments about closely scored papers may require many independent calls. Can AI-score reliability be improved without substantially increasing review costs?

5.1 更细的分数同时改变方差和位置

5.1 Finer Scores Change Both Variance and Location

如果将高质量同行评审视为对论文质量的排序，该排序未必本质上是离散的。然而分数是离散整数。这促使我们检验更细的输出粒度是否能提高模型稳定性。我们固定论文输入和所有其他评审要求，仅将整数协议替换为要求小数部分非零的 1–10 分。Luna 和 Sol 各自对全部 200 篇论文产生五次重复评分。

If high-quality peer review is viewed as a ranking of paper quality, that ranking need not be inherently discrete. Scores, however, are discrete integers. This motivates testing whether finer output granularity improves model stability. We fix paper inputs and all other review requirements, replacing only the integer protocol with a 1–10 score required to have a nonzero fractional part. Luna and Sol each produce five repeated scores for all 200 papers.

两个模型的分数变异性均下降，而均值分别上升 0.693 和 0.266。Luna 相对于人类平均分的 MAE 也下降，而 Sol 的 MAE 基本不变。因此，小数协议在评分尺度上同时改变了位置和变异性。为估计原始整数评审者，我们接下来保留其协议，并利用额外信息改进估计。

Score variability decreases for both models, while their means rise by 0.693 and 0.266, respectively. Luna’s MAE relative to mean human scores also decreases, whereas Sol’s MAE remains largely unchanged. The decimal protocol thus changes both location and variability on the scoring scale. To estimate the original integer reviewer, we next retain its protocol and use additional information to improve estimation.

5.2 用更少推理调用估计同一评审者

5.2 Estimating the Same Reviewer with Fewer Reasoning Calls

除了比较整数评分和小数评分之外，我们还评估了两种简单策略，用于在推理调用次数更少的情况下，估计固定 GPT-5.6-Sol 推理配置的整数评分分布 pi。Quick 从一次非推理调用中提取候选整数评分的 token 概率，对其进行归一化，使用温度 T 调整其集中程度，并通过仿射映射 s↦a+b​s 校准评分位置。映射后的位置被裁剪到 [1,10]，非整数位置上的概率质量被线性分配给两个相邻的评分类别，得到 qQ。Standard 将该估计与两个目标推理评分的经验分布相结合： qS=w​p^2+(1−w)​qQ。

Beyond comparing integer and decimal scoring, we evaluate two simple strategies for estimating the integer score distribution pi of a fixed GPT-5.6-Sol reasoning configuration with fewer reasoning calls. Quick extracts token probabilities for candidate integer scores from one non-reasoning call, normalizes them, adjusts their concentration using temperature T, and calibrates score locations through an affine mapping s↦a+b​s. Mapped locations are clipped to [1,10], and probability mass at non-integer positions is allocated linearly to the two adjacent score categories, yielding qQ. Standard combines this estimate with the empirical distribution of two target reasoning scores: qS=w​p^2+(1−w)​qQ.

我们使用 198 篇具有可用辅助概率的论文，并在 30 次随机论文级划分上评估这两种策略，每次划分包含 99 篇训练论文和 99 篇测试论文。Quick 使用每篇论文的第一个有效非推理输出。其参数 T、a、b 通过在六个温度、13 个偏移量和三个斜率的网格上最小化针对训练论文推理评分的 Brier 损失来选择。在这些参数固定后，Standard 通过枚举将每篇训练论文的五个推理评分划分为两个输入和三个独立目标的所有十种方式，从八个候选权重中选择 w。在测试论文上的评估遵循相同的输入–目标分离方式，所有参数均从训练集固定。

We use the 198 papers with available auxiliary probabilities and evaluate both strategies over 30 random paper-level splits, each containing 99 training and 99 test papers. Quick uses the first valid non-reasoning output for each paper. Its parameters T,a,b are selected by minimizing Brier loss against the training papers’ reasoning scores, over a grid of six temperatures, 13 offsets, and three slopes. With these parameters fixed, Standard selects w from eight candidate weights by enumerating all ten ways to divide each training paper’s five reasoning scores into two inputs and three independent targets. Evaluation on test papers follows the same input–target separation, with all parameters fixed from the training set.

使用独立目标的风险评估。 对于预测分布 q 和独立目标评分 Z1,…,Zm，我们计算

Risk evaluation with independent targets. For a predicted distribution q and independent target scores Z1,…,Zm, we compute

在给定 q 的条件下，其期望为 ‖q−pi‖22。目标评分之间的碰撞率估计 ‖pi‖22，因此来自有限目标样本的变异性不会直接加到预测风险中。对于 n 个直接推理样本，风险为 (1−‖pi‖22)/n，同样使用独立评分的碰撞率进行估计。

Conditional on q, its expectation is ‖q−pi‖22. The collision rate between target scores estimates ‖pi‖22, so variability from the finite target sample is not directly added to prediction risk. For n direct reasoning samples, risk is (1−‖pi‖22)/n, also estimated using the collision rate of independent scores.

Standard 相对于两个直接推理样本将风险降低 0.0388，配对 95% 区间为 [-0.0524, -0.0255]；其与三个推理样本的风险差为 +0.0018 [-0.0057, 0.0095]。辅助概率在两个推理样本之外提供了有用信息，达到接近三个直接推理样本的风险水平。Quick 的风险接近单个推理样本的风险，并且融合比仅替换评分器更有效。表中分别报告了调用组成；实际美元成本取决于输入长度和缓存。

Standard reduces risk by 0.0388 relative to two direct reasoning samples, with a paired 95% interval of [-0.0524, -0.0255]; its risk difference from three reasoning samples is +0.0018 [-0.0057, 0.0095]. Auxiliary probabilities provide useful information alongside two reasoning samples, reaching a risk level close to three direct reasoning samples. Quick’s risk is close to that of a single reasoning sample, and fusion is more effective than replacing the scorer alone. The table reports the call composition separately; actual dollar costs depend on input length and caching.

融合对风险的影响也可以直接表达。如果辅助调用和两个推理评分条件独立，且经验推理分布对 pi 无偏，则 𝔼​‖qS−pi‖22=w2​(1−‖pi‖22)/2+(1−w)2​𝔼​‖qQ−pi‖22。推理项捕捉直接采样误差，而辅助项捕捉校准分布与目标之间的差异。在训练论文上选择权重会在两项之间分配风险；去偏测试集评估则评估该选择在未参与拟合的论文上是否仍然有效。

The effect of fusion on risk can also be expressed directly. If the auxiliary call and the two reasoning scores are conditionally independent, and the empirical reasoning distribution is unbiased for pi, then 𝔼​‖qS−pi‖22=w2​(1−‖pi‖22)/2+(1−w)2​𝔼​‖qQ−pi‖22. The reasoning term captures direct-sampling error, while the auxiliary term captures the discrepancy between the calibrated distribution and the target. Selecting the weight on training papers allocates risk between the two terms; debiased test-set evaluation assesses whether that choice remains effective on papers excluded from fitting.

5.3 独立目标验证

5.3 Independent-Target Validation

一项单独的现有实验为每年五篇论文收集了十个额外的独立推理分数，使用原始五个分数的子集进行预测。Quick 和 Standard 在其他 188 篇论文上拟合。Standard 获得 0.353 的 Brier 分数和 0.19 的 Wasserstein-1 距离；三个直接推理样本分别获得 0.379 和 0.18。独立目标给出的估计模式与主实验相似。这些损失包含有限目标变异性，并与表 5中的去偏风险分开报告；表 6保留了所有五个基线。这项十篇论文的评估使用新的目标分数，并将评估论文与参数训练分开。

A separate existing experiment collects ten additional independent reasoning scores for five papers from each year, using subsets of the original five scores for prediction. Quick and Standard are fitted on the other 188 papers. Standard obtains a Brier score of 0.353 and Wasserstein-1 distance of 0.19; three direct reasoning samples obtain 0.379 and 0.18, respectively. The independent targets give a similar estimation pattern to the main experiment. These losses include finite-target variability and are reported separately from the debiased risk in Table 5; Table 6 retains all five baselines. This ten-paper evaluation uses new target scores and separates evaluation papers from parameter training.

6 我们能信任修订带来的增益吗？

6 Can We Trust Revision Gains?

准确估计一位审稿人的预期分数，并不能确定其他审稿人是否认可其判断。这对 AI 辅助的稿件修订尤为重要：当修订后的论文获得更高分数时，我们能否信任这一增长反映的是有效改进，而非某位审稿人的偏好？通过受控的稿件修订和跨模型评审，我们研究不同审稿人是否对相同的编辑做出一致反应，以及观察到的分歧中有多少可以由评分随机性解释。

Accurately estimating one reviewer’s expected score does not establish whether other reviewers endorse its judgments. This matters particularly for AI-assisted manuscript revision: when a revised paper receives a higher score, can we trust that increase to reflect an effective improvement rather than a particular reviewer’s preferences? Through controlled manuscript revision and cross-model reviewing, we study whether different reviewers respond consistently to the same edits and how much of the observed disagreement can be explained by scoring randomness.

6.1 固定修订内容

6.1 Holding Revisions Fixed

借助更精确的分数估计，我们考察另一个用例：将 AI 分数作为修订反馈。该工作流有三个角色：Writer 根据原始稿件和现有评审生成候选文本编辑；Selector 决定是否接受每个编辑；Scorer 在编辑应用后评估完整稿件。Scorer 不会收到选择理由。修订必须保留事实、实验数据和非文本材料。

With more precise score estimation, we examine another use case: AI scores as revision feedback. The workflow has three roles: the Writer generates candidate text edits from the original manuscript and existing reviews; the Selector decides whether to accept each edit; and the Scorer evaluates the complete manuscript after edits are applied. The Scorer does not receive selection rationales. Revisions are required to preserve facts, experimental data, and non-text materials.

由 Luna 生成的候选编辑由 Luna 或 Haiku 选择，然后由两者评分，形成一个完整的 2×2 交叉设计。Haiku 生成的编辑增加了三种角色组合，共七种配置。每个 Scorer 独立地对原始稿件和修订稿件各评分三次，所有比较使用相同的 199 篇论文。在 Luna 的 9,327 个候选编辑中，有 8,459 个被两个 Selector 同时接受，每篇论文接受集的平均 Jaccard 相似度为 0.919；最终版本仍有 182 篇论文不同。这使我们能够在固定候选编辑的情况下比较选择效应，并在固定最终版本的情况下比较评分效应。

Candidate edits generated by Luna are selected by Luna or Haiku and then scored by both, forming a complete 2×2 crossover. Haiku-generated edits add three role combinations, for seven configurations in total. Each Scorer independently scores the original and revised manuscripts three times each, and all comparisons use the same 199 papers. Of Luna’s 9,327 candidate edits, 8,459 are accepted by both Selectors, with a mean per-paper acceptance-set Jaccard similarity of 0.919; the final versions still differ for 182 papers. This lets us compare selection effects with fixed candidate edits and scoring effects with fixed final versions.

6.2 相同修订带来相反增益

6.2 Opposite Gains for the Same Revisions

对于由 Luna 生成并选出的相同修订版本，Luna 给出的平均增益为 +0.474，Haiku 为 -0.348，相差 0.822 [0.717, 0.928]。将 Selector 替换为 Haiku 后，两者仍分别得到 +0.472 和 -0.296；跨评分器差距仅变化 -0.054 [-0.116, 0.010]。图 4 展示了相同选择条件下评分器响应的分离情况。

For identical revisions generated and selected by Luna, Luna assigns a mean gain of +0.474 and Haiku -0.348, a difference of 0.822 [0.717, 0.928]. Replacing the Selector with Haiku still yields +0.472 and -0.296, respectively; the cross-scorer gap changes by only -0.054 [-0.116, 0.010]. Figure 4 shows the separation in scorer responses under the same selection condition.

Luna 对 Haiku 生成的修订版本也给出正的平均增益，而 Haiku 对其自身选出的修订版本给出的增益为 +0.050。结果是不对称的：Luna 认可两个 Writer 的修订；Haiku 对 Luna 的修订给出负增益，对自身的修订给出接近零的增益。更换 Selector 对 Luna 候选编辑的总体结果影响较小。在这两组 Luna 修订版本中，最终 Scorer 改变了增益的方向。

Luna also assigns positive average gains to Haiku-generated revisions, while Haiku assigns its own selected revisions a gain of +0.050. The result is asymmetric: Luna endorses revisions from both Writers; Haiku assigns negative gains to Luna’s revisions and near-zero gains to its own. Changing the Selector has a smaller effect on the aggregate results for Luna’s candidate edits. Across these two sets of Luna revisions, the final Scorer changes the direction of the gain.

这些对照服务于不同目的。固定最终稿件并更换 Scorer，直接测量对相同文本编辑的不同响应；固定 Writer 的候选编辑并更换 Selector，检验选择是否改变这一差距。两个 Selector 接受了许多相同的编辑。即使最终版本不同，Luna 和 Haiku 在两种选择条件下仍给出相反的总体增益。因此，无需比较内容不同的两个修订版本，即可研究跨评审者的分歧。

These controls serve different purposes. Holding the final manuscript fixed and changing the Scorer directly measures differing responses to the same textual edits; holding the Writer’s candidate edits fixed and changing the Selector tests whether selection changes this gap. The two Selectors accept many of the same edits. Even when final versions differ, Luna and Haiku assign opposite aggregate gains under both selection conditions. Cross-reviewer disagreement can thus be studied without comparing two revisions with different content.

6.3 对原始版本的一致性不能预测对增益的一致性

6.3 Agreement on Originals Does Not Predict Agreement on Gains

两个 Scorer 在原始论文均值上的 Pearson 相关系数为 0.804，但它们在相同修订版本上的论文级增益相关性分别降至 0.415、0.407 和 0.085。增益方向完全一致的论文比例分别为 28.1%、30.2% 和 41.2%，其中包括两者均给出零增益的情况。

The two Scorers have a Pearson correlation of 0.804 on original-paper means, but their paper-level gain correlations for identical revisions fall to 0.415, 0.407, and 0.085. Gain directions agree exactly for 28.1%, 30.2%, and 41.2% of papers, respectively, including cases in which both assign zero gain.

我们通过重复评分估计噪声贡献：原始论文均值的噪声方差为 so2/3，增益的噪声方差为 (so2+sv2)/3。我们从相应的论文间方差中减去这些分量，同时保留跨模型协方差。Luna 修订版本的调整后相关性约为 0.52，仍低于原始论文的 0.843。评分波动解释了相关性下降的一部分原因，而固定修订版本上的相反总体增益直接表明评审者响应存在差异。

We estimate the noise contribution from repeated scores: the noise variance of original-paper means is so2/3, and that of gains is (so2+sv2)/3. We subtract these components from the corresponding between-paper variances while retaining the cross-model covariance. Adjusted correlations for Luna revisions are approximately 0.52, still below 0.843 for original papers. Scoring fluctuations explain part of the correlation decline, while opposite aggregate gains for fixed revised versions directly demonstrate differing reviewer responses.

7 结论

7 Conclusion

将 AI 评分用作研究反馈涉及两种推断：从单个输出推断预期判断，以及从这些判断推断修订收益。重复评审可提高平均分精度，其预算由平均方差和误差容限决定，并进一步针对有限试点数据中的不确定性进行控制。在固定评分协议下，辅助概率结合少量推理分数可改进分布估计；改变输出粒度会同时改变变异性和评分中心。修订交叉进一步表明，即使原论文判断高度相关，相同的修订也可能导致方向相反的收益。分数精度阐明了一位评审员的判断，而对共享修订的跨评审员评估则检验这种反馈是否获得一致认可。

Using AI scores as research feedback involves two inferences: from individual outputs to expected judgments, and from those judgments to revision gains. Repeated reviews improve mean-score precision, with budgets determined by average variance and error tolerance and further controlled for uncertainty in finite pilot data. Under a fixed scoring protocol, auxiliary probabilities combined with a small number of reasoning scores improve distribution estimation; changing output granularity alters both variability and the scoring center. The revision crossover further shows that even highly correlated original-paper judgments can lead to gains in opposite directions for identical revisions. Score precision clarifies one reviewer’s judgment, while cross-reviewer evaluation of shared revisions tests whether this feedback receives consistent endorsement.

局限性

Limitations

采样预算针对固定配置下的新独立分数；模型版本、提示或服务环境的变化会改变目标。置信边界使用完整的 1–10 范围，并在尾部观测有限时偏保守。人类分数来自当前公开记录，PDF 可能是更新版本；两年使用不同的评分类别，相关性不能直接观测潜在质量。校准在一个 API 配置上验证，而修订交叉仅包含两个模型并省略了一种角色组合。修订中的事实保留由编辑协议要求，但尚未通过盲法人工评估合规性和质量收益来验证；使用现有评审生成编辑也可能影响噪声调整中的独立性假设。实验使用公开匿名稿件。研究结果分析评分行为；自动编辑和评分不能取代作者对论文内容的责任。

Sampling budgets target new independent scores under a fixed configuration; changes to model versions, prompts, or service environments change the target. Confidence bounds use the full 1–10 range and are conservative when tail observations are limited. Human scores come from current public records, and PDFs may be updated versions; the two years use different rating categories, and correlations do not directly observe latent quality. Calibration is validated on one API configuration, while the revision crossover includes only two models and omits one role combination. Factual preservation in revisions is required by the editing protocol but has not been validated through blinded human assessment of compliance and quality gains; using existing reviews to generate edits may also affect the independence assumption in noise adjustment. Experiments use public anonymized manuscripts. The findings analyze scoring behavior; automated editing and scoring do not replace authors’ responsibility for paper content.

参考文献

References

• AAAI (2025) AAAI AAAI 推出 AI 驱动的同行评审评估系统。 注：人工智能促进协会 外部链接：链接 引用：§1。
• Chiang and Lee (2023) C. Chiang and H. Lee 深入探讨使用大型语言模型进行自动评估。 载于 计算语言学协会发现：EMNLP 2023，H. Bouamor、J. Pino 和 K. Bali（编）， 新加坡，第 8928–8942 页。 外部链接：链接， 文档 引用：§2。
• Du (2025) S. Du TitleTrap：探究基于 LLM 的科学评审中的呈现偏差。 载于 第 5 届 NLP 系统评估与比较研讨会论文集，M. Akter、T. Chowdhury、S. Eger、C. Leiter、J. Opitz 和 E. Çano（编）， 印度孟买，第 119–125 页。 外部链接：链接， 文档， ISBN 979-8-89176-305-0 引用：§1， §2。
• Garg et al.（2025） M. K. Garg, T. Prasad, T. Singhal, C. Kirtani, M. Mandal, and D. Kumar ReviewEval：一个用于 AI 生成评审的评估框架。 载于 Findings of the Association for Computational Linguistics: EMNLP 2025，C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.)， 中国苏州，pp. 20542–20564。 外部链接：Link， Document， ISBN 979-8-89176-335-7 被引于：§1， §2。
• Hoeffding（1963） W. Hoeffding 有界随机变量之和的概率不等式。 Journal of the American Statistical Association 58（301），pp. 13–30。 外部链接：Document， Link 被引于：§D.3， §4.3。
• Homan et al.（2026） C. M. Homan, F. Korn, D. Pandita, and C. Welty 每个条目需要多少个评分才能进行可靠的显著性检验？。 载于 Findings of the Association for Computational Linguistics: EACL 2026，V. Demberg, K. Inui, and L. Marquez (Eds.)， 摩洛哥拉巴特，pp. 4258–4273。 外部链接：Link， Document， ISBN 979-8-89176-386-9 被引于：§2。
• Li et al.（2026a） L. Li, J. Xiong, C. Zhu, R. Yu, C. Chen, J. Wang, R. Ma, and Z. Lu LLM-as-a-reviewer：基准测试其作为论文评审员的能力、分歧与提示注入抵抗力。 外部链接：2605.25415， Document， Link 被引于：§1， §2。
• Li et al.（2026b） M. Li, C. Wang, X. Li, X. Zeng, D. Li, P. Shi, D. Zhou, and T. Zhou 修辞如何对 AI 评审员进行奖励黑客攻击？剖析 AI 同行评审中的修辞敏感性。 外部链接：2608.08975， Document， Link 被引于：§1， §2。
• Liu et al.（2023） Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu G-eval：使用 Gpt-4 进行 NLG 评估并具有更好的人类对齐。 载于 Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing，H. Bouamor, J. Pino, and K. Bali (Eds.)， 新加坡，pp. 2511–2522。 外部链接：Link， Document 被引于：§2。
• Stureborg et al. (2024) R. Stureborg, D. Alikaniotis, and Y. Suhara 大型语言模型是不一致且有偏见的评估者。 外部链接：2405.01724, Document, Link 引用位置：§2。
• Zou and Thakkar (2025) J. Zou and N. Thakkar 利用 LLM 反馈提升评审质量。 备注：ICLR Blog 外部链接：Link 引用位置：§1。

• AAAI (2025) AAAI AAAI launches AI-powered peer review assessment system. Note: Association for the Advancement of Artificial Intelligence External Links: Link Cited by: §1.
• Chiang and Lee (2023) C. Chiang and H. Lee A closer look into using large language models for automatic evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 8928–8942. External Links: Link, Document Cited by: §2.
• Du (2025) S. Du TitleTrap: probing presentation bias in LLM-based scientific reviewing. In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems, M. Akter, T. Chowdhury, S. Eger, C. Leiter, J. Opitz, and E. Çano (Eds.), Mumbai, India, pp. 119–125. External Links: Link, Document, ISBN 979-8-89176-305-0 Cited by: §1, §2.
• Garg et al. (2025) M. K. Garg, T. Prasad, T. Singhal, C. Kirtani, M. Mandal, and D. Kumar ReviewEval: an evaluation framework for AI-generated reviews. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, pp. 20542–20564. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §1, §2.
• Hoeffding (1963) W. Hoeffding Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association 58 (301), pp. 13–30. External Links: Document, Link Cited by: §D.3, §4.3.
• Homan et al. (2026) C. M. Homan, F. Korn, D. Pandita, and C. Welty How many ratings per item are necessary for reliable significance testing?. In Findings of the Association for Computational Linguistics: EACL 2026, V. Demberg, K. Inui, and L. Marquez (Eds.), Rabat, Morocco, pp. 4258–4273. External Links: Link, Document, ISBN 979-8-89176-386-9 Cited by: §2.
• Li et al. (2026a) L. Li, J. Xiong, C. Zhu, R. Yu, C. Chen, J. Wang, R. Ma, and Z. Lu LLM-as-a-reviewer: benchmarking their ability, divergence, and prompt injection resistance as paper reviewers. External Links: 2605.25415, Document, Link Cited by: §1, §2.
• Li et al. (2026b) M. Li, C. Wang, X. Li, X. Zeng, D. Li, P. Shi, D. Zhou, and T. Zhou How can rhetoric reward-hack AI reviewers? dissecting rhetorical sensitivity in AI-based peer review. External Links: 2608.08975, Document, Link Cited by: §1, §2.
• Liu et al. (2023) Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu G-eval: NLG evaluation using Gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 2511–2522. External Links: Link, Document Cited by: §2.
• Stureborg et al. (2024) R. Stureborg, D. Alikaniotis, and Y. Suhara Large language models are inconsistent and biased evaluators. External Links: 2405.01724, Document, Link Cited by: §2.
• Zou and Thakkar (2025) J. Zou and N. Thakkar Leveraging LLM feedback to enhance review quality. Note: ICLR Blog External Links: Link Cited by: §1.

附录 A 数据与抽样

Appendix A Data and Sampling

A.1 投稿总体与样本构成

A.1 Submission Populations and Sample Composition

本研究分别从 ICLR 2025 和 ICLR 2026 各纳入 100 篇论文。抽样总体分别包含 11,520 篇和 19,474 篇具有公开官方评审意见的投稿，涵盖接收、拒稿、撤稿和 desk-reject 的论文。论文列表在下载 PDF 或运行 AI 评审之前即已固定。AI 评分、PDF 可获得性以及与早期样本的重叠情况均未参与选择。

The study includes 100 papers each from ICLR 2025 and ICLR 2026. The sampling populations contain 11,520 and 19,474 submissions with publicly available official reviews, respectively, including accepted, rejected, withdrawn, and desk-rejected papers. The paper lists were fixed before downloading PDFs or running AI reviews. AI scores, PDF availability, and overlap with earlier samples played no role in selection.

2025 年样本按官方主要学科领域与人类评分均值分箱联合分层。2026 年样本采用第二版抽样设计，在这些维度上增加了投稿状态。

The 2025 sample was stratified jointly by official primary subject area and mean human-score bin. The 2026 sample uses the second version of the sampling design, which adds submission status to these dimensions.

A.2 分层配额与层内抽样

A.2 Stratified Quotas and Within-Stratum Sampling

对于 2025 年，使用 SciPy milp 和 HiGHS 联合求解了 100、300 和 1,000 三个嵌套样本量的整数配额。对于每个学科领域、评分分箱以及非空的领域×评分分箱层，样本数被约束为其总体占比乘以样本量后的向下取整或向上取整值，使计数偏差严格小于一篇论文。附加约束确保了学科领域和非空评分分箱的覆盖。在每个层内，论文按其精确的人类评分均值排序，并列者随机排序，并以随机起点进行系统抽样。先抽取 1,000 的样本，再从该样本中抽取 300，然后从 300 中抽取 100。

For 2025, integer quotas for three nested sample sizes of 100, 300, and 1,000 were jointly solved using SciPy milp and HiGHS. For each subject area, score bin, and nonempty area-by-score-bin stratum, the sample count was constrained to the floor or ceiling of its population share multiplied by the sample size, keeping the count discrepancy strictly below one paper. Additional constraints ensured coverage of subject areas and nonempty score bins. Within each stratum, papers were sorted by their exact mean human score, with random ordering of ties, and selected by systematic sampling with a random start. The sample of 1,000 was drawn first, followed by 300 from that sample and then 100 from the 300.

2026 年的第二版设计使用领域×评分分箱×状态分层。状态类别为 Poster、Oral、拒稿、撤稿和 desk reject。状态配额按最大余数法分配，接收论文总数固定为 27。领域、评分分箱和跨层配额保留向下取整/向上取整约束，层内选择同样在按精确均值排序后使用系统抽样。A.3 节报告最终的状态配额和计数。

The second 2026 design uses area-by-score-bin-by-status strata. Status categories are Poster, Oral, rejected, withdrawn, and desk rejected. Status quotas were assigned by the largest-remainder method, with the total number of accepted papers fixed at 27. Area, score-bin, and cross-stratum quotas retained the floor/ceiling constraints, and within-stratum selection again used systematic sampling after sorting by exact mean score. Section A.3 reports the final status quotas and counts.

每篇论文的人类均值是其有效官方评审分数的未加权平均值。分箱使用精确、未四舍五入的均值进行分配。2025 年的非空分箱范围从 [1,2) 到 [9,10]。2026 年总体包含低于 1 的均值，非空分箱从 [0,1) 到 [8,9)；其 [9,10] 分箱为空。完整分布如下。

The human mean for each paper is the unweighted average of its valid official review scores. Bins are assigned using the exact, unrounded mean. Nonempty bins range from [1,2) to [9,10] in 2025. The 2026 population includes means below 1, with nonempty bins from [0,1) to [8,9); its [9,10] bin is empty. The complete distributions follow.

A.3 学科领域与投稿状态分布

A.3 Subject-Area and Submission-Status Distributions

全部21个官方一级学科领域均有覆盖。该表保留原始领域名称，未合并小领域。

All 21 official primary subject areas are represented. The table retains the original area names without merging small areas.

2025年样本包含36篇录用、41篇拒稿和23篇撤稿论文。2026年样本包含27篇录用、43篇拒稿、26篇撤稿和4篇编辑直接拒稿论文。合计包括63篇录用、84篇拒稿、49篇撤稿和4篇编辑直接拒稿论文。

The 2025 sample contains 36 accepted, 41 rejected, and 23 withdrawn papers. The 2026 sample contains 27 accepted, 43 rejected, 26 withdrawn, and 4 desk-rejected papers. Together, these comprise 63 accepted, 84 rejected, 49 withdrawn, and 4 desk-rejected papers.

配额取整会使稀有类别进入小样本。例如，2025年总体中仅有九篇论文的人类评分均值不低于9，其中一篇被保留在100篇的样本中：样本占比为1%，而总体占比约为0.078%。学科领域和分数区间的覆盖并不意味着覆盖每个小型交叉分层；部分分层的取整配额为零。结果对所选论文赋予相同权重，未按总体占比重新加权。

Quota rounding allows rare categories to enter a small sample. For example, only nine papers in the 2025 population have a mean human score of at least 9, and one is retained in the sample of 100: a sample share of 1% versus a population share of approximately 0.078%. Coverage of subject areas and score bins does not imply coverage of every small cross-stratum; some receive a rounded quota of zero. Results give equal weight to the selected papers, without reweighting to population shares.

A.4 人类评分、PDF版本与录用标签

A.4 Human Scores, PDF Versions, and Acceptance Labels

人类评分取自冻结数据中的公开记录，可能包含反驳后的更新。论文PDF也可能是更新版本。2025年的人类评分选项为1、3、5、6、8和10，2026年为0、2、4、6、8和10。因此，C节还按年份分别报告关联和误差。

Human scores are taken from the public records in the frozen data and may include post-rebuttal updates. Paper PDFs may also be updated versions. Human score options are 1, 3, 5, 6, 8, and 10 for 2025, and 0, 2, 4, 6, 8, and 10 for 2026. Section C therefore also reports associations and errors separately by year.

录用标签由最终公开状态推导：accepted_*标记为1，拒稿、撤稿和编辑直接拒稿论文标记为0。主分析使用全部200篇论文，其中63篇正例、137篇负例。排除撤稿后剩余151篇论文：63篇正例和88篇负例。完整样本与排除撤稿样本在2025年分别包含100篇和77篇论文，2026年分别包含100篇和74篇。预测分数相等的正负例对为AUC贡献0.5。

Acceptance labels are derived from final public status: accepted_* is labeled 1, and rejected, withdrawn, and desk-rejected papers are labeled 0. The primary analysis uses all 200 papers, with 63 positives and 137 negatives. Excluding withdrawals leaves 151 papers: 63 positives and 88 negatives. The full and withdrawal-excluded samples contain 100 and 77 papers in 2025, and 100 and 74 in 2026. Positive-negative pairs with equal predicted scores receive a contribution of 0.5 to AUC.

A.5 各实验共享的样本与论文列表

A.5 Shared Samples and Paper Lists Across Experiments

缺少辅助概率的两篇论文与受Claude图像限制影响的论文不同。完整校准样本包含198篇论文，其中63篇录用、135篇未录用。排除其中48篇撤稿论文后剩余150篇论文，其中63篇录用、87篇未录用。修订实验使用固定的199篇论文共享集：100篇来自2025年，99篇来自2026年。

The two papers missing auxiliary probabilities differ from the paper affected by Claude’s image limit. The full calibration sample contains 198 papers, with 63 accepted and 135 nonaccepted. Excluding its 48 withdrawn papers leaves 150 papers, with 63 accepted and 87 nonaccepted. The revision experiment uses a fixed shared set of 199 papers: 100 from 2025 and 99 from 2026.

评分模型接收匿名化的稿件内容。人类评分、录用状态和其他模型的结果保留在分析元数据中。

Scoring models receive anonymized manuscript content. Human scores, acceptance status, and other models’ results remain in the analysis metadata.

附录 B 评审与修订协议

Appendix B Review and Revision Protocols

B.1 模型与评审设置

B.1 Models and Review Settings

每个实验固定其评审配置。C节和D节对每篇论文使用十个原始稿件评分，而E节的分布估计使用单独存储的每篇论文五个目标推理评分。

Each experiment fixes its review configuration. Sections C and D use ten original-manuscript scores per paper, whereas distribution estimation in Section E uses a separately stored set of five target reasoning scores per paper.

辅助概率调用使用 reasoning.effort="none"、top_logprobs=20 和 max_output_tokens=4096，并请求输出 token 的对数概率。目标推理评分使用相同的手稿输入和评分模式，采用 reasoning.effort="medium" 和 16,000 token 的输出预算。请求中未显式设置温度参数。

Auxiliary probability calls use reasoning.effort="none", top_logprobs=20, and max_output_tokens=4096, and request output-token log probabilities. Target reasoning scores use the same manuscript input and scoring schema, with reasoning.effort="medium" and an output budget of 16,000 tokens. Temperature is not explicitly set in the requests.

每次评审都在独立会话中运行。模型只能访问预加载的手稿文本和图像；工具、网络搜索、记忆和先前上下文均被禁用。分析仅使用通过格式和执行检查的有效结果。

Each review runs in an independent session. Models can access only the preloaded manuscript text and images; tools, web search, memory, and prior context are disabled. Analyses use valid results that pass format and execution checks.

B.2 手稿输入与图像处理

B.2 Manuscript Inputs and Image Processing

每份原始手稿由 text.json、index.json 和 figures/ 表示。文本记录包含稳定的 id、PDF 页码、记录类型和文本。index.json 中的 sequence 指定文本与图像的交替顺序。输入加载器按照此顺序组装全文、表格文本、参考文献和附录。图像在相对路径和 SHA-256 校验后加载。相邻的文本记录可以合并，但不得改变其顺序。

Each original manuscript is represented by text.json, index.json, and figures/. Text records contain a stable id, PDF page number, record type, and text. The sequence in index.json specifies the interleaving of text and images. The input loader follows this sequence to assemble the full text, table text, references, and appendices. Images are loaded after relative-path and SHA-256 checks. Adjacent text records may be merged without changing their order.

所有评分器均接收预加载的文本和图像。Haiku 按相同顺序处理图像，将最长边调整为 1,568 像素并在提交前重新编码，每次调用最多处理 100 张图像。论文 2026-paper_067 包含 224 张图像，超出此限制，因此修订交叉实验使用其余 199 篇论文。

All scorers receive preloaded text and images. Haiku processes images in the same order, resizing the longest side to 1,568 pixels and re-encoding them before submission, with a limit of 100 images per call. Paper 2026-paper_067 contains 224 images and exceeds this limit, so the revision crossover experiment uses the remaining 199 papers.

在 200 个输入包中，PDF 页码标记总计 4,923 页，图像索引条目总计 2,584 个。页数的中位数、第 90 百分位数和最大值分别为 23、38 和 69；相应的图像数量分别为 9、23 和 224。共有 133,967 条文本记录，包含 15,324,311 个字符。页数来自包中的 PDF 页码标记，图像数量来自索引条目，而非单个图或表面板。

Across the 200 input packages, PDF page markers total 4,923 pages and image-index entries total 2,584. The median, 90th percentile, and maximum page counts are 23, 38, and 69; the corresponding image counts are 9, 23, and 224. There are 133,967 text records containing 15,324,311 characters. Page counts come from PDF page markers in the packages, and image counts from index entries rather than individual figure or table panels.

B.3 整数评分任务

B.3 Integer-Scoring Task

评分器扮演 ICLR 审稿人，阅读带有交错图像的完整匿名手稿，评估新颖性、技术正确性、实验证据和局限性，并返回 1 到 10 的整数总体评分。锚点为：1 表示强烈拒稿，3 表示拒稿，5 表示略低于接收阈值，6 表示略高于接收阈值，8 表示接收，10 表示强烈接收。允许其他中间整数。

The scorer acts as an ICLR reviewer, reads the complete anonymized manuscript with interleaved images, evaluates novelty, technical correctness, experimental evidence, and limitations, and returns an integer overall score from 1 to 10. The anchors are 1 for strong reject, 3 for reject, 5 for marginally below the acceptance threshold, 6 for marginally above it, 8 for accept, and 10 for strong accept. Other intermediate integers are permitted.

输出包含总体评分、1 到 5 的置信度、简要总结、最多三条优点、最多三条疑虑以及评分理由。关键判断引用 PDF 页码。评分器不会收到人类评分、最终决定或其他评审意见。置信度不用于对评分加权。

The output contains an overall score, confidence from 1 to 5, a brief summary, up to three strengths, up to three concerns, and a scoring rationale. Key judgments cite PDF page numbers. The scorer does not receive human scores, final decisions, or other reviews. Confidence is not used to weight scores.

B.4 强制小数协议

B.4 Forced-Decimal Protocol

小数实验保持输入、模型和其他要求不变，将整数要求替换为强制非零小数部分。评分字段从 integer 改为 number，范围仍为 1–10。本地验证还会检查评分是否为非整数。每个模型的所有 1,000 个有效评分均满足此要求。

The decimal experiment keeps the inputs, models, and other requirements unchanged, replacing the integer requirement with a mandatory nonzero fractional part. The score field changes from integer to number, while its range remains 1–10. Local validation additionally checks that the score is noninteger. All 1,000 valid scores from each model satisfy this requirement.

一项仅允许小数的早期试点被弃用，并排除在所有分析之外。完整的强制小数实验在相同协议下复用了20篇论文的试点运行，随后纳入其余论文，从而为每个模型的200篇论文各获得五个分数。整数对照逐篇、逐轮与十次评审原始手稿实验中的前五个分数进行核对，重复次数保持一致。

An earlier pilot that merely allowed decimals was discarded and is excluded from all analyses. The full forced-decimal experiment reused the 20-paper pilot run under the same protocol, then incorporated the remaining papers to obtain five scores each for 200 papers per model. The integer controls were verified paper by paper and round by round against the first five scores in the ten-review original-manuscript experiment, matching the number of repetitions.

B.5 Writer、Selector 和 Scorer 的输入

B.5 Inputs to the Writer, Selector, and Scorer

Luna Writer 使用现有的第一份 GPT-5.6-Sol 评审作为反馈；源记录为全部200篇论文标识了此配置。Haiku Writer 使用 Haiku 对原始手稿的第一次评审。因此，两个 Writer 条件遵循各自的反馈流程。在 Writer 固定的情况下，两个 Selector 读取相同的候选差异。在最终手稿固定的情况下，两个 Scorer 读取相同的文本版本。Scorer 不接收候选差异、选择理由、其他评分者的分数，也不接收将该手稿标识为修订版的比较提示。

The Luna Writer uses the first existing GPT-5.6-Sol review as feedback; source records identify this configuration for all 200 papers. The Haiku Writer uses Haiku’s first review of the original manuscript. The two Writer conditions thus follow their respective feedback pipelines. With the Writer fixed, both Selectors read the same candidate diffs. With the final manuscript fixed, both Scorers read the same text version. Scorers receive no candidate diffs, selection rationales, other scorers’ scores, or comparative prompt identifying the manuscript as revised.

Writer 返回要替换的文本记录。Selector 对每个候选编辑给出接受决定和理由，每个候选恰好一个决定。程序在生成最终手稿前验证记录 ID、决定数量和输出格式。

The Writer returns text records to replace. The Selector gives an acceptance decision and a reason for each candidate edit, with exactly one decision per candidate. The program validates record IDs, decision counts, and output format before generating the final manuscript.

Writer 被指示保留事实、数值、数据、证据范围、记录顺序、页码和非文本字段。开头指令和页面标记必须保持不变。返回的文本被合并到候选手稿中，并据此生成块级差异。每条 Selector 评论限制为32个字符；较长的 Claude 评论的截断情况会被记录。最终手稿从原始记录复制，仅将已接受编辑的文本替换为候选文本。F.5 节报告结构检查。

The Writer is instructed to preserve facts, numerical values, data, evidential scope, record order, page numbers, and nontext fields. Leading instructions and page markers must remain unchanged. Returned text is merged into a candidate manuscript, from which block-level diffs are generated. Each Selector comment is limited to 32 characters; truncations of longer Claude comments are recorded. The final manuscript is copied from the original records, replacing only the text of accepted edits with candidate text. Section F.5 reports structural checks.

附录 C 扩展的评分可靠性结果

Appendix C Extended Score Reliability Results

C.1 指标与聚合单位

C.1 Metrics and Aggregation Units

设论文 i 的十个分数为 Yi​1,…,Yi,10，均值为 Y¯i。其无偏的论文内方差为

Let the ten scores for paper i be Yi​1,…,Yi,10, with mean Y¯i. Its unbiased within-paper variance is

论文内方差的均值为 V^=N−1​∑iSi2，此处报告的论文内 SD 为 V^。所有分数的 SD 还包含论文之间均值分数的差异。完全相同的10/10比率是十次观测完全相同的论文所占比例，极差是最高分减去最低分。成对完全一致率先在每篇论文内的 (102)=45 个无序分数对上取平均，再跨论文取平均。

The mean within-paper variance is V^=N−1​∑iSi2, and the within-paper SD reported here is V^. The SD of all scores additionally includes differences in mean scores between papers. The identical-10/10 rate is the fraction of papers with ten identical observations, and the range is the maximum minus the minimum score. Pairwise exact agreement is first averaged over the (102)=45 unordered score pairs within each paper, then across papers.

所有 GPT-6-Sol 论文的分数极差至多为一分，其中113篇有十个完全相同的分数。相应的相同分数计数为 GPT-5.6-Sol 60篇、Luna 65篇。按年份的结果显示相同的配置差异：Sol 的论文内 SD 在2025年为0.2614，在2026年为0.2824。

All GPT-6-Sol papers have a score range of at most one point, and 113 have ten identical scores. The corresponding identical-score counts are 60 for GPT-5.6-Sol and 65 for Luna. Year-specific results show the same configuration differences: Sol’s within-paper SD is 0.2614 in 2025 and 0.2824 in 2026.

C.2 与人类评分及最终接受的关联

C.2 Association with Human Scores and Final Acceptance

Pearson 相关、Spearman 相关、MAE 和 RMSE 将每篇论文的十次评审均值与其公开的人类平均分进行比较。AUC 使用相同的十次评审均值来预测 A.4 节中定义的接受标签。

Pearson correlation, Spearman correlation, MAE, and RMSE compare each paper’s ten-review mean with its public mean human score. AUC uses the same ten-review mean to predict the acceptance labels defined in Section A.4.

2025 年 Spearman 相关系数范围为 0.4017 至 0.5457，2026 年为 0.1572 至 0.2345。全样本 AUC 在 2025 年为 0.6756 至 0.7852，2026 年为 0.6010 至 0.6185。这些按年份的结果使用对应年份的评分和标签。

Spearman correlations range from 0.4017 to 0.5457 in 2025 and from 0.1572 to 0.2345 in 2026. Full-sample AUCs range from 0.6756 to 0.7852 in 2025 and from 0.6010 to 0.6185 in 2026. These year-specific results use the scores and labels for the corresponding year.

C.3 评分集中度与论文级方差分布

C.3 Score Concentration and Paper-Level Variance Distributions

Luna 的评分集中在 2–4。Sol 在 1,042 个案例中给出评分 5，占其 2,000 个评分的 52.1%。5 至 7 分的评分占 Sol 输出的 65.20%，其中大多数为 5 或 6。因此，重复评分的范围与论文之间的数值间隔描述了评分中不同的方面。

Luna’s scores are concentrated at 2–4. Sol assigns a score of 5 in 1,042 cases, or 52.1% of its 2,000 scores. Scores from 5 to 7 account for 65.20% of Sol’s outputs, most of them at 5 or 6. The range of repeated scores and numerical separation between papers therefore describe different aspects of scoring.

C.4 前五次与后五次评审

C.4 First Five Versus Last Five Reviews

这十次评审结合了历史评分与相同配置下的后续运行。该表将第 1–5 轮与第 6–10 轮分开，计算每篇论文内批次均值之间的差异，并对论文进行自助抽样。

The ten reviews combine historical scores with later runs under the same configuration. The table separates rounds 1–5 from rounds 6–10, computes the difference between batch means within each paper, and bootstraps papers.

对于所有三个模型，批次间总体差异的区间均包含零。该分析记录批次位置之间的偏移；D 节中的精度预算关注固定配置下未来独立评分的期望。

For all three models, the interval for the overall difference between batches includes zero. This analysis records shifts between batch positions; the precision budgets in Section D concern the expectation of future independent scores under a fixed configuration.

C.5 全部十次评审的排名可复现性

C.5 Ranking Reproducibility Across All Ten Reviews

对于每篇论文，我们从其十次评审中构建两个互不重叠的 k 个评分块。将这对块视为无序，得到

For each paper, we construct two nonoverlapping blocks of k scores from its ten reviews. Treating the pair of blocks as unordered gives

由此得到的分区数分别为 45、630、2,100、1,575 和 126。对于每个分区，在同一年份的论文之间比较块均值，然后在分区和论文对上取平均。两年共产生 2​(1002)=9,900 个不同的论文对。

The resulting numbers of partitions are 45, 630, 2,100, 1,575, and 126. For each partition, block means are compared between papers from the same year, then averaged over partitions and paper pairs. The two years yield 2​(1002)=9,900 distinct paper pairs.

Same 表示两个块中严格排序相同；Tie 表示至少一个块中出现并列；Reverse 表示严格排序相反。这些类别互斥且完备。

Same denotes the same strict ordering in both blocks; Tie denotes a tie in at least one block; and Reverse denotes opposite strict orderings. These categories are mutually exclusive and exhaustive.

C.6 使用独立分组评分的接近对分析

C.6 Close-Pair Analysis with Independent Grouping Scores

为考察接近的论文对，我们按前五次均值之间的绝对差对论文对进行分组，保留后五次评分用于排名比较。在后五次中，单次评审块枚举全部 10 个分区，两次评审块枚举全部 15 个分区。因此，分组和排名比较使用不同的评分。

To examine close paper pairs, we group pairs by the absolute difference between their first-five means, reserving the last five scores for ranking comparisons. Among the last five, single-review blocks enumerate all 10 partitions, and two-review blocks enumerate all 15 partitions. Grouping and ranking comparisons thus use separate scores.

前五轮均值差异被分为 0、(0,0.4]、(0.4,1]、(1,2] 和大于 2 分这几组。这些差异是有限样本估计值。下表中的 Same、Tie 和 Reverse 衡量的是两个后续独立评审块之间的可复现性，并不把分组估计视为正确排序。

The first-five mean differences are grouped as 0, (0,0.4], (0.4,1], (1,2], and greater than 2 points. These differences are finite-sample estimates. Same, Tie, and Reverse below measure reproducibility between two subsequent independent review blocks, without treating the grouping estimate as the correct ordering.

对于前五轮均值差异位于 (0,0.4] 的论文对，后续单次评审比较在 74.84%–85.01% 的情况下产生并列；若使用两次评审的均值，这一比例仍为 55.73%–68.45%。对于 (1,2] 组，使用两次评审均值的严格同序率达到 97.95%–99.58%。因此，在所有论文对上平均得到的提升，综合了不同评分差距区间的贡献。

For pairs whose first-five mean difference lies in (0,0.4], subsequent single-review comparisons produce ties in 74.84%–85.01% of cases; with two-review means, this remains 55.73%–68.45%. For the (1,2] group, strict same-order rates with two-review means reach 97.95%–99.58%. Improvements averaged across all pairs therefore combine contributions from different score-gap regions.

每篇论文会参与多个论文对，从而在论文对之间产生依赖性。我们精确枚举所有划分并报告描述性比例，而不把 9,900 个论文对视为二项区间的独立观测。

Each paper participates in multiple pairs, creating dependence across pairs. We enumerate all partitions exactly and report descriptive proportions, without treating the 9,900 paper pairs as independent observations for binomial intervals.

C.7 现有评审语料中的语义一致性

C.7 Semantic Consistency in the Existing Review Corpus

现有文本分析语料包含 Luna 和 Sol 各五份原始稿件评审、Haiku 三份，覆盖同一组 199 篇论文。该语料不包含后续第 6–10 轮。我们使用这一固定语料中所有可用的轮次：五份评审产生 10 个模型内论文对，三份产生 3 个，跨模型比较产生 25 或 15 个论文对。

The existing text-analysis corpus contains five original-manuscript reviews each from Luna and Sol and three from Haiku, for a shared set of 199 papers. It excludes the later rounds 6–10. We use every available round in this fixed corpus: five reviews yield 10 within-model pairs, three yield 3, and cross-model comparisons yield 25 or 15 pairs.

每条优点和每条关切均使用 sentence-transformers/all-mpnet-base-v2 编码。对于两个极性相同的列表，我们计算两两余弦相似度矩阵，并使用匈牙利算法进行一对一匹配。匹配余弦值之和除以较长列表的长度，未匹配项贡献为零：

Each strength and concern is encoded with sentence-transformers/all-mpnet-base-v2. For two lists of the same polarity, we compute the pairwise cosine-similarity matrix and use the Hungarian algorithm for one-to-one matching. The sum of matched cosines is divided by the longer list’s length, with unmatched items contributing zero:

语料中所有优点和关切列表均非空。评审对数值先在每篇论文和每种条件下取平均，再跨论文取平均。该表覆盖 15,522 个评审对。置信区间沿用原分析的 5,000 次论文级 bootstrap 重采样。

All strength and concern lists in the corpus are nonempty. Review-pair values are first averaged within each paper and condition, then across papers. The table covers 15,522 review pairs. Confidence intervals retain the original analysis’s 5,000 paper-level bootstrap resamples.

同一模型的重复评审平均而言观点列表更接近，且关切列表比优点列表更接近。现有助手对 40 个匹配观点对的检查发现了共享核心、部分重叠和互补观察；这不构成人工盲评金标准。通用 NLI 也对一些复合技术陈述进行了误分类。因此，我们保留可复现的连续语义度量，而不将自动蕴含阈值转换为实质性一致率。

Repeated reviews from the same model have closer opinion lists on average, and concern lists are closer than strength lists. An existing assistant inspection of 40 matched opinion pairs found shared cores, partial overlaps, and complementary observations; it does not constitute a human blind-evaluation gold standard. General-purpose NLI also misclassified some compound technical statements. We therefore retain the reproducible continuous semantic measure rather than converting automatic entailment thresholds into rates of substantive agreement.

附录 D 评审预算与平均分精度

Appendix D Review Budgets and Mean-Score Precision

D.1 跨论文的平均成功概率

D.1 Average Success Probability Across Papers

我们对一组固定的 N=200 篇论文赋予相同权重。对于论文 i，令 μi 和 σi2 表示在固定配置下得分的均值和方差。令 Y¯i,k 为 k 次未来独立得分的均值，并定义

We give equal weight to a fixed set of N=200 papers. For paper i, let μi and σi2 denote the mean and variance of scores under a fixed configuration. Let Y¯i,k be the mean of k future independent scores, and define

目标是 A⁡(k,ϵ)≥1−β。这是对于均匀选取的样本论文，k 次得分的均值在 ϵ 范围内估计模型期望得分的总体概率。主论文设定 β=0.05，并考虑 ϵ=0.25 和 0.5。

The target is A⁡(k,ϵ)≥1−β. This is the overall probability that the mean of k scores estimates the model’s expected score to within ϵ for a uniformly selected sample paper. The main paper sets β=0.05 and considers ϵ=0.25 and 0.5.

在每篇论文内独立同分布评分的情况下，Var⁡(Y¯i,k)=σi2/k。对失败事件 |Y¯i,k−μi|≥ϵ 应用切比雪夫不等式可得

Under independent, identically distributed scoring within each paper, Var⁡(Y¯i,k)=σi2/k. Applying Chebyshev’s inequality to the failure event |Y¯i,k−μi|≥ϵ gives

This expression gives a sufficient budget when the score scale is known and the mean and variance exist. It requires neither a between-paper score gap nor a normality assumption or an empirical histogram as ground truth.

This expression gives a sufficient budget when the score scale is known and the mean and variance exist. It requires neither a between-paper score gap nor a normality assumption or an empirical histogram as ground truth.

D.2 Planning Budgets from Observed Variance

D.2 Planning Budgets from Observed Variance

With n pilot scores per paper, estimate within-paper variance by V^=N−1​∑iSi2. Substituting V^ into the preceding expression gives the Planning budget. With ten pilot scores, the three configurations have V^ values of 0.140056, 0.148889, and 0.074056. Planning budgets are 45, 48, and 24 at a tolerance of 0.25 points, and 12, 12, and 6 at 0.5 points.

With n pilot scores per paper, estimate within-paper variance by V^=N−1​∑iSi2. Substituting V^ into the preceding expression gives the Planning budget. With ten pilot scores, the three configurations have V^ values of 0.140056, 0.148889, and 0.074056. Planning budgets are 45, 48, and 24 at a tolerance of 0.25 points, and 12, 12, and 6 at 0.5 points.

Planning expresses the budget implied by the observed variance. The next section also accounts for uncertainty in variance estimation from a finite pilot sample.

Planning expresses the budget implied by the observed variance. The next section also accounts for uncertainty in variance estimation from a finite pilot sample.

D.3 An Upper Bound on Mean Variance from Nonoverlapping Pairs

D.3 An Upper Bound on Mean Variance from Nonoverlapping Pairs

Scores lie in [a,b]=[1,10]. Define the symmetric kernel

Scores lie in [a,b]=[1,10]. Define the symmetric kernel

For two independent, identically distributed scores, 𝔼​h​(Y,Y′)=σi2. The unbiased sample variance for each paper also equals the average kernel value over all score pairs:

For two independent, identically distributed scores, 𝔼​h​(Y,Y′)=σi2. The unbiased sample variance for each paper also equals the average kernel value over all score pairs:

Let K=⌊n/2⌋. Randomly permute each paper’s n scores and form K nonoverlapping pairs from the first 2​K scores. Let Ti be their mean kernel value and T=N−1​∑iTi its average across papers. Conditional on the observed scores, every unordered pair has the same probability of selection, so

Let K=⌊n/2⌋. Randomly permute each paper’s n scores and form K nonoverlapping pairs from the first 2​K scores. Let Ti be their mean kernel value and T=N−1​∑iTi its average across papers. Conditional on the observed scores, every unordered pair has the same probability of selection, so

For fixed permutations, there are M=N​K nonoverlapping kernel values. Assuming calls are independent across papers, these values are independent, and their expectations average to V. For any real λ, Jensen’s inequality gives

For fixed permutations, there are M=N​K nonoverlapping kernel values. Assuming calls are independent across papers, these values are independent, and their expectations average to V. For any real λ, Jensen’s inequality gives

For X∈[0,1], the chord bound for the exponential function gives

For X∈[0,1], the chord bound for the exponential function gives

The kernel values may have different expectations. Multiplying the moment-generating functions of the independent variables and applying concavity of the logarithm yields a bound depending only on the overall mean V/B:

The kernel values may have different expectations. Multiplying the moment-generating functions of the independent variables and applying concavity of the logarithm yields a bound depending only on the overall mean V/B:

For v<V, taking λ<0 and optimizing the Chernoff parameter gives

For v<V, taking λ<0 and optimizing the Chernoff parameter gives

where

where

Given the observed V^, let Uα be the root above V^ of

Given the observed V^, let Uα be the root above V^ of

Inverting the lower-tail bound yields

Inverting the lower-tail bound yields

The score range also implies V≤(b−a)2/4=20.25, so the final upper bound can be capped at 20.25. The derivation uses a Chernoff argument for bounded variables and random grouping of a U-statistic; see Hoeffding (1963).

The score range also implies V≤(b−a)2/4=20.25, so the final upper bound can be capped at 20.25. The derivation uses a Chernoff argument for bounded variables and random grouping of a U-statistic; see Hoeffding (1963).

With ten pilot reviews, all 45 pairs per paper, or 9,000 pairs overall, contribute to the unbiased estimate V^, while the concentration bound uses M=200×5=1,000. These quantities serve variance estimation and confidence control, respectively; the overlapping 9,000 pairs are not treated as independent kernel values.

With ten pilot reviews, all 45 pairs per paper, or 9,000 pairs overall, contribute to the unbiased estimate V^, while the concentration bound uses M=200×5=1,000. These quantities serve variance estimation and confidence control, respectively; the overlapping 9,000 pairs are not treated as independent kernel values.

D.4 The Two 95% Levels

D.4 The Two 95% Levels

Choose k⁡(Uα)=⌈Uα/(β​ϵ2)⌉ from the pilot data. For separately collected, independent future scores,

Choose k⁡(Uα)=⌈Uα/(β​ϵ2)⌉ from the pilot data. For separately collected, independent future scores,

The main paper uses α=β=0.05. The first 95% is the confidence level for the variance upper bound constructed from the pilot data; the second is the overall success probability of future mean-score estimation. The budget k counts new independent reviews and excludes the existing pilot reviews.

The main paper uses α=β=0.05. The first 95% is the confidence level for the variance upper bound constructed from the pilot data; the second is the overall success probability of future mean-score estimation. The budget k counts new independent reviews and excludes the existing pilot reviews.

Averaging over both pilot-data and future-score randomness gives a joint success lower bound of (1−α)​(1−β). This equals 0.9025 when α=β=0.05. To require a joint lower bound of at least 0.95 in advance, one can choose α=0.01,β=0.04, giving 0.99×0.96=0.9504. Section D.5 reports the corresponding budgets.

Averaging over both pilot-data and future-score randomness gives a joint success lower bound of (1−α)​(1−β). This equals 0.9025 when α=β=0.05. To require a joint lower bound of at least 0.95 in advance, one can choose α=0.01,β=0.04, giving 0.99×0.96=0.9504. Section D.5 reports the corresponding budgets.

D.5 Simultaneous Control Across Models and Different Tolerances

D.5 Simultaneous Control Across Models and Different Tolerances

Assigning failure probability α/3 to each model’s variance bound gives simultaneous coverage of at least 1−α for all three bounds by Bonferroni’s inequality. Each model’s overall future-score success probability remains 1−β.

Assigning failure probability α/3 to each model’s variance bound gives simultaneous coverage of at least 1−α for all three bounds by Bonferroni’s inequality. Each model’s overall future-score success probability remains 1−β.

Halving the tolerance approximately quadruples the variance-based budget, and a higher success target also increases the budget. All tables round budgets upward to integer review counts.

Halving the tolerance approximately quadruples the variance-based budget, and a higher success target also increases the budget. All tables round budgets upward to integer review counts.

D.6 Five Versus Ten Pilot Reviews

D.6 Five Versus Ten Pilot Reviews

Ten pilot reviews increase the number of independent kernel values available to the bound from 400 to 1,000. Observed variances change little, but the upper bounds tighten: Certified budgets at a half-point tolerance decrease from 53, 55, and 43 to 33, 35, and 24.

Ten pilot reviews increase the number of independent kernel values available to the bound from 400 to 1,000. Observed variances change little, but the upper bounds tighten: Certified budgets at a half-point tolerance decrease from 53, 55, and 43 to 33, 35, and 24.

D.7 A Reference Bound Using Only the Score Range

D.7 A Reference Bound Using Only the Score Range

Without pilot variance estimates, the paper-specific Hoeffding bound gives

Without pilot variance estimates, the paper-specific Hoeffding bound gives

Thus k≥⌈81​log⁡(2/β)/(2​ϵ2)⌉ suffices to control each paper’s failure probability and hence the average failure probability. With β=.05, tolerances of 0.25 and 0.5 points require 2,391 and 598 reviews, respectively. Our primary analysis uses pilot variances and their upper bounds to exploit the observed dispersion of scores.

Thus k≥⌈81​log⁡(2/β)/(2​ϵ2)⌉ suffices to control each paper’s failure probability and hence the average failure probability. With β=.05, tolerances of 0.25 and 0.5 points require 2,391 and 598 reviews, respectively. Our primary analysis uses pilot variances and their upper bounds to exploit the observed dispersion of scores.

Appendix E Score Granularity and Distribution Estimation

Appendix E Score Granularity and Distribution Estimation

E.1 Integer Versus Forced-Decimal Scoring

E.1 Integer Versus Forced-Decimal Scoring

We compare the two protocols within papers, with five scores per protocol: 1,000 observations per model and protocol. Integer scores are exactly the first five from the ten-review original-manuscript experiment. The table also reports year-specific results and both acceptance-label conventions.

We compare the two protocols within papers, with five scores per protocol: 1,000 observations per model and protocol. Integer scores are exactly the first five from the ten-review original-manuscript experiment. The table also reports year-specific results and both acceptance-label conventions.

The forced-decimal protocol raises Luna’s mean score by 0.693 points and Sol’s by 0.266. Their within-paper SDs decrease from 0.390 to 0.329 and from 0.286 to 0.249, respectively. Luna’s MAE decreases in both years. Sol’s MAE changes in opposite directions across years, with both intervals including zero.

The forced-decimal protocol raises Luna’s mean score by 0.693 points and Sol’s by 0.266. Their within-paper SDs decrease from 0.390 to 0.329 and from 0.286 to 0.249, respectively. Luna’s MAE decreases in both years. Sol’s MAE changes in opposite directions across years, with both intervals including zero.

Output granularity affects exact agreement and score range under the integer protocol. If scores switch only between adjacent integers s and s+1, with probabilities 1−π and π, the variance is π⁡(1−π) and the probability of disagreement between two reviews is 2​π​(1−π). This two-point model shows how switching between adjacent levels can produce frequent disagreement within a small range. The decimal experiment directly measures the variance and score-location changes caused by a protocol change, without assuming that lower variance implies a more accurate estimate of an unchanged scoring target.

Output granularity affects exact agreement and score range under the integer protocol. If scores switch only between adjacent integers s and s+1, with probabilities 1−π and π, the variance is π⁡(1−π) and the probability of disagreement between two reviews is 2​π​(1−π). This two-point model shows how switching between adjacent levels can produce frequent disagreement within a small range. The decimal experiment directly measures the variance and score-location changes caused by a protocol change, without assuming that lower variance implies a more accurate estimate of an unchanged scoring target.

E.2 Construction of Quick and Standard

E.2 Construction of Quick and Standard

The target is the score distribution pi​(s) under the fixed GPT-5.6-Sol medium integer configuration, for s∈{1,…,10}. An auxiliary call uses a nonreasoning configuration and extracts the visible probabilities of valid candidate score tokens. Restricting candidates to 1–10 and renormalizing yields ri​(s). The analysis does not assign probability mass to candidates absent from the response.

The target is the score distribution pi​(s) under the fixed GPT-5.6-Sol medium integer configuration, for s∈{1,…,10}. An auxiliary call uses a nonreasoning configuration and extracts the visible probabilities of valid candidate score tokens. Restricting candidates to 1–10 and renormalizing yields ri​(s). The analysis does not assign probability mass to candidates absent from the response.

Quick first adjusts probability concentration:

Quick first adjusts probability concentration:

Here 𝒮i is the valid candidate set returned by the call. It then maps score locations to xs=clip⁡(a+b​s,1,10) and linearly distributes the mass at each location between adjacent integers. Integer locations receive all their mass; otherwise, the weights at ⌊xs⌋ and ⌈xs⌉ are ⌈xs⌉−xs and xs−⌊xs⌋, respectively. Denote the resulting distribution by qQ,i.

Here 𝒮i is the valid candidate set returned by the call. It then maps score locations to xs=clip⁡(a+b​s,1,10) and linearly distributes the mass at each location between adjacent integers. Integer locations receive all their mass; otherwise, the weights at ⌊xs⌋ and ⌈xs⌉ are ⌈xs⌉−xs and xs−⌊xs⌋, respectively. Denote the resulting distribution by qQ,i.

Standard combines Quick with two reasoning scores from the target configuration:

Standard combines Quick with two reasoning scores from the target configuration:

A Quick prediction uses one auxiliary call. Standard uses that same auxiliary call and two reasoning reviews.

A Quick prediction uses one auxiliary call. Standard uses that same auxiliary call and two reasoning reviews.

E.3 Parameter Search and Train–Test Splits

E.3 Parameter Search and Train–Test Splits

The 198 papers are sorted by anonymous ID, and 30 random 99/99 train–test splits are generated without additional stratification by year. The search grid is

The 198 papers are sorted by anonymous ID, and 30 random 99/99 train–test splits are generated without additional stratification by year. The search grid is

Quick has 234 parameter combinations. First, (T,a,b) is selected using Brier loss against five target scores per training paper. Next, all ten partitions of five scores into two inputs and three targets are enumerated, and the mean training loss selects one of eight candidate weights. Test papers do not participate in parameter selection, and Standard’s input and target scores do not overlap.

Quick has 234 parameter combinations. First, (T,a,b) is selected using Brier loss against five target scores per training paper. Next, all ten partitions of five scores into two inputs and three targets are enumerated, and the mean training loss selects one of eight candidate weights. Test papers do not participate in parameter selection, and Standard’s input and target scores do not overlap.

The parameter combinations selected across the 30 splits are as follows.

The parameter combinations selected across the 30 splits are as follows.

At test time, Quick consistently uses the first valid auxiliary output and is evaluated against five independent target scores. Standard enumerates ten partitions per paper, first averaging over partitions, then over that paper’s test-set appearances, and finally equally across the 198 papers. Each paper appears in the test set 8–23 times.

At test time, Quick consistently uses the first valid auxiliary output and is evaluated against five independent target scores. Standard enumerates ten partitions per paper, first averaging over partitions, then over that paper’s test-set appearances, and finally equally across the 198 papers. Each paper appears in the test set 8–23 times.

E.4 Squared Distribution Risk with Independent Targets

E.4 Squared Distribution Risk with Independent Targets

For a predicted distribution q and independent target scores Z1,…,Zm, define

For a predicted distribution q and independent target scores Z1,…,Zm, define

Conditional on q, the expectation of the second term is −2∑sq(s)pi(s) and that of the final term is ∑spi​(s)2. Therefore,

Conditional on q, the expectation of the second term is −2∑sq(s)pi(s) and that of the final term is ∑spi​(s)2. Therefore,

The final term estimates the squared probability sum through collisions between independent target scores, removing variation due to a finite target sample. Individual paper-level risk estimates may be negative; these values are retained when averaging.

The final term estimates the squared probability sum through collisions between independent target scores, removing variation due to a finite target sample. Individual paper-level risk estimates may be negative; these values are retained when averaging.

For a frequency estimator based on n direct scores, expected risk is

For a frequency estimator based on n direct scores, expected risk is

Direct-sampling baselines estimate the numerator from the collision statistic of the five existing scores and divide by n. Risks for 1, 2, 3, and 5 reviews are estimates of expected risk under this sampling law, without treating the five-score histogram as a known target distribution.

Direct-sampling baselines estimate the numerator from the collision statistic of the five existing scores and divide by n. Risks for 1, 2, 3, and 5 reviews are estimates of expected risk under this sampling law, without treating the five-score histogram as a known target distribution.

For an auxiliary prediction conditionally independent of two reasoning scores, and a frequency estimator unbiased for pi, Standard’s risk decomposes as

For an auxiliary prediction conditionally independent of two reasoning scores, and a frequency estimator unbiased for pi, Standard’s risk decomposes as

The cross term vanishes because the reasoning-frequency error has mean zero and is conditionally independent of the auxiliary prediction. Weights are selected on training papers, and the combined predictor’s risk is evaluated on test papers.

The cross term vanishes because the reasoning-frequency error has mean zero and is conditionally independent of the auxiliary prediction. Weights are selected on training papers, and the combined predictor’s risk is evaluated on test papers.

E.5 Pooled and Year-Specific Risks

E.5 Pooled and Year-Specific Risks

Overall Standard risk is 0.0829. Year-specific risks are 0.0932 in 2025 and 0.0727 in 2026, with 99 papers per year; two direct reasoning reviews yield 0.1520 and 0.0914. All intervals hold the original parameters and splits fixed while resampling papers, preserving the same within-paper aggregation order.

Overall Standard risk is 0.0829. Year-specific risks are 0.0932 in 2025 and 0.0727 in 2026, with 99 papers per year; two direct reasoning reviews yield 0.1520 and 0.0914. All intervals hold the original parameters and splits fixed while resampling papers, preserving the same within-paper aggregation order.

E.6 Bootstrap with Parameter Refitting

E.6 Bootstrap with Parameter Refitting

For each paper-level refit bootstrap, we draw 198 papers with replacement from the 198-paper set, record each paper’s multiplicity, and use it as a weight within the original 30 train–test splits. The training side searches all 234 Quick parameter combinations and eight Standard weights again. The test side continues to exclude training papers for that split. Refit results retain the original within-paper averaging over test appearances and are then aggregated using the bootstrap paper weights.

For each paper-level refit bootstrap, we draw 198 papers with replacement from the 198-paper set, record each paper’s multiplicity, and use it as a weight within the original 30 train–test splits. The training side searches all 234 Quick parameter combinations and eight Standard weights again. The test side continues to exclude training papers for that split. Refit results retain the original within-paper averaging over test appearances and are then aggregated using the bootstrap paper weights.

This procedure preserves the original split design while propagating uncertainty from paper resampling and parameter selection. We run 1,000 resamples using frozen scores and probabilities, without new reviews. The original, unresampled fit and paper-level risks match the saved main-paper results to a tolerance of 10−12.

This procedure preserves the original split design while propagating uncertainty from paper resampling and parameter selection. We run 1,000 resamples using frozen scores and probabilities, without new reviews. The original, unresampled fit and paper-level risks match the saved main-paper results to a tolerance of 10−12.

With refitting, the risk difference between Standard and two reasoning reviews is −0.0388, with a 95% interval of [−0.0525, −0.0259]. Relative to three reasoning reviews, it is +0.0018, with an interval of [−0.0062, +0.0091]. The finding that Standard reduces risk relative to two reasoning reviews and approaches the risk of three persists after parameter reselection.

With refitting, the risk difference between Standard and two reasoning reviews is −0.0388, with a 95% interval of [−0.0525, −0.0259]. Relative to three reasoning reviews, it is +0.0018, with an interval of [−0.0062, +0.0091]. The finding that Standard reduces risk relative to two reasoning reviews and approaches the risk of three persists after parameter reselection.

E.7 Acceptance AUC Under the Same Held-Out Design

E.7 Acceptance AUC Under the Same Held-Out Design

All methods use the same 198 papers, parameters, and test splits as the risk analysis. Predictions are distributional expected scores; parameters are not fitted to acceptance labels. Direct-reasoning baselines enumerate all equal-sized subsets of the five scores, while Standard enumerates the ten two-input combinations. AUC is computed on test papers for each combination, then averaged over combinations and the 30 splits.

All methods use the same 198 papers, parameters, and test splits as the risk analysis. Predictions are distributional expected scores; parameters are not fitted to acceptance labels. Direct-reasoning baselines enumerate all equal-sized subsets of the five scores, while Standard enumerates the ten two-input combinations. AUC is computed on test papers for each combination, then averaged over combinations and the 30 splits.

On all 198 papers, Quick exceeds one reasoning review in AUC by +0.0197, and Standard exceeds two reasoning reviews by +0.0131. Excluding withdrawals gives differences of +0.0141 and +0.0115. This table reports point estimates; intervals for squared risk concern a different statistic and are not AUC intervals.

On all 198 papers, Quick exceeds one reasoning review in AUC by +0.0197, and Standard exceeds two reasoning reviews by +0.0131. Excluding withdrawals gives differences of +0.0141 and +0.0115. This table reports point estimates; intervals for squared risk concern a different statistic and are not AUC intervals.

E.8 Independent-Target Validation on Ten Papers

E.8 Independent-Target Validation on Ten Papers

The existing independent-target validation sampled five papers per year, excluding the long-input paper 2026-paper_067 before sampling. Each paper received ten additional GPT-5.6-Sol medium target scores. Predictions use the original five scores. Quick and Standard are fitted on the other 188 papers, yielding T=8,a=0.2,b=0.9,w=0.7.

The existing independent-target validation sampled five papers per year, excluding the long-input paper 2026-paper_067 before sampling. Each paper received ten additional GPT-5.6-Sol medium target scores. Predictions use the original five scores. Quick and Standard are fitted on the other 188 papers, yielding T=8,a=0.2,b=0.9,w=0.7.

For Quick, every available auxiliary output for a paper forms a separate one-call prediction, and losses are averaged without first combining the outputs into a single prediction. Standard uses the first auxiliary output and all two-score combinations from the original five scores. Direct-reasoning methods enumerate all subsets of the corresponding size. All methods first average over prediction combinations within each paper, then equally over the ten papers.

For Quick, every available auxiliary output for a paper forms a separate one-call prediction, and losses are averaged without first combining the outputs into a single prediction. Standard uses the first auxiliary output and all two-score combinations from the original five scores. Direct-reasoning methods enumerate all subsets of the corresponding size. All methods first average over prediction combinations within each paper, then equally over the ten papers.

Brier loss is computed against each of the ten one-hot targets and then averaged, including randomness in the target scores themselves. W1 compares the predicted distribution with the empirical CDF of the ten target scores, in score points. Modal accuracy compares the modes of the prediction and the ten target scores, choosing the smallest score in a tie. Mean-score error and P⁡(Y≥6) error compare the respective means and tail probabilities.

Brier loss is computed against each of the ten one-hot targets and then averaged, including randomness in the target scores themselves. W1 compares the predicted distribution with the empirical CDF of the ten target scores, in score points. Modal accuracy compares the modes of the prediction and the ten target scores, choosing the smallest score in a tie. Mean-score error and P⁡(Y≥6) error compare the respective means and tail probabilities.

In this independent-target set, Standard has Brier loss 0.353, W1 of 0.194 points, and modal accuracy of 69%; three reasoning reviews yield 0.379, 0.178 points, and 77%. These metrics measure the full distribution, cumulative distribution, and most likely score, respectively, and are reported together.

In this independent-target set, Standard has Brier loss 0.353, W1 of 0.194 points, and modal accuracy of 69%; three reasoning reviews yield 0.379, 0.178 points, and 77%. These metrics measure the full distribution, cumulative distribution, and most likely score, respectively, and are reported together.

E.9 Historical Call Costs

E.9 Historical Call Costs

The existing cost audit uses the original script’s historical pretax USD rates per million tokens: 4 for ordinary input, 5 for cache writes, 0.4 for cached input, and 20 for output. Call cost is computed from usage fields as

The existing cost audit uses the original script’s historical pretax USD rates per million tokens: 4 for ordinary input, 5 for cache writes, 0.4 for cached input, and 20 for output. Call cost is computed from usage fields as

Here I denotes input tokens, C cached-input tokens, W cache-write tokens, and O output tokens.

Here I denotes input tokens, C cached-input tokens, W cache-write tokens, and O output tokens.

First and subsequent calls differ in cache state, so the table reports both historical averages. The main paper’s R/Q columns describe call composition; this table gives observed costs for those calls without combining different cache states into a single deployment price. The frozen summaries do not contain consistent end-to-end latency measurements for comparing methods under matched conditions, so no latency estimate is reported.

First and subsequent calls differ in cache state, so the table reports both historical averages. The main paper’s R/Q columns describe call composition; this table gives observed costs for those calls without combining different cache states into a single deployment price. The frozen summaries do not contain consistent end-to-end latency measurements for comparing methods under matched conditions, so no latency estimate is reported.

Appendix F Revision Crossover Experiment

Appendix F Revision Crossover Experiment

F.1 Seven-Cell Design and Paired Scoring

F.1 Seven-Cell Design and Paired Scoring

Three letters denote Writer, Selector, and Scorer, in that order, with L for Luna and H for Haiku. Luna candidates are processed by two Selectors and evaluated by two Scorers, yielding LLL, LHL, LLH, and LHH. Haiku candidates yield HLL, HHL, and HHH; HLH was not collected. All cells use the same 199 papers.

Three letters denote Writer, Selector, and Scorer, in that order, with L for Luna and H for Haiku. Luna candidates are processed by two Selectors and evaluated by two Scorers, yielding LLL, LHL, LLH, and LHH. Haiku candidates yield HLL, HHL, and HHH; HLH was not collected. All cells use the same 199 papers.

Section B.5 describes the Writers’ feedback sources. Each Writer’s candidate set is held fixed across Selectors, and both Scorers read the same final manuscript for a given Writer/Selector combination. Each Scorer independently evaluates the original and revised manuscripts three times each. A Scorer’s original-manuscript baseline is reused across revision conditions. Paper-level gain is defined as

Section B.5 describes the Writers’ feedback sources. Each Writer’s candidate set is held fixed across Selectors, and both Scorers read the same final manuscript for a given Writer/Selector combination. Each Scorer independently evaluates the original and revised manuscripts three times each. A Scorer’s original-manuscript baseline is reused across revision conditions. Paper-level gain is defined as

Positive, zero, and negative gains are determined by the difference between the two sums of three scores, avoiding floating-point changes to direction. Overall gains and cross-Scorer differences use paired paper-level resampling.

Positive, zero, and negative gains are determined by the difference between the two sums of three scores, avoiding floating-point changes to direction. Overall gains and cross-Scorer differences use paired paper-level resampling.

Both years retain opposite average gains for Luna revisions under the two Scorers. For example, year-specific gains for LLL are +0.280 and +0.670, whereas those for LLH are −0.417 and −0.279. The year-specific results for LHL and LHH likewise remain positive and negative, respectively.

Both years retain opposite average gains for Luna revisions under the two Scorers. For example, year-specific gains for LLL are +0.280 and +0.670, whereas those for LLH are −0.417 and −0.279. The year-specific results for LHL and LHH likewise remain positive and negative, respectively.

F.2 Candidate Edits and Selector Agreement

F.2 Candidate Edits and Selector Agreement

The two Selectors independently accept or reject each candidate edit. For accepted sets Ai,Bi, Jaccard similarity is Ji=|Ai∩Bi|/|Ai∪Bi|, defined as 1 when both sets are empty. The table reports candidate counts and acceptance intersections for each Writer.

The two Selectors independently accept or reject each candidate edit. For accepted sets Ai,Bi, Jaccard similarity is Ji=|Ai∩Bi|/|Ai∪Bi|, defined as 1 when both sets are empty. The table reports candidate counts and acceptance intersections for each Writer.

Of Luna’s 9,327 candidates, 8,459 are accepted by both Selectors and 661 receive different decisions. Decisions are identical for 17 papers; the other 182 have different final texts. Candidate acceptance and final scoring occur at different stages: a large overlap in accepted candidates coexists with opposite gains from the two Scorers on fixed final manuscripts.

Of Luna’s 9,327 candidates, 8,459 are accepted by both Selectors and 661 receive different decisions. Decisions are identical for 17 papers; the other 182 have different final texts. Candidate acceptance and final scoring occur at different stages: a large overlap in accepted candidates coexists with opposite gains from the two Scorers on fixed final manuscripts.

Haiku generates 2,587 candidates. Luna and Haiku accept 2,442 and 2,450, respectively, with 2,354 accepted by both. Mean paper-level Jaccard similarity is 0.9198. The Writers produce different numbers of edits, while selection agreement within their respective candidate sets is similar.

Haiku generates 2,587 candidates. Luna and Haiku accept 2,442 and 2,450, respectively, with 2,354 accepted by both. Mean paper-level Jaccard similarity is 0.9198. The Writers produce different numbers of edits, while selection agreement within their respective candidate sets is similar.

F.3 Paper-Level Gain Directions and Cross-Model Differences

F.3 Paper-Level Gain Directions and Cross-Model Differences

Beyond overall gains, we classify each paper’s difference between three-review means as positive, zero, or negative. The table gives complete direction contingency tables for the three shared-manuscript conditions.

Beyond overall gains, we classify each paper’s difference between three-review means as positive, zero, or negative. The table gives complete direction contingency tables for the three shared-manuscript conditions.

For Luna-written, Luna-selected manuscripts, 46 papers have positive Luna gains and negative Haiku gains, and none show the reverse combination. Both gains are positive for 27 papers, negative for 19, and zero for 10. The two strict reversal combinations total 61 papers for Luna-written, Haiku-selected manuscripts and 23 for Haiku-written, Haiku-selected manuscripts.

For Luna-written, Luna-selected manuscripts, 46 papers have positive Luna gains and negative Haiku gains, and none show the reverse combination. Both gains are positive for 27 papers, negative for 19, and zero for 10. The two strict reversal combinations total 61 papers for Luna-written, Haiku-selected manuscripts and 23 for Haiku-written, Haiku-selected manuscripts.

At least one Scorer assigns zero gain in 53.8%, 45.7%, and 70.4% of papers across the three conditions. Direction disagreement can therefore be separated into strict reversals and cases involving zero gain; the full contingency tables preserve this distinction.

At least one Scorer assigns zero gain in 53.8%, 45.7%, and 70.4% of papers across the three conditions. Direction disagreement can therefore be separated into strict reversals and cases involving zero gain; the full contingency tables preserve this distinction.

F.4 Scoring Noise and Correlation Adjustment

F.4 Scoring Noise and Correlation Adjustment

Under a model with fixed manuscripts and independent, zero-mean scoring errors, the noise variance of the three-review original mean is eo,i=So,i2/3, and the noise variance of the gain estimate is

Under a model with fixed manuscripts and independent, zero-mean scoring errors, the noise variance of the three-review original mean is eo,i=So,i2/3, and the noise variance of the gain estimate is

Let vx,vy be the observed across-paper variances for two Scorers, ex,ey their mean noise variances, and c their observed covariance. Keeping the covariance unchanged and subtracting diagonal noise terms gives

Let vx,vy be the observed across-paper variances for two Scorers, ex,ey their mean noise variances, and c their observed covariance. Keeping the covariance unchanged and subtracting diagonal noise terms gives

The original-manuscript correlation is 0.804 before adjustment and 0.843 after it. Observed gain correlations for the two Luna revision conditions are 0.415 and 0.407, rising to 0.521 and 0.518 after adjustment. For Haiku revisions, the observed correlation is 0.085 and the adjusted value is approximately 0.230, with a wide interval. Some bootstrap samples have nonpositive noise-adjusted variances or correlations outside the usual range. The table retains valid-resample counts and out-of-range cases.

The original-manuscript correlation is 0.804 before adjustment and 0.843 after it. Observed gain correlations for the two Luna revision conditions are 0.415 and 0.407, rising to 0.521 and 0.518 after adjustment. For Haiku revisions, the observed correlation is 0.085 and the adjusted value is approximately 0.230, with a wide interval. Some bootstrap samples have nonpositive noise-adjusted variances or correlations outside the usual range. The table retains valid-resample counts and out-of-range cases.

The adjustment assumes independent errors across models and between original and revised manuscripts. Writers use existing first reviews to generate revisions; Section B.5 specifies their actual relationship to the original-score baselines. We use noise subtraction to diagnose the contribution of scoring error. Mean cross-Scorer differences for fixed final manuscripts are computed directly from paired scores and do not depend on this adjustment.

The adjustment assumes independent errors across models and between original and revised manuscripts. Writers use existing first reviews to generate revisions; Section B.5 specifies their actual relationship to the original-score baselines. We use noise subtraction to diagnose the contribution of scoring error. Mean cross-Scorer differences for fixed final manuscripts are computed directly from paired scores and do not depend on this adjustment.

F.5 Structural Checks of Manuscript Packages

F.5 Structural Checks of Manuscript Packages

We compare original and final packages paper by paper for the four Writer/Selector conditions, checking record counts and order, fields other than text, leading instructions and page-marker text, image indices and stored hashes, and the existence of referenced image files.

We compare original and final packages paper by paper for the four Writer/Selector conditions, checking record counts and order, fields other than text, leading instructions and page-marker text, image indices and stored hashes, and the existence of referenced image files.

All four conditions pass these checks. The Luna-written conditions accept 8,857 and 8,722 text blocks; the Haiku-written conditions accept 2,442 and 2,450. Recomputing SHA-256 for all 9,440 image files across the four final-package conditions gives exact matches to the original image-index hashes. This audit covers input-package structure, text-replacement rules, and image files. The study did not separately collect blinded expert labels for factual preservation or quality changes in the revisions.

All four conditions pass these checks. The Luna-written conditions accept 8,857 and 8,722 text blocks; the Haiku-written conditions accept 2,442 and 2,450. Recomputing SHA-256 for all 9,440 image files across the four final-package conditions gives exact matches to the original image-index hashes. This audit covers input-package structure, text-replacement rules, and image files. The study did not separately collect blinded expert labels for factual preservation or quality changes in the revisions.

F.6 Real Revision Examples Selected by a Fixed Rule

F.6 Real Revision Examples Selected by a Fixed Rule

Cases are selected from five direction strata in LL: Luna positive/Haiku negative, both positive, both negative, Luna negative/Haiku positive, and at least one zero. Papers in each stratum are sorted by a fixed hash rule, and the first is selected. The Luna-negative/Haiku-positive stratum is empty. One additional case is selected by the same rule from Luna-positive/Haiku-zero papers in the Haiku-written, Haiku-selected condition.

Cases are selected from five direction strata in LL: Luna positive/Haiku negative, both positive, both negative, Luna negative/Haiku positive, and at least one zero. Papers in each stratum are sorted by a fixed hash rule, and the first is selected. The Luna-negative/Haiku-positive stratum is empty. One additional case is selected by the same rule from Luna-positive/Haiku-zero papers in the Haiku-written, Haiku-selected condition.

Each case shows the first accepted block of suitable display length after sorting edits by page number and text-block ID. Both original and revised text must contain 70–750 characters after whitespace normalization; if no block qualifies, the first accepted block is used. Scores always refer to the complete revised manuscript. The displayed block illustrates the edit and is not assigned responsibility for the whole-paper gain.

Each case shows the first accepted block of suitable display length after sorting edits by page number and text-block ID. Both original and revised text must contain 70–750 characters after whitespace normalization; if no block qualifies, the first accepted block is used. Scores always refer to the complete revised manuscript. The displayed block illustrates the edit and is not assigned responsibility for the whole-paper gain.

F.6.1 Case 1: Luna Positive / Haiku Negative

F.6.1 Case 1: Luna Positive / Haiku Negative

Paper 2025-paper_092; Luna Writer/Luna Selector; 46 papers in this direction stratum. The displayed edit is on PDF page 3, text block 27, diff d0003, and was accepted by this condition’s Selector.

Paper 2025-paper_092; Luna Writer/Luna Selector; 46 papers in this direction stratum. The displayed edit is on PDF page 3, text block 27, diff d0003, and was accepted by this condition’s Selector.

Original:

Original:

3. We demonstrate theoretically how a) SGFT can learn highly performant policies, despite the bias introduced by the SGFT objective and b) how this enables SGFT to overcome compounding errors which plague standard MBRL approaches.

3. We demonstrate theoretically how a) SGFT can learn highly performant policies, despite the bias introduced by the SGFT objective and b) how this enables SGFT to overcome compounding errors which plague standard MBRL approaches.

Revised:

Revised:

3. We analyze how value-transfer bias and model error affect SGFT policies. The resulting guarantee is conditional on uniform closeness of Vsim to the optimal real-world value and improvability under real-world dynamics; neither condition is empirically verified in our experiments.

3. We analyze how value-transfer bias and model error affect SGFT policies. The resulting guarantee is conditional on uniform closeness of Vsim to the optimal real-world value and improvability under real-world dynamics; neither condition is empirically verified in our experiments.

F.6.2 Case 2: Both Positive

F.6.2 Case 2: Both Positive

Paper 2026-paper_005; Luna Writer/Luna Selector; 27 papers in this direction stratum. The displayed edit is on PDF page 1, text block 10, diff d0002, and was accepted by this condition’s Selector.

Paper 2026-paper_005; Luna Writer/Luna Selector; 27 papers in this direction stratum. The displayed edit is on PDF page 1, text block 10, diff d0002, and was accepted by this condition’s Selector.

Original:

Original:

Despite rapid progress, two gaps remain. First, no research has scaled EFMs beyond the levels nec- essary to fully exploit this potential—previous efforts have been limited by computational resources and dataset availability. Second, no systematic, quantitative analysis of EFM scaling behavior has been conducted, leaving fundamental questions unanswered about optimal resource allocation and scaling strategies for neural data—in particular, how to allocate a fixed compute budget across model size and number of training epochs.

Despite rapid progress, two gaps remain. First, no research has scaled EFMs beyond the levels nec- essary to fully exploit this potential—previous efforts have been limited by computational resources and dataset availability. Second, no systematic, quantitative analysis of EFM scaling behavior has been conducted, leaving fundamental questions unanswered about optimal resource allocation and scaling strategies for neural data—in particular, how to allocate a fixed compute budget across model size and number of training epochs.

Revised:

Revised:

Despite rapid progress, two gaps remain. First, previous EFM efforts have explored a narrower range of model and data scales than those considered here, in part because of computational resources and dataset availability. Second, systematic quantitative analysis of EFM scaling behavior remains limited, leaving open questions about resource allocation and how to divide a fixed compute budget between model size and training duration.

Despite rapid progress, two gaps remain. First, previous EFM efforts have explored a narrower range of model and data scales than those considered here, in part because of computational resources and dataset availability. Second, systematic quantitative analysis of EFM scaling behavior remains limited, leaving open questions about resource allocation and how to divide a fixed compute budget between model size and training duration.

F.6.3 Case 3: Both Negative

F.6.3 Case 3: Both Negative

Paper 2025-paper_020; Luna Writer/Luna Selector; 19 papers in this direction stratum. The displayed edit is on PDF page 2, text block 15, diff d0004, and was accepted by this condition’s Selector.

Paper 2025-paper_020; Luna Writer/Luna Selector; 19 papers in this direction stratum. The displayed edit is on PDF page 2, text block 15, diff d0004, and was accepted by this condition’s Selector.

Original:

Original:

We show that each Koopman operator is represented by the exponential of a matrix constructed with Toeplitz matrices and diagonal matrices. This allows us to apply Krylov subspace methods (Gallopou- los & Saad, 1992; Güttel, 2013; Hashimoto & Nodera, 2016) to compute the estimation of Koopman operators with low computational costs. By virtue of the universal property of Toeplitz matrices (Ye & Lim, 2016), we can show the universality of the proposed model with a linear algebraic approach. We also show a generalization bound of the proposed model using a reproducing kernel Hilbert space (RKHS) associated with the Fourier functions. We can analyze both the universality and generalization error with the same framework.

We show that each Koopman operator is represented by the exponential of a matrix constructed with Toeplitz matrices and diagonal matrices. This allows us to apply Krylov subspace methods (Gallopou- los & Saad, 1992; Güttel, 2013; Hashimoto & Nodera, 2016) to compute the estimation of Koopman operators with low computational costs. By virtue of the universal property of Toeplitz matrices (Ye & Lim, 2016), we can show the universality of the proposed model with a linear algebraic approach. We also show a generalization bound of the proposed model using a reproducing kernel Hilbert space (RKHS) associated with the Fourier functions. We can analyze both the universality and generalization error with the same framework.

Revised:

Revised:

We represent each layer by the exponential of a matrix constructed from Toeplitz and diagonal matrices. This structure allows Krylov subspace methods to compute matrix-exponential actions. Using the universal property of Toeplitz matrices (Ye & Lim, 2016), we establish a finite-dimensional function-approximation result. We also analyze generalization using an RKHS associated with Fourier functions, with the bound depending on operator norms and requiring appropriate controls on the model class.

We represent each layer by the exponential of a matrix constructed from Toeplitz and diagonal matrices. This structure allows Krylov subspace methods to compute matrix-exponential actions. Using the universal property of Toeplitz matrices (Ye & Lim, 2016), we establish a finite-dimensional function-approximation result. We also analyze generalization using an RKHS associated with Fourier functions, with the bound depending on operator norms and requiring appropriate controls on the model class.

F.6.4 Case 4: At Least One Zero

F.6.4 Case 4: At Least One Zero

Paper 2026-paper_066; Luna Writer/Luna Selector; 107 papers in this direction stratum. The displayed edit is on PDF page 1, text block 9, diff d0003, and was accepted by this condition’s Selector.

Paper 2026-paper_066; Luna Writer/Luna Selector; 107 papers in this direction stratum. The displayed edit is on PDF page 1, text block 9, diff d0003, and was accepted by this condition’s Selector.

Original:

Original:

Large Language models (LLMs) such as Transformer (Vaswani et al., 2017), BERT (Devlin et al., 2019a), PaLM (Chowdhery et al., 2023), and GPT-4o (Hurst et al., 2024) have demonstrated re- markable capabilities in natural language understanding and generation, which helps to achieve a wide range of tasks such as language translation, sentiment analysis, and question answering. Simi- larly, video Transformers such as TimeSformer (Bertasius et al., 2021), ViViT (Arnab et al., 2021), and Video Swin Transformer (Liu et al., 2022) have illustrated remarkable progress in capturing spatio-temporal dynamics, greatly advancing applications such as video understanding and editing.

Large Language models (LLMs) such as Transformer (Vaswani et al., 2017), BERT (Devlin et al., 2019a), PaLM (Chowdhery et al., 2023), and GPT-4o (Hurst et al., 2024) have demonstrated re- markable capabilities in natural language understanding and generation, which helps to achieve a wide range of tasks such as language translation, sentiment analysis, and question answering. Simi- larly, video Transformers such as TimeSformer (Bertasius et al., 2021), ViViT (Arnab et al., 2021), and Video Swin Transformer (Liu et al., 2022) have illustrated remarkable progress in capturing spatio-temporal dynamics, greatly advancing applications such as video understanding and editing.

Revised:

Revised:

Video Transformers, including TimeSformer (Bertasius et al., 2021), ViViT (Arnab et al., 2021), and Video Swin Transformer (Liu et al., 2022), model spatio-temporal dependencies. This manuscript examines the computational cost of exact block attention and a proposed dynamic approach. The review identified fundamental correctness and dimensionality errors in that approach; the results below therefore define the target computation, not a validated efficient algorithm.

Video Transformers, including TimeSformer (Bertasius et al., 2021), ViViT (Arnab et al., 2021), and Video Swin Transformer (Liu et al., 2022), model spatio-temporal dependencies. This manuscript examines the computational cost of exact block attention and a proposed dynamic approach. The review identified fundamental correctness and dimensionality errors in that approach; the results below therefore define the target computation, not a validated efficient algorithm.

F.6.5 Case 5: Haiku Revision, Luna Positive / Haiku Zero

F.6.5 Case 5: Haiku Revision, Luna Positive / Haiku Zero

Paper 2025-paper_093; Haiku Writer/Haiku Selector; 47 papers in this direction stratum. The displayed edit is on PDF page 1, text block 20, diff d0002, and was accepted by this condition’s Selector.

Paper 2025-paper_093; Haiku Writer/Haiku Selector; 47 papers in this direction stratum. The displayed edit is on PDF page 1, text block 20, diff d0002, and was accepted by this condition’s Selector.

Original:

Original:

To address this gap, in this paper, we curate a comprehensive ex- trapolation benchmark consisting of seven datasets sourced from Matminer (Ward et al., 2018), accompanied by well-defined train/validation/test splits based on real-

To address this gap, in this paper, we curate a comprehensive ex- trapolation benchmark consisting of seven datasets sourced from Matminer (Ward et al., 2018), accompanied by well-defined train/validation/test splits based on real-

Revised:

Revised:

To address this gap, in this paper, we curate a comprehensive extrapolation benchmark consisting of seven tasks derived from four datasets sourced from Matminer (Ward et al., 2018), accompanied by well-defined train/validation/test splits based on real-

To address this gap, in this paper, we curate a comprehensive extrapolation benchmark consisting of seven tasks derived from four datasets sourced from Matminer (Ward et al., 2018), accompanied by well-defined train/validation/test splits based on real-

Appendix G Reproduction Resources

Appendix G Reproduction Resources

The code package includes all data, code, configurations, and external files needed to reproduce the analyses and figures.

The code package includes all data, code, configurations, and external files needed to reproduce the analyses and figures.

Appendix H Ethical Considerations and Responsible Use

Appendix H Ethical Considerations and Responsible Use

Data Sources and Licensing. This study uses publicly available ICLR 2025 and 2026 submissions and associated review metadata obtained through OpenReview. We selected 100 papers from each year for academic analysis of AI-based peer review. The original manuscripts and their contents remain subject to the copyrights and applicable licenses of their respective authors. Our reported findings primarily consist of independently computed statistics and analyses. Any redistribution of source materials or derived artifacts is subject to the applicable licenses and terms of use.

Data Sources and Licensing. This study uses publicly available ICLR 2025 and 2026 submissions and associated review metadata obtained through OpenReview. We selected 100 papers from each year for academic analysis of AI-based peer review. The original manuscripts and their contents remain subject to the copyrights and applicable licenses of their respective authors. Our reported findings primarily consist of independently computed statistics and analyses. Any redistribution of source materials or derived artifacts is subject to the applicable licenses and terms of use.

Use of AI Models. Large language models are the primary research objects in this study. They are used to evaluate manuscripts, generate and select candidate revisions, and assess scoring consistency. These model-generated reviews and revisions constitute experimental outputs rather than independent human assessments. The experimental procedures, statistical analyses, and interpretation of the results remain the responsibility of the authors.

Use of AI Models. Large language models are the primary research objects in this study. They are used to evaluate manuscripts, generate and select candidate revisions, and assess scoring consistency. These model-generated reviews and revisions constitute experimental outputs rather than independent human assessments. The experimental procedures, statistical analyses, and interpretation of the results remain the responsibility of the authors.

Ethical Risks. The study analyzes publicly available research manuscripts and review information without recruiting human participants. Manuscript inputs are anonymized for AI evaluation. Our findings should not be interpreted as evidence that AI-generated reviews can replace human peer reviewers or reliably determine publication decisions. Automated manuscript revisions may also introduce inaccuracies, and their scientific validity requires independent verification.

Ethical Risks. The study analyzes publicly available research manuscripts and review information without recruiting human participants. Manuscript inputs are anonymized for AI evaluation. Our findings should not be interpreted as evidence that AI-generated reviews can replace human peer reviewers or reliably determine publication decisions. Automated manuscript revisions may also introduce inaccuracies, and their scientific validity requires independent verification.

注意转载授权 · 阅读原文。中文正文为 AI 翻译，请以原文为准。

注意转载授权 · 阅读原文