同行评审应约束评审权威：Hanrui Wang 等提出“权威受限的同行评审”

Hanrui Wang ^††thanks: Corresponding author: hanrui_wang@nii.ac.jp    Timo Spinde Affiliation: National Institute of Informatics, Japan Affiliation: Media Bias Group, Japan    Isabella Habereder Affiliation: Media Bias Group, Japan Affiliation: Georg-August-University Göttingen, Germany    Chun-Shien Lu Affiliation: Academia Sinica, Taiwan    Isao Echizen Affiliation: National Institute of Informatics, Japan Affiliation: Media Bias Group, Japan

Abstract

Our position shifts the target of peer-review reform from reducing uncertainty through evaluator improvement or judgment aggregation to constraining evaluative authority under persistent uncertainty.

Scientific peer review aims to assure research quality but also allocates scarce academic opportunities, such as publication in high-impact venues. However, the review system is under growing pressure as submissions increase while reviewer capacity, expertise, scrutiny, and incentives cannot scale accordingly. As peer review operates under persistent evaluative uncertainty, evaluator improvement and judgment aggregation remain structurally limited. We therefore propose authority-constrained peer review: before a consequential judgment is released as decision-relevant, it should withstand structured scrutiny through critique-authority decoupling, independent secondary evaluation, contestability and reversibility, and accountability for unsupported influence. Our aim is to prevent weakly supported judgments from entering final deliberation with the same authority as judgments that have withstood structured scrutiny, while creating incentives for reviewers to justify consequential claims more carefully. For large-scale computer-science conferences, this position offers an institutional response to uneven review quality and reviewer-assignment dependence; more broadly, it provides a governance principle for any domain where uncertain expert judgments allocate scarce academic opportunities.

Project Page summarizes this work and will be updated with future policy and technical developments related to peer review.

1 Introduction

Peer review is a central mechanism for ensuring the quality of scientific findings, but it also serves as a basis for allocating scarce opportunities for publication, funding, and professional recognition siler2015measuring; aczel2025present. However, in large-scale machine learning and AI conferences, the number of submissions under review has increased substantially in recent years (see Figure 1). This growth not only increases reviewing workload, but also places greater pressure on review judgments: these judgments continue to shape consequential decisions siler2015measuring; kravitz2010editorial in a review system where reviewer expertise, review scrutiny, and reviewing incentives cannot scale at the same pace gallo2016influence; huisman2017duration; tite2007peer; breuning2015reviewer; fox2017recruitment; seghier2025paying.

One major response to rising review pressure is to improve evaluators themselves, based on the intuition that better evaluators lead to better reviews aczel2025present; hesselberg2023reviewer; zhuang2025large. Human-centered reforms seek better reviewer training, matching, incentives, transparency, and accountability aczel2025present; leyton2024matching; seghier2025paying; kim2025position; yang2025position. AI-centered reforms aim to improve review quality or accelerate peer review through model optimization, automated review generation, or hybrid human-AI workflows chang2025treereview; lee2025role; zhuang2025large. However, many criteria that peer review relies on, such as novelty and significance, lack reliable ground truth at the time of review lee2015commensuration; hug2024referees; aczel2025present; oxley2025variability: whether the review judgment is correct cannot be directly verified when decisions are made. Peer review, therefore, operates under persistent evaluative uncertainty. As a result, evaluator improvement lacks stable targets against which improvement can be guided sizo2025defining; superchi2019tools; even when a target can be specified, it remains difficult to validate whether overall review quality has been effectively improved sizo2025defining; superchi2019tools; pier2018low.

Taking persistent evaluative uncertainty in peer review as a premise, we shift the institutional focus of reform: constraining the authority of each review judgment before it can influence the final decision. We propose four principles:

• •

  Critique-authority decoupling: the evaluator who provides the substantive critique should not unilaterally convert that critique into a consequential decision signal, such as an acceptance or rejection recommendation.

• •

  Independent secondary evaluation: consequential review judgments should receive independent second-order checks for clarity, evidential support, and interpretive validity.

• •

  Contestability and reversibility: review judgments should remain open to challenge and, where necessary, revision or reversal.

• •

  Accountability for unsupported influence: reviews that significantly affect decisions but are later found to be poorly supported should remain visible in the record.

Our position also differs from the conventional many-signal peer review. Many-signal review relies on multiple reviewers, rebuttals, and decision layers to let judgments complement, correct, or compete with one another pier2018low; hug2024referees. However, each review judgment in a many-signal system can still enter final deliberation as one evaluator’s unilateral judgment and become a consequential input to the final decision hug2024referees; kravitz2010editorial. In contrast, our proposed authority-constrained review requires consequential judgments to withstand structured scrutiny before they are released as decision-relevant signals. The aim of authority-constrained review is not to add more opinions for aggregation, but to prevent weakly supported judgments from entering final deliberation with the same authority as judgments that have withstood structured scrutiny. Constraining authority also creates incentives for reviewers to provide judgments that are more explicit, evidence-based, and defensible.

Taken together, our position does not rely on reducing evaluative uncertainty. Instead, it aims to prevent uncertain judgments from acquiring unchecked authority in final decisions and to encourage more careful review. More generally, it asks how peer-review institutions should govern the authority of expert judgments when those judgments allocate scarce opportunities under persistent uncertainty.

2 Core condition of peer review and its structural consequences

2.1 Standard inconsistency

Review guidelines are intended to ensure that the same evaluation criteria are applied to all submissions hug2024referees, but the standards by which reviewers interpret and apply these criteria remain heterogeneous lee2015commensuration; hug2024referees. Building on prior literature leyton2024matching; luukkonen2012conservatism; weinberg2024reviewer; huisman2017duration; nust2021codecheck; hug2024referees; gallo2016influence, we identify six reviewer-side sources of interpretive subjectivity that shape how evaluative criteria are assessed (see Table 2). Topic familiarity: familiarity with the paper’s specific topic, problem setting, or literature leyton2024matching. Expertise: relevant technical or methodological competence gallo2016influence. Scholarly pursuit: the reviewer’s orientation toward what kinds of research are worth advancing luukkonen2012conservatism. Preferences: intellectual, methodological, or stylistic inclinations weinberg2024reviewer. Reviewer effort: the time and attention invested in the review huisman2017duration. Verification resources: the practical means available to check relevant aspects of the submission nust2021codecheck.

Reviewers may interpret and apply the same criteria differently, making criteria ground truth difficult to determine collectively at the time of decision pier2018low; hug2024referees. Reviewer judgments may therefore diverge not only because submissions differ, but also because reviewers rely on different standards, evidence, and interpretive frames pier2018low; hug2024referees; lee2015commensuration; gallo2016influence; oxley2025variability. We therefore argue that peer review operates under standard inconsistency.

2.2 Trust-dependent opacity

Reviews may be shaped not only by how criteria are interpreted, but also by surrounding incentive conditions bonn2021advancing; lim2025regional; trueblood2025misalignment; michelson2022association; resnik2018conflict; aczel2025present; bianchi2022can; tite2007peer; seghier2025paying. We summarize three incentive conditions that are particularly relevant to peer review. Scarce rewards: the fact that publication records are tied to prestige bonn2021advancing, career advancement lim2025regional; trueblood2025misalignment, visibility bonn2021advancing, and downstream access to resources michelson2022association. Conflict of interest (COI): direct rivalry, shared interests, professional or network ties resnik2018conflict, as well as concerns about retaliation aczel2025present; bianchi2022can. Weak incentives for careful review: the fact that reviewing is often time-consuming and only weakly rewarded tite2007peer; seghier2025paying.

We refer to reviews shaped by such surrounding incentive conditions as incentive-coupled evaluations. As discussed in the literature buenstorf2025analyzing; resnik2018conflict; righi2017miracle; kocak2025ensuring; bianchi2022can, incentive-coupled evaluations may take at least three forms. Strategic distortion: deliberate inflation or deflation of evaluative judgments, for example, through harsher or more favorable assessments of novelty or significance; such distortion may be motivated by scarce rewards or COI buenstorf2025analyzing; resnik2018conflict. Low-effort or AI-substituted review: evaluation based on limited effort or AI substitution, leading to superficial comments, heuristic judgment, or weakly scrutinized review text; this may arise when reviewers prioritize research production over careful reviewing under weak reviewing incentives righi2017miracle; kocak2025ensuring. Cautious non-commitment: hedged or persistently borderline judgment that avoids strong commitment, often reflecting limited evaluative investment or reluctance to impose a clear acceptance or rejection; such judgment may also be reinforced by COI, particularly concerns about retaliation bianchi2022can.

In Table 3, we illustrate couplings between incentive conditions and forms of incentive-coupled evaluation. Because many review criteria have limited or unavailable ground truth, incentive-coupled evaluations are difficult to detect or separate from ordinary criteria-based disagreement resnik2018conflict; lee2015commensuration. Peer review, therefore, relies partly on trust in reviewers’ professional and ethical conduct.

2.3 Criteria-to-decision indeterminacy

As illustrated in Figure 2, peer review involves a two-stage process: evaluative criteria (Table 1) are first translated into reviewer-level judgments (e.g., recommendations and confidence scores), which are then used by editors or chairs to reach final decisions.1

However, neither stage follows explicit or stable decision rules, i.e., there is no standardized way to translate criteria into reviewer judgments, nor to translate those judgments into final decisions hug2024referees; kravitz2010editorial. As a result, evaluations that use the same criteria may still yield different outcomes. We refer to this structural lack of a well-defined mapping as criteria-to-decision indeterminacy.

3 Alternative views

As stated in Section 1, we examine three prominent approaches to improving peer review: many-signal review, human-centered improvement, and AI-assisted review. The following sections analyze what each approach can improve and why reducing uncertainty through judgment aggregation or evaluator improvement remains insufficient under missing criteria ground truth.

3.1 Many-signal review systems

Conventional peer review often uses multiple reviewers, author rebuttals, and layered decision-making by chairs and editors; we refer to this as a many-signal review system.²² 2 https://www.acm.org/publications/policies/roles-and-responsibilities The intuition of many-signal review is that distributing judgment across reviewers and decision layers can reduce the impact of a single reviewer’s idiosyncratic judgment on final decisions pier2018low; hug2024referees.

However, each reviewer in a many-signal system still evaluates under missing criteria ground truth, applies heterogeneous standards, may be influenced by surrounding incentive conditions, and contributes to a decision process without a stable criteria-to-decision mapping. Adding more reviewers, therefore, does not prevent a single review from exerting consequential downstream influence kravitz2010editorial; d2017can. This influence may become significant when negative or cautionary interpretations carry disproportionate weight in subsequent decision-making rozin2001negativity, for example, because early judgments may anchor later evaluation liu2024testing; tversky1974judgment, or because competitive selection settings may make adjudicators more cautious toward uncertain work luukkonen2012conservatism; gallo2018risk.

A further limitation concerns the timing of correction. Many-signal systems allow a judgment to be challenged through other reviews, author rebuttal, reviewer discussion, or chair/editor moderation2, but such correction may occur after the judgment has already begun to shape the decision process kravitz2010editorial; liu2024testing. The key distinction is therefore not whether a judgment can eventually be corrected, but whether it is independently scrutinized before or only after it begins to exert consequential influence.

Moreover, substantial disagreement among reviewers remains common under current many-signal review systems hesselberg2025registered; aczel2025present, suggesting that the number of reviewers typically assigned to a submission is often insufficient to stabilize evaluative variance. Existing studies do not clearly establish how many reviewers would be needed to reduce such variance substantially, nor do they show that greater agreement among a limited number of reviewers would guarantee correctness pier2018low; erosheva2021zero; fogelholm2012panel. Furthermore, the problem of reviewer disagreement is further compounded when review quality is degraded by incentive-coupled evaluations, for example, through low-effort review or cautious non-commitment.

3.2 Human-centered evaluator improvement

Human-centered reforms typically aim to improve peer review through clearer reviewing standards and training hesselberg2023reviewer; yang2025position, improved reviewer matching shah2022challenges; jakobsen2022factors, stronger transparency and accountability mechanisms yang2025position; kim2025position, and stronger incentives or expansion of the reviewer pool seghier2025paying; kim2025position. Their intuition is that peer review can be improved by better human reviewers and more robust review procedures. However, there is no universally accepted reference standard for a “good” review sizo2025defining; superchi2019tools, leaving evaluator improvement without a reliable target.

Improved training, matching, and procedural guidance may reduce some forms of inconsistency hesselberg2023reviewer; jakobsen2022factors; leyton2024matching, but they cannot remove differences in topic familiarity, expertise, scholarly pursuit, and evaluative preferences leyton2024matching; gallo2016influence; luukkonen2012conservatism; weinberg2024reviewer. Judgments about novelty, significance, and fit therefore remain contestable rather than reducible to uniform application of a shared standard lee2015commensuration; li2025developing; hug2024referees.

Furthermore, improved review standards and procedures do not ensure meaningful compliance. Greater transparency, accountability requirements, and procedural expectations may encourage improved review conduct yang2025position; kim2025position; hesselberg2023reviewer, but they do not ensure that reviewers will follow the guidelines consistently in practice bianchi2022can; resnik2018conflict; aczel2025present. Reviewer overload smith2006peer; aczel2025present, difficulty securing willing reviewers tite2007peer; breuning2015reviewer; fox2017recruitment, and weak incentives seghier2025paying; aczel2025present continue to place reviewers under conflicting constraints of time, effort, and care.

Moreover, there is evidence of a sustained increase in the number of submissions (see Figure 1), while it is unlikely that a human-centered reform would lead to a corresponding increase in reviewer capacity. Stronger incentives and author-obligated reviewing may increase reviewer supply, but they cannot generate scarce expertise on demand gallo2016influence; bergstrom2026screening or eliminate reviewer-submission mismatch leyton2024matching; jakobsen2022factors; rogers2023program. Expertise, carefulness, and institutional professionalism are difficult to expand at the pace of rising submission volume and AI-assisted research production aczel2025present; zhang2025exploring; lu2026towards.

3.3 AI-assisted review

AI-centered reforms typically aim to improve peer review through automated review generation chang2025treereview; bougie2025generative; zhu2025deepreview, AI-assisted review support lee2025role; choi2026position; zhuang2025large, and hybrid human-AI evaluation workflows zhuang2025large. Their central intuition is that peer review can be improved through scalable evaluative support that expands review capacity, increases consistency, and enhances the quality of critique zhuang2025large.

However, review corpora kuznetsov2022revise; lin2023moprd; kang2018dataset; dycke2023nlpeer encode historically produced judgments shaped by standard inconsistency and trust-dependent opacity, rather than verified targets of correctness, significance, or future value lee2015commensuration; glanzel2003better; ke2015defining. Synthetic data may broaden the training signal sadallah2025good, but optimization and validation still rely on heterogeneous human judgments, evaluative standards, and heuristic metrics rather than verified criteria ground truth pier2018low; lee2015commensuration; kirtani2025revieweval; glanzel2003better; ke2015defining. AI-assisted review, therefore, inherits the core condition of missing criteria ground truth at the level of data, optimization target, and validation.

Moreover, training and optimization of AI review systems become more challenging as the review stream itself is affected by variability and distortion. The missing criteria ground truth already limits the possibility of clean supervision. Reviewer-side subjectivity and incentive conditions further shape the evaluative signals that enter review corpora, while accelerated scientific output, limited reviewer capacity, and scarce rewards may accelerate the accumulation of inconsistency, bias, and signal degradation in these data kocak2025ensuring; lo2025personal; liang2024monitoring. AI reviews thus face a recursive problem: the data from which these systems learn are themselves produced under the very structural constraints that these systems are expected to overcome.

Existing studies document critical limitations of AI review, including factual unreliability zhou2024llm; huang2025survey, shallow assessment of scientific validity and technical depth lee2025role; choi2026position; zhou2024llm, and security, privacy, bias, and governance concerns hosseini2023fighting; lin2025breaking; kousha2024artificial; zhuang2025large; ghaffar2026ethical. These limitations are relevant but not central to our argument, as many may be partially mitigated by stronger models and more mature institutional safeguards kousha2024artificial; lee2025role; zhuang2025large. Our central concern is instead structural: model improvement may reduce generic LLM weaknesses, but it cannot, by itself, establish reliable performance on the specialized task of peer-review evaluation without a valid reference standard for review quality.

4 Our position: Authority-constrained peer review

We argued that existing approaches share the same problematic logic: they seek better decisions through more signals, better human reviewers, or AI evaluators, without reliable criteria ground truth. Under this logic, uncertain judgments remain difficult to distinguish from well-supported ones, yet uncertain judgments can still acquire unchecked evaluative authority in final deliberation. In contrast, we take such persistent evaluative uncertainty due to missing ground truth, including standard inconsistency, trust-dependent opacity, and criteria-to-decision indeterminacy, as a premise and shift the reform target from improving evaluators to governing evaluative authority. We use evaluative authority to refer to the capacity of a review judgment, score, objection, or recommendation to shape final deliberation or appear in the decision rationale.

4.1 Design principles

Authority-constrained review can be implemented through four principles. First, critique and authority should be decoupled: the evaluator who writes the substantive critique should not automatically convert it into a consequential recommendation or score. Second, consequential judgments should receive independent second-order evaluation of their clarity, evidential support, and interpretive validity before they enter the decision rationale. This is a targeted check of the existing review judgment, not an independent full review of the submission. Third, review judgments should remain contestable and reversible, so that challenges can lead to clarification, narrowing, downgrading, score revision, or removal from the decision rationale. Fourth, unsupported influence should remain accountable: when a judgment significantly shaped a decision but is later found to be poorly supported, the system should record both the judgment and the correction of its authority. These principles make review influence conditional rather than automatic, while encouraging reviewers to make consequential claims more carefully.

Authority-constrained review differs from many-signal review systems not only in how judgments are checked, but also in when they acquire consequential authority. Many-signal review adds signals for aggregation and may correct judgments through later rebuttal, discussion, or moderation, whereas authority-constrained review requires independent scrutiny before a judgment enters final deliberation with full decision relevance. The aim is to expose weak evidential bases, unsupported claims, and unresolved interpretive problems before they exert consequential influence, making authority conditional on prior scrutiny rather than later correction.

4.2 Comparison with alternative views

To clarify the distinctiveness of our position, Table 4 summarizes how authority-constrained peer review governs evaluative authority under the core condition and structural consequences developed in Section 2, in contrast to the three approaches discussed in Section 3. We elaborate on each comparison below.

Core condition. Many-signal, human-centered, and AI-centered approaches are all oriented toward bringing peer review closer to the criteria ground truth: by aggregating more signals, by training better human reviewers, or by optimizing models on review data and preferences. In contrast, our position begins with the assumption that weakly supported, seriously flawed, or otherwise contestable reviews will persist in the absence of criteria ground truth and cannot be eliminated with confidence. We therefore seek to constrain the downstream influence of consequential judgments through independent scrutiny rather than to rely on directly improving their quality. This principle applies to all consequential judgments, although its practical implementation may need to allocate scrutiny unevenly. In an initial or resource-constrained deployment, judgments with greater potential influence on the final decision should therefore be checked more carefully, for example, clear accept or reject judgments that, if unsupported, could substantially shape the eventual outcome.

Standard inconsistency. Standard inconsistency arises primarily from reviewer-side human subjectivity, which cannot be reliably eliminated. Many-signal review responds by adding evaluators under the assumption that a larger number of judgments will reduce variance. Human-centered reform provides clearer, more detailed criteria to ensure more uniform interpretation. However, reviewer pools are usually too small for multiplicity alone to neutralize heterogeneous interpretation. Detailed standards also remain subject to divergent human understandings. Compared with many-signal and human-centered approaches, AI-assisted evaluation can respond more directly to standard inconsistency. A single model may apply a more consistent scoring standard across submissions. But consistent scoring does not imply that AI-assisted review is an optimal solution, given the additional limitations discussed in Section 3.3. Instead of relying on multiple separate reviews, we propose a different form of evaluation diversity, in which evaluators cross-check one another’s interpretations within the same review process. Our position is particularly clear in a human-critique, AI-score arrangement, in which the human evaluator’s decision-making authority is limited, and the AI evaluator helps ensure greater consistency in scoring.

Trust-dependent opacity. Trust-dependent opacity arises because incentive-coupled evaluations cannot be cleanly ruled out under missing criteria ground truth. In a many-signal review system, it is assumed that including additional reviewers mitigates the influence of individual judgments driven by incentives. Human-centered and AI-centered reforms, by contrast, depend more on correcting and mitigating problematic evaluations through improved norms, supervision, or model design. We argue that, given missing criteria ground truth, detection of incentive-coupled evaluations rarely produces evidence strong enough to resolve the issue institutionally. Instead, our position of constrained authority does not aim to eliminate incentive-coupled evaluations, but makes them harder to shape outcomes without resistance.

Criteria-to-decision indeterminacy. Criteria-to-decision indeterminacy is the consequence least directly resolved by existing approaches, including our own. Many-signal review adds evaluative inputs but does not establish a stable guideline for aggregating criteria or reviewer-level judgments into final decisions. Human-centered reform can clarify criteria and procedures, but does not yield a determinate mapping from substantive evaluation to recommendations or from recommendations to final decisions. While AI-centered reform can learn or reproduce historical classifications from previous reviews and decisions, this merely mimics existing patterns rather than eliminating the underlying uncertainty. Our position does not eliminate the criteria-to-decision indeterminacy either. Instead, we limit the extent to which a single review can steer final decisions. In doing so, we limit the extent to which the criteria-to-decision indeterminacy from a single review can affect outcomes.

5 One possible instantiation

Authority-constrained review can be instantiated in different institutional forms. We describe one concrete illustration based on dual-track human–AI mutual supervision. In Track A, an AI system produces a substantive critique without assigning a consequential score, and human supervisors independently assess the critique by recording agreement, disagreement, and important omissions before forming their own judgments. In Track B, human reviewers produce substantive critiques, while an AI system independently assigns the rating most consistent with each critique and scrutinizes its evidential support, interpretive validity, and decision proportionality. The two tracks do not observe each other’s intermediate outputs, and their outputs are provided to the chair or editor for final adjudication rather than automatically aggregated into a decision. This separation operationalizes the principle that the evaluator producing a substantive critique should not unilaterally determine its consequential authority.

A central component of this instantiation is structured scrutiny. Rather than asking a second evaluator to produce another full review, scrutiny targets the basis on which an existing judgment could acquire decision influence. As summarized in Table 5, we distinguish three dimensions that assess whether a judgment is supported, correctly interpreted, and proportionate to its intended decision consequence, without requiring a single ground-truth evaluation.

These three dimensions also provide a basis for interpreting agreement and disagreement. Agreement provides additional support for a judgment, whereas disagreement indicates that it requires closer examination. In the case of disagreement, the chair or editor can first compare the competing evaluations in terms of the three scrutiny dimensions. If needed, independent reviews and the relevant manuscript content can provide further evidence for adjudication. If the disagreement remains unresolved, the judgment may receive reduced authority, require clarification, or be prevented from determining the outcome by itself. Importantly, agreement does not establish that a review is correct, just as disagreement does not establish that it is wrong. Our position aims to prevent unresolved consequential judgments from exercising unchecked authority, rather than to claim that scrutiny signals reveal a “more correct” evaluation.

One concern is that a dual-track design may increase human reviewer workload. The workload implications, however, depend on the institutional realization. An all-human implementation introduces additional human scrutiny, although checking an existing judgment can be lighter than conducting another full review. In the AI–human realization studied here (Track A), some full human reviewing can instead be replaced by supervision of an existing AI critique, potentially resulting in a lighter reviewer-side workload. Chair and editor workload remains uncertain: more structured signals must be considered, but unsupported or conflicting judgments may also become easier to identify. We analyze the workload implications of different realizations in Appendix A.

To examine the operational feasibility of the dual-track human–AI mutual supervision design, we instantiated both tracks on one real borderline submission using four original peer reviews, four recruited human supervisors, one experienced area chair, and ChatGPT-5.1. The study provides preliminary evidence for several aspects of the design. In Track A, all four human supervisors reported that supervising the AI-generated critique was easier than conventional reviewing, including two who voluntarily consulted the original manuscript. In Track B, AI scrutiny produced explicit counter-signals by identifying score–reasoning inconsistencies and potential review-quality concerns, while human and AI judgments also diverged in some cases. These counter-signals remained visible for area-chair adjudication rather than being resolved automatically. The area chair reported neither a clear increase nor decrease in adjudication difficulty, consistent with the uncertain chair/editor workload discussed above. Detailed settings, results, participant feedback, and limitations are provided in Appendix B.

This feasibility study establishes operational plausibility rather than venue-scale effectiveness. It shows that the proposed role separation, structured scrutiny, reciprocal checking, and chair-level adjudication can be instantiated in a complete review process, but evaluating broader effects requires institutionally representative studies across reviewer populations, research fields, incentives, policies, and decision contexts. Building on these open questions, we outline a roadmap for developing and validating authority-constrained review, including improved scrutiny design, suitable peer-review datasets, more reliable counter-signal generation and adjudication, less-biased AI support, safeguards against technical misconduct, and institutionally representative pilot studies. These objectives are detailed in Appendix C.

6 Scope

First, this paper does not argue that scientific evaluation should cease because of uncertainty, nor that current peer review is broken. Our argument is institutional rather than accusatory: under present conditions, peer-review reform cannot safely rely solely on selecting trusted evaluators.

Second, this paper does not argue that AI is ineffective or that improving evaluators is futile. Better reviewer training, incentives, and AI systems may all improve the signals available to the system and, in some settings, substantially improve decision quality. However, such gains should be understood as improvements to evaluative inputs rather than as solutions to evaluative authority. Better signals do not, by themselves, justify unconstrained decision influence.

Third, this paper does not argue that expertise is irrelevant or that all signal sources should be treated identically. Expertise matters, and final responsibility cannot be eliminated. Our claim is narrower and more practical: under missing criteria ground truth, expertise alone should not exempt consequential judgments from structured checking, explicit challenge, and possible reversal.

Fourth, reducing reviewer workload is not the primary claim of authority-constrained review, although it is an important consideration in its practical realization. The central aim is to govern how consequential evaluative influence is exercised under missing criteria ground truth. Workload implications depend on the institutional design, and our human–AI instantiation illustrates how this governance objective can be pursued while also reducing reviewer-side workload.

7 Conclusion

Peer review operates under missing criteria ground truth: many criteria that shape consequential decisions cannot be verified against stable standards at the time of review. This condition limits approaches that seek better decisions primarily by adding signals, improving evaluators, or scaling evaluative support, because these approaches do not by themselves prevent uncertain or weakly supported judgments from acquiring unchecked authority in final deliberation. Our position, therefore, departs from efforts to reduce uncertainty and instead governs the conditions under which review judgments become decision-relevant. Its aim is to ensure that consequential judgments are exposed to scrutiny, challenge, and possible correction before they carry full authority in final decisions. For large-scale computer-science conferences, this position addresses the downstream effects of variable review quality and reviewer-assignment dependence.

More broadly, our argument in this position paper becomes more urgent as AI agents accelerate research production, manuscript preparation, and review assistance. The pressures now visible in computer-science conferences may therefore foreshadow challenges that other fields will face. Computer science is not only a domain under review pressure, but also an early testbed for institutional reform. Venues such as NeurIPS, where scale, AI-assisted research, and peer-review governance already intersect, are well positioned to pilot authority-constraining mechanisms. The practical task is not to wait for perfect evaluators, complete ground truth, or fully settled policies, but to test and refine institutional designs that make consequential review authority more accountable under persistent uncertainty.

Acknowledgments and Disclosure of Funding

This work was partially supported by JSPS KAKENHI Grant JP24H00732, by JST CREST Grants JPMJCR20D3 and JPMJCR2562 including AIP challenge program, and by JST K Program Grant JPMJKP24C2 Japan. It was also funded by the German Federal Ministry of Education and Research (BMBF) through the DAAD (German Academic Exchange Service) and a fellowship by the Hanns-Seidel-Foundation. It was also supported by the Taiwan National Science and Technology Council (NSTC) with Grant 114-2634-F-001-001-MBK, and by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under project number 565115197 (BARI).

References

Appendix A Human Workload Analysis and Comparison Across Peer-Review Designs

The workload implications of authority-constrained review depend on its institutional realization. Compared with conventional independent human review, an all-human realization of mutual supervision increases human workload, whereas the dual-track human–AI realization described in Section 5 can reduce reviewer-side human workload. Table 6 summarizes the workload implications across designs and roles.

Conventional independent human review.

Consider a conventional design in which four human reviewers independently evaluate each submission. Let Wfull denote the workload required for one full human review. The reviewer-side human workload is

This provides the baseline against which we compare realizations of mutual supervision.

All-human mutual supervision.

Authority-constrained review does not depend on AI adoption. One possible realization retains four full human reviewers and assigns an independent human supervisor to scrutinize each review judgment, yielding four final review judgments. Let Wsup denote the workload required to perform this second-order scrutiny. Reviewer-side human workload then becomes

Because Wsup>0, this realization necessarily requires more human effort than conventional independent review. Structured scrutiny can be lighter than a full review because it evaluates an existing judgment rather than the submission from scratch.

Dual-track human–AI mutual supervision.

In the human–AI dual-track realization (Section 5), two full human reviews receive AI supervision, while one AI-generated critique is independently supervised by two humans, again yielding four final review judgments.

Compared with conventional independent review,

Our feasibility study provides preliminary empirical support for this workload condition (Appendix B.2).

Chair and editor workload.

Reviewer-side workload does not determine the workload of chairs or editors. Mutual supervision produces additional structured signals that may require adjudication, but it may also make unsupported or conflicting judgments easier to identify. The net effect therefore remains uncertain; in our feasibility study, the area chair reported neither a clear increase nor decrease in adjudication difficulty (Appendix B.2).

Scaling to venue-level deployment.

Because evaluative uncertainty persists across submissions and reviewers, all consequential judgments should in principle remain subject to scrutiny. Practical deployment, however, may need to allocate scrutiny unevenly when resources are limited or when a venue first pilots the mechanism. Under tighter resource constraints, scrutiny can prioritize judgments with greater potential decision influence, such as clear accept or reject judgments, decision-critical claims, explicitly challenged judgments, or strong cross-review inconsistencies.

Appendix B Feasibility Study of Dual-Track Human–AI Mutual Supervision

In this section, we present a feasibility study illustrating how the dual-track human--AI mutual supervision design can be instantiated on a selected ICLR’25 rejected submission³³ 3 https://openreview.net/forum?id=fFtmpqLFvw. In addition to the four original human reviewers from ICLR’25, we recruited four new human supervisors with diverse backgrounds and one experienced area chair. The study instantiates both tracks end to end and examines the counter-signals, workload implications, and adjudication process.

B.1 Settings

The selected paper from ICLR 2025.

We selected an ICLR’25 rejected submission3 for which we have no COI. The paper received ratings of 3/6/6/8 with corresponding confidences of 4/3/4/4 (maximum 5), following ICLR’s official rating scale:

• •

  10: strong accept, should be highlighted at the conference

• •

  8: accept, good paper

• •

  6: marginally above the acceptance threshold

• •

  5: marginally below the acceptance threshold

• •

  3: reject, not good enough

• •

  1: strong reject

We chose this paper because it combined a borderline average score with substantial disagreement among reviewers. Its average score was 5.75, but the individual ratings ranged from a clear reject to a clear accept. This combination makes it a useful setting for examining how independent scrutiny operates when the aggregate judgment is borderline but the underlying reviewer assessments span substantially different judgments.

Participants.

The study uses review comments from four original anonymous ICLR reviewers and involves four newly recruited human supervisors, one human area chair, and one LLM (ChatGPT 5.1, used at the time of the study).

The four original anonymous ICLR review comments serve as the human critiques in Track B and are independently examined through AI supervision. We did not provide their original ratings or confidence scores to the LLM; these values were used only for post-hoc comparison between the original and AI-inferred judgments.

We recruited four additional human supervisors for Track A: two AI security researchers, one machine learning researcher, and one Ph.D. student in LLM security. Each supervisor independently assessed the same AI-generated critique, assigned a rating to the submission, and provided a justification. They could consult the original manuscript at their discretion.

ChatGPT 5.1 in standard reasoning mode served as the AI system. We disabled memory and used independent interactions so that no prior conversation history affected its outputs. The model participated in both tracks:

• •
• •

  Track B: The AI independently inferred the rating most consistent with each human critique and performed structured scrutiny of that critique, referring to the manuscript when necessary and providing justification (see Section E.2). For this feasibility study, structured scrutiny was operationalized using the provisional failure-mode checklist in Table 7.

Finally, we recruited an experienced area chair, with service across ICML, ICLR, NeurIPS, ACM Multimedia, and AAAI, to perform the final meta-review assessment.

Process.

Table 8 summarizes the inputs and outputs of the study. Detailed prompts and intermediate outputs are provided in Section E. We do not include an author rebuttal or subsequent discussion because the original authors and anonymous ICLR reviewers were not participants in this study.

Questionnaire and additional AI–AI analysis.

After completing the study, we administered a short questionnaire to the four recruited human supervisors and the area chair:

1. 1.

   Does this new protocol make your task easier or more difficult?

2. 2.

   Does this new protocol improve or reduce the quality of your output?

After all four supervisors completed their tasks, we added a post-hoc question:

1. 3.

   Did you refer to the original paper when assigning ratings?

We also conducted an additional AI–AI analysis in which the AI rated and scrutinized its own generated critique. This analysis is not part of the proposed dual-track design; it provides an additional observation about possible self-preference when critique generation and scrutiny are not independently separated.

B.2 Results

Overall.

Table 9 summarizes the review judgments produced in the study. The four human supervisors in Track A assigned ratings of 5/5/5/6 based on the AI-generated critique. In Track B, ratings inferred by the AI from the four original human critiques were 6/8/6/8, compared with the original ICLR ratings of 3/6/6/8. AI scrutiny also produced explicit counter-signals for all four original critiques, flagging combinations of superficiality, inaccuracy, and misguidance for further adjudication. The area chair’s final decision remained reject. These outcomes illustrate that human and AI judgments can diverge while remaining visible for final adjudication rather than being automatically reconciled.

Track A: AI critique, human supervision.

The four human supervisors assigned ratings of 5/5/5/6 based on the shared AI-generated critique, compared with the original ICLR ratings of 3/6/6/8. One possible explanation for the narrower spread is that all four supervisors received the same detailed critique, which enumerated many strengths and weaknesses and may have anchored judgments toward the borderline.

This observation highlights a design challenge for AI-generated critiques. A highly detailed critique can provide broad evidential coverage but may also shape subsequent human judgments through the selection and presentation of issues. Conversely, a shorter critique may reduce such influence while omitting evidence useful for supervision. Designing AI critiques that provide sufficient support without exerting excessive anchoring therefore remains an important question for future work.

Track B: Human critiques, AI supervision.

Track B revealed mismatches between the ratings implied by written critiques and the reviewers’ original consequential judgments. Most notably, for Reviewer #5, the AI inferred a rating of 6 from a critique that had originally supported a rating of 3, while structured scrutiny also flagged that critique as superficial, inaccurate, and misguided. Reviewer #6 showed a smaller but still substantial mismatch, with an inferred rating of 8 compared with the original rating of 6. These cases show how AI supervision can surface review judgments whose assigned influence warrants closer chair-level examination, particularly when the rating appears stronger than the support articulated in the critique.

Meta review.

The area chair ultimately recommended rejection, the same outcome as the original ICLR decision, after considering the outputs produced by the two tracks. The case therefore illustrates how conflicting scrutiny signals can be preserved through the review process and resolved at the chair level without automatic aggregation.

Feedback on the questionnaire.

As summarized in Table 10, all four human supervisors found the supervision task easier than conducting a conventional full review. Two of them voluntarily consulted the original manuscript and still reported lower task difficulty. This provides preliminary empirical support for the reviewer-side workload condition discussed in Appendix A.

Three supervisors perceived an improvement in output quality, while one perceived a reduction. Reviewer #2, who consulted the original manuscript, specifically raised concerns that the AI critique could itself be biased or insufficiently validated. The feedback therefore suggests that lower reviewer-side workload can coexist with reservations about the reliability of the AI critique and the quality of the resulting evaluation.

The area chair selected “Neither” for both task difficulty and output quality, providing no clear indication of reduced or increased chair-level workload. This is consistent with Appendix A, where chair and editor workload remains uncertain because additional structured signals must be adjudicated even as unsupported or conflicting judgments may become easier to identify.

AI supervises AI (outside the proposed mutual supervision design).

As shown in the final row of Table 9, the AI assigned a rating of 8 to its own critique and detected no failure modes. In contrast, independently generated human critiques received several scrutiny flags. This asymmetry suggests a possible self-preference or confirmation effect when critique generation and scrutiny are performed by the same evaluator, further motivating the separation of evaluative roles in the proposed design.

B.3 Discussion — From Critique to Decision Influence

The feasibility study suggests that consequential ratings depend not only on which strengths and weaknesses are identified, but also on how evaluators weight their importance. Track A showed a shared tendency toward borderline judgments from the same detailed critique, whereas Track B revealed differences between the ratings inferred by AI from human-written critiques and the ratings originally assigned by those reviewers. Such variation is consistent with prior observations of differences in human and AI reviewing tendencies russo2025ai and may also reflect broader effects such as negativity bias rozin2001negativity or review cultures that emphasize flaw identification over contribution assessment lee2013bias.

This distinction is central to decision proportionality. Identifying a weakness does not determine how much decision influence that weakness should receive. Authority-constrained review does not recover a ground-truth evaluation or establish which evaluator is correct; instead, it makes the transition from observed evidence to consequential judgment explicit and subject to independent scrutiny before that judgment acquires decision authority.

B.4 Limitations of the Feasibility Study

The study is limited to one submission, four recruited human supervisors, one area chair, and one LLM configuration. It also omits author rebuttal and subsequent interaction with the original anonymous reviewers. In Track A, all supervisors received the same AI-generated critique, so their judgments may share biases introduced by that critique rather than being fully independent. Broader evaluation will therefore require larger and more representative studies across reviewers, models, fields, and institutional settings.

Appendix C Research and Deployment Roadmap

Developing authority-constrained review requires progress across scrutiny design, datasets, methodologies, system security, and institutional deployment. In this section, we outline concrete research objectives for developing and evaluating practical realizations of the position proposed in this paper.

C.1 Prerequisites

Objective I: Develop structured scrutiny.

A central prerequisite is to operationalize the three dimensions of structured scrutiny introduced in Section 5: evidential support, interpretive validity, and decision proportionality. These dimensions provide a conceptual basis for examining consequential review judgments, but practical implementations require methods that can identify relevant problems consistently across submissions, reviewers, and research fields.

The provisional failure modes used in our feasibility study (Table 7) provide one possible operational starting point. Future work should determine how such failure modes relate to the three scrutiny dimensions, which signals can be identified reliably, and which are most useful for subsequent chair or editor adjudication. In particular, decision proportionality requires examining not only whether a criticism is valid, but also whether the identified issue justifies the decision influence assigned to it.

Objective II: Establish datasets for structured scrutiny.

Existing datasets (Section D) primarily repurpose raw records from venues such as ICLR and NeurIPS on OpenReview⁵⁵ 5 https://openreview.net. These datasets overlap substantially and provide limited coverage of high-quality AI reviews, problematic review comments, scrutiny signals, and subsequent adjudication.

Manual annotation is also difficult because many consequential review problems lack reliable ground truth. Program committees rarely establish that a review is objectively overly harsh or overly gentle. Disagreement may instead reflect either a genuine review problem or a legitimate difference in interpretation. LLMs, combined with well-defined failure modes, can generate synthetic problematic reviews at scale demetrio2025gen. A central open question is how to verify that such reviews exhibit the intended failure mode rather than merely differing in style, strictness, or evaluative preference.

Datasets for authority-constrained review should therefore preserve disagreement, provenance, and uncertainty where possible. They should also cover human and AI critiques, consequential ratings, scrutiny signals, and subsequent adjudication.

C.2 Methodologies

Objective III: Develop AI supervision methods.

AI supervision requires two core capabilities. First, the AI should infer the rating implied by a human review comment, allowing inconsistencies between the written critique and the consequential rating to be identified. Existing work shows that this remains difficult when models rely only on review text, particularly for long comments zhou2024llm. Approaches that additionally use paper content or abstracts perform better kang2018dataset; singh2021compare; dycke2023nlpeer; zhang2025replication, suggesting that effective supervision may require access to both the review and the submission.

Second, the AI should identify potential problems in the review itself. Existing studies address individual categories, including overly harsh reviews stelmakh2021catch, unconfident language ghosal2022hedgepeer, superficial assessments guo2023automatic, and impolite tone bharti2024politepeer. More recently, Thakkar et al. thakkar2025can deployed an LLM-based review-feedback system at ICLR 2025 to identify issues such as vague comments, content misunderstandings, and unprofessional remarks, demonstrating the practical feasibility of AI-assisted review supervision at scale. Developing methods that can reliably cover a broader range of review failures and support structured scrutiny therefore remains an important open problem.

Objective IV: Develop less-biased and more reliable AI review support.

Track A relies on AI-generated critiques, making their reliability an important research objective. AI reviews can exhibit systematic biases and evaluator-specific tendencies demetrio2025gen; russo2025ai, while broader concerns about bias in LLM-assisted peer review have also been raised hosseini2023fighting.

Recent approaches such as ReviewRL zeng2025reviewrl use reinforcement learning to improve review quality, factual grounding, and rating accuracy. Similar methods could support more reliable AI critiques when combined with the scrutiny methods in Objective III.

Critique presentation also matters. As suggested by our feasibility study, a detailed AI critique can provide broad evidential coverage while influencing which issues human supervisors attend to. Future work should therefore study both critique quality and how its structure affects downstream human judgments.

C.3 System Security

Objective V: Address technical misconduct.

AI-assisted peer review creates new opportunities for technical manipulation. Relevant threats include:

Prompt injection: explicit or hidden instructions embedded in a submission to influence AI behavior shi2024optimization; ye2024we; collu2026misleading; gibney2025scientists.

Textual adversarial manipulation: small or strategically chosen edits intended to alter AI-generated evaluations raina2024llm; lin2025breaking.

Data poisoning: manipulation of review-related training or update data to bias future model behavior drori2024human.

Implicit manipulation: shaping text, structure, or presentation to exploit known AI-review tendencies without explicit instructions ye2024we.

For authority-constrained review, two threat directions are particularly relevant:

• •

  Manipulating AI scrutiny: influencing AI evaluation of an existing human judgment.

• •

  Manipulating AI critique: influencing AI-generated critiques without substantively improving the submission.

Peer-review-specific defenses remain limited. Potential safeguards include task-specific defenses against prompt injection piet2024jatmo, restricting untrusted instruction channels liu2024formalizing, adversarial testing and red-teaming perez2022red, consistency-based checks for unreliable outputs manakul2023selfcheckgpt, and auditable records of AI-generated critiques and scrutiny signals mokander2024auditing.

These mechanisms should be evaluated specifically in peer-review settings. Transparency also creates a security trade-off: review prompts and procedures should support accountability and contestability without making manipulation unnecessarily easy.

C.4 Deployment

Objective VI: Pilot authority-constrained review.

A crucial next step is to evaluate authority-constrained review in real peer-review settings. A practical near-term deployment can begin with AI scrutiny of existing human reviews, examining evidential support, rating–reasoning consistency, omissions, and potential review failures without replacing human reviewers or chairs. Recent large-scale deployment at ICLR 2025 shows that LLM-generated feedback on existing human reviews can be integrated into a live conference workflow thakkar2025can.

Subsequent pilots can evaluate the full dual-track human–AI mutual supervision design described in Section 5, including reviewer and supervisor workload, chair or editor workload, the usefulness of counter-signals, disagreement resolution, anchoring effects, and robustness to technical misconduct.

We therefore call for conferences, journals, workshops, review platforms, and program committees to collaborate on controlled studies and venue-level pilots. When resources are limited, initial deployment can prioritize judgments with greater potential decision influence.

Appendix D Peer-Review Datasets

We compiled a broad collection of publicly available peer-review datasets, summarized in Table 11. Our review draws on three major surveys lin2023automated; kuznetsov2024can; zhuang2025large as well as individual dataset papers. Because many datasets contain overlapping submissions or reviews, we focus on resources most relevant to authority-constrained review, including datasets with papers, human reviews, AI-generated reviews, rebuttals, or annotations related to review quality and bias. Among the available resources, Gen-Review demetrio2025gen is particularly relevant because it contains both human and AI-generated reviews as well as bias-related annotations, although broader data are still needed for structured scrutiny and adjudication.

Appendix E Full Review Process of Human–AI Mutual Supervision

E.1 Track A: AI Critique, Human Supervision

E.2 Track B: Human Critiques, AI Supervision

E.3 Meta Review

注意转载授权 · 阅读原文