Auditing AI peer reviewers: a benchmark on real scientific papers

Auditing AI peer reviewers:
a benchmark on real scientific papers

Íñigo Zubeldia Affiliation: DAMTP, University of Cambridge Affiliation: Kavli Institute for Cosmology, University of Cambridge Email: iz221@cam.ac.uk    Boris Bolliet Affiliation: Kavli Institute for Cosmology, University of Cambridge Affiliation: Cavendish Astrophysics, University of Cambridge    Francisco Villaescusa-Navarro    Pablo Villanueva-Domingo Affiliation: Computer Vision Center, Universitat Autònoma de Barcelona    Miles Cranmer Affiliation: DAMTP, University of Cambridge Affiliation: Polymathic AI

Abstract

LLM-based peer review is now used at production scale, but its evaluation remains limited. We introduce a benchmark that injects objective, localisable errors of four types (general issues, mathematics, numerics, citations) at three severity levels into real papers and measures each reviewer’s detection rate by error type and severity, together with its false-positive rate on clean copies. We also present Skepthical, a multi-agent reviewer that verifies citation-backed claims by retrieving the cited papers, mathematics with computer algebra, and numerical claims with executed code. We apply the benchmark to two corpora of 15 recent arXiv preprints each, one in astrophysics and one in machine learning (ML), injecting a total of 357 errors, and evaluate twelve reviewer systems: six single-LLM reviewers, two Claude Code configurations, three publicly available reviewers, and Skepthical. The choice of model has the largest impact on detection rate: given the same prompt, three frontier LLMs differ by about 50 percentage points in total detection on both corpora. Tools and scaffolding help when they contribute what the model cannot supply on its own. In particular, on astrophysics, Claude Code improves on a single Claude call in mathematics and numerics detection, and enforced citation retrieval gives Skepthical the highest citation error detection rate of all systems by a large margin. This citation lead, however, is not observed on ML, where the models appear to know the literature better and Skepthical’s citation detection rate is statistically indistinguishable from that of the best single LLM. False-positive rates are below 10% for most systems, but vary with prompt, model, and scaffolding, with mathematics flags the main source of false positives. The evaluation pipeline, the injected errors, and the adjudication logs are released as a benchmark.

1 Introduction

Peer review is a cornerstone of the scientific process. With AI tools becoming increasingly used in research and AI scientists starting to conduct research at scale [3, 13, 26, 18, 19, 25, 22, 7], the need for reliable and scalable peer review is greater than ever. In parallel, LLM-based peer review has reached production scale in 2026, with a growing ecosystem of AI reviewers and review venues [16, 1, 8, 20, 2, 5, 4, 28]. Despite the publication of the first AI-reviewer benchmarks [10], methodology for evaluating these AI reviewers has generally lagged behind. Evaluation is most often framed either as alignment with human-written reviews [11, 5, 16] or as synthetic error injection on Machine Learning (ML) papers only [20, 2, 24, 6], with important issues such as false-positive rates or error detection as a function of the error severity not being typically investigated.

This paper addresses these concerns with two primary contributions. First, we introduce an evaluation pipeline for reviewer systems that injects errors into a paper’s LaTeX source at three severity levels (subtle, clear, blatant) across four error types (general issues, mathematics, numerics, citations), and then measures i) the error detection rate for each severity level and error type and ii) the false-positive rates (on a copy with no injected errors). The verdicts are adjudicated by a cross-family LLM panel (Anthropic + OpenAI + Google) following recent LLM-as-judge prescriptions [15, 21, 27], with additional mechanical verifiers for mathematics and numerics. Our ‘dose-response’ design extends perturbation-based behavioural evaluation [17, 29] with graded severity levels inspired by [14]. We note that our benchmark focuses on whether reviewers can detect objective, localisable issues (e.g., numerical and mathematical errors, misrepresented citations, internal inconsistencies), leaving aside questions such as significance and novelty assessment or overall review quality, which are harder to quantify.

Second, we introduce Skepthical, an LLM-powered paper reviewer comprising four specialist modules: a citations module that retrieves and reads the cited papers, a mathematics module that verifies mathematical claims using computer algebra, a numerics module that verifies numerical results through executed code, and a general-issues module consisting of an ensemble of cross-family LLMs. Unlike prior reviewers that rely entirely on LLMs, Skepthical verifies LLM-flagged issues where possible: the mathematics and numerics modules use computer algebra and executed code, respectively, while the citations module checks the claims directly against the retrieved PDFs of cited papers.

We apply our evaluation pipeline to two corpora of 15 recent astrophysics arXiv preprints and 15 ML preprints, benchmarking Skepthical alongside eleven additional reviewer systems: six single-LLM reviewers, two Claude Code configurations, and three publicly available external AI reviewers (the Stanford Agentic Reviewer [16], OpenReviewer-8B [8], and the PaperAudit-Bench detector [20]). To our knowledge, our application to the astrophysics corpus is the first controlled comparison of AI peer-reviewer systems on a real, non-ML scientific paper corpus. We release the evaluation pipeline, together with the paper lists, injection catalogues, and full adjudication logs, as a public benchmark at github.com/inigozubeldia/skepthical-benchmark.

Table 1 positions this work against prior AI peer-review systems and benchmarks across nine evaluation axes. No prior work grades severity levels or uses a cross-family adjudication panel; only one [11] covers papers beyond ML/NLP; only one [6] provides false-positive control; and only one [2] integrates mechanical verifiers. This work is the first to combine all five. Unlike in some previous work, our review adjudication is LLM-based (with additional mechanical checks for mathematics and numerics), although we check the performance of our pipeline against a human annotator on one paper.

2 Benchmark design

2.1 Reviewer systems

We evaluate the following twelve reviewer systems.

• •

  Skepthical (full pipeline description in Appendix A). Skepthical is a multi-agent reviewer with four parallel specialist modules: (i) general issues, which uses two cross-family LLMs (Claude Opus 4.7 and Gemini 2.5 Pro) to generate two independent reports that are then merged by a third-family model (GPT-5.1), alongside a free-form reading of the paper by a reasoning model (GPT-5.4); (ii) mathematics, in which the paper’s key mathematical statements are extracted and internally-verifiable claims are verified mechanically with either wolframscript or pint (the former for symbolic equality, the latter for dimensional analysis); (iii) numerics, in which the consistency of the numerical statements in the paper is verified with Python code; and (iv) citations, in which every key factual statement relying on external references is verified with an LLM with the cited papers provided in its context. To our knowledge, Skepthical is the first AI peer reviewer that combines mechanical verification of mathematics and numerics with retrieval-based verification of factual citations. It is publicly accessible at skepthical-ai.org. The configuration evaluated below is the public release frozen on 2026-04-21, one day before the 2026-04-22 arXiv submissions from which the astrophysics corpus is sampled (§2.2). No prompt or model choice was tuned in response to the papers or results reported in this paper.

• •

  Single-LLM reviewers. We consider two configurations of a single-LLM reviewer: a free-form configuration, run with a one-sentence prompt (‘Write a referee report for this scientific paper as if you were a journal referee’), and a structured configuration, run with the full operating instructions of all of Skepthical’s agents compiled into one 978-line prompt (report-merging and tool-output adjudication agents removed and with analogous mechanical-verification-free instructions given for mathematics and numerics). Both configurations are evaluated for three different models (Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro Preview, hereafter Gemini 3.1 Pro), giving six single-LLM reviewers in total. We further evaluate a concise version of the structured configuration in Appendix H.

• •

  Claude Code. We use the frontier general-purpose coding agent Claude Code [46], running with Claude Opus 4.7 in a sandbox with the following allowed tools: Read, Bash, Grep, Glob, python3 (with NumPy, SymPy, SciPy, pint), wolframscript, and arXiv access. As for the single-LLM reviewers, we consider two configurations (sharing model, sandbox, and tools): a free-form configuration, run with the same one-sentence referee prompt as in the free-form single-LLM reviewers, and a structured configuration, run with a prompt containing Skepthical’s full operating instructions.

• •

  Three publicly available external reviewers. (i) The Stanford Agentic Reviewer at paperreview.ai [16] (closed-source, internals undocumented, queried on 2026-05-01; only run on the astrophysics corpus); (ii) OpenReviewer-8B [8], an open-weights Llama-3.1-8B fine-tuned on ≈79K ICLR / NeurIPS reviews, self-hosted and invoked with the published ICLR-template prompt; and (iii) the PaperAudit-Bench detector [20], an open-weights Qwen3-14B-based reviewer, run in Deep mode. We do not retrain or re-prompt any of these reviewers.

Together, the twelve systems allow for three comparisons and one external reference point. The first comparison is the effect of the prompt, as within the single-LLM group and within the Claude Code group, the free-form and structured configurations differ only in the prompt. The second is the effect of the agentic loop with tool-use: at each prompt level, the Claude Code configuration can be compared with the Claude Opus 4.7 single-LLM configuration with the same prompt, so the difference between them is not the instructions but the ability to perform several LLM calls, run code, read files, and fetch cited papers. The third is the effect of agentic scaffolding: structured Claude Code and Skepthical share the same operating instructions, but in Skepthical those instructions are executed by dedicated specialist agents that are enforced to run, whereas Claude Code has a much less strict scaffolding. Finally, the three external reviewers show how publicly available systems that an author could use today perform on the same benchmark, and the two open-weights reviewers also test whether small models fine-tuned on review data can compete with frontier systems. Two caveats apply: i) Skepthical uses models from all three providers, so its margin over any single-provider system mixes the effect of the pipeline with that of cross-provider ensembling, and ii) none of these comparisons is a capacity-matched ablation of Skepthical. Compute resources and asset licences are summarised in Appendix M.

2.2 Corpora

We evaluate on two corpora of 15 arXiv preprints each, one in astrophysics and one in ML. Each corpus consists of submissions to a single arXiv category on a single recent day, capped at 20 pages to limit cost. The astrophysics corpus comprises 15 of the 24 astro-ph submissions of 2026-04-22 within the page cap (median 13 pages), whereas the ML corpus comprises 15 of the 74 eligible cs.LG submissions of 2026-07-15, drawn by stratified random sampling over eight subfields (median 9 pages, range 3–19). Both chosen days postdate the knowledge cut-off of every model we use. Both paper lists are given in Appendix B.

Astrophysics was chosen for its abundant formal mathematics and numerics, two key aspects that our benchmark evaluates, and because most existing AI-reviewer evaluations have run on ML papers [20, 2, 11, 8, 5]. ML was added as the second corpus i) to test whether our conclusions hold beyond a single field and ii) because it is the field on which the two fine-tuned open-weights external reviewers were trained, so their performance can be measured in-distribution as well as out of it. We note that no component or prompt in any of the reviewer systems is field-specific.

2.3 Error injection

For each paper, four versions are reviewed: one clean copy (used to assess false positives) and three error-injected copies, each carrying one error of each of four types (general issues, mathematics, numerics, citations) at a chosen severity (S1 subtle, S2 clear, or S3 blatant). Across the three injected copies of a paper, every (error type, severity) combination appears exactly once (see Appendix C.1). Spreading the 12 injections across three copies (rather than concentrating them in one) keeps the per-copy error density small, which we take to be more representative of a typical real paper. Each corpus thus leads to 15 clean and 45 error-injected copies, with the astrophysics corpus containing a total of 15×12−3=177 injected errors (one paper had too few equations to admit mathematics injections) and its ML counterpart, 180.

Representative examples of the three severity levels, by error type, are as follows (full taxonomy with examples in Appendix C). Citations run from a 1σ value shift misquotation (S1) through a significant methodology misattribution (S2) to a reversed cited conclusion (S3); numerics from an abstract–body second-decimal drift (S1) through a digit swap or percent-of error (S2) to a structural inconsistency such as percentages summing above 100 % (S3); mathematics from a sign flip on a subleading term (S1) through an operator swap or normalisation drop (S2) to a dimensional absurdity (S3); and general issues from a typo (S1) through a methods–results inconsistency or phantom-evidence insertion (S2) to an abstract–body contradiction (S3). We note that the taxonomy may not fully reflect the distribution of naturally occurring errors in papers and that severity levels are designed to be calibrated within each error type but not across them, so a given severity label may not represent the same detection difficulty across error types.

Each injection is performed by an agent (Claude Sonnet 4.6 for the astrophysics corpus, Gemini 3.1 Pro for the ML corpus) that takes as input the full paper LaTeX source and returns a suggested edit. The suggested edit is accepted only if it passes a mechanical validator (Appendix C); otherwise an injection of the same type is re-attempted.

2.4 Adjudication pipeline

Once the reviewer systems have processed all 60 paper copies for each corpus (45 error-injected and 15 clean), we apply an adjudication pipeline to every resulting review to assess (i) which injected errors were detected on the error-injected copies (detection matching), and (ii) which of the issues raised on the clean copies are genuine false positives (false-positive adjudication).

Detection matching. For each (paper, injected copy, reviewer), the review plus the error-injection records are given to a three-judge cross-family panel (Claude Sonnet 4.6, GPT-5.4, and Gemini 3 Flash Preview, hereafter Gemini 3 Flash). Each judge returns a verdict for each injection ({detected, confidence, matching_evidence, reason}); the final verdict is the 3-way majority. Low-confidence or 2-1 ambiguous verdicts trigger a PDF-attached escalation pass, in which a single judge (Claude Sonnet 4.6) adjudicates the injection with the full paper PDF in context.

Across the 4,084 per-error-type adjudications in the two corpora (2,104 astrophysics, 1,980 ML), 93.5% reach unanimous first-round agreement (3-0) and 6.5% are 2-1 splits. The PDF-attached escalation fires on 34.9% of injections (38.7% astrophysics, 30.8% ML), with the dominant trigger being panel uncertainty rather than disagreement. Of the escalated injections, 92.3% confirm the first-round majority, and 73.6% return at high confidence. Aggregating across both stages, 97.3% of all adjudications end with a verdict that was either accepted at first round or confirmed under PDF re-judgement, only 2.7% are reversed by the escalation pass, and 91.6% reach unanimous agreement across every judge that voted (first-round panel plus, where applicable, the escalation pass). Only 7.4% of adjudications retain a low-confidence verdict after escalation. The adjudication is therefore highly self-consistent (judge-level agreement statistics in Appendix D).

False-positive adjudication on clean copies. Each review is processed by an agent (GPT-4.1) tasked with identifying clear positives (as opposed to low-confidence or open-to-interpretation points) and classifying them into four types: citations, mathematics, numerics, and general issues. These detections are then passed into a type-dependent pipeline. Citation detections are assessed by a two-judge cross-family panel with the cited PDFs attached (GPT-5 and Claude Sonnet 4.6; if needed, tie-break with Gemini 3.1 Pro). Numerics detections are checked mechanically: GPT-4.1 turns each one into a concrete numerical check to be carried out, a second independent GPT-4.1 call confirms that the numbers involved appear in the paper, and the check is then run in Python. Mathematics detections are checked with Skepthical’s mathematics module (which uses wolframscript and pint). Finally, general issues detections are adjudicated with a 3-judge cross-family panel with PDF attached (Claude Sonnet 4.6, GPT-5.4 with medium reasoning, and Gemini 3.1 Pro). Outcome verdicts are confirmed-issue, fp, or ambiguous.

3 Results and Discussion

3.1 Detection rates of injected errors

Detection by error type.

Table 2 gives total and per-error-type detection rates on both corpora, with 95 % CIs as ± half-widths from nboot=2000 paper-level bootstrap replicates (the scheme used for every interval in this section). In the following, we call a difference between two systems significant when a paired paper-level bootstrap puts less than 5% probability on its sign reversing (Appendix G). The two highest totals on each corpus belong to Skepthical and structured GPT-5.5: 70.6±6.2 and 61.3±6.6%, respectively, on astrophysics, a significant difference; and 76.7±5.6 and 81.1±5.3% on ML, where the difference is not significant. On general issues, the highest rate is Skepthical’s on astrophysics (82.2%; next is structured GPT-5.5 at 72.7%, a difference short of significance) and is shared by Skepthical and structured GPT-5.5 on ML (84.4% each). On mathematics, the highest is structured GPT-5.5’s on both corpora, by a small, non-significant margin on astrophysics (75.6 vs 73.8% for Skepthical and free-form GPT-5.5) and a larger one on ML (91.1 vs 77.8% for Skepthical) that falls just short of significance. On numerics the highest is structured Claude Code’s on astrophysics (91.1%, obtained by writing and running ad-hoc Python checks; see below) and is shared by Skepthical and structured GPT-5.5 on ML (91.1% each). On citations, Skepthical, the only system in which PDF retrieval is enforced, is at 46.7% against at most 26.7% for any other system on astrophysics, whereas on ML structured Claude Opus 4.7, structured GPT-5.5, and structured Claude Code are at 57.8–60.0% and Skepthical at 53.3%; we examine the reason for this behaviour below.

External reviewers.

The two ML-tuned open-weights reviewers detect almost no injections on either corpus (≤2.3% on astrophysics; 1.7 and 2.8% on ML, their home field). The Stanford Agentic Reviewer, run on astrophysics only, reaches 40.7%, 21 percentage points (pp) below the strongest single LLM and 30​pp below Skepthical.

Detection by severity.

Figure 1 adds the third axis, injected severity, on both corpora, with one column per architectural family and one line per provider (single-LLM columns) or per system (agentic and external columns). Detection rises from subtle to blatant in most individual error-type curves and in the average over error types of almost every system. The strongest systems detect a large fraction of the subtle injections (57–73% at S1 for Skepthical, structured GPT-5.5, and structured Claude Code), whereas the weakest configurations miss most errors even when blatant (the open-weights reviewers below 5%). The ordering of systems is essentially the same at every severity level, so the comparisons that follow hold across the scale.

Prompt content.

As shown by the per-provider rows of Table 2 and per-provider curves of Figure 1, on the same free-form prompt, three frontier LLMs differ by up to 48.6​pp in total detection on astrophysics (GPT-5.5 60.5%, Claude Opus 4.7 36.7%, Gemini 3.1 Pro 11.9%) and by 52.8​pp on ML (63.9, 43.9, and 11.1%). Mathematics and numerics drive this discrepancy.

The structured prompt narrows the cross-provider spread considerably by boosting the performance of the weakest models: it has a very large impact for Gemini on both corpora (+25.4​pp on astrophysics, +40.0​pp on ML) and a smaller but significant one on Claude (+16.6 and +28.3​pp). For GPT-5.5, the effect of the structured prompt is near-negligible on astrophysics (+0.8​pp), but substantial on ML (+17.2​pp). In general, we can conclude that detection depends strongly on both the model family and the prompt content

The same prompt change inside the Claude Code group is the cleanest probe of what an operating-instructions prompt causes. With the model (Claude Opus 4.7), sandbox, allowed tools, and arXiv access held fixed, the only difference between the two configurations is the prompt, which moves detection by +32.2​pp on astrophysics and +38.9​pp on ML. The full output of the astrophysics runs, which we have analysed, shows how the agent uses its available resources. (i) The structured agent used tools extensively (median 9 agent turns, max 26), whereas the free-form agent simply read the PDF and replied. (ii) The structured agent wrote and ran 21 multi-line Python scripts (using NumPy, SymPy, SciPy, math, and mpmath), whereas the free-form agent wrote none, although it knew it could. (iii) Neither used wolframscript, although the structured prompt encourages its use. In summary, with everything else fixed, the prompt alone drives qualitatively different tool use.

Agentic scaffolding.

Holding the structured prompt fixed, going from a single Claude Opus 4.7 call to Claude Code to Skepthical takes detection from 53.3 to 61.0 to 70.6% on astrophysics and from 72.2 to 73.3 to 76.7% on ML. The first step adds tools at fixed model and prompt, and its gain on astrophysics comes from the error types that a script can check: numerics and mathematics, with +12.0 and +16.4​pp (both significant); on ML, where there are fewer mathematical and numerical statements to check, there are statistically insignificant changes.

Going from Claude Code to Skepthical changes more than scaffolding, as Skepthical draws on three LLM providers. With this caveat in mind, Skepthical’s gains over Claude Code on astrophysics are on citations (+20.0​pp) and general issues (+26.7​pp), both significant, while Claude Code is ahead on numerics (by 11.1​pp, significant); on ML, no difference is significant. The citation result is the clearest case for enforcing a verification step rather than offering one: Claude Code had the same arXiv access, but the execution traces show paper retrieval never took place, whereas in Skepthical retrieval is mandatory. On astrophysics this puts Skepthical 20​pp ahead of structured Claude Code on citations; on ML it does not (53.3 vs 60.0%, not significant). A possible explanation, developed in the next paragraph, is that the models are familiar enough with the ML literature to judge many cited claims without retrieval from the source, so that enforced retrieval adds little there.

Finally, compared with the best single-LLM reviewer, structured GPT-5.5 (the highest total among the six single-LLM configurations on both corpora), Skepthical is +9.3​pp on astrophysics (significant) and −4.4​pp on ML (not significant). The multi-agent pipeline therefore outperforms the best single-LLM reviewer only on the corpus where its enforced steps make a difference, citation retrieval on astrophysics in particular; on ML the two are statistically indistinguishable on overall detection rate. We note, however, that against the concise structured GPT-5.5 of Appendix H, which reaches 68.9% on astrophysics and 76.1% on ML, the margin is +1.7 and +0.6​pp, neither significant.

Citation detection across fields.

On ML, the structured single LLMs have a higher citation detection rate than on astrophysics (all three providers; GPT-5.5 from 18 to 58%), while Skepthical barely changes (47 to 53%). We attribute this to two factors, as discussed in more detail in Appendix J. First, more of the ML citation errors can be detected by having a general knowledge of the field, e.g., a well-known method attributed to the wrong paper (14 of 45, against 4 of 45 on astrophysics). Second, the models appear to know the ML literature better than the astrophysics one, since they also detect more of the errors that we classified as needing retrieval from the cited paper. Skepthical, which reads the cited paper in every case, changes little. In conclusion, retrieval makes a difference when the model does not already know the literature.

Mechanical verification.

The detection results understate the contribution of the mechanical verifiers. A frontier LLM matching Skepthical’s 80% numerics detection rate is not the same finding as Skepthical delivering 80% recall through tool-mediated deterministic computation. Skepthical’s checks invoke deterministic tools (Python calculations, wolframscript, pint, and arXiv-PDF retrieval) on every checked claim and are auditable in source, while the single-LLM equivalent is a single forward pass whose correctness on any given equation is opaque and non-reproducible. Claude Code with the structured prompt sits between the two: it has the same tool affordances as Skepthical, but only invokes them on a fraction of instances (and never wolframscript, despite the prompt’s encouragement), so its detections are mechanically grounded only on the instances where the agent chose to run a verification script.

Further results.

Six further analyses are reported in the appendices: a test for judge–reviewer family overlap (Appendix E); a human check of the detection adjudication for one paper (Appendix F); a concise structured-prompt configuration (Appendix H); the attribution of Skepthical’s detections to its modules (Appendix I); a test of whether errors in the same copy suppress each other’s detection (Appendix K); and the agreement between reviewer systems on which errors they detect (Appendix L).

3.2 False positives

We define a reviewer’s false-positive (FP) rate as the fraction of the flags it raises on the 15 clean copies of a corpus that the adjudication panel judges clearly false (§2.4). Figure 2 shows, per system and corpus, the FP and ambiguous counts and the FP rate with its paper-level bootstrap 95 % CI. Note the rate depends on the count and the total flag count, so a short bar can carry a high rate and vice versa.

False-positive rates are generally low: apart from the two open-weights reviewers, every system is below 9% on astrophysics, and six of nine are below 10% on ML, the exceptions being the three structured single LLMs (11.5–22.7%). Skepthical has a low FP rate on both corpora, 2.8% on astrophysics (106 flags) and 4.0% on ML (99 flags), against 3.4–22.7% for the structured single LLMs. The lowest rates overall belong to free-form single LLMs, which raise almost no mathematics or numerics flags: Claude Opus 4.7 on astrophysics (1.0%) and Gemini 3.1 Pro on ML (3.5%).

The effect of the structured prompt on the FP rate is mixed: on astrophysics it narrows the spread across providers, lowering GPT-5.5’s rate (8.7→3.4%) and raising Claude’s (1.0→5.2%); on ML it raises every provider’s rate; and for Claude Code it lowers the rate on both corpora. Three things, however, are consistent across reviewers and corpora. First, mathematics flags are the least reliable: pooled over the frontier systems, they are judged false at 30% on astrophysics and 67% on ML, against 3% for general-issue flags, and they account for half of all false positives on both corpora. Second, the structured prompt roughly halves the number of general-issue flags, which are rarely wrong, so it can raise the FP rate even when the number of false positives falls (GPT-5.5 on ML: 12→9 false positives on 131→69 flags). Finally, the two open-weights detectors perform poorly on both corpora, featuring the highest rates.

4 Limitations

The benchmark measures one part of peer review: whether a reviewer detects objective, localisable defects (citation support, mathematical and numerical correctness, internal consistency) and whether it raises false alarms on clean papers. The benchmark does not address issues such as novelty, significance, adequacy of experiments, missing controls, interpretive judgement, or alignment with human-written reviews, as they are significantly harder to quantify and, in general, lack a clear ground truth. Therefore, our results should not be interpreted as holistic measurements of review quality.

Each corpus is small: 15 papers from one day in one field, capped at 20 pages. The rates reported here are therefore estimates for those two slices, and longer manuscripts are not represented. As already noted, severity labels are calibrated within each error type but not across them. In addition, each injected copy carries four errors, and a blatant one might draw attention away from a subtle one in the same copy. Because severities were assigned independently for each error type, the severity of the other errors in a copy varies at random. We find that whether an error is detected does not depend measurably on how severe its neighbours are (Appendix K). However, this test cannot ascertain whether the mere presence of other errors has an effect; that would require copies carrying a single error, which was beyond the cost constraints of this work.

Adjudication is LLM-based for detection matching and for general-issue and citation false positives. Due to resource constraints, the blinded human check covers just one paper and detection only (Appendix F). The comparisons involving Skepthical are not capacity-matched: it uses more calls and three providers, and we did not run a same-provider ensemble of equal cost, so its gains cannot be attributed to the pipeline design alone.

For the external reviewers, two specific fairness caveats apply. The Stanford reviewer is hard-capped at the first 15 pages of the PDF; the same page truncation is applied to OpenReviewer-8B’s input; and PaperAudit-Bench Deep mode chunks natively and is given the full paper.

Exact reproduction of any system here requires the frozen commercial model versions, which will be deprecated. The benchmark, however, remains reusable, since the evaluation pipeline, the paper lists, the injections, and the adjudication records are released. Finally, three risks accompany deployment of tools like Skepthical at scale: false confidence from uneven per-error-type detection, adversarial gaming of known weaknesses, and concentration of review infrastructure on a few LLM providers. The design choices here (cross-family ensembling, auditable verification of mathematics, numerics, and citations) mitigate these risks but do not remove them.

5 Conclusion

In this paper we have introduced an evaluation pipeline for AI peer reviewers that injects errors of four types and three severity levels into real papers and measures the detection rate as a function of error type, severity, prompt, and reviewer architecture, together with the false-positive rate on clean copies. We have applied it to two corpora of 15 real arXiv preprints each, one in astrophysics and one in ML, and benchmarked twelve reviewer systems: six single-LLM reviewers, two Claude Code configurations, three publicly available external reviewers, and Skepthical. Skepthical is a multi-agent reviewer, also introduced here, with a reference-attached citation verifier, a computer-algebra-backed mathematics auditor, a code-backed numerical auditor, and a cross-family general-issues ensemble. The evaluation pipeline, together with the injection catalogues, and the full adjudication logs, is released as a benchmark for the AI-reviewer landscape at github.com/inigozubeldia/skepthical-benchmark, and Skepthical is publicly available at skepthical-ai.org.

Our main conclusions are as follows.

1. 1.

   The choices of model and prompt make a difference. Given the same one-sentence prompt, three frontier LLMs differ by about 50​pp in total detection on both corpora, mostly on mathematics and numerics. The longer structured prompt narrows this spread by lifting the weaker models. For Claude Code, with model and tools fixed, the same prompt change alone triggers extensive tool use and raises detection by over 30​pp on both corpora.

2. 2.

   Tools and scaffolding help when they contribute what the model cannot supply on its own. In particular, on astrophysics, Claude Code improves significantly on a single Claude call in mathematics and numerics detection, and enforcing citation retrieval gives Skepthical a large citation lead there. On ML, however, where the models appear to know the literature better, enforced retrieval does not lead to a significant improvement.

3. 3.

   Detection rises with severity for every capable system, whereas the two fine-tuned open-weights reviewers have a negligible detection rate.

4. 4.

   False-positive rates are generally low, below 10% for most systems apart from the open-weights reviewers, but depend on prompt, model, and scaffolding. Mathematics flags are the least reliable and account for half of all false positives; Skepthical keeps a low rate on both corpora.

Beyond the AI-reviewer landscape itself, peer review remains a cornerstone of the scientific process for both human-authored and increasingly AI-conducted research [3, 13, 26, 18, 19, 25, 22, 7]. In this regard, we see Skepthical, and tools like it, both as standalone reviewing infrastructure for human-authored research and as components that AI scientists could integrate as an internal self-audit layer guiding autonomous research.

Acknowledgments and Disclosure of Funding

The authors would like to thank Antony Lewis, Licong Xu, Thomas Borrett, and Raul Jimenez for useful discussions. IZ and BB thank the Infosys–Cambridge AI Centre for support. This work was partially funded by an unrestricted gift from Google to BB. Part of the development of Skepthical was carried out while IZ was at the Kavli Institute for Theoretical Physics (KITP); his stay was supported in part by grant NSF PHY-2309135 to the KITP. MC is supported by an AI2050 Early Career Fellowship from Schmidt Sciences and by the Isaac Newton Trust; Polymathic AI is funded by the Simons Foundation and Schmidt Sciences. The authors declare no competing interests.

References

• [1] Bianchi, F., Queen, O., Thakkar, N., Sun, E., & Zou, J. (2025) “Exploring the use of AI authors and reviewers at Agents4Science.” Nature Biotechnology, December 2025. https://doi.org/10.1038/s41587-025-02963-8.
• [2] Biswas, J., Schoepp, S., Vasan, G., Opipari, A., Zhang, A., Hu, Z., Joseph, S., Lease, M., Li, J. J., Stone, P., Wagstaff, K. L., Taylor, M. E., & Jenkins, O. C. (2026) “AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot.” arXiv:2604.13940.
• [3] Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023) “Autonomous chemical research with large language models.” Nature 624, 570–578.
• [4] Chang, Y., Li, Z., Zhang, H., et al. (2025) “TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review.” arXiv:2506.07642.
• [5] D’Arcy, M., Hope, T., Birnbaum, L., & Downey, D. (2024) “MARG: Multi-Agent Review Generation for Scientific Papers.” arXiv:2401.04259.
• [6] Dycke, N., & Gurevych, I. (2025) “Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework.” arXiv:2508.21422.
• [7] Hambardzumyan, K., Baldwin, N., Toledo, E., et al. (2026) “AIRA_2: Overcoming Bottlenecks in AI Research Agents.” arXiv:2603.26499.
• [8] Idahl, M., & Ahmadi, Z. (2025) “OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews.” Proceedings of NAACL 2025, System Demonstrations.
• [9] Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) “AgentReview: Exploring Peer Review Dynamics with LLM Agents.” Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1208–1226.
• [10] Li, R., Gu, J.-C., Kung, P.-N., Xia, H., Liu, J., Kong, X., Sui, Z., & Peng, N. (2025) “LLM-REVal: Can We Trust LLM Reviewers Yet?” arXiv:2510.12367.
• [11] Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., et al. (2023) “Can large language models provide useful feedback on research papers? A large-scale empirical analysis.” arXiv:2310.01783.
• [12] Liu, R., & Shah, N. B. (2023) “ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing.” arXiv:2306.00622.
• [13] Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., & Ha, D. (2024) “The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.” arXiv:2408.06292.
• [14] Luo, X., Jiang, X., & Wu, J. (2026) “MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors.” arXiv:2604.06846.
• [15] Panickssery, A., Bowman, S. R., & Feng, S. (2024) “LLM evaluators recognize and favor their own generations.” NeurIPS 2024 (oral).
• [16] Jiang, Y., & Ng, A. (2025). Stanford Agentic Reviewer. Stanford ML Group. https://paperreview.ai; technical overview at https://paperreview.ai/tech-overview. Accessed 2026-05-01.
• [17] Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) “Beyond Accuracy: Behavioral Testing of NLP Models with CheckList.” ACL 2020.
• [18] Schmidgall, S., et al. (2025) “Agent Laboratory: Using LLM Agents as Research Assistants.” arXiv:2501.04227.
• [19] Swanson, K., Wu, W., Bulaong, N. L., Pak, J. E., & Zou, J. (2025) “The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies.” Nature 646, 716–723.
• [20] Tu, S., Ma, Y., Lin, J., Zhang, Q., Lan, X., Li, J., Xu, N., Li, L., & Zhao, D. (2026) “PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review.” arXiv:2601.19916.
• [21] Verga, P., Hofstätter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., & Lewis, P. (2024) “Replacing judges with juries: Evaluating LLM generations with a panel of diverse models (PoLL).” arXiv:2404.18796.
• [22] Villaescusa-Navarro, F., et al. (2025) “The Denario project: Deep knowledge AI agents for scientific discovery.” arXiv:2510.26887.
• [23] Weng, Y., Zhu, M., Bao, G., Zhang, H., Wang, J., Zhang, Y., & Yang, L. (2025) “CycleResearcher: Improving Automated Research via Automated Review.” The Thirteenth International Conference on Learning Representations (ICLR).
• [24] Xi, S., Rao, V., Payan, J., & Shah, N. B. (2025) “FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers.” arXiv:2511.21843.
• [25] Xu, L., et al. (2025) “Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery.” arXiv:2507.07257.
• [26] Yamada, Y., et al. (2025) “The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search.” arXiv:2504.08066.
• [27] Ye, J., et al. (2024) “Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge.” arXiv:2410.02736.
• [28] Zeng, S., Tian, K., Zhang, K., et al. (2025) “ReviewRL: Towards Automated Scientific Review with RL.” arXiv:2508.10308.
• [29] Zhou, Z., Liu, S., Ning, M., et al. (2024) “Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist.” arXiv:2407.08733; ICLR 2025.
• [30] Zhu, M., Weng, Y., Yang, L., & Zhang, Y. (2025) “DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process.” Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 29330–29355.
• [31] Baes, M. (2026) “The core of the problem: Physical limits of the core-Sérsic model.” arXiv:2604.20344.
• [32] Carrillo-Monteverde, A., et al. (2026) “Photon and neutrino fluxes from spheroidal dwarf galaxies in a decaying DM model.” arXiv:2604.20086.
• [33] Chen, J.-M., et al. (2026) “Fast-Cooling Synchrotron in Decaying Magnetic Fields: Implications for the GRB Spectral Distribution.” arXiv:2604.20142.
• [34] Chen, D.-C., et al. (2026) “Sub-Neptunes Show a Stronger Correlation with Cold Jupiters than Super-Earths Especially in Metal-rich Systems.” arXiv:2604.20203.
• [35] Garg, M., et al. (2026) “Chaotic migration of LISA Extreme Mass Ratio Inspirals in a turbulent accretion disk: effect on waveform de-phasing.” arXiv:2604.20971.
• [36] Geng, Y., et al. (2026) “Measuring Tianlai’s primary beam using sky model.” arXiv:2604.20262.
• [37] He, K., et al. (2026) “The Evolution of the SFR–M⋆ Relation at 0.1<z<4: Environmental and Morphological Dependencies.” arXiv:2604.20411.
• [38] Kakimoto, T., et al. (2026) “DeepDive: Simultaneous Formation of Massive Quiescent Galaxies in High-Redshift Galaxy Proto-clusters.” arXiv:2604.21007.
• [39] Kinney, W. H. (2026) “Geodesic Completeness in General Cosmological Scenarios.” arXiv:2604.20809.
• [40] Lella, A. (2026) “Multimessenger probes of Axions from Compact Objects.” arXiv:2604.21077.
• [41] Matsuo, M., et al. (2026) “Interaction between nuclear clusters and superfluid phonons in the neutron-star inner crust.” arXiv:2604.20725.
• [42] Padgett, M. J., & Kephart, T. W. (2026) “Ultra High Energy Cosmic Rays from the Local Void.” arXiv:2604.21099.
• [43] Quelquejay Leclere, H. (2026) “Probing Supermassive Black Hole Mergers with Pulsar Timing Arrays.” arXiv:2604.20975.
• [44] Slater, R., et al. (2026) “Cold molecular gas distribution and kinematics in the low-metallicity dusty starburst of Mrk 996 resolved with ALMA.” arXiv:2604.20080.
• [45] Wang, Z., et al. (2026) “Cracking Gravitational Wave Multiple Ringdown Modes in Space.” arXiv:2604.20914.
• [46] Anthropic. (2025). Claude Code. https://claude.ai/code.
• [47] Zang, B. et al. (2026) “HIVE-3D: Hierarchical Voxel Enhancement for High-Quality 3D Scene Generation.” arXiv:2607.13468.
• [48] Yang, E. et al. (2026) “Deformable State Estimation for Autonomous Surgical Tissue Retraction Under Partial Observability.” arXiv:2607.13475.
• [49] Chethan Reddy, G. P. (2026) “ExTernD: Expanded-Rank Ternary Decomposition Ternary LLM PTQ with Accuracy Approaching Any Quantization Level.” arXiv:2607.13511.
• [50] Tran, L. (2026) “FastCentNN: Accelerating Centroid Neural Network with Entropy Proxy.” arXiv:2607.13613.
• [51] Deb, S. et al. (2026) “STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle.” arXiv:2607.13618.
• [52] Kam, C. et al. (2026) “Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations.” arXiv:2607.13749.
• [53] Wang, W. et al. (2026) “Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0.” arXiv:2607.14004.
• [54] Gürsoy, M. E. et al. (2026) “Lighthouse RL: Sample-Efficient Circuit Optimization via Strategic Reset Points.” arXiv:2607.14008.
• [55] Grillmeyer, D. et al. (2026) “Improving Wind and Solar Power Prediction with Efficient Wrapper-based Feature Selection: An Empirical Study.” arXiv:2607.14024.
• [56] Amati, G. et al. (2026) “Low-Latency Relay Selection in NR-V2X Vehicular Communications via Graph Isomorphism Networks with Edge Features.” arXiv:2607.14176.
• [57] Hasan, M. M. et al. (2026) “XCT-SAM: Sequential Parameter-Efficient Domain Adaptation of SAM for Industrial XCT Defect Segmentation.” arXiv:2607.14287.
• [58] Gaytan-Villarreal, D. et al. (2026) “Real-Time Detection of Charge Jumps in Superconducting Qubits with a Convolutional Neural Network.” arXiv:2607.14293.
• [59] Makridis, G. et al. (2026) “Towards a Unified Multidimensional Explainability Metric: Evaluating Trustworthiness in AI Models.” arXiv:2607.14315.
• [60] Soltanalian, M. (2026) “Random Parameter Noise Does Not Make Exact ReLU Verification Easy.” arXiv:2607.14375.
• [61] Amati, G. et al. (2026) “AI-Driven Multi-Hop Relay Selection for Smart Urban NR-V2X Networks via Learning-to-Optimize Graph Neural Networks.” arXiv:2607.20554.

Appendix A Skepthical pipeline detail

Skepthical reviews a paper with a pipeline of specialised agents rather than with a single LLM call. The paper is processed in parallel by four modules, one each for general issues, mathematics, numerics, and citations, and their outputs are merged at the end into a single structured review (Figure 3). Table 3 lists the model used by every agent in the configuration run in this work.

A.1 General-issues module

The general-issues module produces the backbone of the review. It has two parts: a structured stage, in which two LLMs from different families are combined so that the issues raised do not depend on a single family, and a parallel free-form reading of the paper by a single reasoning model.

The first stage is the total_reviewer ensemble: two parallel calls, one on Claude Opus 4.7 and one on Gemini 2.5 Pro, each returning a summary, a list of strengths, and three lists of issues by severity (major, minor, and very minor) in a fixed JSON format. Both calls use the same operating instructions, which require a coverage checklist of five categories (general, logical consistency, evidence and citations, presentation and clarity, reproducibility), a consistency pass across sections on named choices and re-used quantities, an audit of rhetoric and scope intended to catch sweeping generalisations and contradictions in direction or kind, and a phantom-evidence audit in which every reference to a figure, table, or other reported result must be matched to the corresponding object in the paper. The two outputs are then merged by report_merger (GPT-5.1), which removes duplicate issues, combines the strengths, and harmonises the severity levels across the two calls. The merger belongs to a third family (OpenAI), so that neither of the two input families is favoured at the merge step [15].

In parallel with the structured ensemble, the unstructured_reviewer agent (GPT-5.4, high reasoning) reads the paper without a checklist. Its purpose is not coverage, which the structured pass already provides, but to bring extended reasoning to a workflow whose other parts run without it. This serves two ends. First, the free-form pass can raise concerns about the paper as a whole that fall outside the checklist. Second, it helps to establish the relative importance of the issues found by the structured calls, which are not necessarily reported in order of importance. This pass runs on the third frontier family (OpenAI), so that, taken together, the structured and unstructured parts of the general-issues module span Anthropic, Google, and OpenAI.

A.2 Mathematics module

The mathematics module audits the paper’s mathematics with a five-stage pipeline. The first stage extracts the mathematical statements in the paper, the second cross-checks them, the third verifies them mechanically (if applicable), and the fourth and fifth interpret and aggregate the results.

In stage 1, the maths_extractor agent (Gemini 2.5 Pro, high reasoning) extracts every mathematical statement that the paper presents as central (e.g., an equation, a derivation chain, a limit or series identity, a dimensional or normalisation constraint, a definition, or an inequality) into a structured record. The record contains the LaTeX of the statement, the verbatim source snippet, its location, the kind of statement, a flag marking whether units matter, and an executable plan for checking it. When a derivation chain contains an asserted equation, that equation is also extracted on its own, so that the derivation steps go to mechanical verification while the asserted starting equation receives a separate knowledge audit. The extractor must record every numbered equation it does not extract in an excluded_equations list, with a reason. If that list is empty while some equations remain uncovered, the extraction is repeated with reinforced instructions.

Stage 2 is a cross-family check of the extracted statements. For each of them, the maths_transcription_checker agent (Claude Opus 4.7) verifies two things. First, that the code prepared for the mechanical check in stage 3 faithfully reproduces the statement as it appears in the paper. Second, for any equation that the paper asserts rather than derives, such as a definition or a standard result quoted from the literature, that the equation matches the canonical form of the quantity it describes. This second check targets errors in prefactors, exponents, scaling factors, and subleading terms, which a mechanical check of the paper’s own derivation cannot detect, since the derivation can be internally consistent while starting from a wrong equation. We note that stages 1 and 2 run on different model families, which makes it less likely that an error of the extracting model, for instance a misread sign or exponent, is shared by the model that checks its work.

In stage 3, the statements are verified mechanically. Symbolic equalities, limits, series identities, and solvability queries are checked with wolframscript. When a statement contains free symbols, it is also evaluated at ten random points of their parameter space. SymPy serves as a fallback if wolframscript crashes. In addition, pint performs the dimensional analysis of the statements flagged as unit-critical in stage 1. Each check returns one of six outcomes: confirmed, refuted, inconclusive, error, skipped, or unavailable.

In stage 4, the maths_interpreter agent (Claude Opus 4.7) combines the outcome of the stage-2 check and the stage-3 mechanical verdict for each statement into a final judgement of pass, fail, or uncertain. Finally, in stage 5, the maths_aggregator agent (Gemini 2.5 Pro, high reasoning) compiles the per-statement judgements into the mathematics audit and writes a mathematics audit report that is attached to the final review.

A.3 Numerics module

The numerics module verifies numerical statements with code. First, two extractor agents from different model families, numerics_extractor_a (Claude Opus 4.7) and numerics_extractor_b (GPT-5.5, high reasoning), read the PDF independently and each returns a list of numerical statements to be verified. Every statement is labelled with one of eleven kinds of check (agreement between abstract and body, a table-row total, a table-column total, percentages that should sum to one hundred, a percentage of a stated quantity, a ratio, a difference, an integer equality, a constant repeated across the paper, a rounded value, and a sum that should equal a stated total). To make sure that the statement refers to a number that actually appears in the paper, the extractor must also record where it found it. It does so by quoting a verbatim passage of at least eight consecutive words from the paper that contains the number, together with the number itself exactly as printed, without units or LaTeX markup.

Second, a deterministic Python step compares the two lists and keeps only the statements on which both extractors agree, that is, those for which the quoted passage, the kind of check, and the printed number all match. Statements found by only one extractor are set aside in a separate list of unverified items. Each surviving statement is then re-read by the numerics_rereader agent (GPT-4.1). This agent is given only the quoted passage and the kind of check, not the number reported by the extractors, and is asked to state which number is printed in that passage. The statement is accepted for verification only if the number the re-reader reports matches the one both extractors gave. This removes a whole class of errors in which an extractor misreads or invents a number.

Third, the accepted statements are verified by Python code, with no LLM involved. For each of the eleven kinds of check there is a fixed function that performs the corresponding arithmetic on the extracted numbers, for instance summing a table row and comparing the result with the printed total. When a check fails, a fixed rule assigns the severity according to the kind of check and the size of the discrepancy. Some problems cannot be found in this way, and a single LLM call is used for them. The numerics_reasoner agent (GPT-5.5) reads the full PDF once, with the results of the mechanical checks given to it so that it does not repeat them, and looks for two further kinds of problem: a quantity that is given different values in distant parts of the paper, and numerical errors that both extractors missed. Its findings are marked in the audit report as not mechanically verified. Each of them must be anchored to a verbatim quote from the paper, and a finding whose quote cannot be found in the paper is discarded. Finally, the numerics_reporter agent (GPT-4.1) compiles the mechanical and reasoner findings into a numerics audit report that is attached to the final review.

A.4 Citations module

The citations module checks the factual statements in the paper that rely on external references against the cited papers themselves, which it retrieves. It runs in two passes. In the first, the statement_extractor agent (GPT-5.1) reads the full text of the paper and returns the N most important such statements. N grows with the length of the paper, at one statement per ten thousand characters, up to a maximum of ten in order to bound the cost. Each statement is returned together with the reference identifier exactly as it appears in the paper (a BibTeX key, a numeric label, or an author-year string). Two further agents then connect the statements to the cited papers. First, the reference_extractor agent (GPT-4.1) copies the entries of the paper’s reference list verbatim. Then, the reference_resolver agent (GPT-5 mini) maps each in-text identifier to its entry in the reference list and from there to an arXiv identifier, resolving differences in numbering and in author-year forms (a smaller model suffices for this lookup step). The PDFs of the cited papers are then downloaded from the arXiv using the identifiers obtained in this way.

The statements are then verified. For each of them, the paper_verification_agent agent (Gemini 2.5 Pro, high reasoning) receives the statement together with the PDF of the cited paper and returns one of five verdicts: supported, weakly supported, partially supported, contradicted, or unrelated. The agent is instructed to use only the attached paper, which limits the influence of what the model may already know, or believe, about the cited work.

In the second pass, the statement_extractor agent is run again, this time with the first-pass statements and their verdicts given to it so that it does not repeat them, and asked for up to three further statements that look suspicious. These are verified in the same way by the paper_verification_agent agent, and their verdicts are labelled as coming from the second pass, so that the two passes can be weighted separately later in the pipeline.

A.5 Coordination, assembly, and deployment

The four modules run concurrently, on four worker threads inside the Skepthical worker process. Once all four have finished, an assembly step builds the review in a fixed order as follows: it merges the two total_reviewer outputs through the report_merger agent, adds the paper title, appends the mathematics and numerics audit reports, folds in the free-form pass through the unstructured_report_merger agent, and finally inserts the per-statement verdicts of the citations module. The assembled review is then compiled to markdown and PDF.

Skepthical is deployed on Google Cloud Run and is accessible at skepthical-ai.org. Submitted papers are placed in a queue and processed by worker instances that run in parallel, so throughput grows with the number of workers. A typical 12-page astrophysics paper takes approximately 24 minutes.

Appendix B Corpus

Both corpora are listed in Table 4, which gives each paper’s primary category; 6 of the astrophysics papers and 8 of the ML papers are cross-listed to the corpus category. Of the 60 astro-ph submissions of 2026-04-22, 24 were within the 20-page cap, of which 7 could not be processed by the injection toolchain and 2 were injected but not evaluated, leaving the 15 evaluated papers.

Of the 151 cs.LG submissions of 2026-07-15 (cross-lists included), 81 were within the page cap and 74 had LaTeX source that compiled unmodified and contained enough material for all four error types, as judged by an LLM from the PDF (Claude Sonnet). The 15 evaluated papers were drawn from these 74 by stratified random sampling over eight subfields assigned from the titles alone, with quotas of one to three papers per subfield. This screening was not infallible: two drawn papers turned out to have no displayed mathematics and were replaced by the next paper in their subfield. The ML papers are shorter than the astrophysics ones (median 9 against 13 pages) and contain fewer displayed equations (median 5 against 12).

Appendix C Error injection design and taxonomy

The injection pipeline has two pieces. First, a severity-balanced 3-copy schedule fixes how the twelve (error type, severity) combinations are distributed across the three injected copies of each paper, so that every combination is chosen exactly once. Second, a combination registry of 33 sub-types, distributed across those twelve (error type, severity) combinations, with two or three sub-types per combination, determines what each combination looks like in practice, with a verbatim mechanism block, an eligibility filter, and a fixed output format (a Pydantic schema) that the LLM-driven proposer must conform to.

C.1 Severity-balanced 3-copy schedule

Each paper features in four versions: a clean copy that carries the original LaTeX source, and three injected copies, each bearing one error per error type (four errors per copy). Across the three injected copies, every (error type, severity) combination is hit exactly once. A schematic schedule for one paper:

Reading down a column gives a permutation of {S1,S2,S3}, so that each error type sees each severity exactly once across the three copies. The permutations are drawn independently for each error type, so the severities of the four errors in a copy are independent of one another.

C.2 Injection error sub-types with verbatim examples

Table 5 gives the operating definition of each (error type, severity) combination used for the astrophysics corpus, summarising the severity scale per error type from S1 (subtle) through S2 (clear) to S3 (blatant). The injection registry refines this taxonomy into 33 sub-types distributed across the twelve combinations, with two or three sub-types per combination (e.g. Citations ⋅ S1 admits both “quantitative-qualifier mis-quote” and “methodological-qualifier drift”). On the ML corpus, the same registry was used with one substitution: since ML equations often carry no physical units, the mathematics S3 sub-type ‘dimensional absurdity’ was replaced by ‘shape or axis non-conformability’ (a tensor-shape or index mismatch).

For each sub-type to be injected into a paper, an LLM-driven proposer (Claude Sonnet 4.6 on the astrophysics corpus, Gemini 3.1 Pro on the ML corpus) is given the full paper LaTeX, a verbatim mechanism block, an eligibility filter, and a fixed output format (a Pydantic schema), and asked to return a single atomic edit. Each proposed edit then passes through a mechanical validator. The validator accepts an edit only if the text to be replaced occurs exactly once in a source file that is part of the compiled document, the replacement actually changes the text, and no citation key is added, removed, or altered; the edited copy must also compile.

Here we give one verbatim example per sub-type, all drawn from the astrophysics corpus, grouped by severity within each error type.

C.2.1 Citations

S1 (subtle).

Quantitative-qualifier mis-quote. Paper: 2604.20203. Original: “Here we adopt the result of [cite], P⁡(CJ,[Fe/H]>0)=14.3−1.8+2.0%, as their definition of cold Jupiters in mass and semimajor axis is same to that in this work.”. Injected: “…P⁡(CJ,[Fe/H]>0)=16.5−1.8+2.0%, as their definition…”. The cited R21 paper reports 14.3−1.8+2.0%; the injection nudges the value upward by +2.2​pp (∼1.1​σ) while the cite key is preserved byte-for-byte.

Methodological-qualifier drift. Paper: 2604.20975. Original: “We model it using the Schechter-like parametric model introduced in [cite],”. Injected: “We model it using the parametric model introduced in [cite],”. The dropped functional-form qualifier (“Schechter-like”) is explicit in the cited paper and load-bearing for the model class.

S2 (clear).

Conditional-claim drop / over-generalisation. Paper: 2604.20086. Original: “stacked analyses using IceCube [cite] data have shown that neutrino searches from dSphs can provide competitive constraints on the dark matter annihilation cross section for several leptonic channels.”. Injected: same sentence with the trailing qualifier “for several leptonic channels” dropped, generalising the cited result from leptonic channels to all annihilation channels.

Methodology category mis-attribution. Paper: 2604.20080. Original: “as observed in giant molecular clouds [cite]”. Injected: “as predicted by hydrodynamic simulations of giant molecular clouds [cite]”. Solomon et al. 1987 is purely observational; recasting the citing claim as “predicted by hydrodynamic simulations” mis-categorises the cited methodology.

S3 (blatant).

Direction reversal. Paper: 2604.21099. Original: “[cite] limit of E∼5×1019 eV (or GZK cutoff) is the upper bound on the energy for cosmic-ray protons …”. Injected: “…is the lower bound on the energy for cosmic-ray protons …”. The polarity of the cited GZK conclusion is reversed (upper → lower bound).

Topic mis-attribution. Paper: 2604.20344. Original: “the conclusion reached previously for the Nuker model [cite]”. Injected: “the convergence criterion established for numerical Abel-transform inversions in triaxial stellar systems [cite]”. The reattributed topic (numerical Abel-transform inversions in triaxial systems) is unrelated to the cited paper’s actual subject (the Nuker model).

C.2.2 Numerics

S1 (subtle).

Abstract↔body second-decimal drift. Paper: 2604.20080. Original (abstract): “The global CO(2–1)/CO(1–0) ratio is low (R21≈0.29–0.34)”. Injected (abstract): “…(R21≈0.29–0.35)”. The abstract’s headline R21 upper bound is perturbed in the second decimal; the body (Sect. 3.5 and Table 5) still reports R21=0.34.

Table-row total slight mismatch. Paper: 2604.20411. Original (Table 1, 0.7≤z<1.0 row, galaxy-count column): 47279. Injected: 47269. The row count is reduced by 10; the abstract still reports the unmodified mass-complete sample size of ∼ 290k.

Repeated-constant slight drift. Paper: 2604.20971. Original: “turbulence normalization C (=360 in the aforementioned HD study)”. Injected: “turbulence normalization C (=370 in the aforementioned HD study)”. The constant C printed in Sec. 4.1 is bumped by +10; Sec. 3 (Eq. 6) and the Conclusion still print C=360.

S2 (clear).

Headline-number digit swap. Paper: 2604.20142. Original (abstract): “…remains centered around α≈−1.5”. Injected (abstract): “…remains centered around α≈−5.1”. A digit transposition −1.5→−5.1 in the abstract; the body, table, and figure continue to report values centred near −1.5.

Percent-of arithmetic inconsistency. Paper: 2604.20914. Original: “∼200 times speedup, reducing the computation from about 13 hours to 4 minutes”. Injected: “…from about 13 hours to 6 minutes”. The same sentence advertises a ∼200× speedup and a 13-hour baseline; 13​h/200=3.9​min, so the printed runtime no longer matches the printed ratio.

Integer-equal sample-size disagreement. Paper: 2604.20411. Original (body): “∼290k galaxies from the latest COSMOS2020 catalog”. Injected (body): “∼29k galaxies from the latest COSMOS2020 catalog”. The abstract still says “mass-complete sample of approximately 290,000 galaxies”—a factor-10 discrete disagreement between abstract and body.

S3 (blatant).

Percentages do not sum to 100. Paper: 2604.20344. Original: “A substantial fraction of the formal core–Sérsic parameter space is therefore physically inadmissible.”. Injected: “A substantial fraction (approximately 40%) of the formal core–Sérsic parameter space is therefore physically inadmissible, while the remaining 75% corresponds to physically acceptable models.”. The split “40% inadmissible + 75% admissible” totals 115%, structurally violating part-of-whole.

Order-of-magnitude mismatch. Paper: 2604.20086. Original (abstract): “in the Milky Way and in 14 dwarf spheroidal galaxies”. Injected (abstract): “in the Milky Way and in 140 dwarf spheroidal galaxies”. The body still states “we analyzed 14 different dSphs” and Table 1 lists 14 rows: a factor-10 mismatch between abstract and body.

Ratio arithmetic blatantly wrong. Paper: 2604.20971. Original: “symmetric mass ratio q/(1+q)2≈q because q≪1”. Injected: “symmetric mass ratio q/(1+q)2≈q/4 because q≪1”. The leading-order approximation in the small-q limit is replaced by a value four times smaller, numerically wrong by a factor of four for any small q.

C.2.3 Mathematics

S1 (subtle).

Subleading-term sign flip. Paper: 2604.20344. Original: Ib​exp⁡[−b​(R/Re)1/m+b​(Rb/Re)1/m]. Injected: Ib​exp⁡[−b​(R/Re)1/m−b​(Rb/Re)1/m]. The sign of the subleading +b​(Rb/Re)1/m term inside the core-Sérsic sharp-transition exponential is flipped from + to −, making the limiting form inconsistent with its general-form derivation elsewhere in the same paper.

Coefficient slight perturbation. Paper: 2604.20086. Original: Ω2​(ϕ)=1+2​ξ​M​κ2​ϕ. Injected: Ω2​(ϕ)=1+4​ξ​M​κ2​ϕ. The numerical prefactor on the linear term of the conformal factor doubles (2→4).

Index / bound shift. Paper: 2604.20142. Original: +1E∑n=1∞(1+4​E)−nn2. Injected: +1E∑n=0∞(1+4​E)−nn2. The lower bound of the Klein–Nishina kernel sum is shifted from n=1 to n=0, introducing a 1/0 singularity at n=0 absent from the original Jones (1968) expansion.

S2 (clear).

Term deletion. Paper: 2604.20080. Original: log⁡αCO=0.7−1.5​[(12+log⁡O/H)−8.7]. Injected: log⁡αCO=0.7−1.5​[(12+log⁡O/H)]. The metallicity zero-point term “−8.7” is deleted from the Amorin+2016 log⁡αCO calibration; the formula remains valid LaTeX but no longer reproduces the published values at the solar anchor.

Operator swap. Paper: 2604.20262. Original: S=1N​∑(i,j)∈𝒮Vi,jcal​e2​π​iλ​n→⋅(r→j−r→i). Injected: S=1N​∏(i,j)∈𝒮Vi,jcal​e2​π​iλ​n→⋅(r→j−r→i). The outer summation in the beamforming definition is swapped for a product, turning a coherent sum over baselines into a product of phased visibilities.

Prefactor / normalisation drop. Paper: 2604.20411. Original: log⁡SFR=S0−log⁡[(10M⋆/10M0)−γ1+(10M⋆/10M0)γ2]. Injected: log⁡SFR=S0−12​log⁡[(10M⋆/10M0)−γ1+(10M⋆/10M0)γ2]. A spurious 1/2 prefactor is inserted in front of the log on the RHS of the modified Lee+15 main-sequence parameterisation, breaking the stated reduction to the unmodified form when γ2=0.

S3 (blatant).

Dimensional absurdity. Paper: 2604.20411. Original: log⁡SFR=S0−log⁡[(10M⋆/10M0)−γ1+(10M⋆/10M0)γ2]. Injected: log⁡SFR=S0+10M⋆−log⁡[(10M⋆/10M0)−γ1+(10M⋆/10M0)γ2]. A bare additive term 10M⋆ (dimensions of M⊙ since M⋆ is logarithmic) is inserted into a relation whose other terms are dimensionless.

Wrong-but-plausible formula. Paper: 2604.20809. Original: γ2=11−v2. Injected: γ2=11−v2. The canonical Lorentz factor γ2 is replaced by the structurally similar but incorrect 1/1−v2.

Patently false limit claim. Paper: 2604.20203. Original: 1B⁡(a,b)​Pa−1​(1−P)b−1. Injected: 1B⁡(a,b)​Pa−1​(1−P)1−b. In the Beta-distribution PDF, the exponent b−1 is flipped to 1−b, reversing the boundary behaviour at P→1 from the correct limit of 0 (for b>1) to a divergence.

C.2.4 General issues

S1 (subtle).

Typo / repeated word / homophone swap. Paper: 2604.20975. Original: “thus have only a negligible effect on the detection efficiency”. Injected: “thus have only a negligible affect on the detection efficiency”. Classic homophone swap (“effect”→“affect”); spell-checkers do not catch it.

Rendering artefact. Paper: 2604.20086. Original: “In section \ref{sec:conclusions}, we provide our conclusions.”. Injected: “In section \ref{sec:conclusion}, we provide our conclusions.”. The conclusions section has label sec:conclusions; the injected reference points at sec:conclusion (no trailing ‘s’), which compiles fine but renders as “section ??” in the PDF.

Local notation drift. Paper: 2604.21099. Original (Sec. 4 displayed equation): fM​M​(E)=IM​M​(E)/Iv​(E). Injected: FM​M​(E)=IM​M​(E)/Iv​(E). The surrounding paragraphs continue to call the same magnetic-monopole flux fraction fM​M.

S2 (clear).

Broken cross-reference. Paper: 2604.21077. Original: “…the region labeled Low-Energy SNe in Fig. \ref{fig:boundsgag}”. Injected: “…in Fig. \ref{fig:CoolingBound}”. The reference target is shifted from the figure that actually contains the “Low-Energy SNe” region to a different (cooling-bound) figure; the cited figure no longer matches the surrounding claim.

Methods↔Results inconsistency. Paper: 2604.20809. Original: “where τ is the duration of the expanding phase.”. Injected: “where τ is the duration of the contracting phase.”. The same τ is identified as the duration of the expanding phase elsewhere in Sec. 2 (and Sec. 3), so the local rewrite contradicts the surrounding text.

Phantom-evidence insertion. Paper: 2604.20262. Original: “To prevent cross-coupling between feeds, we selected those with L>1.8​m.”. Injected: “…with L>1.8​m (shorter baselines showed visibly degraded phase stability in our preliminary checks).”. The parenthetical claim about “visibly degraded phase stability in our preliminary checks” has no corresponding entry in Results, the Appendix, or any figure.

S3 (blatant).

Abstract↔body contradiction. Paper: 2604.20914. Original (abstract): “With high fidelity, we achieve a ∼200-fold speedup for a simulated ringdown signal including six modes, providing a viable and scalable route for multi-mode ringdown analysis in the space context.”. Injected (abstract): “…however, the computational cost grows steeply with the number of modes, which is likely to restrict the approach to moderate mode counts in the space context.”. The body (Fig. 2 and surrounding text) explicitly states the opposite—the FIREFLY computational cost “increases only mildly with N, demonstrating its scalability”.

Sweeping unsupported generalisation. Paper: 2604.20971. Original (Conclusion): “Hence, this work strongly motivates MHD simulations of gas disks embedded near-coalescence EMRIs to extract maximum science in the LISA era.”. Injected (Conclusion): same sentence with an appended clause: “…, and further demonstrates that turbulence-driven stochastic torques will generically dominate orbital evolution across all classes of compact binary inspirals embedded in any astrophysical accretion disk, regardless of the mass ratio or turbulence source.”. The added clause extrapolates from a single LISA EMRI / accretion-disk scenario to “all classes of compact binary inspirals …regardless of the mass ratio or turbulence source”—unsupported by the paper’s actual scope.

Methodology vs experiment contradiction. Paper: 2604.20262. Original: “The length of these baselines are all 3.2m.”. Injected: “The length of these baselines are all 3.2m. The visibilities from all 12 baselines are then jointly combined into a single stacked likelihood, which suppresses per-baseline systematics and ensures that the recovered N-S beam parameters are consistent across the full set of selected baselines.”. The Methods section is augmented with a clause asserting a “stacked likelihood” over all 12 baselines; the actual Results pipeline operates per-baseline rather than on a stacked likelihood.

Appendix D Panel-internal agreement

Two LLM-judge panels feature in the adjudication pipeline (§2.4): the detection-matching panel (Claude Sonnet 4.6, GPT-5.4, Gemini 3 Flash), which decides, for each injection, whether a given review detected it, and the general-issues false-positive panel (Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro), which classifies each general-issue flag raised on a clean copy as confirmed issue, false positive, or ambiguous. The rates in the main text rest on these panels, so we report here how well the judges within each panel agree with one another, on both corpora. For the matching panel we use the first-round votes only, so that the PDF-attached escalation pass is not mixed in.

Figure 4 shows the pairwise Pearson r between judges on their binary votes. The matching panel agrees closely, r=0.90–0.91 on astrophysics (n=2,104 pair-verdicts) and 0.90–0.93 on ML (n=1,980): the task is a lookup against an enumerated injection record, and high agreement is what one expects when an objective key exists. The false-positive panel agrees far less, r=0.32–0.58 on astrophysics (n=650–660) and 0.37–0.53 on ML (n=716): whether a flagged point is a real but minor issue or a spurious one is ambiguous when no key is available, which is why this task is given to a three-judge panel that sees the full PDF. The two corpora differ little in this respect: a paper-level bootstrap puts every ML–astrophysics difference within ±0.12, and none of the 95% intervals excludes zero. We also computed Cohen’s κ for every pair. It never differs from r by more than 0.01 on the matching panel or 0.03 on the false-positive panel, so the choice of agreement measure does not affect these conclusions.

Appendix E Adjudication-panel family-pairing test

Several reviewer systems share an LLM provider family with one of the three judges of the detection-matching panel. Following [15], this raises a self-preference concern: a judge might vote “detected” more readily on output from its own family than a judge from another family would on the same injection, which would inflate the detection rates of those reviewer systems. We test this for both corpora on the first-round panel votes themselves, comparing the judges within each injection, rather than by re-adjudicating with a substitute panel.

To define the test, let A, O, G denote the Anthropic (Sonnet 4.6), OpenAI (GPT-5.4), and Google (Gemini 3 Flash) judges. For each of the eight reviewer systems built on a single LLM family (the six single-LLM configurations and the two Claude Code configurations), let rf(s)∈[0,1] denote the first-round detection rate of judge f on system s’s injections. With self⁡(s) the family of system s and disj⁡(s)={A,O,G}∖{self⁡(s)},

A positive Δs means that the same-family judge detects more often than the other two. A judge that is more generous overall, voting “detected” more often on every system’s output, would produce a positive Δs for all systems of its own family without any self-preference. To separate the two, we also report an adjusted version Δ~s, in which each judge’s rate on a system is measured relative to that judge’s rate pooled over all reviewer systems. These pooled rates are given in Table 6, which shows that the three judges are close to one another, within about 3​pp on both corpora, with OpenAI slightly the most generous. Skepthical and the external reviewers are excluded from Δs, since Skepthical uses agents from all three judge families and the external reviewers either cannot be attributed to one family (Stanford) or share a family with no judge (open-weights). We note that all CIs in this appendix are 95% paper-level bootstrap intervals.

Table 7 reports Δs and Δ~s for each of the eight systems on each corpus. With under 200 injections per system, the interval of a single system is wide, and most of them include zero, so no single row can establish or rule out self-preference on its own. The test therefore rests on the mean of Δs over the eight systems, whose bootstrap interval is given in the last row of each block.

This mean is −1.37​pp (95% CI [−2.41,−0.39]) on astrophysics and −0.17​pp ([−0.97,+0.66]) on ML; with the adjustment it is −1.31 ([−2.36,−0.34]) and −0.05 ([−0.73,+0.65]). Self-preference would make these means positive. They are not: on ML the mean is consistent with zero, on astrophysics it is slightly negative, and in no case does the interval reach beyond +0.7​pp. On both corpora, then, judges do not favour reviewers of their own family by any amount that could inflate a detection rate. The slightly negative astrophysics value should not be read as judges disfavouring their own family either: it is about 1​pp, only two of the eight per-system intervals exclude zero, and effects unrelated to family (e.g., differences in how thoroughly each judge reads, judge style priors, the randomness of a single panel run) could produce a difference of that size.

The split panels allow for a second test. On the 2-1 splits, the injections on which one judge disagreed with the other two, we ask how often the lone dissenter is the judge of the reviewer’s own family. If every judge dissents equally often, that share is one third. Self-preference would push it below one third on the 2-1 detected panels, where two judges found the error and the same-family judge would be expected to be among them, and above one third on the 1-2 not-detected panels, where only one judge found it and self-preference would make that one the same-family judge. Table 8 shows that the same-family judge is the lone dissenter in 25.2% of the 107 astrophysics splits and 23.2% of the 108 ML splits, and that on the not-detected panels, where self-preference would raise the share above one third, it is 16.9% on astrophysics and 24.3% on ML. Neither corpus shows the pattern that self-preference would produce.

Skepthical cannot be tested in this way, because its reviews are written by agents from all three families, so every judge is a same-family judge and there is none left to compare against. The question for Skepthical is nonetheless answered indirectly. If a judge favoured reviews written by models of its own family, this would appear as a positive Δs for the systems built on that family: the four Anthropic-based systems for the Anthropic judge, and two systems each for the OpenAI and Google judges. It appears for none of the three judges, on either corpus, so there is no sign of the mechanism that could favour Skepthical.

Two related concerns remain outside this test. First, the test can only detect a preference specific to one judge. A preference that is shared by all three, for instance for a certain style of review prose, would affect every reviewer system alike and cannot be detected by this test. Second, the human ground truth against which the panel can be checked is limited to the one-paper check of Appendix F.

Finally, as an additional test, we ask how much the results depend on the size of the panel. In particular, we recompute every system’s detection rate with one judge family removed from the first-round panel, taking the majority of the remaining two votes and resolving 1-1 ties either strictly (not detected) or leniently (detected). On astrophysics, the same three systems stay on top whichever judge is removed, and with strict ties their order is also unchanged. In the worst case for Skepthical (Google removed, strict ties) its rate falls from 70.6 to 63.3%, a change of 7.3​pp (CI [3.9,11.5]), and its lead over structured GPT-5.5 shrinks from 9.3 to 2.6​pp. On ML, structured GPT-5.5 stays in the first position whichever judge is removed, Skepthical and structured Claude Code, 3.3​pp apart at baseline, swap second and third place in two of the six variants, and Skepthical’s largest change is 5.6​pp (CI [2.2,10.0]). With lenient ties, Skepthical’s rate stays within 1.7​pp of its baseline in every variant on both corpora. Per-system rates under each variant are released with the benchmark.

The script that produces every result in this appendix is released with the benchmark; it makes no LLM calls and runs in about a minute.

Appendix F Human adjudication

The detection rates in this paper are computed from the verdicts of an LLM panel. As a validation of these verdicts, one of the authors annotated every detection decision for one astrophysics paper, without knowledge of the panel’s verdicts or of which system had written each review.

The paper, 2604.20975, was drawn at random from the 15-paper corpus. Its three injected copies carry four injections each, and each copy was reviewed by the 12 systems, giving 36 reviews and 144 adjudications, each on whether a given review detected a given injection. Twelve of the 144 decisions concern three reviews for which no panel verdict was recorded, leaving 132 comparisons.

On these 132, the annotator and the panel agree on 127 (96.2%; Cohen’s κ=0.914). The five disagreements are all in the same direction: the panel credited a detection that the annotator rejected. The panel’s detection rate on this paper is therefore slightly higher than the annotator’s, 34.8 against 31.1%.

Three limitations apply. First, the check covers a single astrophysics paper. Annotating every review of a paper against every injection is labour-intensive, and extending it to further papers or to the ML corpus was beyond the resources of this work, so the agreement reported here is evidence about the panel on one paper rather than a corpus-wide validation. Second, with a single annotator, inter-annotator agreement cannot be measured, so a disagreement with the panel cannot be attributed to the panel rather than to the annotator. Third, the check covers detection only: the false-positive panel, which Appendix D shows to be the less reliable of the two, was not checked against a human.

Appendix G Significance tests: configuration comparisons and the severity trend

In this appendix we present the results of the tests behind two kinds of statements in §3.1: that a given reviewer configuration detects more than another one, and that detection rises with injected severity. Both use only the per-injection verdicts and severity labels already computed, resampling the 15 papers of a corpus (nboot=10,000, against 2000 for the intervals of §3.1) or permuting severity labels within papers; no new LLM calls are made.

Comparisons between configurations.

For each comparison made in §3.1, we resample the 15 papers with replacement, recompute the two rates on the same resampled papers, and take their difference. Table 9 gives the difference on the full corpus and the fraction of resamples in which its sign reverses, for the total rate and for each error type. We recall that the main text calls a difference significant when this fraction is below 0.05. Four readings of the table are used there. First, the structured prompt raises the total rate significantly for Claude Opus 4.7 and Gemini 3.1 Pro on astrophysics, for all three providers on ML, and within Claude Code on both corpora; GPT-5.5 on astrophysics is the one case where it does not. Second, giving the model tools helps only with the structured prompt and only on astrophysics, where the gain is significant on numerics and mathematics; on ML no change is significant, and with the free-form prompt Claude Code detects significantly less than the single call on both corpora. Third, Skepthical is significantly ahead of structured Claude Code on citations and general issues on astrophysics and significantly behind it on numerics, with no significant difference on ML. Fourth, against structured GPT-5.5, Skepthical is significantly ahead on astrophysics on the total rate and on citations, and not significantly different on ML in any column.

Severity trend.

The question is whether detection rises from S1 to S3. For each error type, we pool the detection counts of the reviewers with non-negligible detection (all but OpenReviewer-8B and PaperAudit-Bench Deep), so that each paper contributes one rate per severity level rather than a binary outcome per reviewer, and the number of tests falls from one per reviewer and error type (48 on astrophysics, 44 on ML) to four. We then test for an upward trend with Page’s L statistic (mid-ranks for ties), obtain the p-value by permuting the severity labels within papers, and apply the Benjamini–Hochberg correction at α=0.05 across the four tests (Table 10). On astrophysics, the upward trend is significant for citations only; for numerics it falls just short (raw p=0.026 against a corrected threshold of 0.025), and for mathematics and general issues the rate at S2 is lower than at S1, so there is no monotone trend to detect. On ML, the trend is significant for citations, numerics, and mathematics, and general issues again show a lower rate at S2 than at S1 (71, 45, and 81% at S1, S2, S3). The lower S2 rate for general issues has a simple cause: the S1 injections of that type (a typo or a repeated word) are easier to spot than the S2 ones (a Methods–Results inconsistency or phantom evidence). Testing each reviewer and error type separately on the binary outcomes (same correction) rejects one combination on astrophysics, free-form GPT-5.5 on numerics (42→100%), and two on ML, the same combination (47→100%) and structured Claude Opus 4.7 on citations (27→80%); with 9–15 papers per combination and a correction over all of them these tests are too conservative to resolve trends, and the pooled tests are the ones to read.

The scripts to reproduce the results in this appendix are released with the benchmark and run in under a minute on a single CPU.

Appendix H Structured-concise prompt baseline

The structured single-LLM configuration of §2.1 uses Skepthical’s full operating instructions as its prompt. Its gain over the free-form prompt can therefore come from two sources that the main comparison cannot separate: simply telling the model which four kinds of error to look for, and more detailed instructions on how to look for them. To separate the two, we wrote a 471-word ‘concise structured’ prompt that does only the first. This prompt asks the reviewer to examine citation support, numerical consistency, mathematical correctness, and logical or cross-section consistency, with a brief example of each. We ran it with the three single-LLM providers and inside Claude Code on all 60 copies of each corpus.

Table 11 shows the results next to those of the free-form and structured prompts. The concise prompt captures most of the detection gain of the structured one. Over the free-form prompt it raises total detection for every provider, by 8–16​pp on astrophysics and 12–38​pp on ML (all significant except Gemini on astrophysics, which falls just short), and inside Claude Code it accounts for 24.3 of the 32.2​pp between the free-form and structured prompts on astrophysics and 32.3 of 38.9​pp on ML. For the single LLMs, the structured prompt adds little over the concise one, and not uniformly: +0.8, −7.6, and +16.4​pp for Claude, GPT-5.5, and Gemini on astrophysics, +6.3, +5.0, and +2.2​pp on ML. Averaged over the three providers, the only significant per-type gain of the structured prompt on astrophysics is on general issues (+14.0​pp), whereas on ML it gains on citations (+16.0) and general issues (+12.8) and loses on mathematics (−8.8). Inside Claude Code, by contrast, the step from the concise to the structured prompt is significant on both corpora (+7.9 and +6.6​pp) and concentrated in citations and general issues.

The strongest concise control is GPT-5.5, and Skepthical’s total detection is not significantly different from it: 70.6 against 68.9% on astrophysics (+1.7​pp) and 76.7 against 76.1% on ML (+0.6​pp). The margin reported in §3.1 against the structured GPT-5.5 therefore depends on the comparison prompt. Per error type, Skepthical remains significantly ahead of concise GPT-5.5 on citations and general issues on astrophysics and significantly behind it on numerics and mathematics. The two corpora also differ in how the prompts affect the spread between providers: on ML the concise prompt already reduces the spread in total detection, from 52.8 to 27.2​pp (30.0 with the structured prompt), whereas on astrophysics only the structured prompt does (48.6, 48.0, and 24.0​pp for the three prompts).

On astrophysics, the concise prompt’s gain in detection comes with many more false positives. Its clean-copy false-positive rate is 17–27% for the single LLMs, against 1–9% with the free-form prompt and 3–6% with the structured one, and 16.7% inside Claude Code against 7.4 and 3.5%. The excess consists mostly of mathematics flags. On ML, on the other hand, the two structured prompts have similar false-positive rates (11–21% concise structured, 11–23% structured).

Appendix I Module attribution

A natural question about a multi-module reviewer such as Skepthical is which of its modules its detections come from. The rates of Table 2 credit Skepthical as a whole, so we attributed its astrophysics detections to modules by examining the reviews themselves. Each Skepthical review contains a general review and separate citation, mathematics, and numerics audit sections written by the specialist modules. For every injection of those three types, we recorded whether the corresponding specialist section identifies the injected error and whether the review identifies it anywhere, which is what the panel judges and Table 2 reports. This locates where the supporting evidence appears, although it does not say what would happen if a module were removed. The attribution was carried out by Claude in a single pass.

The attribution results are as follows. The citation audit identifies 11 of the 21 citation errors the review finds. The main failure mode is that the citing claim was never selected for checking. The numerics audit identifies 27 of the 36 numerics errors the review finds (75%); its misses are statements never selected for a check, so claim selection is again its main failure. The mathematics audit identifies 7 of the 31 mathematics errors the review finds. Most mathematics detections thus come from the general-issues ensemble reading the whole paper. These results also argue against the explanation that Skepthical performs well merely because the benchmark’s error types mirror its module names: one specialist section supplies most of its corresponding detections, one about half, and one a minority.

Appendix J Citation errors across the two corpora

On astrophysics, Skepthical detects 46.7% of the injected citation errors and no other system more than 26.7%. On ML, by contrast, two of the three structured single-LLM reviewers reach 57.8% and Skepthical 53.3%. In this appendix, we examine the origin of this discrepancy. We recall that the single LLMs receive one call with the paper attached and no tools, so they entirely rely on their pre-trained knowledge to detect the citation injections. Skepthical, on the other hand, retrieves and reads the cited papers.

This difference may arise because the ML corpus contains more citation errors that can be caught from familiarity with the field alone, because the models are more familiar with the ML literature, or through a combination of the two. To separate these contributions, we classified each of the 45 injected citation errors of each corpus into three groups. The first group contains errors that a reader familiar with the field could reject without opening the cited paper, for instance a well-known method attributed to the wrong paper. The second contains errors for which the cited paper is needed, for instance a dropped condition or a changed number. The third contains errors that could fall either way. The classification was made in a single pass by Claude. The two corpora differ markedly in this respect: errors of the first group are more than three times as common on ML as on astrophysics (14 against 4 of the 45).

This difference in composition, however, explains only part of the single LLMs’ gain. Taken together, the three structured single LLMs detect 16.7% of the citation errors on astrophysics and 47.4% on ML, and had the ML errors been distributed across the groups in the astrophysics proportions they would still have detected 39.6%. About a quarter of the gain therefore comes from the composition and the rest from higher detection within every group. The largest increase is in the first group, where knowing the field is enough. Skepthical’s rate, by contrast, is within a few points of its astrophysics value once the composition is accounted for. In short, the single LLMs gained on ML both because more of the errors can be caught from familiarity with the field and because they are more familiar with it, whereas Skepthical, which retrieves the cited paper in every case, changed little.

Appendix K Error coupling within a copy

A reviewer reading an injected copy encounters four injected errors at once. A natural question to ask, which we examine in this appendix, is whether the detection of one error could depend on the severity of the other errors present in the same copy. A blatant error, in particular, may draw attention away from a subtle one. The most direct test would place each injected error in a copy of its own, which requires 12 injected copies per paper rather than three. For five papers alone this would amount to 60 additional reviewed copies, at a cost comparable to that of the entire astrophysics benchmark, so this experiment was not carried out.

Instead, we used the randomisation already present in the benchmark. Because severity was assigned independently for each error type, an error of a given severity appears beside three neighbours whose severities vary at random from copy to copy, so the effect of the neighbours can be read from the existing verdicts. For every verdict on every injected error of both corpora, we recorded whether the error was detected and the highest severity among the other three errors in its copy, taking together the verdicts of the systems of Table 2 and of the concise structured configurations of Appendix H.

We find no measurable dependence on the neighbours. Over both corpora and all severities combined, the detection rate is 47, 50, and 45% when the most severe neighbour is subtle, clear, and blatant. The difference between a blatant and a clear neighbour, the only comparison with enough errors behind it, is −4.9​pp with a 95% paper-level bootstrap interval of [−11.8,+1.2], and the interval includes zero within each severity of the target error and within each corpus as well. A logistic model that controls for the target error’s own severity, its type, the reviewer system, and the paper likewise finds no significant effect in either corpus.

We note that this test cannot determine whether the mere presence of other errors has an effect, because every copy carries four. Answering this question would require analysing the single-error copies described above, which is beyond the resources of this work.

Appendix L Inter-reviewer agreement

Per-system detection rates (Table 2) leave open the question of overlap: two systems with the same rate can detect the same errors or entirely different ones, and two systems with rates near zero can agree by chance. This appendix reports the pairwise agreement between reviewers on the astrophysics corpus, which shows which features of architecture and prompt lead different systems to detect the same errors. Figure 5 shows the pairwise Pearson r between the binary per-injection detection vectors of six reviewer systems, overall and per error type: Skepthical, single-LLM (free-form), single-LLM (structured), Claude Code (free-form), Claude Code (structured), and the Stanford Agentic Reviewer. For the two single-LLM entries, an error counts as detected when any of the three providers detects it. The values lie between 0.2 and 0.54: the reviewers are heterogeneous, with similar overall detection rates but visibly different sets of detected errors.

The highest pair is single-LLM (free-form) and Stanford (r=0.54), consistent with the hosted Stanford reviewer behaving like a free-form referee. Single-LLM (structured) and Claude Code (structured) are at r=0.48: a shared prompt produces moderately similar detections even between a single call and an agentic loop with tools. Skepthical and Claude Code (structured) are at r=0.27 overall and at r≈0.04 on citations and numerics, the error types on which each is the best of all systems: two top systems whose detections are essentially independent there.

This independence is above chance. Pearson r on binary vectors measures agreement beyond what the two rates alone imply: two systems that detect 80 and 91% of the numerics errors already coincide on about 75% of them for that reason, so r≈0 means that their detections coincide no more often than that, not that the systems disagree. As a check we also computed Cohen’s κ. On the overall matrix it lies at or slightly below r for every pair, by at most 0.10, as expected for binary votes with unequal rates, and the conclusions drawn above with r hold with κ as well. Skepthical and structured Claude Code still agree only weakly overall (κ=0.27) and hardly at all on citations and numerics (κ≈0.04). The single-LLM free-form and Stanford reviewers, the most similar pair under r, remain among the most similar under κ (0.48, against a maximum of 0.49).

Beyond per-injection agreement (Figure 5), we also test convergence at the paper level. For each (paper, reviewer) we compute the reviewer’s detection rate on that paper (out of 9 or 12 injections) and form the 15×12 rate matrix. The mean pairwise Pearson r between reviewer columns is +0.39 (n=45 pairs, restricted to the ten systems with non-negligible detection rates; the two open-weights baselines are excluded). Reviewers thus agree more on which papers are difficult in aggregate than on which specific injections each catches. The within-paper picture is the complement: a Pearson chi-square dispersion test on per-paper detection counts gives a mean ratio of 1.04 across the same ten systems, indicating that within a paper detections behave approximately as independent Bernoulli draws across injections, so finding one injection does not strongly predict finding another one in the same paper.

Appendix M Compute and cost

M.1 Compute resources

The benchmark uses three classes of compute. (i) Hosted-API LLM calls to OpenAI, Anthropic, and Google serve every Skepthical agent (Table 3), the six single-LLM reviewers, the two Claude Code configurations, and the detection-matching and false-positive adjudication panels; the local CPU footprint of these calls is negligible (orchestration drivers pinned to 10 cores via taskset) since the work is dominated by external API latency. (ii) Skepthical itself runs as a Google Cloud Run worker (1 vCPU and 2 GiB of memory per task), with a per-paper wall time of approximately 24 minutes (Appendix A). (iii) The two self-hosted open-weights baselines, OpenReviewer-8B (bfloat16, ∼16 GB) and the PaperAudit-Bench Qwen3-14B detector (bfloat16, ∼28 GB weights plus key-value cache), run sequentially on a single NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU (96 GB VRAM). The host workstation is an AMD Ryzen Threadripper PRO 9995WX (96 cores, 192 threads, 502 GiB RAM) with two such GPUs attached; only one of the two was used for the open-weights baselines.

M.2 Cost estimates

The corpus run uses three billing modes: hosted-API metered calls (for the six single-LLM reviewers, every Skepthical agent, and the adjudication panels), Claude Code Max flat-rate subscription (for the two Claude Code configurations), and self-hosted or free-hosted operation with no API charge (the Stanford Agentic Reviewer, OpenReviewer-8B, and the PaperAudit-Bench detector).

Table 12 shows the API spend incurred by the reviewer systems on the astrophysics corpus. Note that the Claude Code costs are estimates derived from the associated token usage (as they are run on a flat-rate subscription). Two observations follow. First, the per-paper cost spread across hosted-API systems is roughly sevenfold ($0.34 to $2.41), with Skepthical at the top end ($2.41) reflecting its multi-agent design. Second, within Claude Code (model, sandbox, and tools held fixed), swapping the bare prompt for the structured operating instructions raises per-paper cost by ∼5× and per-paper wall-clock by ∼6×.

These per-reviewer figures understate what the benchmark cost to build and run, since they exclude the injection of the errors and the adjudication of every review. Table 13 gives the complete logged API spend on each corpus, split into error injection, reviewer execution, detection adjudication, and false-positive adjudication, for the systems of Table 2 together with the concise structured configurations of Appendix H.

M.3 Assets and licences

The mechanical-verification stack uses Wolfram Engine (proprietary, with a free developer licence from Wolfram Research), SymPy (BSD-3-Clause), and pint (BSD). The two open-weights baselines are used under their published licences: OpenReviewer-8B (maxidl/Llama-OpenReviewer-8B) under the Llama 3.1 Community License, as a Meta-Llama-3.1-8B-Instruct fine-tune; and the PaperAudit-Bench Qwen3-14B detector (mayiwen/PaperAudit_Qwen3_14B_sft_rl) under Apache-2.0. The hosted LLMs from OpenAI, Anthropic, and Google (identifiers in Table 3 and §2.1) are accessed under each provider’s terms of service. The two 15-paper corpora consist of arXiv preprints used for research purposes only, with each paper credited individually in Appendix B; the papers themselves are not redistributed, and the release identifies them by their arXiv identifiers.

注意转载授权 · 阅读原文