Do AI Scientists Know What's New?

   Xiayu Zhao    Encheng Xie    ChengXiang Zhai Affiliation: University of Illinois, Urbana-Champaign Affiliation: {miri3, xiayuz3, encheng2, czhai}@illinois.edu

Abstract

A major challenge facing AI scientific discovery pipelines is how to determine when a generated idea is meaningfully novel. Ideally, novelty checks in these pipelines must be able to reject non-novel ideas while not rejecting too many novel ideas, and they should also be able to distinguish which parts of a proposal are non-novel, as opposed to opaquely discarding entire proposals. We design an analytical framework for evaluation of novelty checks with temporal ground truth at both the proposal- and claim-level, using four datasets of 1,060 proposals built from published papers, reviewer disputes, and generated ideas matching ones the field went on to publish. Auditing existing novelty checks with this framework, we find that checks with high rejection rates on non-novel work also tend to have high false alarms on ideas that are later realized by human researchers, and observe that checks built to evaluate proposals as a whole can point to claim-level prior art only by rejecting the whole proposal. We therefore introduce ClaimNov, a fine-grained, interpretable claim-level novelty module. ClaimNov catches 97% of disguised papers at a 1.5% false-alarm rate, and its claim-level novelty profile cuts the LLM judge’s false alarms from 11.8% to 1.5%. Because its verdict rests on claim-level evidence, ClaimNov also keeps proposals that build on established contributions, rejecting only 3% of ICLR-accepted proposals with a reviewer-disputed contribution and keeping 91 of 100 novel proposals with one spliced prior contribution, against 76 for the strongest existing check. Code and data are available at https://github.com/emmatliu/claimnov.

1 Introduction

The development of AI scientific discovery systems is an active, fast-moving research field (Lu et al., 2026a; Yamada et al., 2025; Gottweis et al., 2026; Schmidgall et al., 2025), and such systems have begun to produce results that look like genuine, highly exciting discoveries (Wang et al., 2025; Sothanaphan, 2026; Yoon et al., 2026). This is strong evidence that LLMs can be creative, motivating their use in hypothesis generation (Wang et al., 2024; Si et al., 2025; Liu et al., 2026a; O’Neill et al., 2025) and in end-to-end discovery pipelines that generate, filter, and even execute ideas autonomously or semi-autonomously (Li et al., 2025; Baek et al., 2025; Weng et al., 2025b; Ghareeb et al., 2025; Tang et al., 2025).

One important facet of creativity, however, is novelty (Ismayilzada et al., 2025; Lu et al., 2026b). Most AI scientist pipelines employ some kind of novelty check (e.g., searching against Semantic Scholar and using an LLM judge to determine if any results are too close) which in turn decides which generated ideas will go on to receive compute and human attention. Using a novelty check only at the level of a whole proposal has three limitations. (1) Judging the novelty of a whole proposal is challenging even for human experts; the annotation itself is expensive and annotators often disagree about what counts as novelty (Si et al., 2025). (2) Deployed checks (Lu et al., 2026a; Li et al., 2025) that reject or pass a research proposal as a whole without considering its parts subtly deviate from the tradition of human science, which is incremental (Uzzi et al., 2013). (3) Whole-proposal checks also make it harder, if not impossible, to understand whether they are actually identifying the right prior work when they reject proposals. We therefore argue that the right unit to consider is the claim, not the proposal, since most papers, including ones that humans consider highly novel, are not novel in literally every contribution they make (Wang et al., 2017). With this motivation, we address the limitations of previous work and make the following contributions to the understanding and improvement of novelty checks in AI scientist pipelines:

• •

  We introduce ClaimNov, a fine-grained, interpretable novelty module that treats claims as the basic unit of novelty.

• •

  We propose an evaluation framework for novelty checks in end-to-end AI scientist pipelines which draws upon temporal ground truth from the published record and from reviewer comments at a whole-proposal and claim-level, with four datasets totaling 1,060 proposals (§4.4).

• •

  We audit seven existing novelty checks alongside ClaimNov. The checks that reject the most non-novel work also reject the most work later realized by human researchers, and checks that judge proposals as a whole can only identify claim-level established work by rejecting an entire proposal.

• •

  ClaimNov catches 97% of disguised papers while rejecting only 1.5% of later realized ideas. It correctly preserves partially novel work, keeping 91 of 100 proposals with a spliced prior contribution against 76 for the strongest existing check, and given the same attribution ranking, it keeps more correctly attributed proposals.

2 Related Work

Traditional measures of scientific novelty. Zhao and Zhang (2025) categorize scientific novelty metrics by whether they draw on citation data, textual data, or both. Bibliometric measures use unusual or first-time combinations of cited work (Uzzi et al., 2013; Wang et al., 2017; Foster et al., 2013) or related ideas such as disruptiveness (Wu et al., 2019); text-based measures consider semantic distance (Shibayama et al., 2021; Jeon et al., 2023) or new concepts and new question–method combinations (Mishra and Torvik, 2016; Luo et al., 2022); some measures also combine the two (Amplayo et al., 2018). Traditional measures are often validated against external signals such as expert judgments and later impact (Bornmann et al., 2019; Fontana et al., 2020; Arts et al., 2021). Our evaluation focuses on text-based measures, as does our introduced module ClaimNov, which does not consider bibliometric data.

Literature-grounded novelty assessment. Early on, IR and NLP researchers studied novelty detection at the sentence and document level, as information not covered by previously seen documents (Soboroff and Harman, 2005; Ghosal et al., 2018; Ghosal et al., 2022), and patent prior-art searches focused on naming anticipating documents (Piroi et al., 2011). LLMs have since enabled approaches that reason about the claims of existing work: retrieving prior work to revise an idea (Wang et al., 2024), judging a paper against retrieved neighbors (Lin et al., 2025), reranking retrieved papers by idea-specific facets (Shahid et al., 2025), scoring ideas against a literature set (Moussa et al., 2026; Qiao et al., 2026), and comparing atomic content units or contribution claims against full-text evidence (Ai et al., 2024; Zhang et al., 2026; Afzal et al., 2026). The typed contribution claims used in ClaimNov draw on scientific information extraction (Luan et al., 2018; D’Souza et al., 2021), facet-level views of ideas (Radensky et al., 2026) and claim verification (Wadden et al., 2020). Benchmarks for these LLM-based systems score them against expert annotations or review text (Schopf and Färber, 2026; Wu et al., 2026), against future papers (Amplayo et al., 2019), or under controlled, counterfactual changes (Liu and Zhai, 2026). Critiques of LLM novelty judges find that they prefer generated research questions that experts rate lower (Sinhahajari et al., 2026) and that their ratings disagree with hindsight against later papers (Jiang, 2026). We specifically evaluate novelty checks as deployed inside AI-scientist pipelines, and also focus on counterfactual changes rather than alignment with explicit human novelty labels, akin to how paraphrase attacks test text detectors (Krishna et al., 2023).

AI-scientist systems. Pipelines with a research-ideation stage almost always include a novelty check for generated ideas, although these checks can vary widely in mechanism (Appendix A). Some are hard filters grounded in retrieval, including AI Scientist’s agentic search loop (Lu et al., 2026a), CoI-Agent’s reject-and-regenerate loop (Li et al., 2025), the retrieval filter of Si et al. (2025), and AutoResearchClaw’s lexical check (Liu et al., 2026b). Others are advisory or score-based, with or without explicit retrieval (Yamada et al., 2025; Baek et al., 2025; Gottweis et al., 2026; Schmidgall et al., 2025; Weng et al., 2025b; Weng et al., 2025a); still others directly rank generated ideas against one another (Ghareeb et al., 2025; Su et al., 2025; Tang et al., 2025; Keya et al., 2025). Work on automatic peer review usually judges full papers, most recently with LLM judges (Kang et al., 2018; Wang et al., 2020; Taechoyotin and Acuna, 2025; Chen et al., 2025); we instead focus specifically on novelty.

3 Novelty Module

We introduce ClaimNov, an interpretable, claim-level novelty module for research proposals that can be dropped directly into any AI for scientific discovery pipeline with a novelty check. As shown in Figure 2, ClaimNov extracts typed claims from research proposals, retrieves relevant same-dimension claims from a pool of claims extracted from prior art, scores the entailment per extracted claim, aggregates into a novelty score, and passes the full claim profile to a verdict judge for a binary novel or non-novel decision. For validation experiments, see Appendix O, and for exact prompts, see Appendix C.

3.1 Claim Extraction

During the claim extraction stage, ClaimNov takes the title and description from the research proposal P and produces, with an LLM (gpt-4o-mini set to temperature 0), a list of contribution claims CP={c1,…,cm}.

Each claim ci is typed with a dimension dc and an attribute-value pair (ca,cv). Dimensions are drawn from {Problem, Methodology, Resource, Result}, while an attribute represents the axis along which two works can be compared. For instance, for a Result claim, the attribute may be classification accuracy on CIFAR-10, and the value may be an improvement from 30.1% to 76.9%. We aim to extract a small number of claims per proposal (2.4–2.7 on average, the prompt allowing 1–4), based on conventions in AI research. We process each paper in the reference pool Π through this stage once, using its abstract as the description text, to produce pool claims CΠ.

3.2 Claim Retrieval

During the claim retrieval stage, ClaimNov takes a claim c from the previous stage’s list of claims, embeds its text together with its paper’s title with text-embedding-3-large, and produces a set of relevant pool claims R⁡(c) for comparison to that claim.

To obtain such a set, we retrieve the k-nearest (by cosine similarity) pool claims of the same dimension, keeping at most one claim per pool paper. We determine k by pool size, taking k=10 as the minimum based on an independent retrieval validation (see Appendix O), and for larger pools we use the heuristic k=0.2​nd, where nd is the number of pool papers with a claim in that dimension, capped at 40. This gives k=10 for the disguise pools and k≈35 for the anchor pools (§4.4).

3.3 Claim Entailment

During the claim entailment stage, ClaimNov judges the entailment of every pair (c,c′) for c′∈R⁡(c) with an LLM (gpt-4o-mini at temperature 0), giving titles and abstracts of pool papers as context.

The judge rates the extent to which the pool claim entails or covers the proposal claim on an ordinal scale ℓ⁡(c,c′)∈{1,…,5}: 1 unrelated, 2 shared topic only, 3 partial coverage with important details differing, 4 strong coverage with minor details missing, and 5 full coverage of the claim’s substantive content. We map these levels onto the unit interval by taking e⁡(c,c′)=(ℓ⁡(c,c′)−1)/4∈[0,1].

3.4 Novelty Aggregation

During the novelty aggregation stage, we first take the sorted eis from the previous stage and combine them into one coverage score a⁡(c)∈[0,1] for each claim as a rank-discounted mean:

Here, α=0.5, which balances the importance of the strongest match against the weaker ones, and kc=min⁡(k,|R⁡(c)|), which prevents a claim with very few relevant pool claims from having an inflated score.

We then combine claim scores into a proposal-level score using dimension weights wd (Problem 0.4, Result 0.3, Methodology 0.2, Resource 0.1) and per-claim importance weights imp⁡(c), which are uniform for proposals and set to each claim’s share of support passages when scoring a full paper. The dimension weights encode a prior about contribution types in AI research; they can be tuned to adapt ClaimNov to other fields.

Amax​(P) is the coverage if every claim were fully covered. N⁡(P) summarizes the novelty profile in one number; the full profile is provided as context to the novelty verdict stage below.

3.5 Novelty Verdict

Finally, the novelty verdict stage of ClaimNov assembles previous evidence produced by the module into a profile Φ⁡(P) which is evaluated by a stronger judge (claude-sonnet-4-6). Φ⁡(P) lists the proposal’s extracted claims with matched prior claims, their sources (and their abstracts), and the entailment levels.

From the judge, we receive a verdict of pass or fail. If the proposal fails, ClaimNov also outputs the prior art paper p∗∈Π, its subset of claims S⊆CP, and a one- or two-sentence reason. For the exact prompt, please see Appendix C.

4 Experimental Setup

We explain the shared inputs and outputs for each check, which existing checks we evaluate, the judge models and their temporal cutoffs, the test data we collect, and the metrics we define.

4.1 Evaluation Contract

Because each of the AI scientist pipelines and standalone scientific novelty evaluations we survey (Appendix A) uses slightly different schema, literature search, and verdict formats, we standardize the evaluation contract to allow for better comparison.

We pass as input to each check a research proposal consisting of a title plus a 150–400 word description as well as a reference pool, which is a list of prior art papers given as {id,title,abstract}. Checks using search are sandboxed. For more details on how we adapt proposals to native input formats, see Table 8. For output, we consider three layers: L1, the verdict verdict∈{pass,fail} with reasons; L2, a novelty score N⁡(P)∈[0,1]; and L3, a novelty profile, every claim with its matched prior claims, their entailment levels, and the papers they come from. For ClaimNov, L3 determines both L2 and L1 (§3.4, §3.5). The existing checks we evaluate only emit L1.

4.2 Evaluated Systems

We consider four kinds of novelty check. (1) Novelty checks that are a clear-cut component in some end-to-end AI-scientist pipeline: AI Scientist v1 (check_idea_novelty), CoI-Agent (check_novel), Si et al. (retrieve_novelty_score), and ARC’s lexical filter. (2) Novelty checks we have “extracted” from pipelines that make some kind of novelty judgment but which are not clearly plug-and-play in the same way: ResearchAgent’s Originality rating (a proposal passes at a rating of 4 or 5 out of 5), ScholarEval’s pairwise contribution scoring (a proposal passes when its mean score against the top-8 pool papers is positive), and AI-Scientist-v2’s ideation reflection. The v2 ideation reflection has no final verdict, so we append a final verdict to the prompt. (3) A claim-level reference baseline after the style of OpenNovelty (Zhang et al., 2026), which judges 1–3 extracted contributions against their BM25 top-10 and rejects when every contribution is refuted. (4) ClaimNov as described in §3 and its ablations. Appendix A gives our selection criteria and the full survey of 17 pipelines. Searches embedded in novelty checks are replaced by dense retrieval over the pool; see Appendix B for details.

4.3 Judge Models and Temporal Validity

Because six of the seven novelty checks we evaluate use an LLM as a judge at some point (and ClaimNov also does so), we fix a cutoff T= March 2026 and exclusively evaluate models with earlier cutoffs. This mitigates the risk of models recognizing post-T papers from training data. We run each check with Sonnet 4.6 (training cutoff January 2026), which is the same model ClaimNov uses as the final verdict judge (for earlier, lightweight stages, ClaimNov uses gpt-4o-mini). As we explain in more detail in §4.4, we use Sonnet 4.6 to generate ideas, disguise existing papers, and rewrite disputed papers as research proposals, and Opus 4.6 (training cutoff August 2025) as the labeler.

4.4 Data Collection

We collect data and establish ground truth for two levels of non-novelty. In both cases, we aim to collect data that falls after our temporal cutoff of T= March 2026, to avoid LLM judge awareness.

Already-done ideas (n=400, whole-proposal level).

For this data, novelty checks should correctly reject a disguised proposal if the source paper is in the pool, but should not reject when the source is removed (making the proposal novel relative to the pool). We collect n=100 ICML 2026 spotlight papers, filtering for papers where the v1 arXiv appeared after T, and disguise each paper as a new research proposal at four increasingly challenging to spot levels of disguise. Rung 1 swaps the voice, rung 2 paraphrases, rung 3 reframes the paper with a different motivation, reordered contributions, and an unnamed method, and rung 4 restructures it adversarially from a different theoretical starting point with generalized numbers. Because rungs 3 and 4 may depart significantly from the original paper, we accept them only when a verifier (Sonnet 4.6) confirms that the core contributions are the same. To ensure the disguise is challenging, we require that at most 2% of the disguise’s 6-grams appear in the source, regenerating up to three times; for more details and prompts, see Appendix D. We then form a reference pool from the source plus up to 19 pre-T papers closest to the proposal by embedding, which we take from the source’s reference list and Semantic Scholar recommendations. We drop five sources whose pools contain only the source, leaving 380 disguises (all 400 in Appendix K).

Claim-disputed ideas (n=160, claim level).

For this data, novelty checks should correctly find a prior work B that establishes a specific claim. We use a regex prefilter and gpt-5-nano classification to mine 8.3k ICLR 2022–23 reviews in which a reviewer identified a paper A’s contribution as already established by some prior paper B. We then rewrite each A in proposal voice with Sonnet 4.6 and the rung-1 prompt provided in Appendix D and plant B among 19 pre-T papers to serve as distractors. We also record the final ICLR decision (62 accepted, 96 rejected, 2 unknown) as an additional axis of analysis of novelty check performance. Of n=200 pairs of papers (A,B), we keep n=160 with usable pools, and we annotate a sample for genuine disputes (see Appendix F). Unlike the other sets, these papers predate T and may appear in a judge’s training data. In this case the ground truth is reviewer attribution (and not some post-T event), so we consider this acceptable.

Later-realized ideas (n=400, whole-proposal level).

For this data, novelty checks should correctly pass the proposal, because the pool does not contain the idea and researchers went on to realize some of its contributions. We collect n=30 anchor papers, where each anchor paper was published in NeurIPS 2025 proceedings and has at least 50 citing papers that fall after T. We exclude benchmarks, datasets, and surveys (which may be cited primarily for use), then generate new proposals based on the remaining anchor abstracts. We construct a relevant works pool (median 174 papers, capped at 400 papers) for each anchor from its reference list plus any citing papers that come before T, as well as a follow-up pool F of the citing papers that come after T (median 124 papers). We use each anchor paper to generate n=100 ideas from Sonnet 4.6, producing n=3000 proposals total. For each of these proposals, Opus 4.6 writes a contribution list, then labels each contribution as established or not by the 8 papers in the pool nearest to the idea (by text-embedding-3-small over title and abstract) and, separately, by the 8 nearest papers in F. We identify later-realized ideas as proposals with at least one contribution established in F and none established in the pool. From the 407 proposals of this kind generated from 26 of the 30 anchors, we draw 400 at random to constitute this dataset. We provide labeling prompts in Appendix E.

Hybrid proposals (n=100, claim level).

For this data, novelty checks should correctly pass the proposal and correctly name the paper whose contribution it contains. We take n=100 later-realized (and thus assumed novel) ideas from the previous dataset and, for each, a pool paper B which was moderately similar (ranked 5–15 by cosine similarity), then ask Sonnet 4.6 to rewrite these ideas so that B’s core contribution is also included. Sonnet 4.6 verifies that the splice preserved the original contributions while adding the new one.

4.5 Metrics

We evaluate based on two levels of granularity – the whole-proposal level and the claim level – motivated by the idea that we want to reject proposals that are wholly non-novel, but not proposals that simply contain some existing claims.

For the whole-proposal level, we evaluate checks’ ability to catch non-novel ideas as well as their false alarm rate, how often they reject ideas that are later realized and validated by the human researcher community. We measure false alarms both on the later-realized ideas (FA) and, as a second set of assumed novel ideas, on the disguises with their source removed from the pool (Src. out). From these, we compute informedness J=catch−FA and the pool-flip 𝚫=catch−Src. out, or the drop in rejection rate when the disguised proposal’s source paper is removed from the pool.

For both whole-proposal and claim level, we evaluate checks’ ability to attribute claims to prior art. For whole-proposal, we observe whether checks name the source paper for disguises, and for claim level, we report the share of the n=160 claim-disputed proposals for which a check’s top-ranked paper is the prior art B. We also report rejection rates on this set by ICLR decision, with the motivation that one disputed claim does not a non-novel proposal make.

5 Results

We find that ClaimNov is both sensitive and specific, catching disguised copies of published work while keeping ideas that researchers went on to publish (§5.1). Because its verdict rests on claim-level evidence, we also find that ClaimNov preserves proposals in which a single contribution is established, so more correctly attributed proposals can reach the full AI-scientist pipeline (§5.2, §5.3). We report full results in Tables 1–3, with 95% intervals and paired exact McNemar tests in Appendix J.

5.1 Whole-Proposal Evaluation

On whole proposals (Table 1), all checks except ARC and ScholarEval catch at least 338 of 380 disguises, but most checks continue to reject after the source is removed. For instance, Si et al. catches all 380 but also rejects 359 when the source is no longer present in the pool. All checks except ClaimNov and CoI-Agent also reject a large share of ideas the field went on to publish. Si et al. rejects 96%, ResearchAgent 98.5%, AI Scientist v1 52%, and AI-Scientist-v2 39%. ARC and ScholarEval reject 0.0%, but do so by rejecting almost nothing.

ClaimNov and CoI-Agent are clear outliers in their ability to balance the two. Both catch almost every disguise (367 and 368 of 380, respectively) while rejecting few later-realized ideas (1.5% and 3.8%). ClaimNov has the highest J (+0.95) and Δ (+0.92), whereas the next best pipeline checks after CoI-Agent are at +0.56 and +0.76. The per-contribution OpenNovelty-style baseline does catch more disguises (379, p<.001) but false-alarms on 11.5% of later-realized ideas and rejects 38 disguises with the source removed (p=.001 against ClaimNov). While clustering tests by source and anchor shows ClaimNov and CoI-Agent are statistically level, ClaimNov’s verdict call costs a quarter of CoI-Agent’s check call with the same judge ($0.010 against $0.039), with entailment cost growing with pool size (Table 9, Appendix M).

The ClaimNov ablations (Appendix H) show the value of each stage of the module. Without the claim-level novelty profile, the verdict judge reads the nearest abstracts instead, and its false-alarm rate rises from 1.5% to 11.8% as a result. When we instead use a per-claim verdict rule, which fails a proposal if any central claim appears entailed, we catch all 380 disguised proposals, but the false-alarm rate rises to 56.8% and we wrongly reject 197 of the 380 proposals whose source has been removed. This reinforces the idea that (1) claims are the appropriate unit for understanding a research proposal and (2) a research proposal is not one large claim; its claims should be considered separately. On generated ideas that a whole-idea labeler matched to one pool paper, ClaimNov rejects 18%, against 55% for the judge without the claim-level novelty profile (Appendix L). These ideas are noisier than disguises, and we audit a sample of them by hand, together with later-realized ideas, in Appendix F.1. Results with other verdict judges are given in Appendix K.

5.2 Claim-Level Prior Art

We evaluate ClaimNov and the seven existing checks on claim-level data (Table 2) and find that ClaimNov correctly treats a disputed claim as a claim-level finding, and not a reason to reject an entire research proposal. It rejects only 3% of accepted ICLR proposals, the lowest rate of any check that catches disguises (10% for CoI-Agent and 47–81% for Si et al., AI Scientist v1, and ResearchAgent). All of the whole-proposal checks we evaluate name prior art only alongside a rejection, so Si et al. names B for 47% by rejecting about 80% of proposals, and CoI-Agent passes 89%, meaning it can name B for only the remaining 11%.

Considering this design choice in existing checks, we separate attribution from the verdict by pairing each verdict with the same attribution ranking. ClaimNov’s top-rated profile paper is B for 21% of proposals, while the pool paper nearest to the whole proposal by dense embedding is B for 37%. Intuitively, this is because the B named by a reviewer is often closest to the proposal as a whole, which embedding similarity captures well, whereas ClaimNov ranks papers by how fully they cover a single extracted claim. Extraction failures further exacerbate this issue (Appendix G). We therefore pass the dense ranking to every verdict, and ClaimNov then passes the proposal and names B for 36% of proposals, against 31% for CoI-Agent (p=.016). Given the same attribution, ClaimNov’s verdict keeps more proposals building on established work.

5.3 Hybrid Proposals

Evaluating ClaimNov alongside the existing checks on the hybrid proposals (Table 3) allows us to more clearly distinguish ClaimNov’s advantages. We fix the correct behavior by construction, as each proposal is a later-realized novel idea which should not be rejected, and we control proper attribution by splicing in a contribution from a known paper B. The other checks continue to behave roughly according to established traits. ARC and ScholarEval pass all 100 and name nothing, Si et al. names B for 63 but rejects all 100, and ResearchAgent rejects 98. CoI-Agent, the closest check on whole proposals, passes 76 and names B for only 11 of the 100, all among the 24 it rejects.

ClaimNov, on the other hand, passes 91 of 100 (p=.004 against CoI-Agent). Because a hybrid states B’s contribution explicitly, BM25 over the whole proposal ranks B first for 74. Given this shared ranking (similarly to the isolation we perform in §5.2), ClaimNov keeps 67 hybrids with B named, against 53 for CoI-Agent (p=.004). The OpenNovelty-style baseline keeps 64, but this is at the cost of 11.5% false alarms on later-realized ideas (Table 1). When ClaimNov successfully extracts the spliced claim as its own claim (as opposed to missing it or absorbing it into a different claim), the entailment judge rates B at level 4 or 5, and the verdict stage properly reads one established claim among several novel ones as a proposal that builds on prior work. We consider these ratings reliable, as 13 of the 15 papers rated at level 4 or above are B, and this entailment level is reached by only 9 of 400 later-realized ideas (Appendix J).

6 Conclusion

We establish an evaluation framework with temporal ground truth at both the proposal and the claim level, and use it to audit seven existing novelty checks. We find that the checks which reject the most non-novel work also tend to reject the most work the field went on to publish, and observe that checks built to judge proposals as a whole name claim-level prior art only when they reject. We therefore introduce ClaimNov, a drop-in claim-level novelty module, and show it rejects non-novel ideas, keeps novel and partially novel ones, and, given the same attribution ranking, keeps more correctly attributed proposals. Beyond filtering, a specific claim-level check like ClaimNov could serve as a reward when training AI-scientist agents, since it does not penalize the incremental follow-ups most science consists of, and its profile tells the agent which of its contributions are new. Do AI scientists know what’s new? Based on current novelty checks, they largely do not, rejecting much of what the field goes on to publish or practically nothing at all. However, ClaimNov enables scientific novelty to be judged claim by claim against specific prior papers, thus potentially making AI scientists much better judges of novelty.

Limitations

Our work employs several heuristics in how we consider sets of proposals or papers to be novel or non-novel, and this is not as clean as if we had a “true” dataset. We also focus exclusively on AI research. In practice, this is where most AI-scientist work also concentrates, so this aligns well, but if we want AI-scientist pipelines to be more widely applicable, then assumptions like the ones we make about dimension weights (problem contributions being more valuable than other contributions, for instance) might not generalize and would require further tuning for different fields, thus impeding adoption. We also consider only English-language research, which can further skew generalization.

Because our pools are fixed lists of prior work rather than open search, consisting of 20 papers for disguises and disputed pairs and up to 400 for generated ideas, we have limited ability to evaluate the retrieval of novelty checks. Additionally, we identify later-realized ideas with an LLM labeler, although we do cross-check this with a different model family labeler and with a small-scale human audit in Appendix F.1. Our disguised proposals and generated ideas are generated by the same model (Sonnet 4.6) that ultimately judges, which is a known source of possible bias for LLMs-as-judges (Zheng et al., 2023; Panickssery et al., 2024).

The temporal construction of our datasets also decays, since running the same analysis with more recent LLM judges (with training cutoffs post-T) will require all materials to be rebuilt at a later T. Although this is simple mechanically and our scripts support it, this still takes time and money. Finally, one important aspect of novelty modules in AI-scientist pipelines is how they affect in-the-loop generation: when the module is actively rejecting and causing regeneration of proposals, how does that affect the final committed set of ideas, and what are the compute and token costs involved? We leave this question to future work because (1) it is highly computationally expensive compared to the static question we examine and (2) establishing ground truth for ideas produced in response to a rejection is harder still and may require extensive human annotation, for which we have limited resources. We do run a small pilot (Appendix M) and report the rejection and cost dynamics of the loop.

Ethical Considerations

Our aim with this work is to encourage the development of scientific novelty modules in AI scientist pipelines which are better aligned with human understandings of novelty, which we believe is important to the science produced by these pipelines. We recognize that development of AI for scientific discovery capabilities has several concerning implications for current review processes and for the human experience of science in general. ClaimNov, like any automated judge, should be used carefully.

We do not release any intellectual property directly; we release no direct paper text, only pointers and code for reproduction of our results, and we do not release ICLR review text, but rather the code that re-mines the claim disputes from OpenReview. We follow the terms of use of both arXiv and OpenReview licenses. Additionally, while we use human annotation to validate some LLM labels, the annotation consists only of labels on published papers and generated ideas, and we collect no survey responses or personal information. It is not human-subjects research and falls outside IRB review.

Acknowledgments

We thank Lucy Zhou for her work on an earlier version of this project. Miri Liu was supported by the Amazon AI PhD Fellowship and by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Department of Energy Computational Science Graduate Fellowship under Award Number DE-SC0026617. This work is supported in part by the National Science Foundation under Grants 2229612 and 2433308. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the United States Government or any agency thereof.

References

• Afzal et al. (2026) Osama Mohammed Afzal, Preslav Nakov, Tom Hope, and Iryna Gurevych. 2026. Beyond "not novel enough": Enriching scholarly critique with llm-assisted feedback. Preprint, arXiv:2508.10795.
• Ai et al. (2024) Lin Ai, Ziwei Gong, Harshsaiprasad Deshpande, Alexander Johnson, Emmy Phung, Ahmad Emami, and Julia Hirschberg. 2024. Novascore: A new automated metric for evaluating document level novelty. Preprint, arXiv:2409.09249.
• Amplayo et al. (2018) Reinald Kim Amplayo, SuLyn Hong, and Min Song. 2018. Network-based approach to detect novelty of scholarly literature. Information Sciences, 422:542–557.
• Amplayo et al. (2019) Reinald Kim Amplayo, Seung-won Hwang, and Min Song. 2019. Evaluating research novelty detection: Counterfactual approaches. In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), pages 124–133, Hong Kong. Association for Computational Linguistics.
• Arts et al. (2021) Sam Arts, Jianan Hou, and Juan Carlos Gomez. 2021. Natural language processing to identify the creation and impact of new technologies in patent text: Code, data, and new measures. Research Policy, 50(2):104144.
• Baek et al. (2025) Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2025. ResearchAgent: Iterative research idea generation over scientific literature with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6709–6738.
• Bornmann et al. (2019) Lutz Bornmann, Alexander Tekles, Helena H. Zhang, and Fred Y. Ye. 2019. Do we measure novelty when we analyze unusual combinations of cited references? a validation study of bibliometric novelty indicators based on F1000Prime data. Journal of Informetrics, 13(4):100979.
• Chen et al. (2025) Shiping Chen, Duncan Brumby, and Anna Cox. 2025. Envisioning the future of peer review: Investigating llm-assisted reviewing using chatgpt as a case study. In Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work, CHIWORK ’25, New York, NY, USA. Association for Computing Machinery.
• D’Souza et al. (2021) Jennifer D’Souza, Sören Auer, and Ted Pedersen. 2021. Semeval-2021 task 11: Nlpcontributiongraph - structuring scholarly nlp contributions for a research knowledge graph. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), page 364–376. Association for Computational Linguistics.
• Fontana et al. (2020) Magda Fontana, Martina Iori, Fabio Montobbio, and Roberta Sinatra. 2020. New and atypical combinations: An assessment of novelty and interdisciplinarity. Research Policy, 49(7):104063.
• Foster et al. (2013) Jacob G. Foster, Andrey Rzhetsky, and James A. Evans. 2013. Tradition and innovation in scientists’ research strategies. Preprint, arXiv:1302.6906.
• Ghareeb et al. (2025) Ali Essam Ghareeb, Benjamin Chang, Ludovico Mitchener, Angela Yiu, Caralyn J. Szostkiewicz, Jon M. Laurent, Muhammed T. Razzak, Andrew D. White, Michaela M. Hinks, and Samuel G. Rodriques. 2025. Robin: A multi-agent system for automating scientific discovery. Preprint, arXiv:2505.13400.
• Ghosal et al. (2018) Tirthankar Ghosal, Vignesh Edithal, Asif Ekbal, Pushpak Bhattacharyya, George Tsatsaronis, and Srinivasa Satya Sameer Kumar Chivukula. 2018. Novelty goes deep. a deep neural solution to document level novelty detection. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2802–2813. Association for Computational Linguistics.
• Ghosal et al. (2022) Tirthankar Ghosal, Tanik Saikh, Tameesh Biswas, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Novelty detection: A perspective from natural language processing. Computational Linguistics, 48(1):77–117.
• Gottweis et al. (2026) Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Petar Sirkovic, Artiom Myaskovsky, Grzegorz Glowaty, Felix Weissenberger, Alessio Orlandi, Dan Popovici, Anil Palepu, Keran Rong, Ryutaro Tanno, Khaled Saab, Fan Zhang, Jacob Blum, Andrew Carroll, Kavita Kulkarni, Nenad Tomašev, and 32 others. 2026. Accelerating scientific discovery with co-scientist. Nature, 655(8122):487–496.
• Ismayilzada et al. (2025) Mete Ismayilzada, Antonio Laverghetta Jr., Simone A. Luchini, Reet Patel, Antoine Bosselut, Lonneke van der Plas, and Roger Beaty. 2025. Creative preference optimization. Preprint, arXiv:2505.14442.
• Jeon et al. (2023) Daeseong Jeon, Junyoup Lee, Joon Mo Ahn, and Changyong Lee. 2023. Measuring the novelty of scientific publications: A fastText and local outlier factor approach. Journal of Informetrics, 17(4):101450.
• Jiang (2026) Bo Jiang. 2026. Hindsight: Evaluating llm-generated research ideas via future impact. Preprint, arXiv:2603.15164.
• Kang et al. (2018) Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (peerread): Collection, insights and nlp applications. Preprint, arXiv:1804.09635.
• Keya et al. (2025) Farhana Keya, Gollam Rabby, Prasenjit Mitra, Sahar Vahdati, Sören Auer, and Yaser Jaradeh. 2025. Sci-idea: Context-aware scientific ideation using token and sentence embeddings. Preprint, arXiv:2503.19257.
• Krishna et al. (2023) Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Preprint, arXiv:2303.13408.
• Li et al. (2025) Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Yu Rong, Deli Zhao, Tian Feng, and Lidong Bing. 2025. Chain of ideas: Revolutionizing research via novel idea development with LLM agents. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 8971–9004.
• Lin et al. (2025) Ethan Lin, Zhiyuan Peng, and Yi Fang. 2025. Evaluating and enhancing large language models for novelty assessment in scholarly publications. In Proceedings of the 1st Workshop on AI and Scientific Discovery: Directions and Opportunities, pages 46–57.
• Liu et al. (2026a) Haokun Liu, Sicong Huang, Jingyu Hu, Yangqiaoyu Zhou, and Chenhao Tan. 2026a. Hypobench: Towards systematic and principled benchmarking for hypothesis generation. Preprint, arXiv:2504.11524.
• Liu et al. (2026b) Jiaqi Liu, Shi Qiu, Mairui Li, Bingzhou Li, Haonian Ji, Siwei Han, Xinyu Ye, Peng Xia, Zihan Dong, Meng Chen, Congyu Zhang, Letian Zhang, Guiming Chen, Haoqin Tu, Xinyu Yang, Lu Feng, Xujiang Zhao, Haifeng Chen, Jiawei Zhou, and 17 others. 2026b. Autoresearchclaw: Self-reinforcing autonomous research with human-ai collaboration. Preprint, arXiv:2605.20025.
• Liu and Zhai (2026) Miri Liu and ChengXiang Zhai. 2026. An axiomatic benchmark for evaluation of scientific novelty metrics. Preprint, arXiv:2604.15145.
• Lu et al. (2026a) Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. 2026a. Towards end-to-end automation of AI research. Nature, 651:914–919.
• Lu et al. (2026b) Li-Chun Lu, Miri Liu, Pin Chun Lu, Yufei Tian, Shao-Hua Sun, and Nanyun Peng. 2026b. Rethinking creativity evaluation: A critical analysis of existing creativity evaluations. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6329–6352.
• Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. Preprint, arXiv:1808.09602.
• Luo et al. (2022) Zhuoran Luo, Wei Lu, Jiangen He, and Yuqi Wang. 2022. Combination of research questions and methods: A new measurement of scientific novelty. Journal of Informetrics, 16(2):101282.
• Mishra and Torvik (2016) Shubhanshu Mishra and Vetle I. Torvik. 2016. Quantifying conceptual novelty in the biomedical literature. D-Lib magazine : the magazine of the Digital Library Forum, 22.
• Moussa et al. (2026) Hanane Nour Moussa, Patrick Queiroz Da Silva, Daniel Adu-Ampratwum, Alyson East, Zitong Lu, Nikki Puccetti, Mingyi Xue, Huan Sun, Bodhisattwa Prasad Majumder, and Sachin Kumar. 2026. Scholareval: Research idea evaluation grounded in literature. Preprint, arXiv:2510.16234.
• O’Neill et al. (2025) Charles O’Neill, Tirthankar Ghosal, Roberta Răileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciucă. 2025. Sparks of science: Hypothesis generation using structured paper data. Preprint, arXiv:2504.12976.
• Panickssery et al. (2024) Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076.
• Piroi et al. (2011) Florina Piroi, Mihai Lupu, Allan Hanbury, and Veronika Zenz. 2011. Clef-ip 2011: Retrieval in the intellectual property domain. In Conference and Labs of the Evaluation Forum.
• Qiao et al. (2026) Shuofei Qiao, Yunxiang Wei, Xuehai Wang, Bin Wu, Boyang Xue, Ningyu Zhang, Hossein A. Rahmani, Yanshan Wang, Qiang Zhang, Keyan Ding, Jeff Z. Pan, Huajun Chen, and Emine Yilmaz. 2026. Innoeval: On research idea evaluation as a knowledge-grounded, multi-perspective reasoning problem. Preprint, arXiv:2602.14367.
• Radensky et al. (2026) Marissa Radensky, Simra Shahid, Raymond Fok, Pao Siangliulue, Tom Hope, and Daniel S. Weld. 2026. Human-llm compound system for scientific ideation through facet recombination and novelty evaluation. Preprint, arXiv:2409.14634.
• Schmidgall et al. (2025) Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. 2025. Agent laboratory: Using llm agents as research assistants. Preprint, arXiv:2501.04227.
• Schopf and Färber (2026) Tim Schopf and Michael Färber. 2026. Is this idea novel? an automated benchmark for judgment of research ideas. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 4716–4727.
• Shahid et al. (2025) Simra Shahid, Marissa Radensky, Raymond Fok, Pao Siangliulue, Daniel S. Weld, and Tom Hope. 2025. Literature-grounded novelty assessment of scientific ideas. In Proceedings of the Fifth Workshop on Scholarly Document Processing, pages 96–113.
• Shibayama et al. (2021) Sotaro Shibayama, Deyun Yin, and Kuniko Matsumoto. 2021. Measuring novelty in science with word embedding. PLOS ONE, 16(7):e0254034.
• Si et al. (2025) Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2025. Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers. In International Conference on Learning Representations.
• Sinhahajari et al. (2026) Soumitra Sinhahajari, Navonil Majumder, and Soujanya Poria. 2026. On the limits of llm-as-judge for scientific novelty assessment. Preprint, arXiv:2606.12071.
• Soboroff and Harman (2005) Ian Soboroff and Donna Harman. 2005. Novelty detection: The TREC experience. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 105–112, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
• Sothanaphan (2026) Nat Sothanaphan. 2026. Resolution of erdős problem #728: a writeup of aristotle’s lean proof. Preprint, arXiv:2601.07421.
• Su et al. (2025) Haoyang Su, Renqi Chen, Shixiang Tang, Zhenfei Yin, Xinzhe Zheng, Jinzhe Li, Biqing Qi, Qi Wu, Hui Li, Wanli Ouyang, Philip Torr, Bowen Zhou, and Nanqing Dong. 2025. Many heads are better than one: Improved scientific idea generation by a llm-based multi-agent system. Preprint, arXiv:2410.09403.
• Taechoyotin and Acuna (2025) Pawin Taechoyotin and Daniel Acuna. 2025. Remor: Automated peer review generation with llm reasoning and multi-objective reinforcement learning. Preprint, arXiv:2505.11718.
• Tang et al. (2025) Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. 2025. Ai-researcher: Autonomous scientific innovation. Preprint, arXiv:2505.18705.
• Uzzi et al. (2013) Brian Uzzi, Satyam Mukherjee, Michael Stringer, and Ben Jones. 2013. Atypical combinations and scientific impact. Science, 342(6157):468–472.
• Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. Preprint, arXiv:2004.14974.
• Wang et al. (2017) Jian Wang, Reinhilde Veugelers, and Paula Stephan. 2017. Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 46(8):1416–1436.
• Wang et al. (2024) Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024. SciMON: Scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 279–299.
• Wang et al. (2020) Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. 2020. ReviewRobot: Explainable paper review generation based on knowledge synthesis. In Proceedings of the 13th International Conference on Natural Language Generation, pages 384–397, Dublin, Ireland. Association for Computational Linguistics.
• Wang et al. (2025) Yongji Wang, Mehdi Bennani, James Martens, Sébastien Racanière, Sam Blackwell, Alex Matthews, Stanislav Nikolov, Gonzalo Cao-Labora, Daniel S. Park, Martin Arjovsky, Daniel Worrall, Chongli Qin, Ferran Alet, Borislav Kozlovskii, Nenad Tomašev, Alex Davies, Pushmeet Kohli, Tristan Buckmaster, Bogdan Georgiev, and 3 others. 2025. Discovery of unstable singularities. Preprint, arXiv:2509.14185.
• Weng et al. (2025a) Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2025a. Cycleresearcher: Improving automated research via automated review. Preprint, arXiv:2411.00816.
• Weng et al. (2025b) Yixuan Weng, Minjun Zhu, Qiujie Xie, Qiyao Sun, Zhen Lin, Sifan Liu, and Yue Zhang. 2025b. Deepscientist: Advancing frontier-pushing scientific findings progressively. Preprint, arXiv:2509.26603.
• Wu et al. (2019) Lingfei Wu, Dashun Wang, and James A Evans. 2019. Large teams develop and small teams disrupt science and technology. Nature, 566:378–382.
• Wu et al. (2026) Wenqing Wu, Yi Zhao, Yuzhuo Wang, Siyou Li, Juexi Shao, Yunfei Long, and Chengzhi Zhang. 2026. NovBench: Evaluating large language models on academic paper novelty assessment. In Findings of the Association for Computational Linguistics: ACL 2026, pages 32103–32133.
• Yamada et al. (2025) Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. 2025. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. Preprint, arXiv:2504.08066.
• Yoon et al. (2026) Peter H Yoon, Januka S Athukoralage, Emmanuel Ameisen, Eric Kauderer-Abrams, Nicholas T Perry, and Matthew G Durrant. 2026. Autonomous ai agents discover reverse transcriptases with tandem repeat arrays. bioRxiv.
• Zhang et al. (2026) Ming Zhang, Kexin Tan, Yueyuan Huang, Yujiong Shen, Chunchun Ma, Li Ju, Xinran Zhang, Yuhui Wang, Wenqing Jing, Jingyi Deng, Huayu Sha, Binze Hu, Jingqi Tong, Changhao Jiang, Yage Geng, Yuankai Ying, Yue Zhang, Zhangyue Yin, Zhiheng Xi, and 4 others. 2026. OpenNovelty: An LLM-powered agentic system for verifiable scholarly novelty assessment. Preprint, arXiv:2601.01576.
• Zhao and Zhang (2025) Yi Zhao and Chengzhi Zhang. 2025. A review on the novelty measurements of academic papers. Scientometrics, 130:727–753.
• Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36.

Appendix A Pipeline Survey

Table 4 surveys the novelty judgments embedded in AI-scientist pipelines, and Table 6 lists standalone novelty judges. Table 6 shows what each pipeline passes downstream after its check, which shapes the evaluation contract (§4.1). Table 7 gives the systems we include and exclude. We include a system when (1) its code is available, (2) it has a separable judgment call, (3) that judgment takes a research proposal as input, (4) we can control the reference pool or sandbox its retrieval, and (5) its judge can be replaced with one whose training cutoff predates T. Table 8 shows how the shared proposal is passed to each included check, and Table 9 gives the measured cost of the final judge calls.

Appendix B Adapter Details

We pass the same (proposal, pool) pair to each check. For all checks that do some kind of search, we sandbox the search, ranking the pool by cosine similarity (text-embedding-3-large) to the query (checks can still use their native query style and specified depth k). In all cases, we set the final judge to Sonnet 4.6 at temperature 0 (Table 10). In the OpenNovelty-style baseline, we reimplement the per-contribution design of Zhang et al. (2026) with BM25 retrieval to avoid sharing a retriever with ClaimNov. BM25 throughout is rank_bm25’s BM25Okapi with default parameters (k1=1.5, b=0.75) over lowercased alphanumeric tokens with stopwords removed, and the scientific embeddings are SciNCL (malteos/scincl) and SPECTER2 (allenai/specter2_base with the proximity adapter).

Appendix C ClaimNov Prompts

The exact ClaimNov prompts.

Appendix D Disguise Prompts

We provide the exact prompts used to disguise proposals for each rung, as well as the prompt used for the verifier for rungs 3 and 4. Note the rung-1 prompt is also used to rewrite claim-disputed papers (§4.4). These are run with Sonnet 4.6.

Appendix E Later-Realized Labeling Prompts

We provide exact prompts used to label the later-realized set, run with Opus 4.6.

Appendix F Label Validation

For n=100 pairs of papers A and B where an LLM identified a prior-art dispute, two annotators independently mark (i) whether the dispute is genuine, (ii) whether the extracted B is the correct prior art, and (iii) the dispute type. They achieve (i) 81/100 raw agreement with Cohen’s κ=0.62, (ii) 73/100 and κ=0.47, and (iii) 52/100 and κ=0.40, and we drop the dispute-type labels due to the low kappa value. A third annotator then marked the 63 rows with any disagreements, blinded to the labels of the first two annotators. By 2-of-3 majority we find that we have 59/100 genuine disputes with 67/100 having correctly-identified B. We use the 40 pairs that have a genuine dispute and correct B as the label-clean subset, 36 of which have usable pools (Table 12). All three annotators have graduate-level education in English in the relevant research field.

We give the annotation instructions below. The audit of later-realized and sorter-matched ideas uses the second set.

F.1 Audit of generated ideas

We audit 60 generated ideas by hand with the second set of instructions above, one annotator per item. These are 45 later-realized ideas (25 at random, 10 that the whole-idea Opus pass called already done, and 10 that a low-false-alarm check rejected) and 15 sorter-matched ideas. This labeling is extremely strenuous, since each item requires reading an idea against 20 abstracts and deciding whether any one paper does substantially the same thing, a judgment on which experts often disagree (Si et al., 2025). Annotators confirm 28 of the 45 later-realized ideas as novel relative to the pool (a stratum-weighted 60% of all 400) and judge 9 of the 15 sorter-matched ideas already done. Table 13 gives each check’s rejections on both groups of later-realized ideas. On the 9 sorter-matched ideas judged already done, ClaimNov with fixed retrieval depth rejects 1 and the w/o-profile judge 7, which also rejects 3 of the 6 judged novel.

Appendix G Attribution by Dimension and Relation Type

Table 14 breaks down the claim-level attribution of §5.2 by the disputed claim’s dimension and by the relation type in the review.

Appendix H ClaimNov Ablations

We explain the ablations we evaluate for ClaimNov. Each aims to ablate one stage, leaving the rest unchanged (Table 15).

Appendix I Disguises Caught by Rung

Table 16 breaks down Table 1 by disguise rung. Aside from ARC (which we expect to have such behavior since it’s a fully lexical check), we don’t observe significant fluctuation in catch for different checks across the rungs.

Appendix J Intervals and Paired Tests

In Table 17 we give 95% intervals for Table 1. In Table 18 we give exact McNemar tests. Finally, in Table 19 we give repeated runs and clustered tests. Main-table rows come from one run per check on the first-party Anthropic API. Later Claude runs (repeats, profile-plus-abstracts, anchor-in, larger pools) use Sonnet 4.6 on Amazon Bedrock. Tables 20 and 21 score attribution as retrieval.

Appendix K Robustness

In this section, we test some assumptions to better understand if our main results depend on our specific labels, judge model, pool sizes, or the particular data we collected.

Because our later-realized labels come from one Opus labeler, we recompute false alarms under three alternative labelers in Table 23, specifically (1) a labeler from a different model family (gpt-5), (2) the ideas both Opus and gpt-5 agree on, and (3) the full set without the n=101 ideas a whole-idea Opus labeling pass considered previously done. For all of these labeling paradigms, ClaimNov’s false-alarm rate stays between 1.3% and 1.5%.

We check whether the claim-level novelty profile we provide for our final verdict is only helpful to our particular judge (Sonnet 4.6) by rerunning this stage with two non-Anthropic judges, specifically gpt-oss-120b (117B parameters) and Llama 4 Maverick (17B active, 400B total) in Table 24. These are weaker models, so catch does drop, but we still observe as with Sonnet that providing the profile cuts false alarms and source-out rejections relative to the base judge reading nearest abstracts.

We check in Table 25 whether baselines can match the judged checks. These are thresholds on lexical or embedding similarity, a judge that sees only the single nearest paper, and the extracted score-based checks at their best threshold. The baselines are not competitive with ClaimNov’s J.

Because our disguise pools hold only 20 papers, in Table 26 we artificially increase the pool size to ≈200 to investigate if our catch can survive more realistic pool sizes. While catch does fall on the hardest rung, source-out rejections stay near zero.

We also perform a handful of smaller checks that further support these findings (Table 22). Asked with no pool, Sonnet 4.6 names B first for at most 1 of the 160 claim-disputed proposals, so recall of pre-T papers does not drive that set. Appendix F.1 reports a human audit of 60 generated ideas.

Reproducibility.

We release all code and prompts at https://github.com/emmatliu/claimnov, including the scripts that build each dataset and run each check, together with the per-item records behind our tables. We release the generated ideas themselves. For source papers, pool papers and ICLR reviews, we release identifiers and titles rather than paper or review text, and the released code re-mines the claim disputes from public OpenReview reviews. We do not release the disguised or hybrid proposals because they restate published papers, but we do release code to rebuild them.

Appendix L Natural Already-Done Ideas

A whole-idea Opus sorter, reading the eight nearest pool papers, matched 837 of the 3,000 generated ideas to a paper that already does the idea. The claim-level labeler overruled it on 101 (these are in the later-realized set), and 407 of the remaining 736 have the shipped profile (Table 27). The set is noisier than the disguises because the sorter read exactly the abstracts the w/o-profile ablation reads. Appendix F.1 reports a human audit of 15 of these ideas.

Appendix M In-the-Loop Pilot

We plug each check into a generate–check–regenerate loop on two anchors (In-Context Edit and MoBA, 178–185-paper pools, 20 ideas each). Every arm starts from the same first idea, and a rejection returns its reason and cited paper to the generator, with up to three regenerations (Table 28). A pairwise review-score proxy (Si et al., 2025) separates committed from unchecked ideas on only 3 seeds for ClaimNov and 1 for CoI-Agent, too few to compare the arms, so we leave in-the-loop evaluation to future work.

Appendix N Case Study

In Table 29, we trace two proposals through ClaimNov. The first rewrites the ICLR 2022 Spotlight Amortized Tree Generation for Bottom-up Synthesis Planning and Synthesizable Molecular Design, whose reviewer attributed its MDP formulation to Barking up the right tree, with that paper in the pool. The second is a rung-4 disguise of Ratio-Variance Regularized Policy Optimization with the source in the pool.

Appendix O Component Validations

We explain the validations we performed on each component in §3.

O.1 Claim extraction

We extract typed claims from the abstract with gpt-4o-mini (temperature 0, JSON mode) using a GEPA-tuned prompt, scored against a 20-paper, three-annotator ground truth (Table 30). When we extract from introductions rather than abstracts, F1 drops from 0.577 to 0.490, so in ClaimNov we use abstracts. gpt-5-mini has equivalent performance on extraction but performs poorly as the entailment judge (Table 35).

O.2 Same-dimension retrieval

To validate same-dimension retrieval, we embed each claim’s title plus its attribute–value text with text-embedding-3-large, then rank its cosine similarity with same-dimension pool claims. In Table 31 we score recall of the reviewer-named work on 150 ICLR 2022–23 reviewer-tell pairs with A’s own references as distractors, and in Table 32 we look at the value of staying within the same dimension. Because we find that only 3 of the 159 disputes are not in same dimension, we keep same dimension retrieval rather than retrieving over all types of claims.

Retrieval depth.

From above validation we set depth as k=max⁡(10,0.2​nd) capped at 40. Table 33 checks this heuristic on the n=837 sorter-matched ideas (pools of up to 400 papers, median 174). In this set, using this depth rule means that only 5 of the 837 matched papers are never retrieved.

O.3 Graded entailment judge

In Tables 34–37, we evaluate and calibrate the graded entailment stage of ClaimNov. Table 34 compares the value of added context (specifically, titles and abstracts) to the value of higher model tiers; we find that context is comparatively more valuable. Table 35 investigates if the graded score alone can separate reviewer-attested novel claims (n=66) from reviewer-attested established claims (n=159) across six judge tiers and three retrieval settings, but we find that embedding-only similarity is a strong baseline here. Table 36 compares the judge’s use of the 1–5 entailment scale with that of three human annotators on n=50 pairs; we find every tier agrees with the human majority more than the annotators agree with each other. Table 37 tests whether our graded score can successfully rank a claim a reviewer specifically said was novel as higher than the sibling claims from the same paper (n=86 papers and n=233 claims); we find that it does so above chance, 59% of the time. All in all, we observe that the graded score can order claims well, but that there isn’t a good way to set a pass/fail novelty threshold based off the graded score.

O.4 Verdict

Calibration.

We use n=20 disputed pairs, n=32 later-realized decoys, and n=106 already-done ideas to try and calibrate a particular novel or non-novel threshold for N⁡(P), as we show in Table 38. We consider a knockout entailment level, relation labels, and a per-candidate question; the verdict stage we ultimately adopted considers novelty once over the whole profile (§3.5), while the per-candidate question is the per-claim ablation of Table 1.

Appendix P AI Disclosure Statement

LLM agents (e.g., Claude Code) were used to assist with experiment and analysis code, to advise on paper structure and clarity, to generate results tables and draft appendix text, and to prepare released data and code files. All scientific contributions, including the ClaimNov methodology, experimental design, and data collection approaches, were devised by the authors.

注意转载授权 · 阅读原文