Tracing the Influence of Retrieved Literature on Scientific Ideation in LLMs

Abstract

Large language models (LLMs) are increasingly used to generate research ideas from scientific literature. Existing systems combine literature retrieval with search, generation, and refinement, making it difficult to determine how the retrieval strategy itself shapes the ideas generated. We isolate this contribution by comparing five retrieval strategies and a No Literature baseline under a shared generation protocol, using 45,000 ideas across 15 machine learning topics and ten open-weight models. Relative to no literature basline, all five retrieval strategies reduce diversity among ideas generated from the same set of papers by 5.6%–17.9%, while most increase diversity by 10.2%–18.1% when ideas from different sets are pooled. Effects on idea quality and novelty are modest on average but vary across topics. We further apply a JSD-based attribution method to trace the influence of individual input papers on each idea’s motivation, core idea, approach, and validation. In 79.3% of ideas, only one or two distinct papers receive the highest attribution scores across these four parts. These leading sources often recur across ideas generated from the same set of papers, and groups with more varied source attribution contain more diverse ideas (r=0.70). These findings motivate various sets of literature and the use of attribution to guide the choice of sources for subsequent ideas.

1 Introduction

Scientific ideation requires creativity, commonly defined in terms of originality and effectiveness (Runco and Jaeger, 2012). Reflecting these requirements, research on LLM-based scientific ideation commonly evaluates individual ideas in terms of novelty, feasibility, and scientific value (Si et al., 2025; Eger et al., 2026; Chen et al., 2026b). A dynamic perspective on creativity further emphasizes potential originality and effectiveness and considers the entire ensemble of products generated by a creative process (Corazza, 2016). Recent work also incorporates diversity into LLM creativity evaluation and explicitly models and optimizes scientific ideation workflows (Sen et al., 2026). Evaluating an ideation system therefore also requires considering the range of research directions produced across generations (Figure 1).

Literature retrieval is one of the key steps in this process, determining which papers a model receives during ideation. Existing systems combine retrieval with their own generation, search, and refinement procedures (Wang et al., 2024b; Baek et al., 2025; Xiong et al., 2026; Wang et al., 2026). However, this integration makes it difficult to isolate the effect of literature selection from other stages of the ideation process. This distinction matters because different sets of papers may steer the model toward different research directions, even when the resulting sets of ideas have similar average quality or novelty. In particular, it remains unclear whether changing the supplied papers produces more varied ideas than generating multiple ideas from the same papers.

To address this question, we isolate literature selection under a shared generation protocol (Figure 1). For each research topic, each retrieval strategy provides multiple sets of papers, and we generate several ideas using each set as context. Following recent work (Hou et al., 2026), we evaluate generated ideas along three complementary dimensions: quality, novelty, and diversity. Quality and novelty characterize individual ideas, whereas diversity captures the range of directions represented by a set of ideas. We compare diversity among ideas generated from the same papers with diversity among all ideas generated from different sets of papers.

We complement this set-level analysis with context attribution based on input erasure (Li et al., 2017), adapting methods developed for retrieval-augmented generation (Cohen-Wang et al., 2024; Li et al., 2026a) to scientific ideation. We remove retrieved papers one at a time and measure changes in the model’s predictions for the same generated idea.

Our study¹¹ 1 We will release our code and data upon acceptance. examines 45,000 research ideas across five retrieval strategies and a No Literature baseline, 15 machine learning topics, and ten open-weight models. Our main findings are:

Retrieval narrows diversity among ideas generated from the same papers but can broaden it across different paper selections. Relative to No Literature, all five retrieval strategies reduce within-set diversity by 5.6%–17.9%, while four increase diversity by 10.2%–18.1% when ideas generated from different literature sets are considered together. Differences in mean quality and novelty are comparatively modest.

Dependence on retrieved literature is concentrated and recurrent. Our attribution analysis identifies at most two of the eight retrieved papers as leading sources for 79.3% of ideas, with a single leading source for 39.5%. Among ideas generated from the same papers, the leading paper for a given component is shared by 38.7%–43.2% of idea pairs. Groups with more varied core-idea source dependence also contain more diverse ideas (r=0.70).

Retrieval’s average diversity advantage shrinks across model updates, while its quality advantage grows. Across matched model updates, retrieval’s average diversity advantage decreases by approximately 0.030, while its relative quality advantage increases by approximately 0.249. These changes differ substantially across model families.

2 Idea Generation and Attribution

We separate variation associated with retrieved literature from variation in ideas generated under the same context and variation across models. For the same research topic, we compare (i) ideas generated from one set of papers, (ii) ideas generated from different sets under the same model, and (iii) ideas generated from the same set across models. These comparisons follow a shared generation protocol across retrieval strategies.

2.1 Idea Generation Protocol

Let 𝒫 denote the paper pool, the set of papers available to the retrieval strategies. Given a research topic q, a retrieval strategy R constructs a literature set 𝒟⊆𝒫, a set of papers supplied together as context for idea generation:

A model θ receives the research topic q and literature set 𝒟 through a common instruction template I and generates a research idea y:

Each idea y consists of a title and four components: motivation, which states the research problem and why it matters; core idea, which states the central contribution; approach, which describes how the idea would be implemented; and validation plan, which outlines how it would be evaluated.

For each literature set 𝒟, we independently generate several ideas using the same model and refer to them as a generation group. Because the topic, papers, instruction, and model are fixed within a group, differences reflect variation among ideas generated under the same conditions. We compare these groups across literature sets, retrieval strategies, and models.

2.2 Retrieval Strategies

For each topic q, a retrieval strategy selects a literature set 𝒟 of eight papers from the shared corpus. We compare five strategies that differ in how they use semantic relevance and citation structure:

1. R1

   Semantic Top-K. Papers are ranked by semantic similarity to q, following semantic retrieval used in scientific ideation (Wang et al., 2024a). We sample eight papers from the top 100 candidates to obtain different literature sets.

2. R2

   Most-Cited Top-K. We prioritize highly cited papers among topic-relevant candidates, following citation-based selection in ResearchAgent (Baek et al., 2025), and sample eight papers from the top 100.

3. R3

   Semantic + Entity + Co-citation. We adapt SciPIP (Wang et al., 2024b), combining semantic similarity with scientific entities and co-citation relations.

4. R4

   Core-Paper Reference Expansion. We select a core paper and construct 𝒟 from the paper and its references, following ResearchAgent (Baek et al., 2025).

5. R5

   Chain of Ideas. We follow backward references and forward citations to construct a sequence of related papers across time (Li et al., 2025).

No Literature. As a baseline, we set 𝒟=∅ and provide only the research topic.

2.3 Component-Level Attribution

Finally, we examine how individual retrieved papers shape different components of a generated idea. Following context-ablation approaches (Cohen-Wang et al., 2024; Li et al., 2026a), we remove one retrieved paper at a time and rescore the same generated text, keeping its preceding tokens fixed.

Let 𝒟 denote the retrieved literature set and 𝒟−k the same set with paper dk removed. At token position j, with generated prefix z<j held fixed, we compare the next-token distributions

We compute sensitivity separately for the motivation, core idea, approach, and validation components. Let 𝒯c denote the token positions belonging to component c. The score for paper dk and component c is

where JS denotes Jensen–Shannon divergence. Larger values indicate greater sensitivity to removing dk while the generated text is held fixed.

For comparisons across papers, we normalize the scores within each component,

and define the paper with the largest wk,c as the leading paper for that component. We compare these distributions across components and across ideas generated from the same literature set to measure source concentration, source recurrence, and variation in source dependence. Figure 2 shows an example of a generated idea.

3 Experimental Setup

We now instantiate our study design with concrete topics, evaluated models, and evaluation metrics. Full evaluation configurations, sampling procedures, and statistical protocols are provided in Appendix A.

3.1 Topics and Literature

We study 15 machine learning topics derived from the primary areas of ICLR 2026, merging closely related areas. We collect papers first submitted to arXiv between January 1 and July 23, 2026. For each topic q, we rank papers by semantic relevance and retain the top 2,000. After deduplication across topics, the shared corpus contains 24,836 papers. Retrieval uses each paper’s title and abstract.

3.2 Models and Generation

We evaluate ten open-weight models from four families, using the 2025 models Qwen3-2507 (30B), Gemma 3 (27B), Nemotron 3 Nano (30B), and GLM-4 (32B) for the main retrieval comparison. Temporal comparisons with the corresponding 2026 releases and the 2024 Qwen and Gemma checkpoints are reported in the Appendix D. Open weights enable access to the next-token distributions required for paper-deletion analysis (Section 2.3). We select the 2025 models for the main comparison so that papers in the 2026 corpus postdate their release, reducing the likelihood of prior exposure to these papers during training.

All models receive the same ordered literature sets, with papers represented by titles and abstracts. We generate 45,000 ideas across all models and conditions, each containing a title, motivation, core idea, approach, and validation. Additional results for closed-source models and for reasoning-enabled generation are reported in Appendices F and  G, respectively.

3.3 Evaluation Metrics

We evaluate quality, novelty, and diversity as complementary dimensions of creative generation. Individually novel ideas can still concentrate on closely related questions or methods. We therefore assess both individual ideas and the breadth of the generated collection. Below, we define the corresponding evaluation measures.

Diversity.

We measure semantic diversity as the mean pairwise cosine distance between text-embedding-3-large embeddings (OpenAI, 2024) of each idea. Within-set diversity averages this measure over five-idea generation groups, each generated by the same model from the same literature set 𝒟. Overall diversity applies the same measure to all 50 ideas per model–topic–condition combination. We use these measures to compare variation within generation groups with variation across ideas pooled from different groups.

Quality.

We adopt the validity, feasibility, significance, and clarity dimensions from InnoEval (Qiao et al., 2026). GPT-5.6-Sol scores each dimension on a 1–10 scale, and their arithmetic mean gives the quality score. We evaluate five ideas per model–topic–condition combination.

Novelty.

GPT-5.6-Luna scores novelty using the five-point RINoBench rubric (Schopf and Färber, 2026). Local novelty compares an idea with its supplied literature 𝒟 and is evaluated for all literature-conditioned ideas. Search-assisted global novelty compares an idea with additional related papers retrieved through web search. It is evaluated on one idea per model–topic–condition combination, including No Literature.

Full agreement analyses and judge-selection results are reported in Appendix C. Evaluation prompts are provided in Appendix B; judge-agreement results and analyzes are reported in Appendix C.

Human evaluation.

Three PhD students with an NLP background evaluate 20 ideas sampled from the NLP topic using the same rubric as the LLM judge. For each idea, both the annotators and the judge receive the eight retrieved input papers and the five most recent papers retrieved by GPT-5.6-Sol with web-search tool. The judge’s Spearman correlations with mean human ratings are 0.307 for quality, 0.680 for local novelty, and 0.504 for global novelty. Appendix C.2 provides the evaluation details and agreement between human annotators.

4 Findings

4.1 How does retrieved literature shape scientific exploration?

We first compare the five retrieval strategies across the four 2025 models and 15 research topics. Figure 3 reports diversity within a fixed literature set and across all literature sets, together with quality and novelty.

Retrieval narrows diversity among ideas generated from the same papers but can broaden it across different sets of papers.

All five retrieval strategies reduce diversity among ideas generated from the same papers relative to No Literature, with decreases of 5.6%–17.9% (Figure 3a). When ideas generated from different sets of papers are considered together, R1–R4 increase overall diversity by 10.2%–18.1% relative to No Literature, while R5 remains close to the baseline (−0.8%). The gap between overall and within-set diversity is largest for R4 (+0.078) and smallest for R5 (+0.035). R5 also has the lowest mean distance between the centroids of its input paper sets (0.046, compared with 0.066–0.115 for R1–R4). After accounting for topic and retrieval strategy, this distance is positively associated with overall idea diversity (r=0.503) and the gap between overall and within-set diversity (r=0.356). These associations suggest that selecting semantically different sets of papers can help broaden exploration.

Differences in exploration are larger than differences in average idea quality and novelty.

Retrieval produces comparatively modest changes in proposal-level scores (Figure 3b,c). R2 obtains the highest mean quality (5.9) and local novelty (3.3), but no retrieval strategy consistently dominates across quality and novelty. Mean global-novelty scores vary little across retrieval strategies. Strategies with substantially different diversity can therefore have similar average proposal-level scores.

Retrieval effects vary across research topics.

The aggregate results vary substantially across topics (Appendix H). The strategy producing the highest quality or novelty differs across topics, indicating that no single literature retrieval strategy is uniformly preferable. The difference between within-context and across-context diversity is more consistent across topics than the ranking of individual retrieval strategies. Figure 4 illustrates this difference for Vision, Language, Audio, and Multimodal Applications. Ideas generated without literature occupy a comparatively concentrated region, whereas different retrieved literature sets shift generations toward different regions of the projected space.

The within- versus across-context difference extends across idea components.

The same pattern is observed when diversity is computed separately for the title, motivation, core idea, approach, and validation plan (Appendix E.1). For every retrieval strategy, overall diversity exceeds diversity among ideas generated from the same literature set. The difference is largest under R4, where, for example, motivation diversity increases from 0.274 within a literature set to 0.369 across sets. The same pattern therefore holds across all proposal components.

4.2 How do retrieval effects change across model updates?

The preceding analysis holds the generation model fixed when comparing literature contexts. We next examine whether the effect of retrieval itself changes as the underlying model changes. We compare matched 2025 and 2026 checkpoints from four model families while holding topics and ordered literature sets fixed.

Retrieval provides less additional diversity in newer models.

Diversity under No Literature increases by 0.027 from 2025 to 2026, whereas average diversity under retrieval changes little (Figure 12a). Consequently, the average difference between retrieval and No Literature changes from +0.015 in 2025 to −0.015 in 2026. This 0.030 reduction in the retrieval effect is driven primarily by greater diversity without retrieved literature, rather than by a corresponding decline in retrieval-conditioned diversity.

The relative effect on quality changes in the opposite direction.

Mean quality decreases by 0.343 under No Literature, compared with an average decrease of approximately 0.094 under retrieval. The quality difference between retrieval and No Literature consequently increases by approximately 0.249. Across these model updates, retrieved literature therefore provides less additional variation but a larger relative advantage in quality.

Changes are model-family specific.

The changes differ across model families (Figure 12b). Gemma gains in quality while decreasing in local novelty, whereas GLM and Nemotron gain in local novelty while decreasing in quality; Nemotron also becomes less diverse. Qwen shows the clearest increase in exploration, with gains in both diversity and local novelty. Search-assisted global novelty does not change significantly for any family. Improvements in one dimension of scientific ideation therefore do not imply corresponding improvements in the others.

5 How do generated ideas depend on retrieved papers?

The preceding analyses show that changing the literature broadens the idea space and that the magnitude of this effect varies across models. We now turn to the fixed-context setting to examine how generated ideas depend on the individual papers within a retrieved set.

We randomly select five of the 15 topics and increase generation from five to 20 ideas per fixed model–literature input. Figure 5 summarizes source reuse across components and ideas, along with the association between source variation and idea diversity. Figure 6 illustrates how the leading paper differs across components within one idea. Appendix I compares attribution with human judgments and model-reported sources. Additional results across model updates and a qualitative study are presented in Appendices D and K, respectively.

A few papers lead across components and recur across ideas.

In 79.3% of ideas, one or two papers lead across motivation, core idea, approach, and validation; a single paper leads all four in 39.5% (Figure 5a). Across ideas sharing the same literature, 38.7%–43.2% of pairs share a leading paper for a given component, over three times the 12.5% expected under uniform selection (Figure 5b). Source dependence is therefore concentrated on a few papers, with the same leading sources recurring across ideas generated from the same papers.

More varied source dependence accompanies greater idea diversity.

Groups with more varied core-idea source dependence contain more diverse ideas (Pearson r=0.70, 95% CI [0.56,0.81]; Figure 5c). Figure 6 shows how source dependence can shift across components of the same idea. P1 (McCoy, 2026) leads the motivation (69.5%), while P7 (Shafran et al., 2026) leads the core idea (69.6%) and approach (84.8%). Validation is more distributed across P7 (45.7%), P1 (21.6%), and P3 (21.5%). The leading source therefore changes across components, from linguistic framing in the motivation to the analysis method in the core idea and approach, with a more distributed pattern in validation.

Presentation position strongly influences attribution.

Even after shuffling the same eight papers, the first displayed paper received 21.7%–34.5% of attribution across four models, compared with 12.5%–16.4% for the next-highest position in each model. Doubling title–abstract length was also associated with a 4.62-percentage-point increase in attribution share, while paper type and topic relevance showed no clear independent associations.

6 Related Work

Literature-grounded scientific ideation.

Scientific ideation systems retrieve literature to guide idea generation and refinement (Eger et al., 2026). SciMON and ResearchAgent combine retrieval with iterative refinement (Wang et al., 2024a; Baek et al., 2025), while SciPIP combines semantic, entity, and co-citation retrieval (Wang et al., 2024b). Chain of Ideas and Graph2Idea organize literature through developmental chains and graph relations (Li et al., 2025; Li et al., 2026b). FlowPIE couples feedback-guided patent literature exploration with idea evolution (Wang et al., 2026); EvoGens uses retrieval-guided mutation and semantic-aware crossover (Li et al., 2026c). We compare retrieval strategies under a shared generation protocol across models and successive releases.

Scientific exploration and evaluation.

NoveltyBench and Artificial Hivemind document limited diversity in open-ended generation (Zhang et al., 2025; Jiang et al., 2025). In scientific ideation, Tang and Yang (2026) find that generated ideas remain concentrated around seed literature. Heuresis finds that search strategies change quality, diversity, and novelty without expanding the observed quality–novelty frontier (Antoniades et al., 2026), while IDEAgent counts distinct ideas meeting quality thresholds (Gumma et al., 2026). We distinguish diversity within a fixed paper set from diversity across retrieved sets and evaluate idea quality and novelty. Novelty judgments use supplied papers and, following evidence-grounded approaches such as InnoEval (Qiao et al., 2026), additional literature retrieved through web search.

Attributing generation to retrieved literature.

ALCE evaluates whether cited evidence supports generated claims (Gao et al., 2023). ContextCite and ARC-JSD estimate source influence through context ablations, measuring changes in response likelihood and output distributions, respectively (Cohen-Wang et al., 2024; Li et al., 2026a). We remove one paper at a time while holding the generated idea fixed, measuring sensitivity separately for motivation, core idea, approach, and validation to trace literature dependence across idea components.

Implications for literature-grounded ideation.

Our results suggest several practical considerations. Generating more ideas from the same literature does not recover the breadth obtained by varying the literature across generations (§4). Retrieval strategies should therefore be evaluated over sets of ideas, since similar average quality or novelty can coincide with substantial differences in diversity (§4). Recurring dependence on the same papers is also associated with more concentrated idea sets (§5), although whether deliberately varying source dependence can broaden exploration remain to be tested. Finally, retrieval effects change across model updates (§4.2): newer models may exhibit greater baseline diversity without retrieved literature while retaining a relative quality advantage from retrieval. Retrieval strategies should therefore be re-evaluated as the underlying generation model changes, rather than assumed to transfer unchanged across model generations.

7 Conclusion

We study how retrieved literature shapes scientific ideation under a controlled generation protocol. Changing the literature broadens the range of research directions explored, while ideas from a fixed context remain more concentrated. This effect is stronger for diversity than for average quality or novelty. Source dependence is also concentrated on a small number of retrieved papers, with recurring leading sources across generations and substantial sensitivity to presentation position. These patterns vary across model updates.

The literature supplied to a model affects both individual ideas and the distribution of research directions produced across generations. Future work should evaluate literature-grounded ideation at the level of idea sets, considering the range of directions explored alongside quality and novelty.

Limitations

Our diversity measures rely on distances between text embeddings, which do not necessarily correspond to substantive differences in research questions, underlying mechanisms, or methods. Ideas with different wording or framing may still share the same scientific approach. Search-assisted global novelty depends on the coverage of retrieved prior work: relevant studies may be missed, leading to overestimated novelty. Quality and novelty judgments also depend on the LLM judges, and agreement across providers alone does not establish agreement with domain experts. Quality and global novelty are evaluated on limited subsets, further limiting the precision of comparisons, particularly when score differences are small.

The experiments cover 15 ICLR-derived topics, a fixed literature corpus, and selected open-weight model families, with each literature set providing the titles and abstracts of eight papers. Results may differ for other disciplines, model scales, full-text inputs, or iterative ideation systems. Papers published after a checkpoint’s release may still contain concepts encountered during training. Within-family comparisons also combine changes in training and model design, so release year alone cannot explain generational differences.

Our attribution analysis removes papers one at a time and compares model predictions for the same idea text, with preceding tokens held fixed. The resulting scores measure sensitivity to individual papers; they do not establish an idea’s scientific origin or capture all interactions among papers. The association between variation in source use and idea diversity is observational. We evaluate ideas without implementing them, so their practical scientific value remains untested.

Ethics statement

Our study uses publicly available scientific literature and model-generated research ideas. Generated ideas may reproduce prior work, contain scientific errors, or suggest harmful applications, and require expert assessment before implementation. Literature selection may also overrepresent highly cited work and established research communities. Responsible use should preserve source provenance, credit prior contributions, and assess the potential consequences of pursuing an idea.

Reproducibility statement

Section 2 defines the generation protocol. Section 3 specifies the corpus construction, model comparisons, and evaluation measures. Section 2.3 defines the attribution method. Appendix A details checkpoint identifiers, generation and judge settings, evaluation subsets and sampling procedures, and statistical protocols. Reproduction requires preserving the ordered literature contexts shared across checkpoints and the idea subsets used for each evaluation. Details of artifact licensing and our release plan are provided in Appendix J.

AI Use Statement

In this study, we used LLMs for research idea generation, quality and novelty evaluation, and attribution analysis, and text embedding models to measure idea diversity. These uses are part of the experimental protocol described in the paper.

We also used generative AI tools to assist with manuscript editing, coding, and figure design. We did not use these tools to formulate this study’s research questions or hypotheses, interpret its experimental results, or develop its scientific claims. All AI-assisted edits and figures were reviewed and revised by the authors. We take responsibility for the final content of this work, including any text or artifacts produced with the aid of generative AI.

References

• Antoniades et al. (2026) Antonis Antoniades, Deepak Nathani, Ritam Saha, Alfonso Amayuelas, Ivan Bercovich, Zhaotian Weng, Vignesh Baskaran, Kunal Bhatia, and William Yang Wang. 2026. Heuresis: Search strategies for autonomous AI research agents across quality, diversity and novelty. arXiv preprint arXiv:2606.25198.
• Baek et al. (2025) Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2025. Researchagent: Iterative research idea generation over scientific literature with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 6709–6738. Association for Computational Linguistics.
• Chen et al. (2026a) Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, and Ge Liu. 2026a. Langflow: Continuous diffusion rivals discrete in language modeling. Preprint, arXiv:2604.11748.
• Chen et al. (2026b) Zhiyu Chen, Keyu Zhao, Jigao Fu, Dong Liang, Yanbiao Wu, Jiaoyang Li, Haidong Xue, Xinhua Zeng, Yuanyi Zhen, Fengli Xu, and Yong Li. 2026b. Ideation Arena: Evaluating LLM generated research ideas with battle-style human expert assessment. Preprint, arXiv:2608.29696. Accepted to the EMNLP 2026 main conference.
• Cohen-Wang et al. (2024) Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander Madry. 2024. Contextcite: Attributing model generation to context. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.
• Corazza (2016) Giovanni Emanuele Corazza. 2016. Potential originality and effectiveness: The dynamic definition of creativity. Creativity Research Journal, 28(3):258–267.
• Eger et al. (2026) Steffen Eger, Yong Cao, Jennifer D’Souza, Andreas Geiger, Christian Greisinger, Stephanie Gross, Yufang Hou, Brigitte Krenn, Anne Lauscher, Yizhi Li, Chenghua Lin, Nafise Moosavi, Wei Zhao, and Tristan Miller. 2026. Transforming science with large language models: A survey on ai-assisted scientific discovery, experimentation, content generation, and evaluation. ACM Comput. Surv. Just Accepted.
• Gao et al. (2023) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore. Association for Computational Linguistics.
• Gumma et al. (2026) Varun Gumma, Navonil Majumder, Soumitra Sinhahajari, and Soujanya Poria. 2026. Ideagent: Agentic quality-diversity search for research idea generation. Preprint, arXiv:2607.22375.
• Hou et al. (2026) Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu, Bhiman Kumar Baghel, Anneliese Brei, Ximing Lu, Meng Jiang, Faeze Brahman, Snigdha Chaturvedi, Haw-Shiuan Chang, Daniel Khashabi, and Xiang Lorraine Li. 2026. Creativityprism: A cross-domain evaluation framework for large language model creativity. Trans. Mach. Learn. Res., 2026.
• Jiang et al. (2025) Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, and Yejin Choi. 2025. Artificial hivemind: The open-ended homogeneity of language models (and beyond). In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.
• Kinney et al. (2023) Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, David Graham, Fangzhou Hu, and 29 others. 2023. The Semantic Scholar open data platform. arXiv preprint arXiv:2301.10140.
• Li et al. (2017) Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Understanding neural networks through representation erasure. Preprint, arXiv:1612.08220.
• Li et al. (2025) Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Yu Rong, Deli Zhao, Tian Feng, and Lidong Bing. 2025. Chain of ideas: Revolutionizing research via novel idea development with LLM agents. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 8971–9004. Association for Computational Linguistics.
• Li et al. (2026a) Ruizhe Li, CHEN CHEN, Yuchen Hu, Yanjun Gao, Xi Wang, and Emine Yilmaz. 2026a. Attributing response to context: A jensen–shannon divergence driven mechanistic study of context attribution in retrieval-augmented generation. In International Conference on Learning Representations, volume 2026, pages 146805–146837.
• Li et al. (2026b) Xu Li, Hanzhe Tu, and Xun Han. 2026b. Graph2idea:retrieval-augmented scientific idea generation with graph-structured contexts. CoRR, abs/2606.09105.
• Li et al. (2026c) Xu Li, Hanzhe Tu, Xinyi Li, Kuncheng Zhao, Xun Han, and Zhonghui Liu. 2026c. Evogens: A population-based heuristic search framework for scientific idea generation. Preprint, arXiv:2605.30961.
• McCoy (2026) R. Thomas McCoy. 2026. Not-so-strange love: Language models and generative linguistic theories are more compatible than they appear. Preprint, arXiv:2605.10061.
• OpenAI (2024) OpenAI. 2024. New embedding models and API updates.
• Qiao et al. (2026) Shuofei Qiao, Yunxiang Wei, Xuehai Wang, Bin Wu, Boyang Xue, Ningyu Zhang, Hossein A. Rahmani, Yanshan Wang, Qiang Zhang, Keyan Ding, Jeff Z. Pan, Huajun Chen, and Emine Yilmaz. 2026. Innoeval: On research idea evaluation as a knowledge-grounded, multi-perspective reasoning problem. Preprint, arXiv:2602.14367.
• Runco and Jaeger (2012) Mark A Runco and Garrett J Jaeger. 2012. The standard definition of creativity. Creativity research journal, 24(1):92–96.
• Schopf and Färber (2026) Tim Schopf and Michael Färber. 2026. Is this idea novel? an automated benchmark for judgment of research ideas. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, LREC 2026, Palma de Mallorca, Spain, May 13-15, 2026, pages 4716–4727. ELRA Language Resource Association / ACL.
• Sen et al. (2026) Tan Min Sen, Zachary Choy Kit Chun, Syed Ali Redha Alsagoff, Nadya Yuki Wangsajaya, Banerjee Mohor, Swaagat Bikash Saikia, and Alvin Chan. 2026. Automated creativity evaluation of language models across open-ended tasks. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23139–23173, San Diego, California, United States. Association for Computational Linguistics.
• Shafran et al. (2026) Or Shafran, Shaked Ronen, Omri Fahn, Shauli Ravfogel, Atticus Geiger, and Mor Geva. 2026. From directions to regions: Decomposing activations in language models via local geometry. Preprint, arXiv:2602.02464.
• Shi et al. (2026) Chuancheng Shi, Shangze Li, Wenjun Lu, Wenhua Wu, Cong Wang, Zifeng Cheng, Fei Shen, and Tat-Seng Chua. 2026. Tracerouter: Robust safety for large foundation models via path-level intervention. Preprint, arXiv:2601.21900.
• Si et al. (2025) Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2025. Can llms generate novel research ideas? A large-scale human study with 100+ NLP researchers. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
• Tang and Yang (2026) Yixuan Tang and Yi Yang. 2026. Ai research agents narrow scientific exploration. Preprint, arXiv:2605.27905.
• Wang et al. (2024a) Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024a. Scimon: Scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 279–299. Association for Computational Linguistics.
• Wang et al. (2026) Qiyao Wang, Hongbo Wang, Longze Chen, Zhihao Yang, Guhong Chen, Hamid Alinejad-Rokny, Hui Li, Yuan Lin, and Min Yang. 2026. Flowpie: Test-time scientific idea evolution with flow-guided literature exploration. Preprint, arXiv:2603.29557.
• Wang et al. (2024b) Wenxiao Wang, Lihui Gu, Liye Zhang, Yunxiang Luo, Yi Dai, Chen Shen, Liang Xie, Binbin Lin, Xiaofei He, and Jieping Ye. 2024b. Scipip: An llm-based scientific paper idea proposer. CoRR, abs/2410.23166.
• Xiong et al. (2026) Xiaoyu Xiong, Yuqi Ren, and Deyi Xiong. 2026. EvoSci: A bio-inspired multi-agent framework for the evolution of scientific discovery. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9846–9878, San Diego, California, United States. Association for Computational Linguistics.
• Zhang et al. (2025) Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang, and Daphne Ippolito. 2025. Noveltybench: Evaluating language models for human-like diversity. In Proceedings of the 2nd Conference on Language Modeling (COLM).

Appendix A Experimental Protocol

A.1 Models and Comparisons

Table 1 lists the ten open-weight models and three proprietary models used in the study. The main comparison of retrieval strategies uses the four 2025 open-weight models. We compare each with its 2026 counterpart from the same family, with additional 2024–2026 comparisons for Qwen and Gemma. Models receive the same topics, literature sets, paper order, and generation task for each matched comparison.

A.2 Topics, Corpus, and Literature Sets

We construct 15 research topics from the 21-area ICLR 2026 taxonomy by merging related areas and excluding “Other Machine Learning.” Table 2 lists the topics and their constituent areas. Eligible papers were first submitted to arXiv between January 1 and July 23, 2026. For each topic, we select the 2,000 most semantically relevant papers. Deduplicating these selections yields a shared corpus of 24,836 papers. Each literature set contains eight papers, represented by their titles and abstracts.

We use the five retrieval strategies defined in Section 2: Semantic Top-K, Most-Cited Top-K, Semantic + Entity + Co-citation, Core-Paper Reference Expansion, and Chain of Ideas. For Semantic Top-K and Most-Cited Top-K, each literature set is sampled from the strategy’s top 100 candidates. The other strategies select papers through their respective retrieval procedures. The No Literature condition supplies only the research topic.

A.2.1 Retrieval Implementation

We implement the five strategies in Section 2.2 on the shared corpus P. Let Pq denote the 2,000 papers retrieved for topic q. We encode the topic name and each paper’s title and abstract with text-embedding-3-large and compute cosine similarity between their 3,072-dimensional embeddings. Citation counts and reference links come from a fixed Semantic Scholar snapshot; citation-based expansion uses links within P. Each strategy produces ten eight-paper inputs per topic, whose contents and order are shared across models.

R1: Semantic Top-K.

We retain the 100 papers in Pq with the highest topic similarity and sample eight uniformly without replacement. We repeat sampling to obtain ten distinct sets, redrawing duplicate sets while allowing individual papers to recur across sets. Selected papers retain their semantic ranking in the input.

R2: Most-Cited Top-K.

We rank papers in Pq by citation count, using semantic rank and then arXiv ID to break ties. After deduplicating by Semantic Scholar work identity, we retain the first 100 papers and apply the sampling procedure used for R1. Selected papers retain their citation ranking in the input.

R3: Semantic + Entity + Co-citation.

We combine three candidate sets. The semantic set contains the five papers most similar to the topic. For entity retrieval, GPT-5.4-mini extracts up to five scientific entities per topic and twelve per paper, covering tasks, methods, datasets, and evaluation measures. Orthographic variants, singular/plural forms, and explicitly defined abbreviations are normalized. Entities from the topic and the five semantic candidates are expanded with up to ten additional entities that each co-occur with an initial entity in at least seven corpus papers, preferring those with lower document frequency. Matching papers form the entity set. We cap this set at 500 papers by successively halving the expansion count. If the initial entities alone exceed this cap, we retain the 500 most topic-relevant papers matching the initial entities and the original expansion. For each semantic or entity candidate, the co-citation set adds the paper with the most shared citing papers in P, requiring at least two shared citing papers.

We merge the candidates and apply single-link clustering at cosine similarity 0.8. Clusters are ranked by their most topic-relevant member. For each input, we uniformly sample eight of the first 100 clusters without replacement and one paper uniformly from each selected cluster. We redraw duplicate sets and present papers in cluster-rank order.

R4: Core-Paper Reference Expansion.

We rank potential cores by topic similarity, considering Pq before the remaining corpus and prioritizing papers with references in P. We select ten distinct works mapped to Semantic Scholar, one core per input. Each core is paired with up to seven references, ranked by topic similarity. Remaining positions are filled from the semantic ranking of Pq, followed by P if needed, excluding already selected works. Inputs present the core first, then its references, followed by the additional semantic selections.

R5: Chain of Ideas.

We sample two distinct anchors without replacement from mapped works among the 50 most topic-relevant papers in Pq, with probabilities proportional to nonnegative topic similarity. Each anchor starts a chain of up to four papers. At each step, we compare the best-scoring reference at the backward end with the best-scoring citing paper at the forward end and extend toward the higher-scoring candidate, excluding works already in that chain and the other anchor. Expansion ends at four papers or when no eligible neighbor remains. We merge the two chains, deduplicate by work identity, and fill remaining positions with the most topic-relevant unselected mapped works in P. Inputs are ordered by publication year and then arXiv ID.

A.3 Generation and Sampling

The ten open-weight models generate ideas at temperature T=1. Each idea is generated in a separate call and contains five fields: title, motivation, core idea, approach, and validation plan.

For each model, topic, and retrieval strategy, we use ten literature sets and generate five ideas from each set, giving 50 ideas per condition. The No Literature condition generates 50 ideas per model and topic using the same topic-only prompt. Across 15 topics and six conditions, this gives 4,500 ideas per model and 45,000 across the ten open-weight models. Of these, 37,500 use retrieved literature and 7,500 belong to No Literature.

For the expanded attribution analysis, each group contains 20 ideas generated by the same model from the same literature set.

A.4 Evaluation Coverage

Table 3 summarizes evaluation coverage for the open-weight models. Quality is evaluated along the four dimensions defined in Section 3: clarity, feasibility, significance, and validity. Each dimension receives a score from 1 to 10, and their mean gives the overall quality score.

Local and search-assisted global novelty use the five-point rubric described in the same section. Local novelty is assessed relative to the supplied papers. Global novelty is assessed against literature found through web search and also covers No Literature ideas. For judge agreement, participating judges evaluate the same ideas.

A.5 Diversity Measures

We concatenate each idea and embed the text with text-embedding-3-large. For N unit-normalized embeddings {𝐳i}i=1N, diversity is the mean pairwise cosine distance:

For each model, topic, and condition, overall diversity uses all 50 ideas. Within-set diversity is computed among ideas generated from each literature set and then averaged across the ten sets. For No Literature, groups share the same topic-only prompt. We also measure the mean cosine distance between ideas generated from different literature sets.

Additional measures include the mean distance from each idea to its nearest neighbor and the Vendi Score.

A.6 Tracing Literature Influence

We compare the generating model’s predictions for the same idea text with all eight papers present and with each paper omitted in turn. For idea i, paper k, and component c, let Ai,k,c denote the token-averaged JSD score defined in Equation 4. We normalize these scores to obtain the attribution distribution for each component:

when the denominator is positive. The leading paper is the paper with the largest attribution share. For rank profiles, we sort the shares separately for each idea and component before averaging.

To examine variation among ideas generated from the same literature set, we use groups of 20 ideas from the same model. Within each group, we count the distinct leading papers for the core-idea component. For ideas i and j, the total variation distance between their core-idea attribution distributions is

We average these distances across idea pairs within each group and correlate the resulting variation with the group’s diversity, computed over all 20 ideas using Equation 6. We also compute the correlation after subtracting the corresponding model–topic–strategy mean from both quantities.

A.7 Temporal Comparisons and Statistical Analysis

Changes across model releases.

We compare 2025 and 2026 models within each of the four matched families, using the same topics and literature sets. For each retrieval strategy, we average metric differences across the four families and 15 topics. Family-level summaries average across topics and the five retrieval strategies.

For metrics also defined under No Literature, the retrieval effect is the difference between a retrieval condition and the corresponding No Literature baseline. We compare changes in this difference across model releases. Families and topics receive equal weight in aggregate results.

Directions associated with literature sets.

For each literature set, we subtract the mean embedding of the model’s 50 No Literature ideas from the mean embedding of ideas generated from that set. We compare the cosine similarity of these directions across model releases, using either the same literature set or different sets within the same family, topic, and retrieval strategy. This analysis covers the four strategies with sufficient distinct literature sets. Similarities are computed in the original embedding space; PCA is used for visualization.

Uncertainty and multiple comparisons.

We use paired-topic bootstrap resampling to estimate the confidence interval for the change in the retrieval effect on diversity. Each sampled topic retains its linked conditions and model releases. Temporal comparisons use Benjamini–Hochberg correction, with significance defined by adjusted q<0.05.

Global novelty is compared at the model–topic–condition level because the evaluated literature sets are not fully matched across model releases. For correlations, we report 95% confidence intervals before and after subtracting the corresponding model–topic–strategy means.

Appendix B Prompts for LLM-as-a-Judge Evaluation

Figures 7–10 show the prompts used to evaluate quality, local novelty, and global novelty. Double-braced fields mark the inputs supplied for each evaluation. The local novelty prompt follows RINoBench (Schopf and Färber, 2026) and is also used to compare judge scores with the benchmark’s reference scores.

Appendix C Human Evaluation and Agreement Across LLM Judges

C.1 Human Annotation Instructions and Interfaces

Human annotators used the same rubrics and scoring scales as the LLM judges for quality, local novelty, and global novelty. For literature attribution, they identified and ranked retrieved papers according to their connections to each idea component. Figure 11 shows the instructions and annotation interfaces.

C.2 Human Evaluation Results

Three PhD students from our research group, all with backgrounds in NLP, evaluated 20 ideas on NLP. Participation was voluntary, and annotators received no additional compensation. They were informed that their annotations would be used for research and agreed to this use.

For quality, local novelty, and global novelty, we compute Spearman’s rank correlation between the mean human rating per idea and the score from the primary LLM judge used for that metric. Table 4 also reports the mean pairwise correlation among the three annotators.

C.3 Agreement on Quality

We compare GPT-5.6-Sol, Gemini-3.1-Pro-Preview, and Claude-Opus-5 on a balanced sample of 900 ideas spanning ten generating models, 15 topics, and six conditions. All judges evaluate the same ideas using the same rubric.

Table 5 reports quadratically weighted Cohen’s κ and two intraclass correlations: ICC(A,1) measures absolute score agreement, while ICC(C,1) measures consistency after accounting for fixed differences in judges’ mean scores.

C.4 Agreement on Local Novelty

Agreement across judges.

We compare GPT-5.6-Luna, Gemini-3.8-Flash, and Claude-Haiku-4.5 on 750 ideas from ten generating models, 15 topics, and five retrieval strategies. Each judge evaluates the same ideas against the same retrieved papers on a 1–5 scale. Table 6 reports ICC(A,1) and pairwise weighted Cohen’s κ.

Novelty prediction on RINoBench.

We evaluate novelty prediction on the 277-item RINoBench test split (Schopf and Färber, 2026), using its ideas, related papers, and reference scores. Table 7 compares archived benchmark predictions with six GPT-5.6 configurations that vary in model and reasoning effort. These include GPT-5.6-Luna with medium reasoning, which we use for local novelty evaluation.

C.5 API Costs

The total estimated API cost of LLM-based evaluation, including development and validation runs, is $857.93 (Table 8). Costs combine recorded API charges with estimates based on usage and published API prices.²² 2 https://developers.openai.com/api/docs/pricing, accessed September 24, 2026. Reused judgments are counted once.

Appendix D Additional Results Across Model Updates

Table 9 extends the analysis in Section 4.2 across successive model releases. Figure 12 also reports changes in leading-paper counts and reuse.

Retrieval’s diversity advantage narrows after 2025.

For Gemma and Qwen, the mean diversity difference between retrieval and No Literature rises from +0.016 in 2024 to +0.029 in 2025, then falls to −0.005 in 2026 (Table 9). The 2025–2026 decline also appears across all four families, whose mean difference changes from +0.015 to −0.015. Over this update, diversity increases under No Literature while changing little under retrieval, reducing the additional variation supplied by retrieved papers.

The relative quality effect increases from 2025 to 2026.

For Gemma and Qwen, the mean quality difference between retrieval and No Literature changes from −0.129 in 2024 to −0.200 in 2025 and approximately zero in 2026. Across all four families, this difference increases from −0.169 to +0.081 between 2025 and 2026. Mean quality falls by 0.094 under retrieval and by 0.343 under No Literature, yielding a 0.249 increase in retrieval’s relative quality effect. Retrieved literature therefore preserves more of the earlier models’ quality as its diversity advantage declines.

The largest shifts occur in Qwen and GLM.

From 2025 to 2026, Qwen and GLM have the largest estimated decreases in retrieval’s diversity effect (−0.049 and −0.060) and the largest increases in its relative quality effect (+0.396 and +0.415). Gemma’s corresponding changes are −0.017 and +0.005, while Nemotron’s are +0.007 and +0.183.

Changes in leading-paper reuse differ across families.

Qwen’s gains in diversity and local novelty accompany a 13.3-percentage-point decrease in core-idea leading-paper reuse (Figure 12c,d). Gemma’s quality gain accompanies an increase in core-idea reuse of 14.5 percentage points, while its mean number of distinct leading papers per idea falls from 1.94 to 1.66. GLM shows less reuse and more leading papers per idea, alongside higher local novelty. Qwen and Nemotron show little change in their leading-paper counts, despite opposite changes in both reuse and diversity: reuse decreases and diversity increases for Qwen, whereas reuse increases and diversity decreases for Nemotron.

Appendix E Detailed results on diversity, quality, and semantic distance

We examine diversity across idea components, quality scores by dimension, and semantic distances to the input literature across the four 2025 models and 15 research topics. No Literature denotes generation without retrieved literature. R1–R5 denote Semantic Top-K, Most-Cited Top-K, Semantic + Entity + Co-citation, Core-Paper Reference Expansion, and Chain of Ideas, respectively.

E.1 Diversity across idea components

Overall diversity exceeds within-set diversity in every component.

This pattern holds for all five retrieval strategies across titles, motivations, core ideas, approaches, and validation plans (Figure 13). R4 has the largest gap between the two measures: 0.084 when using the title and core idea together, and 0.056–0.095 across individual components. For motivations, its within-set and overall diversity are 0.274 and 0.369, respectively.

R2 produces the highest overall diversity across components.

R2 ranks first for titles (0.507), motivations (0.396), core ideas (0.473), approaches (0.455), and validation plans (0.417). R5 exceeds No Literature in overall title diversity (0.450 vs. 0.431), while its other four components remain at or below their No Literature means.

E.2 Quality across evaluation dimensions

Significance scores are higher and feasibility scores are lower with retrieval.

Every retrieval strategy has a higher mean significance score than No Literature, increasing from 7.05 to 7.13–7.29, while mean feasibility falls from 6.06 to 5.47–5.84 (Figure 14). R5 shows this pattern most clearly: its mean significance is 7.28, while its feasibility is 5.47, the lowest among the six conditions.

R2 has the highest mean composite quality.

R2 achieves a composite score of 5.91, compared with 5.83 for No Literature. It also has the highest mean validity (4.45), significance (7.29), and clarity (6.08). Mean composite scores range from 5.72 to 5.91 across retrieval strategies.

E.3 Semantic distance to the input literature

R2 yields the largest distances across all three measures.

The three measures give the same strategy ranking: R2, R3, R1, R4, and R5 (Figure 15). Under R2, mean distances are 0.341 to the literature-set centroid, 0.367 to the nearest paper, and 0.517 when averaged over the input papers. R5 has the smallest corresponding distances: 0.308, 0.333, and 0.462.

Appendix F Results of closed-source models

Table 10 reports quality, diversity, and novelty for the three closed-source models in the 2026 evaluation under no-lit and R1–R5. GPT-5.6 Luna achieves the highest aggregate quality under every strategy (6.94–7.17). GPT-5.6 Luna and Gemini 3.5 Flash-Lite attain their highest aggregate quality under no-lit (7.17 and 6.33), while Claude Haiku 4.5 performs best under R4 (6.33).

R3 yields the highest local novelty for all three models, tying with R4 for Claude Haiku 4.5. The highest overall diversity occurs under R4 for GPT-5.6 Luna (0.342), R1 for Gemini 3.5 Flash-Lite (0.400), and R2 for Claude Haiku 4.5 (0.378). Thus, the strategies associated with the highest quality, diversity, and novelty differ across models and metrics.

Appendix G Reasoning-Enabled Ideation

G.1 Setup

We compare the four 2026 models with and without reasoning, using matched topics, literature sets, and prompts. Both modes use T=1; reasoning uses top-p=0.95 and an 8,192-token cap, and the baseline uses default top-p and a 3,600-token cap. Evaluation samples are paired by model, topic, condition, and sample index (Table 11). Local novelty uses a balanced subset comprising the first two literature sets and five ideas per set. Global novelty uses the first four jointly evaluated sample indices per cell for the first two topics in Table 2. Metrics and judges follow Section 3.3, with title and core-idea embeddings as in Appendix E.1. Means weight models and topics equally; confidence intervals use 10,000 paired topic-bootstrap resamples.

G.2 Results and Implications

Reasoning does not remove the need to vary the literature.

With reasoning enabled, all five retrieval strategies reduce within-set diversity relative to No Literature by 17.1%–24.5% (Figure 16a). Pooling ideas from different literature sets gives R1–R4 higher mean diversity than No Literature by 2.5%–5.9%; R5 remains 6.6% below it. The gap between overall and within-set diversity increases under every retrieval strategy when reasoning is enabled, from 0.077 to 0.090 on average across R1–R5. R4 retains the largest gap (0.101 without reasoning; 0.110 with reasoning). This separation also holds across all five individual idea fields, with gaps of 0.039–0.119 under reasoning. Thus, the concentration of ideas around a fixed literature context persists even when the model reasons before answering. For an ideation pipeline seeking broader exploration, enabling reasoning leaves literature selection as an important design choice: different paper sets provide variation that repeated generation from the same papers does not recover.

Better individual proposals need not form a more diverse collection.

Quality increases for all four models (Table 12), with the mean rising from 5.711 to 6.038 (Δ=+0.328; 95% CI [0.291,0.365]). Clarity contributes the largest improvement (+0.633), followed by validity (+0.314), significance (+0.216), and feasibility (+0.148). Yet mean overall diversity falls from 0.424 to 0.413 across the six conditions. This contrast appears in three of the four models: Qwen, GLM, and Nemotron gain quality while losing diversity. Gemma gains both, so the trade-off is not universal. The observation separates proposal development from exploration: higher-rated individual ideas can still cover a narrower range of directions. Reasoning should therefore be evaluated using both individual-idea scores and collection-level diversity; an increase in quality alone does not establish a broader scientific search.

Local novelty gains do not establish novelty against the broader literature.

Local novelty increases for all four models, from 3.164 to 3.374 on average (Δ=+0.209; 95% CI [0.170,0.246]). The two-topic search-assisted global-novelty evaluation shows a different pattern: the mean changes from 2.286 to 2.214, with an increase for Qwen (+0.250) and decreases for Gemma (−0.271), Nemotron (−0.104), and GLM (−0.167). The local result shows greater novelty relative to the supplied papers.

Appendix H Retrieval Effects Across Topics

R2 achieves the highest mean diversity in 10 of the 15 topics and the highest mean quality in five, with No Literature included in both comparisons. Figures 17 and 18 report topic-level scores and changes relative to the corresponding baselines, averaged across the four 2025 models.

Diversity–quality trade-offs differ across topics.

In scientific and cognitive applications, all five retrieval strategies reduce diversity and increase quality relative to No Literature. R3 achieves the highest quality, with a gain of 0.538 and an increase in feasibility of 1.00 point. In reinforcement learning, all five strategies increase diversity and reduce quality, with feasibility decreasing by 0.80–1.25 points. R4 achieves the highest local novelty and quality among the retrieval strategies in this topic, but its quality score remains 0.275 below No Literature.

Local and global novelty yield different strategy rankings.

In representation, transfer, meta, and continual learning, R2 scores higher than R5 on local novelty (3.190 versus 2.795). The ordering reverses in the exploratory global novelty evaluation, where R5 scores 2.50 and R2 scores 1.75. The two metrics use different references: local novelty compares each idea with its supplied literature set, while global novelty uses literature found through web search. R2 also has higher quality than R5 in this topic (6.05 versus 5.58).

Appendix I Checking Paper Attribution

We examine whether these attribution scores provide meaningful information about the relationship between retrieved papers and generated ideas, using human judgments and model-reported sources as complementary evidence. We also examine disagreements and discuss why evaluating attribution requires distinguishing different kinds of paper contributions.

How we compute attribution.

For each generated idea, we remove one input paper at a time and measure how the model’s predictions change when scoring the same idea text. Larger changes give higher attribution scores. We compute scores from content tokens and express them as percentages that sum to 100% across the eight input papers within each of the idea’s four parts.

I.1 Comparison with Human Judgments

The Spearman correlation between attribution scores and human ratings is 0.454. For comparison, the correlation between human raters is 0.360, computed on the same samples using the same procedure. Agreement is therefore limited, including among human raters.

These results motivate a finer-grained classification of what each paper contributes. Our scores distinguish papers and idea parts, but do not explicitly distinguish contribution types: a paper may motivate a problem, supply a method, or inform evaluation. Human raters may also weigh these roles differently. Evaluating these contributions separately could help explain the disagreement.

I.2 Reported Sources Usually Include the Highest-scoring Paper

We also compare attribution with sources reported by the model, either after it has generated an idea or while it is generating one. For each idea part, we check whether the reported sources include the paper with the highest attribution score.

When sources are requested after generation, this paper is reported in 77.58% of idea parts. Using another idea’s source report instead lowers this proportion to 52.97% (Table 13). The comparison uses ideas generated from the same eight input papers and matches corresponding parts.

The 24.61-percentage-point difference shows that the correspondence depends on the specific idea, beyond a tendency to report the same papers across ideas. The difference is positive for all four idea parts. Requesting sources during generation produces a similar pattern: 71.89% with the idea’s own report, compared with 48.36% with another idea’s report.

I.3 Why Attribution and Reported Sources Can Disagree

A relevant method paper can be omitted.

In Figure 19a, GLM-4 proposes using Indra representations and relationships between samples to align outputs across modalities. P6, The Indra Representation Hypothesis for Multimodal Alignment, describes this method and receives 82.73% of the attribution score. Yet the model reports P4, P7, and P8, which concern image generation and visual reasoning and together receive only 10.25%. This case suggests that attribution can identify a method-level connection missed by the model’s source report.

Almost equal scores can count as a disagreement.

In Figure 19b, Nemotron-3 proposes adjusting a decision threshold as prediction error is estimated. The idea combines elements of P3, which studies robust threshold selection, and P4, which studies errors between predicted and actual distributions. P4 receives 36.73% of the score and P3 receives 36.63%. The model reports P2 and P3, but not P4. Our comparison therefore counts this as a disagreement, even though the reported P3 scores only 0.10 percentage points below P4.

These examples show why checking only the highest-scoring paper gives an incomplete picture. Understanding a disagreement also requires examining what the papers contribute and how far apart their scores are.

I.4 Generate Ideas with Citations

We also examine whether requesting sources during generation changes the ideas themselves (Table 14). The average number of distinct highest-scoring papers across an idea’s four parts increases from 1.750 to 2.014. Ideas generated from the same inputs also become more similar: their average distance falls by 10.02%. However, neither difference is statistically significant across the five topics (both Holm-adjusted p=0.375).

Appendix J Licenses and Artifact Release

The Qwen checkpoints and Gemma 4 used in this study are distributed under the Apache License 2.0, and the GLM checkpoints under the MIT License. Gemma 2 and Gemma 3 are distributed under the Gemma Terms of Use. Nemotron 3 Nano and Nemotron Cascade 2 are distributed under the NVIDIA Nemotron Open Model License and the NVIDIA Open Model License, respectively.

Paper titles and abstracts were obtained from arXiv, whose metadata are available under the CC0 1.0 Universal Public Domain Dedication.³³ 3 https://info.arxiv.org/help/license/index.html This metadata license does not extend to article full texts, which remain subject to their individual licenses. Citation metadata were obtained from Semantic Scholar (Kinney et al., 2023). Access to its API is governed by the Semantic Scholar API License Agreement, while the supplied data and any underlying third-party content remain subject to their applicable licenses.⁴⁴ 4 https://www.semanticscholar.org/product/api/license

Appendix K A Case Study of Literature Attribution

We present five ideas generated by Gemma 4 (31B) from the same eight-paper literature set using Most-Cited Top-K. Figure 20 shows their component-level attribution. We examine Ideas 3 and 5 in detail; Figures 21 and 22 show their complete text. Figures 23 and 24 reproduce the input abstracts of LangFlow (Chen et al., 2026a) and TraceRouter (Shi et al., 2026).

Manual annotation.

Blue and gold highlights mark manually identified textual correspondences with LangFlow and TraceRouter, respectively.

Different papers lead different components of Idea 3.

Idea 3 proposes path-level control of continuous diffusion language generation. Its core idea explicitly connects “path-level intervention techniques” to TraceRouter and applies them to “ODE trajectories of continuous diffusion models.” The approach proposes combining a flow-matching model, sparse autoencoders, and a trajectory router to suppress selected paths during sampling.

TraceRouter leads the core idea (67.6%) and approach (60.1%), while LangFlow leads motivation (52.4%) and validation (65.5%). The highlighted passages connect the intervention method to TraceRouter and the continuous-generation setting to LangFlow. The validation component also names perplexity and generative perplexity, both reported in the LangFlow abstract.

The leading paper differs across ideas.

TraceRouter leads the core idea in Ideas 1–4. In Idea 5, LangFlow accounts for 72.1% of the core-idea attribution, and TraceRouter accounts for 13.3%.

Idea 5 proposes semantic steering with the Fisher information metric along a continuous diffusion trajectory. It explicitly names LangFlow in both motivation and approach. The highlighted correspondences with LangFlow concern the continuous-generation setting and perplexity-based evaluation.

注意转载授权 · 阅读原文