ReviewPilot：面向系统综述筛查与数据抽取的多智能体系统

Zhaoqian Xue ^††thanks: Corresponding authors: Zhaoqian Xue and Lingyao Li.    Shaoze Zhou Affiliation: Florida International University    Xuexin Li Affiliation: Vanderbilt University    Junting Mu Affiliation: University of California, Irvine    Haozheng Liu Affiliation: New York University    Zhongyin Wang Affiliation: University of Illinois Urbana-Champaign    Yichun He Affiliation: Broad Institute of MIT and Harvard    Renkai Ma Affiliation: University of Cincinnati    Hao Zhou Affiliation: University of South Florida    Sarah Bratt Affiliation: University of Arizona    Siyuan Ma Affiliation: Vanderbilt University Medical Center    Chen Chen Affiliation: Florida International University    Lingyao Li²²footnotemark: 2 Affiliation: University of Arizona

Abstract

Systematic literature reviews follow the PRISMA process of screening scholarly works and extracting evidence from included studies. Deep research tools based on large language models (LLMs) can search and summarize the literature but do not record which studies are excluded or where extracted values come from. We develop a conversational multi-agent system called ReviewPilot that automates the PRISMA workflow from paper screening to information extraction. Researchers can tailor criteria and schemas through dialogue, and a project memory keeps their configurations and revisions. Each screening decision comes with its reason and each extracted value with its source. On epidemiological expert reviews, ReviewPilot outperforms a single-prompt LLM and AgentSLR across three LLMs, with screening macro F1 up to 0.776 and extraction F1 up to 0.788. Case studies in biomedicine and human-computer interaction show close agreement with human annotators, suggesting that ReviewPilot can transfer beyond epidemiology. Code is available at this link.

1 Introduction

Systematic literature reviews depend on a sequence of judgments, including eligibility check, full-text retrieval, and information extraction (Higgins et al., 2023; Page et al., 2021). Such a process if labor-intensive. Searches often result in high recall, so reviewers need to read thousands of titles and abstracts to find a few eligible studies (van de Schoot et al., 2021). A review can take over a year on average from registration to publication (Borah et al., 2017). Automation can reduce this workload, but researchers still need to inspect decisions and document the process based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 process (Page et al., 2021).

Large language models (LLMs) are now widely used to search and summarize literature. Deep research tools such as ChatGPT (OpenAI, 2025), Gemini (Citron, 2024), and Perplexity Research (Perplexity, 2025) can generate a synthesized report. These tools help researchers explore a topic but do not follow the PRISMA process. They do not show which papers are screened or excluded and why, or where each extracted value comes from. Other review tools only limit these decisions to a single stage, such as ASReview (van de Schoot et al., 2021) and LGAR (Jaumann et al., 2025) for screening, and RobotReviewer (Marshall et al., 2017) for extraction, or an agent-based system for a few stages like AgentSLR (Padarha et al., 2026).

However, designing an AI-assisted literature review workflow involves two key considerations. First, screening criteria and field definitions need operational instructions. Domain terms and borderline cases need to be iteratively interpreted consistently for screening and coding. Second, outputs must reflect the current configuration. Researchers refine criteria and fields as a review progresses (Higgins et al., 2023); confirmed settings and corrections must therefore be retained, and affected downstream outputs identified. We implement these requirements within a workflow that records the counts and inclusion-exclusion criteria needed for PRISMA-guided workflow.

We present ReviewPilot, a conversational multi-agent system that addresses these challenges from search through information extraction (Figure 1). Researchers define eligibility criteria and an extraction schema through natural dialogue with the system, and a Prompt Agent turns them into review-specific guidance. Agents screen records and extract information, delivering screening decisions with their reasons and extracted values linked to source passages in the papers. The result is a step-by-step record of how papers are filtered, with the records kept and excluded at each stage, and a table of the information extracted from each retrieved paper. Our contributions are as follows.

• •

  PRISMA-guided workflow. ReviewPilot coordinates search, screening, retrieval, and extraction while recording counts, exclusion reasons, and available source for inspection (Section 3).

• •

  Review-specific configuration and revision. Researchers can set criteria and schemas through dialogue. Then, agents draft screening guidance and coding rules. Researchers can edit these instructions and confirm. Project memory design further retains these configurations and corrections (Sections 3.2 and 3.6).

• •

  System-level evaluation. On PERG, ReviewPilot achieves higher screening macro F1 and extraction F1 than a single-prompt LLM and AgentSLR under the reported settings (Tables 1 and 2). Two case studies document completed workflows and screening agreement with human annotators (Section 5).

2 Related Work

LLM multi-agent systems.

Multi-agent frameworks divide tasks among LLM agents with distinct roles (Guo et al., 2024b). For instance, AutoGen supports configurable conversations that combine LLMs, tools, and human input (Wu et al., 2024). MetaGPT encodes standard operating procedures as prompt sequences to coordinate software development and check intermediate results (Hong et al., 2024). Our system, ReviewPilot, leverages the potential of multi-agent workflow to review stages.

Deep research and literature synthesis. Deep research tools in ChatGPT (OpenAI, 2025), Gemini (Citron, 2024), and Perplexity (Perplexity, 2025) iteratively search and analyze sources to produce reports. ChatGPT or Gemini can link citations and allows users to revise its research plan. For scientific literature, OpenScholar combines passage retrieval, cited generation, and iterative self-feedback (Asai et al., 2026), while AutoSurvey retrieves papers, constructs an outline, and delegates subsection drafting to specialized LLMs (Wang et al., 2024b). These systems can synthesize reports but do not document the paper filtering process based on PRISMA 2020 (Page et al., 2021).

Systematic review tools. Some tools support individual review stages. ASReview prioritizes records for human screening through active learning from relevance judgments (van de Schoot et al., 2021); LGAR ranks abstracts zero-shot using graded, criterion-based LLM scores and dense reranking (Jaumann et al., 2025). RobotReviewer extracts trial characteristics and supports risk-of-bias assessment with sentences linked to the source document (Marshall et al., 2017).

Integrated workflows extend assistance across stages. AgentSLR pairs an epidemiological literature review workflow with stage-specific evaluation and describes an interface for expert verification of extracted values (Padarha et al., 2026). Its published workflow uses predefined eligibility criteria and domain-specific extraction schemas. We compare against it in Section 4. Ren et al. (2026) translate user criteria into screening rules and use multi-agent cross-review for clinical-trial screening. Their extraction workflow combines source-derived table standardization, iterative correction, and targeted human review. The commercial Elicit workflow proposes editable criteria and extraction fields and lets users return to earlier steps (Byun, 2025); it also provides evidence quotes, exclusion reasons, and PRISMA 2020 flow diagrams (Vardhaan and Mohanty, 2026). These workflows already combine stage automation, customization, evidence access, and human oversight.

To address these gaps, we develop ReviewPilot, a PRISMA-guided multi-agent system that coordinates literature search, screening, full-text retrieval, and information extraction. Researchers can tailor eligibility criteria and extraction schemas through dialogue, while agents generate screening guidance and per-field coding rules. Project memory retains configurations and corrections to support revision across stages (Sections 3.2 and 3.6). We evaluate the complete system on PERG and examine its workflows and screening agreement with human annotators in two case studies.

3 ReviewPilot Development

ReviewPilot is a conversational multi-agent system that covers five stages of a systematic literature review: search, screening, full-text retrieval, information extraction, and semantic categorization (Figure 2). Researchers can configure the review through dialogue or direct editing, inspect each stage’s outputs, and can override screening decisions or correct extracted values.

3.1 Architecture and Orchestration

A Lead Agent coordinates four specialized agents and two tools through action contracts (Appendix A.1). The Search Condition Agent designs the search strategy, and the Prompt Agent writes review-specific instructions (Section 3.2). The Screening Agent deduplicates and assesses records, and the Extraction Agent extracts structured information. The Literature Search Tool executes queries, and the PDF Download Tool retrieves full texts. The Lead Agent categorizes values itself.

The Lead Agent starts a stage only when its prerequisites are met and the outputs it depends on are current. It reports completion only after checking the expected artifacts and their required fields. Agent skills are structured Markdown instructions specifying procedures for search strategy, screening, prompt design, extraction, and categorization. A fixed mapping appends the matching skill to the LLM system prompts for these tasks (Appendix G).

3.2 Review-Specific Prompt Design

The Prompt Agent writes the review-specific parts of the screening and extraction instructions, following the prompt-design skill. After the eligibility criteria are confirmed, it writes screening guidance 𝒢 from the review objectives, criteria, search concepts, and up to 200 candidate records (titles and abstract openings). This guidance defines the review focus, domain terms, include and exclude categories, and tie-breaking rules. After screening, it drafts an extraction schema from the objectives, the requested fields, and the abstract openings of up to three included papers. Once the schema is confirmed, it writes the coding rules 𝒞 from up to 25 included papers, using their abstract openings and full-text sentences selected for each field.

The guidance and the coding rules are each drafted and then revised in a second LLM pass against the skill’s checklist (Madaan et al., 2023). Reviewer decisions and reference extraction values are not among the generation inputs. Researchers can edit the guidance, schema, and rules and must confirm each before the stage that uses it runs (Appendices G.3, G.4.1, and G.5.1).

3.3 Search & Screening

Search.

The Search Condition Agent identifies the concepts required for eligibility and proposes alternative terms for each concept. It builds a Boolean query by joining alternatives with OR and required concept groups with AND (Appendix A.2). Analytical dimensions that do not determine eligibility are omitted from the required groups. Researchers refine the concepts and query through dialogue or direct editing and review the sources, date range, and per-source limits. The Literature Search Tool then searches titles and abstracts in PubMed, arXiv, and OpenAlex. The saved query, settings, counts, and errors document the search as run.

Screening. After date filtering and deduplication (Appendix A.3), the model assesses each record pi, its title and available abstract, against the confirmed inclusion criteria ℐ, exclusion criteria ℰ, and screening guidance 𝒢:

where ri is a one-sentence rationale naming the deciding rule and ℓi is a qualitative likelihood label. Its five levels are definitely exclude, probably exclude, uncertain, probably include, and definitely include. The screening skill instructs the model to assess the study’s own work against the review focus and criteria, not topics mentioned only as background (Appendix G.4). Among these five levels, a record is excluded only when ℓi is probably exclude or definitely exclude. Records with the other labels are retained for full-text review. Records are also retained when the model call fails. The prompt requests the deciding rule and a verbatim title or abstract excerpt. An exclusion with an unmatched rule or unlocated excerpt is flagged for human review, but the record remains excluded.

3.4 Paper Retrieval & Information Extraction

Paper retrieval.

The PDF Download Tool tries a cascade of publisher, open-access, and preprint sources for each retained record. It checks PDF validity and compares opening-page text with the title. These heuristics do not guarantee retrieval success (Appendix A.4), so it generates a report to record the success or failure retrieval for each paper, and researchers can add missing PDFs manually.

Information extraction. The confirmed schema 𝒮={(fj,τj,δj)}j=1k defines each field by its name, type, and description (Appendix A.5). The coding rules 𝒞 specify how borderline values are coded. ReviewPilot places a full-text eligibility field first in the drafted schema, with options from the confirmed criteria. The model judges whether the paper remains eligible and names the criterion behind an exclusion. In a PDF assessment, an empty or unrecognized eligibility value does not exclude the paper. Excluded papers are omitted from categorization, and their exclusion reasons are recorded for PRISMA reporting.

For each available PDF, the Extraction Agent sends up to the first 400,000 characters of its text, with page markers, together with the schema and coding rules to the model (Appendix G.5). For each field, the prompt requests a value vj, verbatim excerpts, and page numbers. It requires an empty or null value when the supplied text neither states nor necessarily implies a value. Responses are checked for field presence, types, and value constraints. A failed check is recorded as an extraction failure for that paper. Passing these checks does not establish that the values are correct. ReviewPilot searches the normalized PDF text for the returned excerpts. A field with at least one located excerpt is marked verified, and the first located excerpt is stored with the page on which it was located. Otherwise, the field is unverified. Excerpt location establishes textual presence, not semantic support for the value or coverage of every value in a multi-value field.

For papers whose PDF was not retrieved, researchers can enable assessment from the abstract. When the selected model supports web search, it can seek web evidence for fields the abstract leaves open. Accepted values require located excerpts, and web pages must pass paper-identity checks. These outputs retain source labels and any source URLs without PDF page numbers and are reported separately from full-text assessments. Fields without evidence remain unavailable, and papers whose eligibility remains unavailable are not categorized.

3.5 Semantic Categorization

The Lead Agent categorizes extracted values only for papers retained after eligibility check. For a selected field, it proposes a set of categories with short descriptions (Appendices A.6 and G.6). The plan supports single-label or multiple-label assignment, and an Other category for values that do not fit. Researchers review and edit the plan before the Lead Agent assigns labels to each paper. Papers with no value for the field are marked unavailable.

3.6 Memory Design

Researchers work in a conversation panel and a stage-specific canvas. A chat request that changes criteria or the schema is saved as a draft, and confirmation is a canvas action (Appendices A.7 and H). Project memory stores the conversation history, current configurations, confirmed configuration snapshots, and each stage’s status and PRISMA counts.

In conversation, the Lead Agent receives its task instructions, the current request, the saved project state, and the conversation history. Its instructions give saved state precedence over superseded requests and distinguish drafts from confirmed configurations. Researchers reuse a previous project’s configuration by importing it as a draft that again requires confirmation. The system does not retrieve memories across projects automatically or learn preferences from corrections.

To override a screening decision, researchers select a confirmed criterion or guidance rule and give a reason. They also give reasons when correcting extracted values. Original decisions and values remain in review history. A change to a confirmed configuration or reviewed record marks affected downstream outputs as outdated until rerun.

4 Experiments & Results

We evaluate screening and structured information extraction separately against the expert annotations in PERG, a dataset released with AgentSLR (Padarha et al., 2026). Extraction uses the same expert-included papers for all methods, independently of each method’s screening predictions.

4.1 Experiment Setup

Data and reference annotations.

PERG is released under the CC BY 4.0 license. Its annotations originate from systematic reviews conducted by the Pathogen Epidemiology Review Group at Imperial College London for WHO priority pathogens (Cuomo-Dannenburg et al., 2024; Nash et al., 2024; Doohan et al., 2024; Morgenstern et al., 2025; McCain et al., 2025; Naidoo et al., 2026). Screening uses 5,614 title–abstract records for Lassa, Nipah, SARS, and Zika, with 854 expert inclusions (15.2%). All methods apply the five inclusion and six exclusion criteria specified in AgentSLR’s PERG template (Appendix B). All records are evaluated without date filtering or deduplication.

Extraction uses 178 annotated papers (79 for Ebola, 6 for Lassa, 36 for SARS, and 57 for Zika). We evaluate five fields: model type, stochastic or deterministic status, code availability, transmission routes, and interventions (Table B.2). PERG annotates each transmission model in a paper. Reference values are aggregated to the paper level by taking the union across models. Code availability is true if the paper reports available code for any model.

Models and evaluation metrics. We use gpt-6-luna through the OpenAI API,²² 2 https://developers.openai.com/api/docs and DeepSeek V4.1 Flash and GLM-5.3-Flash through Together AI.³³ 3 https://docs.together.ai/intro The single-prompt LLM uses one instruction containing the screening criteria, or the study objective and field definitions for extraction. It retains uncertain screening cases and requests field-level excerpts for extraction, without ReviewPilot’s skills, coding rules, or schema validation (Appendix C).

We rerun AgentSLR using its public screening implementation and transmission-model extraction pipeline. Its prompts, parser, and internal full-text screening are preserved, as are the tool and provenance loops of its extraction stage. It reads the same PDF text as the other methods. We adapt AgentSLR’s API client and replace its original high reasoning setting (Table D.3). Extraction and gpt-6-luna screening use medium reasoning. Screening with the Together AI models uses provider defaults. Papers rejected by its internal screening remain in the evaluation with empty predictions.

ReviewPilot follows the screening and extraction procedures in Sections 3.3 and 3.4. It uses a fixed five-field extraction schema, with review-specific screening guidance and coding rules generated by the Prompt Agent. All methods receive the same papers and use the same reference annotations.

Prompt Agent uses the same model as its downstream stage. The agent receives the review objectives, criteria or field definitions, and samples of evaluation records or papers. Expert decisions and reference extraction values are excluded from the agent’s inputs. ReviewPilot’s screening procedure and prompt-design skill are refined using evaluation scores from these PERG pools. The pools are therefore not held out from method development.

We report one complete run per method and model. Decoding settings and output limits differ across methods and are recorded with each run (Appendix D). Transport errors and incomplete responses trigger retries. ReviewPilot also retries unparseable screening outputs. GLM-5.3-Flash exhausted its 65,536-token output limit on one paper. That paper is excluded from all three methods’ results on this model, leaving 177 papers. All table entries are single-run point estimates and do not characterize variation across repeated model runs.

Screening metrics. We pool the records and compute precision, recall, and F1 separately for the include and exclude classes. Each metric is then macro-averaged over the two classes. The review share, (TP+FP)/N, is the proportion of records retained for full-text review, where TP and FP denote retained expert-included and expert-excluded records, and N is the pool size. Macro recall averages recall for both classes and does not measure include-class recall alone.

Extraction metrics. Predicted and reference values are normalized with a fixed synonym mapping shared by all methods. At the paper–field level, matching values count as true positives, additional predicted values as false positives, and reference values absent from predictions as false negatives. Under AgentSLR’s empty-field convention, a field empty in both prediction and reference counts as one true positive. This convention applies to model type, stochastic or deterministic status, and code availability. For paper-level scoring, AgentSLR’s per-model predictions are aggregated identically to reference values. Its native model-matching evaluation is not used. Counts are pooled across papers and the five fields to compute precision, recall, and F1. AgentSLR’s additional fields are not scored.

The verified-correct rate is computed over paper–field pairs with a reference value. A pair counts as verified-correct when its complete normalized prediction matches the reference set and at least one returned excerpt is located in the paper. All methods use the same excerpt matcher. Excerpt location does not establish semantic support for the prediction (Gao et al., 2023) or coverage of every value in a multi-value field.

Cost estimation. We estimate API costs from returned token usage and the prices in Table D.1. Costs are reported in USD per 1,000 screened records or per extraction paper. Screening costs include ReviewPilot’s guidance generation, whereas extraction costs exclude coding-rule generation.

4.2 Screening Results

ReviewPilot has the highest screening macro precision, recall, and F1 on each model in Table 1. Its macro F1 exceeds AgentSLR’s by 5.3–10.3 points and the single-prompt LLM’s by 7.0–19.0 points. ReviewPilot retains 26–29% of records, versus 33–42% for AgentSLR and 35–54% for the single-prompt LLM. The lower share describes the number of records sent to full-text review. It does not establish that fewer expert inclusions are missed or that human review time decreased. ReviewPilot’s reported API cost is lower than AgentSLR’s on each model, while the single-prompt LLM has the lowest reported API cost.

4.3 Extraction Results

ReviewPilot has the highest pooled precision, recall, and F1 on each model in Table 2. Its F1 exceeds AgentSLR’s by 3.7–4.7 points and the single-prompt LLM’s by 1.5–1.7 points. ReviewPilot’s verified-correct rate reaches 0.703–0.714, compared with 0.405–0.449 for AgentSLR and 0.528–0.561 for the single-prompt LLM. More paper–field pairs therefore combine a complete normalized match to the reference set with at least one excerpt located in the paper. Semantic verification is still required to assess whether the excerpts support the predicted values. ReviewPilot’s reported extraction cost is lower than AgentSLR’s and higher than the single-prompt LLM’s on each model. Appendix E gives the screening results by pathogen and extraction results by field.

5 Case Studies

We apply ReviewPilot to reviews of LLM applications in biomedicine and human–computer interaction (HCI).

5.1 Complete Review Workflows

Both projects used gpt-6-luna and searched PubMed, arXiv, and OpenAlex, taking up to 1,000 results per source. Searches covered publications dated 1 January 2023 to 1 October 2026. Figure 3 and Figure F.1 in Appendix F.4 trace the workflows, outputs, and actions.

HCI.

The 2,241 records yielded 752 categorized studies (Figure 3); 153 reports were not retrieved, and full-text assessment excluded 75 of 827 reports. The researcher reinstated three excluded records, including a system evaluation of an interactive tool. The examples contrast this reversal with a rapid-review exclusion and pair GPT-4 and 43 participants with source excerpts. Before categorization, the researcher regrouped the proposed application categories. Studies could receive multiple labels.

Biomedicine.

The 3,000 records yielded 1,341 categorized studies (Figure F.1). Screening retained 1,705 records, including four exam evaluations reinstated by the researcher. Of these, 336 reports were not retrieved and seven not extracted. Full-text assessment excluded 21 of 1,362 reports. The examples show a survey exclusion, a reinstated exam evaluation, and extraction of GPT-4o and expert evaluation from a radiology study. The researcher kept the five proposed task categories. ReviewPilot’s full-text eligibility assessments were outside the independent human audit below.

5.2 Human Agreement and Inspection

Screening.

For each topic, two annotators from our research team with expertise in HCI or biomedicine independently labeled titles and abstracts without seeing ReviewPilot’s output (criteria in Appendix F). Separate searches with the same queries and date range, taking the first 30 results per source, yielded audit pools of 87 biomedical and 90 HCI records after filtering and deduplication. ReviewPilot screened each pool in a separate run with gpt-6-luna. Its κ against each annotator exceeded inter-annotator agreement in biomedicine and, in HCI, fell on either side of it (Table F.1).

Extraction inspection.

Source checks in the complete workflows corrected three field values per topic (Table F.2). HCI corrections restored participant counts and distinguished human experiments from assessments of agents. Biomedical corrections removed unsupported task, data-source, or adaptation labels. Missing excerpts were added. The selected inspections provide qualitative error analysis. Appendix F gives sampling and schemas.

6 Discussion and Conclusion

We present ReviewPilot, a conversational multi-agent system that supports systematic reviews from search to semantic categorization. Researchers confirm criteria and schemas, a Prompt Agent writes review-specific guidance and coding rules for applying them, and a project memory retains configurations and corrections. Across three LLMs, ReviewPilot achieves higher screening macro F1, pooled extraction F1, and verified-correct rates on PERG than a single-prompt LLM and AgentSLR.

These gains should be read with care. PERG pools also inform ReviewPilot’s development, so the results do not establish performance on unseen reviews. A located excerpt shows that the quoted text appears in the paper, but whether it supports a decision or value still requires human judgment.

The case studies show why memory design matters. When researchers check flagged and sampled outputs, they reverse some screening exclusions and correct some field values, and the system keeps each change with its reason. ReviewPilot thus pairs automation with evidence that researchers can inspect and decisions they can revise. Future work should measure how these features affect review effort and decision quality, and extend the evaluation to reviews with other evidence types.

Limitations

Broad validation across reviews, evidence types, and review teams remains incomplete. ReviewPilot’s screening procedure and prompt-design skill are refined using scores from the same PERG pools used for evaluation, so the reported gains do not establish performance on unseen reviews. Expert decisions and reference values are excluded from the Prompt Agent’s inputs. Quantitative extraction covers five fields from epidemiological transmission models. The case-study screening audits use small, non-random pools labeled by our research team, and the selected extraction inspections provide qualitative error analysis. Full-text eligibility decisions are not independently audited. We report one run per method and model without confidence intervals. These estimates do not characterize variation across repeated runs. Macro-averaged screening metrics do not isolate include-class sensitivity, which matters for avoiding missed studies (Cohen et al., 2006; O’Mara-Eves et al., 2015). AgentSLR’s original high reasoning setting is replaced with medium reasoning or provider defaults. Excerpt verification establishes textual presence, but does not assess semantic support or whether excerpts cover every value in a multi-value field (Gao et al., 2023).

Search coverage is limited to PubMed, arXiv, and OpenAlex and constrained by the query and per-source result limits. Full-text extraction depends on successful PDF retrieval. Researchers can add PDFs manually, but the reported workflows do not separately quantify retrieval outcomes by publication access status. The contribution of subscription barriers to unretrieved reports therefore remains unquantified in the reported results. Access constraints may affect which studies enter extraction and categorization. Assessments based on abstracts or web sources are not separately validated. These sources may lack the evidence needed to populate every field.

Future evaluations should use reviews held out from method development, cover further evidence types and extraction schemas, and independently audit full-text eligibility and semantic support for extracted values. Repeated runs should report uncertainty and include-class recall. Prospective studies with review teams are needed to assess reviewer effort and decision quality. Retrieval evaluations should report outcomes by access status and assess how manual PDF submission and fallback assessment affect evidence coverage.

ReviewPilot also relies on hosted models whose behavior may change across versions (Chen et al., 2024). Its outputs require human verification, and the system is intended to support the independent review procedures recommended by Cochrane (Higgins et al., 2023).

References

• Asai et al. (2026) Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’Arcy, David Wadden, Matt Latzke, Jenna Sparks, Jena D. Hwang, Varsha Kishore, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, and 9 others. 2026. Synthesizing scientific literature with retrieval-augmented language models. Nature, 650(8103):857–863.
• Borah et al. (2017) Rohit Borah, Andrew W. Brown, Patrice L. Capers, and Kathryn A. Kaiser. 2017. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open, 7(2):e012545.
• Byun (2025) Jungwon Byun. 2025. Introducing Elicit systematic review. Elicit. Published February 20, 2025. Accessed October 11, 2026.
• Chen et al. (2024) Lingjiao Chen, Matei Zaharia, and James Zou. 2024. How is ChatGPT’s behavior changing over time? Harvard Data Science Review, 6(2).
• Citron (2024) Dave Citron. 2024. Try Deep Research and our new experimental model in Gemini, your AI assistant. Google, The Keyword. Published December 11, 2024. Accessed October 11, 2026.
• Cohen et al. (2006) A. M. Cohen, W. R. Hersh, K. Peterson, and Po-Yin Yen. 2006. Reducing workload in systematic review preparation using automated citation classification. Journal of the American Medical Informatics Association, 13(2):206–219.
• Cohen (1960) Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46.
• Cuomo-Dannenburg et al. (2024) Gina Cuomo-Dannenburg, Kelly McCain, Ruth McCabe, H Juliette T Unwin, Patrick Doohan, Rebecca K Nash, Joseph T Hicks, Kelly Charniga, Cyril Geismar, Ben Lambert, Dariya Nikitin, Janetta Skarp, Jack Wardle, Mara Kont, Sangeeta Bhatia, Natsuko Imai, Sabine van Elsland, Anne Cori, and Christian Morgenstern. 2024. Marburg virus disease outbreaks, mathematical models, and disease parameters: a systematic review. The Lancet Infectious Diseases, 24(5):e307–e317.
• Doohan et al. (2024) Patrick Doohan, David Jorgensen, Tristan M. Naidoo, Kelly McCain, Joseph T. Hicks, Ruth McCabe, Sangeeta Bhatia, Kelly Charniga, Gina Cuomo-Dannenburg, Arran Hamlet, et al. 2024. Lassa fever outbreaks, mathematical models, and disease parameters: a systematic review and meta-analysis. The Lancet Global Health, 12(12):e1962–e1972.
• Gao et al. (2023) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore. Association for Computational Linguistics.
• Gartlehner et al. (2024) Gerald Gartlehner, Leila Kahwati, Rainer Hilscher, Ian Thomas, Shannon Kugley, Karen Crotty, Meera Viswanathan, Barbara Nussbaumer-Streit, Graham Booth, Nathaniel Erskine, Amanda Konet, and Robert Chew. 2024. Data extraction for evidence synthesis using a large language model: A proof-of-concept study. Research Synthesis Methods, 15(4):576–589.
• Guo et al. (2024a) Eddie Guo, Mehul Gupta, Jiawen Deng, Ye-Jean Park, Michael Paget, and Christopher Naugler. 2024a. Automated paper screening for clinical reviews using large language models: Data analysis study. Journal of Medical Internet Research, 26:e48996.
• Guo et al. (2024b) Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. 2024b. Large language model based multi-agents: A survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 8048–8057. International Joint Conferences on Artificial Intelligence Organization.
• Higgins et al. (2023) Julian P. T. Higgins, James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J. Page, and Vivian A. Welch, editors. 2023. Cochrane Handbook for Systematic Reviews of Interventions. Cochrane. Version 6.4, updated August 2023.
• Hong et al. (2024) Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations, volume 2024, pages 23247–23275.
• Jaumann et al. (2025) Christian Jaumann, Andreas Wiedholz, and Annemarie Friedrich. 2025. LGAR: Zero-shot LLM-guided neural ranking for abstract screening in systematic literature reviews. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7910–7927, Vienna, Austria. Association for Computational Linguistics.
• Khraisha et al. (2024) Qusai Khraisha, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. 2024. Can large language models replace humans in systematic reviews? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Research Synthesis Methods, 15(4):616–626.
• Lieberum et al. (2025) Judith-Lisa Lieberum, Markus Toews, Maria-Inti Metzendorf, Felix Heilmeyer, Waldemar Siemens, Christian Haverkamp, Daniel Böhringer, Joerg J. Meerpohl, and Angelika Eisele-Metzger. 2025. Large language models for conducting systematic reviews: on the rise, but not yet ready for use—a scoping review. Journal of Clinical Epidemiology, 181:111746.
• Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, volume 36, pages 46534–46594. Curran Associates, Inc.
• Marshall et al. (2017) Iain Marshall, Joël Kuiper, Edward Banner, and Byron C. Wallace. 2017. Automating biomedical evidence synthesis: RobotReviewer. In Proceedings of ACL 2017, System Demonstrations, pages 7–12, Vancouver, Canada. Association for Computational Linguistics.
• McCain et al. (2025) Kelly McCain, Anna Vicco, Christian Morgenstern, Thomas Rawson, Tristan M. Naidoo, Sangeeta Bhatia, Dominic P. Dee, Patrick Doohan, Keith Fraser, Anna-Maria Hartner, et al. 2025. A systematic review of Zika virus disease: epidemiological parameters, mathematical models, and outbreaks. medRxiv. Preprint.
• Morgenstern et al. (2025) Christian Morgenstern, Thomas Rawson, Isobel Routledge, Mara Kont, Natsuko Imai-Eaton, Janetta Skarp, Patrick Doohan, Kelly McCain, Rob Johnson, H. Juliette T. Unwin, et al. 2025. Severe acute respiratory syndrome (SARS) mathematical models and disease parameters: a systematic review. The Lancet Microbe, 6(9):101144.
• Naidoo et al. (2026) Tristan Naidoo, Christian Morgenstern, Patrick Doohan, Rhys Earl, Thomas Rawson, Richard J. Sheppard, Joseph T. Hicks, Sreejith Radhakrishnan, Rob Johnson, Anna-Maria Hartner, et al. 2026. A systematic review of Nipah virus disease epidemiological parameters, outbreaks and mathematical models. The Lancet Infectious Diseases, 26(10):e409–e422.
• Nash et al. (2024) Rebecca K. Nash, Sangeeta Bhatia, Christian Morgenstern, Patrick Doohan, David Jorgensen, Kelly McCain, Ruth McCabe, Dariya Nikitin, Alpha Forna, Gina Cuomo-Dannenburg, et al. 2024. Ebola virus disease mathematical models and epidemiological parameters: a systematic review. The Lancet Infectious Diseases, 24(12):e762–e773.
• O’Mara-Eves et al. (2015) Alison O’Mara-Eves, James Thomas, John McNaught, Makoto Miwa, and Sophia Ananiadou. 2015. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews, 4(1):5.
• OpenAI (2025) OpenAI. 2025. Introducing deep research. OpenAI. Published February 2, 2025. Accessed October 11, 2026.
• Padarha et al. (2026) Shreyansh Padarha, Ryan Othniel Kearns, Tristan Naidoo, Lingyi Yang, Łukasz Borchmann, Piotr Błaszczyk, Christian Morgenstern, Ruth McCabe, Sangeeta Bhatia, Philip H. Torr, Jakob Foerster, Scott A. Hale, Thomas Rawson, Anne Cori, Elizaveta Semenova, and Adam Mahdi. 2026. Evaluating AI-based scientific knowledge synthesis with epidemiological systematic reviews. arXiv preprint arXiv:2603.22327.
• Page et al. (2021) Matthew J Page, Joanne E McKenzie, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, et al. 2021. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372:n71.
• Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(85):2825–2830.
• Perplexity (2025) Perplexity. 2025. Introducing Perplexity Deep Research. Perplexity. Published February 14, 2025. Accessed October 11, 2026.
• Priem et al. (2022) Jason Priem, Heather Piwowar, and Richard Orr. 2022. OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv preprint arXiv:2205.01833.
• Ren et al. (2026) Zexin Ren, Zixuan Zhao, Qiyun Li, Yawen Wu, Lanjing Wang, Renjie Luo, Yi Xu, Qing Guo, Jin Shi, En Xie, Feifang Hu, and Qian Shi. 2026. Systematic literature reviews with two multi-agentic systems and human-in-the-loop. arXiv preprint arXiv:2607.21920.
• Scherbakov et al. (2025) Dmitry Scherbakov, Nina Hubig, Vinita Jansari, Alexander Bakumenko, and Leslie A. Lenert. 2025. The emergence of large language models as tools in literature reviews: a large language model-assisted systematic review. Journal of the American Medical Informatics Association, 32(6):1071–1086.
• van de Schoot et al. (2021) Rens van de Schoot, Jonathan de Bruin, Raoul Schram, Parisa Zahedi, Jan de Boer, Felix Weijdema, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, Albert Harkema, Joukje Willemsen, Yongchao Ma, Qixiang Fang, Sybren Hindriks, Lars Tummers, and Daniel L. Oberski. 2021. An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence, 3(2):125–133.
• Vardhaan and Mohanty (2026) Hamsa Vardhaan and Prem Swaroop Mohanty. 2026. Elicit systematic review: Now built for PRISMA 2020. Elicit. Published May 6, 2026. Accessed October 11, 2026.
• Wang et al. (2023) Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023. Can ChatGPT write a good Boolean query for systematic review literature search? In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1426–1436. ACM.
• Wang et al. (2024a) Shuai Wang, Harrisen Scells, Shengyao Zhuang, Martin Potthast, Bevan Koopman, and Guido Zuccon. 2024a. Zero-shot generative large language models for systematic review screening automation. In Advances in Information Retrieval: 46th European Conference on Information Retrieval, ECIR 2024, Proceedings, Part I, Lecture Notes in Computer Science, pages 403–420. Springer Nature Switzerland.
• Wang et al. (2024b) Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. 2024b. AutoSurvey: Large language models can automatically write surveys. In Advances in Neural Information Processing Systems, volume 37, pages 115119–115145. Curran Associates, Inc.
• Wu et al. (2024) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. 2024. AutoGen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling.
• Zhang et al. (2025) Barry Zhang, Keith Lazuka, and Mahesh Murag. 2025. Equipping agents for the real world with Agent Skills. Anthropic. Published October 16, 2025. Accessed October 12, 2026.

Appendix Contents

Appendix A ReviewPilot Implementation Details

This appendix gives the implementation details behind Section 3, in the order of the main text.

A.1 System and Orchestration

The system uses a Starlette backend and a JavaScript interface. A project-level setting selects the model used for search setup, conversation, screening, and extraction. Each action contract specifies a component’s inputs and expected outputs. Before reporting a stage as complete, the Lead Agent checks that the stage’s expected artifacts exist and contain their required fields. Each skill is stored as a SKILL.md file in the Agent Skills format (Zhang et al., 2025). A fixed mapping assigns the skills to the LLM tasks named in Section 3.1 and appends the selected skill to the task’s system prompt.

A.2 Search

For a research description d, the Search Condition Agent forms the Boolean query

where 𝒢req​(d) is the set of required eligibility groups and 𝒯g​(d) contains the alternative terms of group g. Wang et al. (2023) evaluate Boolean queries of this form written by ChatGPT for systematic review searches. The Literature Search Tool applies source-specific syntax for PubMed, arXiv, and OpenAlex (Priem et al., 2022) to the query while preserving its Boolean grouping, and it records the executed query, settings, counts, and errors separately for each source.

A.3 Screening Preprocessing

The Screening Agent first filters records by publication date. Records with missing dates, or with date intervals that overlap the search range, are retained. It then removes exact and near duplicates. Exact matching normalizes title case, punctuation, and whitespace. Near-duplicate detection compares title and author signatures with term frequency–inverse document frequency (TF-IDF) cosine similarity, computed with scikit-learn (Pedregosa et al., 2011), at a default threshold of 0.70. Records with distinct nonempty DOIs or PubMed identifiers are kept separate. Each removal is recorded with the relevant reason.

A.4 Full-Text Retrieval

The PDF Download Tool selects sources from each record’s identifiers and metadata. Routes include direct links, publisher pages, DOI resolution, PubMed Central, Europe PMC, Unpaywall, arXiv, bioRxiv, and medRxiv, with request throttling, retries, and a browser fallback that can be disabled. The identity check compares the text of the PDF’s opening page with the expected title.

A.5 Extraction

In the schema 𝒮, fj, τj, and δj denote a field’s name, JSON-compatible type, and description. Fields may also have value constraints. The Extraction Agent reads the PDF text with pypdf and inserts a page marker before the text of each page. The schema checks cover field presence, declared types, and value constraints. Proposed excerpts are matched against normalized PDF text, and a located excerpt is recorded with the page on which it is found.

A.6 Categorization

For a selected field fj with nonempty extracted values 𝒱j, the Lead Agent proposes the category plan 𝒦j=fLLM​(fj,sample⁡(𝒱j),m), where sample⁡(𝒱j) takes the first 150 distinct values in record order, each shortened to 300 characters, and m selects single-label or multiple-label assignment. It has at most five categories, fewer when the field has few distinct values, and includes an Other category within that limit when some values fit none. Returned labels are normalized when they match approved names. Multiple-label responses containing other labels are rejected. Papers with no value for the field are marked as unavailable. Saved outputs keep the field, mode, category descriptions, and per-paper assignments.

A.7 Interface, Memory, and Revision

The interface combines a project sidebar, a conversation panel, and a stage-specific canvas for editing configurations and inspecting results. For example, the chat request add a field for sample size edits the schema draft. Researchers can run screening or extraction on samples of one to five papers, kept separate from full-run outputs. In conversation, the Lead Agent’s context is 𝒫chat=𝒫task⊕𝒮project⊕ℋproject, which concatenates task instructions and the current request with the saved project state and the conversation history. Configurations that can be imported from a previous project are its search setup, criteria, schema, and category plan. Edits submitted against an older project revision are rejected.

Appendix B PERG Data, Criteria, and Extraction Fields

Table B.1 gives the PERG records and papers used in Section 4. Table B.2 describes the five evaluated extraction fields.

Screening criteria.

AgentSLR’s screening template states the PERG criteria, and all three methods use these 11 rules: AgentSLR runs its published template unchanged, and ReviewPilot and the single-prompt baseline receive the same rules as an inclusion list and an exclusion list. There are five inclusion criteria: (1) the target pathogen; (2) English; (3) peer-reviewed original research, with systematic reviews and meta-analyses given special consideration; (4) human subjects, or animal studies that report transmission or vector parameters; and (5) at least one of ten content types (outbreak details, a transmission model, transmission, timing, severity or genetic parameters, overdispersion, seroprevalence, risk factors, or vector parameters). There are six exclusion criteria: (1) another pathogen; (2) non-English; (3) conference abstracts, posters, or correspondence; (4) in-vitro work only; (5) animal-only studies without transmission parameters; and (6) laboratory outbreaks.

Appendix C Baselines

The single-prompt LLM sends one request per record or paper. Earlier evaluations of LLM screening (Guo et al., 2024a; Wang et al., 2024a; Khraisha et al., 2024) and data extraction (Gartlehner et al., 2024; Khraisha et al., 2024) also prompt the model directly; Lieberum et al. (2025) and Scherbakov et al. (2025) review this work. It receives the same records, papers, eligibility rules and extraction schema as ReviewPilot, but no skill, review-specific guidance, coding rules or schema validation. Its prompts are reproduced verbatim below. Shaded text marks slots. For screening, the criteria slots hold the published AgentSLR rules (Appendix B). AgentSLR runs the published screening template unchanged. Table D.3 lists every change made to its public implementation.

Appendix D Run Configuration

Table D.1 lists the models and the prices used for cost estimates, and Table D.3 the request settings of each method.

Appendix E Additional Results

Tables E.2 and E.2 break down Tables 1 and 2 by pathogen and by field, with the same metrics. All numbers come from the runs in Tables 1 and 2. No model is called again.

Appendix F Case-Study Details

Figure F.1 shows the biomedical workflow corresponding to the HCI workflow in Figure 3. Tables F.1 and F.2 report screening agreement and researcher inspections, respectively. Tables F.3 and F.4 specify the eligibility criteria and extraction schemas. Both complete workflows assessed full-text eligibility under the screening criteria. The independent annotators did not assess full-text eligibility.

Extraction inspections.

HCI sampling covered papers with participant counts, technical evaluations without users, Other application areas, and fine-tuning or RAG. A consistency check selected papers with user-facing methods and missing participant counts. The HCI coding rules included human output evaluators in participant counts and counted participants shared across studies only once. Corrections included restoring reported counts and distinguishing human experiments from evaluations of agents. Biomedical sampling covered expert evaluation, clinical notes, and other tasks. Corrections removed labels based on incidental mentions: tool use for output formatting, benchmark questions absent from the evaluated data, and patient communication outside the actual task. These checks examined values against full-text evidence and added missing excerpts.

Appendix G ReviewPilot Prompts

This appendix lists every prompt that ReviewPilot’s agents send to the LLM (§3, §4). Following the order in Figure 2, it covers the Lead Agent’s chat prompts, the Search Condition Agent, the Prompt Agent, the Screening Agent with the PERG screening guidance, the Extraction Agent with the PERG coding rules, and the Lead Agent’s categorization prompts. The Literature Search Tool and the PDF Download Tool make no LLM calls. A skill is appended to its agent’s system prompt inside a <reviewpilot-agent-skill> block. Shaded text marks slots filled for each review or record. The prompt text is reproduced verbatim. The colours, the labels on the left of each box and the check and cross marks are added for reading.

G.1 Lead Agent

The Lead Agent answers questions about the project in chat and turns the researcher’s chat requests into edits of the screening criteria and of the extraction schema, which are saved as drafts for the researcher to confirm. These prompts carry no skill. The Lead Agent’s categorization prompts are in Appendix G.6.

G.2 Search Condition Agent

The Search Condition Agent writes the concept strategy from the research request and revises it when the researcher asks in chat. The initial prompt is shown as it is sent when a project starts from a chat request, where the request may also set sources, record limits and dates. A response that fails validation is retried once with the reason appended. The Prompt Agent uses the same retry.

G.3 Prompt Agent

The Prompt Agent writes the screening guidance, the extraction schema and the coding rules. The screening guidance and the coding rules each take two calls, first to draft and then to review the draft against the skill’s checklist.

G.4 Screening Agent

The system prompt is an opening sentence followed by the screening skill.

G.4.1 PERG Review Focus and Domain Rules

The Prompt Agent wrote these for each pathogen from the review objectives, the confirmed criteria and the candidate titles and abstracts, without reviewer labels. They are inserted into the screening prompt before the inclusion criteria. These are the gpt-6-luna versions. For DeepSeek V4.1 Flash and GLM-5.3-Flash, the Prompt Agent writes its own guidance using each model.

G.5 Extraction Agent

The system prompt is an opening sentence followed by the evidence instruction and the extraction skill. The user prompt carries the research scope, the finalized schema, the task instruction, the coding rules and the paper text. For a paper without full text, the abstract step appends a note to the system prompt and sends the title and abstract in place of the paper text. The web step then asks for the fields that are still empty.

G.5.1 PERG Coding Rules

The Prompt Agent wrote these for each pathogen from the review objectives, the confirmed schema and the included papers (abstracts and the full-text sentences that speak to each field), without gold values. They are inserted into the extraction prompt before the paper text, using the rules for the paper’s pathogen. These are the gpt-6-luna versions. For DeepSeek V4.1 Flash and GLM-5.3-Flash, the Prompt Agent writes its own coding rules using each model.

G.6 Categorization

The Lead Agent categorizes the extracted values itself, with the Categorization Skill appended to each system prompt. One call plans the categories from every paper’s value of the selected field. One call per paper then assigns it to one category or to several.

Appendix H System Interface

Figures H.2–H.8 show ReviewPilot at each stage of the biomedical review in Figure F.1. Each screen has the project sidebar, the stage canvas under the five-step progress bar, and the conversation with the Lead Agent. The screenshots are taken from the finished project, and nothing on the screen is edited. Activity cards in the conversation record each stage as it finished, so the screening card reports the totals from before the researcher reinstated four records (1,701 included and 1,014 excluded), while the canvas and the PRISMA flow show the final counts. Full-Text Retrieval and Information Extraction are marked as partial because 336 papers had no PDF and seven are not extracted.

注意转载授权 · 阅读原文