RubricAtlas：从专家报告中挖掘类型级评分标准以评估深度研究智能体

Conference: ; ; CCS: Information systems Evaluation of retrieval resultsCCS: Information systems Test collectionsCCS: Information systems Web searching and information discovery

Delin Wang , Sinuo Wang Affiliation: StepFun, China Affiliation: University of Adelaide, Adelaide, Australia , Rui Sun Affiliation: StepFun, China , Beidi Luan Affiliation: StepFun, China , Qiming Shi Affiliation: StepFun, China Affiliation: Zhejiang University, Hangzhou, China , Jing Li Affiliation: StepFun, China , Daxin Jiang Affiliation: StepFun, China and Zuo Bai email: baizuo@stepfun.com Affiliation: StepFun, China Affiliation: FinStep

© none

Abstract.

Deep research agents conduct multi-step Web searches and synthesise retrieved information into analytical reports. Evaluating these outputs requires checking which analyses they deliver, beyond the presence of relevant facts. Query-specific binary rubrics make individual requirements explicit, but task-level checklists may leave the overall analytical structure underexamined. We introduce RubricAtlas, which derives type-level analytical requirements from expert reports on different objects within the same research type and organises detailed checks around recurring analytical questions. The requirements give related tasks a shared standard for what to analyse, without prescribing their answers. Our framework links binary checkpoints to report passages and evaluates completeness separately from argument validity. Built from 3,079 Chinese financial research reports, RubricAtlas covers 99 research types. Reference-report selection experiments show that individual reports cover only about half of their type’s analytical questions, and that selecting requirements from different reports changes evaluation scores and agent rankings. Across ten agents, two judge pipelines achieve checkpoint agreement of κ = 0.71 and agent-ranking agreement of Kendall τ = 0.87. Controlled deletions reduce scores on the affected analyses, while padding produces no detectable increase. These findings support type-level rubrics for comparing the analytical completeness of Web deep research outputs, with individual decisions traceable to report evidence.

Keywords: 

deep research agents, agentic search, rubric-based evaluation, LLM-as-a-judge, Web agents, expert reports

1. Introduction

Deep research agents carry out multi-step Web searches, read retrieved pages and synthesise findings into analytical reports (Nakano et al., 2021; Du et al., 2025; Gou et al., 2025). Unlike short-answer Web tasks (Wei et al., 2025a; Mialon et al., 2023), these reports often require more than factual retrieval, including comparisons, explanations and examination of the conditions behind their conclusions. A report on an oil and gas producer, for example, may list production, reserves and costs without comparing the firm with peers or explaining whether its advantage can persist. Including relevant facts is not the same as completing the relevant analysis.

A common approach to evaluating deep research outputs is to use query-specific binary rubrics: requirements written for an individual task and judged one by one as met or unmet (Sharma et al., 2025; Akyürek et al., 2025; Li et al., 2026). These rubrics make individual requirements explicit. However, an evaluation centred on checklist items can foreground task-specific details while leaving the report’s overall analytical structure underexamined. The issue is not whether decisions are binary, but whether they reveal which analyses the details collectively support and which remain incomplete. We therefore organise evaluation around analytical questions, with detailed checks specifying what each analysis should deliver. To ground this organisation beyond the choices made for one query or reference report, we derive recurring requirements from expert reports on different research objects.

Expert reports provide examples of how professionals analyse different objects. We define a research type as a research-question family within an industry, such as studies of oil and gas producers’ operations and value realisation. Within a type, reports may attribute advantages to different resources or distribution networks, but repeatedly ask: which verifiable capabilities distinguish the firm from its peers, and under what conditions can that advantage persist? By combining these reports, we identify recurring requirements: name the capability, compare it with named peers using comparable metrics, and examine what could sustain or erode the advantage. These requirements describe what to analyse, not what to conclude. They reflect the source reports’ professional practice, rather than a universal checklist for every request.

Our framework combines two designs (Figure 1). Type-level induction derives reusable analytical requirements across research objects. A question-centred hierarchy organises their evaluation into questions, required results (Targets) and binary checkpoints. For example, a question asks which capabilities give the firm an advantage; one required result is a peer comparison; its checkpoints ask whether the report names peers and provides comparable metrics. The details are thus evaluated as parts of an analysis, not as isolated facts. Passages are assigned to questions, checkpoint decisions cite the corresponding report blocks, and scores are aggregated within the hierarchy. This produces both a report score and a profile of completed and missing analyses. Completeness measures whether the required analytical content is present, not whether its facts or reasoning are correct. Argument validity is assessed separately.

We construct RubricAtlas from 3,079 Chinese financial research reports issued by 64 institutions, yielding 99 research types, 921 questions and 8,160 checkpoints. Our reference-report selection experiments examine how the requirements used to evaluate these outputs depend on the expert reports behind them. We measure the fraction of a type’s questions covered by each report and average within types; the median across types is 54%. Evaluating only questions supported by each task’s source report raises completeness scores by 10.5 points and gives Kendall τ = 0.60 with the full-atlas agent ranking. Thus, selecting one report’s analyses changes both the coverage of the evaluation and its outcomes. These experiments restrict the frozen atlas; they do not reconstruct query-specific rubrics.

Our contributions are threefold:

• •

  Type-level induction of analytical requirements. We derive recurring analytical requirements across research objects and construct RubricAtlas for financial research, with source provenance, tasks and evaluation artifacts.

• •

  Question-centred, evidence-linked report evaluation. We organise detailed checks around analytical questions and retain question-level profiles alongside report scores. Two end-to-end judge pipelines achieve checkpoint agreement of κ = 0.71 and agent-ranking agreement of Kendall τ = 0.87. Controlled deletions lower scores on targeted questions; padding produces no detectable increase. A first-author audit examines sampled source mappings and checkpoint decisions.

• •

  Reference sensitivity and Web-agent profiles. We measure reference-selection effects on coverage, scores and rankings, and compare ten Web research agents on consensus and differentiating analyses. We examine associations with browsing, length and source-report exposure. Generic validity rules remain separate diagnostics because cross-judge agreement is limited.

2. Related Work and Design Requirements

Four requirements guide the framework: professional grounding, question-centred organisation and type-level reuse, attributable decisions, and reproducible comparisons. Table 1 compares the source aggregation used by closely related evaluations.

D1 Grounding. A standard should represent recurring expert analyses, not one document’s choices. Existing rubrics are written per instance (Sharma et al., 2025; Akyürek et al., 2025), extracted from one reference (Li et al., 2026), or mined from one survey per query (Yifei et al., 2025).

Multi-reference methods aggregate content for one input: Pyramid weights units by reference frequency (Nenkova and Passonneau, 2004), and TREC evaluates vital and okay nuggets (Voorhees, 2003). Nuggets can be weighted by assessor agreement (Lin and Demner-Fushman, 2006) or extracted and evaluated using LLMs (Mayfield et al., 2024; Pradeep et al., 2025). We abstract analyses across different objects within a type.

D2 Organised and reusable. Requirements should be organised around analytical questions, so that individual decisions reveal which analyses are complete or missing. This organisation should be reusable across tasks within a research type. Checklists turn holistic judgments into binary decisions (Cook et al., 2024; Lee et al., 2025; Wei et al., 2025b); instance-specific evaluators use rubrics or skill sets (Ye et al., 2024; Kim et al., 2024), and rubrics also serve as rewards (Gunjal et al., 2025; Shao et al., 2025). DEER and FinDeepResearch provide top-down task-level templates (Han et al., 2025; Zhu et al., 2025). We retain binary checks but organise them under questions derived bottom-up from expert reports across research objects.

D3 Attributable. Judgments should identify the unmet requirement or supporting evidence. Factuality evaluation decomposes text into atomic claims (Min et al., 2023); Mind2Web 2 verifies task-specific rubric trees against cited pages (Gou et al., 2025), and ReportLogic evaluates reasoning (Zhao et al., 2026). Our checkpoints cite report blocks retrieved verbatim by the program, not reconstructed by the judge.

D4 Reproducible. Comparisons should remain stable across judges. LLM judges exhibit position, verbosity and self-preference effects (Zheng et al., 2023; Wang et al., 2024; Panickssery et al., 2024; Dubois et al., 2024) and criterion confusion (Hu et al., 2024). IR studies examine assessor disagreement, incomplete judgments, topic sampling and metric discrimination (Voorhees, 2000; Zobel, 1998; Voorhees and Buckley, 2002; Sakai, 2006). Reproducibility differs from measurement validity (Jacobs and Wallach, 2021).

Evaluation of Web research. Web benchmarks assess short answers (Wei et al., 2025a; Mialon et al., 2023) and deep research reports through rubrics or references (Gou et al., 2025; Du et al., 2025; Li et al., 2026). Financial benchmarks cover numerical question answering (Chen et al., 2021), research reports (Jin et al., 2026b; Zhu et al., 2025; Jin et al., 2026a) and search (Hu et al., 2025). We focus on the analytical completeness of reports produced through Web search and synthesis.

FinResearchBench (Sun et al., 2025) combines report-level logic-tree metrics with LLM judgments. FinResearchBench II (Luan et al., 2026) generates rubrics from multiple systems’ reports for one query, then filters for judge consistency and distinguishability. Structured evaluation is therefore not unique to our approach: existing methods also use rubric trees, logic trees and templates. Our distinction is how the analytical structure is grounded and reused. Expert reports on different objects supply type-level questions and required results, under which evidence-linked checks are organised. The consensus tier denotes analyses in a majority of source reports, not agreement among judges.

3. Evaluation Framework

The framework separates deriving an analytical standard from applying it. Type-level construction identifies recurring analyses across research objects; question-centred evaluation determines what each report delivers. Here, analytical structure means the organisation of questions and required results, not the full logical coherence of a report’s reasoning. Evaluation aligns report passages to this structure and judges completeness and argument validity independently.

Research types. A research-question family is a recurring kind of analysis. Combining it with an industry defines a research type, such as operations and value realisation of oil and gas producers. A task specifies a firm, industry or policy within that type.

Analytical requirements. The rubric is a tree: dimension → question → required result → checkpoint. Questions identify analyses, required results specify their expected outputs, and checkpoints assess the content needed to deliver those outputs. Each checkpoint decision therefore remains attached to an analytical requirement. A canonical question represents an analysis found in the type’s reports. Its support records the reports and institutions addressing it; its tier is consensus if a majority of reports address it and differentiating otherwise. A required result (Target) specifies what an answer must deliver without naming a company or date. Binary checkpoints state acceptance and non-acceptance conditions. A Target may have a negative criterion identifying a substitute that misses the intended analysis, such as industry averages instead of named-peer comparisons. Requirements are constructed once per type, not per task.

Evidence alignment and completeness. The judge selects passage IDs; the program retrieves original text, preventing insertion of new material. Target evaluation checks content in these passages, citing blocks for yes and explaining no. Missed evidence during alignment can lower scores. Factual and inferential correctness are not assessed; Target requirements provide context, not extra scoring criteria.

In the current implementation, a Target’s completeness is its checkpoint pass rate; a triggered negative criterion sets it to 0. Question completeness T (0–1) is the mean of its Targets. Report completeness T (0–100) is the weighted mean of question completeness (differentiating 20, consensus 10) ×100. Averaging within Targets and questions prevents extra checkpoints from increasing an analysis’s report-level weight. Differentiating questions were model-screened and author-verified for analytical depth. The higher weight emphasises this depth and was fixed before examining agent evaluation results; alternatives appear in §7. Question, Target and checkpoint profiles locate incomplete analyses alongside report scores.

Argument validity. This layer concerns evidence, inference and quantitative reasoning. Flaw definitions can be shared, with question-specific conditions. General evaluation tests generic rules on actual claims independently of Targets and scores (§7.4).

A worked example. Figure 1 illustrates a consensus question for oil and gas producers: which verifiable capabilities give the firm an advantage over comparable firms, and can that advantage persist? It is supported by 14 of 21 reports from 13 institutions, each on a different firm. Source questions concern a gas block’s resource advantage or a sales network’s contribution to stability. Its 3 Targets and 9 checkpoints require capabilities with checkable metrics, named-peer comparisons on like-for-like metrics, and conditions sustaining or eroding the advantage. The second Target’s negative criterion excludes industry averages in place of named peers. A report can thus receive credit for naming a capability but not for a peer comparison it never makes. Providing capability metrics does not substitute for comparing peers or examining persistence conditions: the separate Targets preserve these omissions within the same analytical question.

Output. We evaluate completeness end to end (§6–7); the separate validity profiles diagnose one generic-rule implementation (§7.4).

4. Constructing RubricAtlas for Financial Research

RubricAtlas derives a shared analytical structure through report classification, question alignment and Target abstraction. Fixed AI workflows and prompts govern aggregation and applicability screening. Source questions determine the analyses to cover; answering passages supply each analysis’s content requirements.

Corpus and research types. The corpus comprises 3,079 Chinese financial research reports on the Chinese market, issued by 64 institutions from January to September 2026. It covers 31 industries and eleven genres, including company coverage, earnings reviews, industry deep dives and strategy reports.

An LLM decomposes reports into dimension → question → conclusion trees. Reports are grouped first by genre × industry (331 cells), then by research-question family × industry. The 13 families include firm operations and value realisation, industry cycle tracking and policy transmission. Of 106 candidates, 3 with heterogeneous branches were deferred and 4 excluded by a predefined discrimination criterion. The 99 retained pools contain a median of 30 reports (10–60; Appendix E).

Question alignment. Within each type, an LLM maps source questions to canonical questions, background or out-of-scope content; a second LLM checks the mappings. Records preserve supporting reports and source questions. Questions below 3 supporting reports are removed unless judged to have distinct analytic value.

Targets and checkpoints. For each canonical question, answering passages are decomposed into data used → reasoning → conclusion, abstracted into company- and date-independent Targets, and divided into checkpoints. We add negative criteria for plausible substitutes. Before scoring, an AI reviewer checked each criterion for one verifiable requirement, clear wording and no adjacent redundancy (Akyürek et al., 2025); authors reviewed all flagged items. Scores use this frozen atlas, unchanged by the later author audit.

Applicability screening. The workflow excludes requirements outside the industry and research-question family, and those specific to an individual object. It also checks consistency between retained requirements and the requests’ analytical scope. These filters produce type-level standards rather than report-content unions. Requirements are frozen before scoring; request checks match tasks to these standards without regenerating rubrics or adapting them to agent outputs.

Scale and support. RubricAtlas contains 99 trees, 921 questions (503 consensus, 418 differentiating), 2,784 Targets, 8,160 checkpoints and 2,520 negative criteria. Median question support is 14 reports and 9 institutions; 94% of questions have at least three institutions and 79% at least five. The largest institution’s median support share is 25%.

Median pool coverage is 68% for consensus and 28% for differentiating questions. The latter still have median support from 8 reports and 6 institutions (Table 2), representing recurring analyses rather than isolated contributions.

Tasks. A seeded hash selects one report per type and converts its topic into a conclusion-free request, yielding 99 tasks. Checks cover research type, object scope, analytical scope and readability, with recorded adjustments (Appendix E). Requests are matched to the type-level requirements, not used to regenerate them.

Resource contents and access. The resource includes trees with provenance, rules, tasks, prompts, checkpoint judgments, scoring and analysis code, and a datasheet (Gebru et al., 2021). The materials will be made publicly available after the review process. Re-scoring uses judgments and code; independent re-judging and source verification also need report text and source materials. A verbatim-overlap check sampled 800 criteria, extracted 3,881 spans of 20 normalised characters and searched 3,228 source texts: no matches, with a positive control detected.

5. Experimental Setup

Agents and harness. The ten LLMs in Table 4 run as deep research agents in a fixed ReAct harness (Yao et al., 2023). WebSearch returns Google results via Serper; WebFetch reads pages with Jina and extracts goal-relevant content with gpt-5. The prompt sets information cutoff 2026-10-09, requires citable sources and prohibits other institutions’ research conclusions as evidence. Each agent runs once per task with provider-default decoding, up to 1,000 turns and 128k output tokens per turn. Tasks and reports are Chinese.

Evaluation sets. One run failed, leaving 989 reports. Agent and browsing comparisons use 98 common tasks (980 reports), covering 912 of 921 atlas questions. Checkpoint agreement uses available paired judgments. Agent scores equally average common-task report scores; tier scores are averaged after within-report computation, so global tier counts cannot reconstruct totals.

Judges and protocol. Judge A is DeepSeek-V4.1-Flash; B is GPT-6-luna. Calls use temperature 0 and are grouped by dimension, with decisions assigned to questions. Evidence alignment selects block IDs for programmatic text retrieval. Target evaluation returns checkpoint and negative-criterion decisions, citations for each yes and reasons for each no. General evaluation runs separately. Neither judge accesses the Web.

The API parameter reasoning_effort is none for alignment, medium/low for A’s Target/General calls and high/high for B’s. Model-specific labels are not a shared budget scale: we compare configured pipelines, not matched-budget model substitution. Settings were fixed in pilots before finalisation; token use and alternatives appear in Appendix D. Failed jobs are listed separately, not scored zero.

A is also an evaluated backbone; B shares the tree builders’ GPT family. Both judges expose sensitivity to these overlaps but cannot rule out family bias. All agents share the gpt-5 page extractor, whose effect is not isolated. Task resampling quantifies sampling uncertainty; score intervals add a shared repeat-run variance proxy (§6.3), not independently estimated B variance.

6. Assessing the Resource and Evaluation Protocol

We assess two aspects of the approach: which analytical requirements the resource represents, and how consistently evaluation applies them. Reference-report selection analyses examine requirement coverage and dependence on particular expert reports; the author audit examines sampled mappings and decisions; cross-judge comparisons and controlled content changes assess the evaluation protocol.

6.1. Author audit

The first author, a finance graduate student, annotated blank forms without AI assistance or access to model judgments: 300 checkpoint decisions and 297 source mappings for 50 canonical questions. Agreement with A/B is 74.3%/71.7% (κ = 0.49/0.43). Mappings are 77.8% faithful, 15.2% partial and 7.1% not faithful. These checks concern checkpoint application and question grouping, not every Target abstraction or task applicability.

The sample overrepresents judge disagreements; evidence is the union of both judges’ aligned passages. Agreement cannot be directly compared with the full-corpus judge–judge rate. All 300 labels are retained as final judgments. This domain-informed author audit is not independent expert validation; sampling, confusion matrices and exploratory preferences appear in Appendix I.

6.2. Reference-report selection and analytical coverage

How does reference selection change requirement coverage and agent comparisons? Using the full atlas as an internal reference, we retain questions supported by selected reports and re-score frozen main-run judgments (§5). Reports and checkpoint decisions stay fixed; rubrics are not re-extracted (Appendix G). These comparisons measure reference sensitivity, not which requirements best fit each task.

Coverage. Averaging reports within types, then taking the median across types gives 54% coverage (type means 14–83%); task-source reports cover 57%. The gap is not solely expert disagreement: genre medians are 73% for company coverage and 33% for retrospective reviews; the shortest and longest thirds cover 50% and 63%.

Removing the measured report from support and dropping questions below 3 reports leaves median coverage at 53%. The other half covers 56% of questions selected from one random half of a pool; the within-half threshold retains more common questions. A single report thus represents only part of a type’s recurring analyses, even excluding its contribution to question selection.

Score shifts. Source-supported subsets raise mean completeness by 10.5 points (task-level 95% confidence interval (CI) [7.1, 12.1]), with increases on 77% of tasks. Size-matched random subsets give 1.2 (200 draws; maximum 4.5). Source subsets contain 71% consensus questions, versus 55% overall; consensus scores are higher. Composition matching gives 5.6 (maximum 8.6 across 200 draws), still 4.9 below source subsets, which exceed every matched draw. The remainder is consistent with tasks eliciting source-emphasised analyses, but object-specific applicability is not isolated.

Ranking sensitivity. Single-report subsets have lower ranking agreement and more leader changes than size-matched random questions (Table 3). Task-source subsets give 0.60, versus 0.81 for size-matched subsets and mean 0.74 (5th percentile 0.56) for composition-matched subsets. Size matching alone does not isolate source dependence: composition explains part of the ranking change. Source subsets change the task-level leader on 50% of tasks, versus 45% for size-matched random subsets (Appendix H). Both reductions affect task-level recommendations, motivating aggregate comparisons.

Aggregation across reports. Three and ten reports cover 79% and 96% of the atlas (Figure 2), with ranking correlations 0.79 and 0.95, near size-matched controls (0.89, 0.97). Single-report small pools (10–20) give τ = 0.59 and 45% coverage, but sensitivity is non-monotonic (21–39: 0.79; 40–60: 0.65). About ten reports recover much of the full pool’s coverage and ranking: an internal planning reference, not an external sufficiency threshold.

Sensitivity to source reports. Disjoint random half-pools give τ = 0.84 across 200 splits (0.76 with three supporting reports within each half), always with the same leader. Task resampling gives 0.79 against the full ranking, a similar scale of variation.

Removing task-source support drops 14 of 912 questions across 98 common tasks below threshold; re-scoring gives τ = 0.96, with model score changes at most 0.27 points. The aggregate evaluation is therefore not driven by the single report used to construct each task. This supports type-level standards; new-task applicability is discussed in §8.

6.3. Cross-judge consistency and repeat runs

B independently re-evaluates all 989 reports, including alignment. The 81,469 paired checkpoints agree at 85.4%, with Cohen’s κ = 0.71 (95% CI [0.70, 0.72], report-clustered) and Jaccard overlap 0.75 for yes decisions. Question-level T correlation is 0.83 over 9,196 questions.

B gives fewer yes decisions (48% versus 56%) and scores reports 8.0 points lower on average, but rankings agree at τ = 0.87 (95% CI [0.56, 0.96], task resamples). Both judges have the same first two agents, top-four set and bottom-three order (Table 4, Figure 3a). Of A’s 36 pairs separated by over one point, 1 reverses under B; none of 28 separated by over two points reverses. None of B’s 34 pairs separated by over one point reverses under A. Judge choice shifts absolute levels more than clearly separated agents’ ordering. Negative criteria rarely trigger (A 262, B 151 of 25,164); agreement is 99.0% (κ = 0.39).

A DeepSeek repeat run on 100 stratified reports (10 tasks × ten agents) uses the same model and effort but condensed negative-criterion wording, correlating at 0.96 with the main run. Its standard deviation (SD) proxy is 4.1 points per report and 0.42 for a 98-task mean. Both judges use this estimate, mixing run and wording variation. Alternative settings correlate highly, but nominal high effort need not increase tokens (Appendix D).

6.4. Response to deletion and padding

We perturb 30 reports (ten tasks × three agents) and re-evaluate each whole report. Deleting all blocks answering one question lowers its T from 0.64 to 0.21 under A and from 0.57 to 0.19 under B. Median relative reductions are 71% and 83%, with decreases on 20/27 and 22/27 target questions¹¹ 1 Perturbation targets were drawn before the atlas was frozen; 27 of the 30 targeted questions are in the resource atlas and are analysed here.. Questions sharing no deleted block change by 0.033 and 0.028 on average, close to unchanged re-evaluation (0.025 and 0.029). The response is concentrated on questions whose content was removed, linking the score change to an identifiable missing analysis rather than only a reduction in report length; Appendix D also reports changes on questions sharing deleted blocks.

Adding 40% irrelevant text from another research family or duplicating paragraphs gives no detectable net increase. A’s unchanged re-evaluation itself shifts scores by −2.5 points. Relative to each judge’s unchanged re-evaluation, irrelevant/duplicated effects are +1.1 [−0.9, 3.2] and +0.5 [−1.3, 2.5] under A, +0.8 [−1.4, 3.0] and 0.0 [−2.1, 2.1] under B (Figure 3c). With 30 reports, only effects above about 3 points are excluded. A’s observed length slope implies 3.1 points, so modest rewards remain compatible with the intervals.

7. Evaluating Deep Research Agents

We compare the final reports of Web research agents through completeness totals and analytical tier profiles. We then examine associations with browsing, report length and exposure to source reports. Generic-rule profiles remain separate from completeness.

7.1. Main results

Leading agents and unresolved margins. Claude leads under both judges (92%, 84% first-place resamples), followed by GLM. Their gaps of 2.0 and 1.0 points remain unresolved: 95% intervals [−0.7, 4.7], [−1.4, 3.4] include shared repeat-run variance. Ranks 3–7 span 1.5 and 0.8 points, separated from Kimi, Gemini and MiniMax at ranks 8–10 by 5.0 and 3.6 points under A/B.

Claude remains first under consensus/differentiating weights 1:1, 1:3 and 2:1, inverse-frequency and frequency weighting (τ 0.73–0.96). Frequency weighting emphasises recurring source analyses (0.78); consensus-only scoring puts GLM first (0.60; Appendix B). The evaluated analyses affect rankings even with a stable leader.

Distinct analytical profiles. Similar consensus scores can conceal differences on differentiating analyses. Under A, GLM, Doubao and Claude are within half a point on consensus questions, leading in 39%, 34% and 21% of resamples. Claude leads differentiating questions by 2.4 points under A (86% first-place resamples) and 0.8 under B (55%).

Differentiating scores have greater dispersion across agents: 4.1 versus 2.8 points under A (difference 1.3, 95% CI [0.33, 2.3]) and 3.1 versus 2.1 under B (0.9, [0.00, 1.9]). Dispersion is the population SD across evaluated agents. B’s interval reaches zero, so the stronger statistical evidence comes from A. Reporting both tiers shows which groups of analyses separate agents with similar totals. These profiles summarise coverage of recurring requirements, not the coherence or validity of the agents’ reasoning.

7.2. Browsing, length and completeness

Claude makes 53 searches and 50 page reads per report; Gemini makes 13 and 5. Portals and financial media supply 29% of reads, official sources 20%. Within tasks, log searches and reads correlate with T at 0.26 and 0.24 under A, 0.18 and 0.16 under B. Official-source share is weakly associated (r = 0.07).

With agent/task fixed effects and task-clustered standard errors (SEs), A’s page-read coefficient is 1.2 points per log unit (SE 0.55), versus 1.1 for searches (SE 0.9, not significant). Adding log length gives page coefficient 1.0 (SE 0.54) and length coefficient 8.6 (SE 1.8). B’s page coefficient is 0.3 (SE 0.57, not significant). Browsing evidence is stronger under A; page reads and length correlate at r = 0.44 within tasks; these observational data do not isolate their contributions.

Longer reports score higher: 9.2 points per log length unit under A (SE 1.7), 5.7 under B (SE 1.3). This association does not identify a causal length effect; padding is tested in §6.4. Length adjustment leaves Claude first under both judges, with Mimo-V2.6-Pro second under A and GLM under B (τ = 0.73, 0.87 with main rankings). Length-matched win rates are 52% for Claude and 60% for GLM (Appendix B); Claude’s lead does not carry over to matching, although it remains first after regression adjustment.

Judge differences and self-preference. Between-judge score shifts are associated with length. DeepSeek’s length-adjusted shift does not establish self-preference; Appendix B reports estimates and intervals.

7.3. Do agents retrieve the source reports?

Research-report sites account for 1.7% of 15,798 successful reads and occur in 17% of generated reports’ trajectories. Sell-side content appears in 31% of reports, with scores +0.5 points above task means, consistent with browsing associations. Task-source titles appear in search results or pages in 53 of 980 trajectories (5.4%, 95% CI [4.2, 7.0]); exposed reports show no detectable gain (−0.7 points, [−3.5, 2.1]). Source-title exposure is uncommon with no detected benefit. The prompt prohibits other institutions’ conclusions, but its effect is not isolated.

7.4. Generic validity-rule diagnostics

Completeness records delivered analyses, not their validity. General evaluation applies ten argument-flaw rules covering facts, evidence and reasoning (G01–G10; Appendix A) to aligned material in separate calls (A low, B high effort). It reports claim blocks, basis blocks and reasons per question, without Target requirements, scores or treating missing content as a violation.

Limited agreement on individual rules. Cross-judge κ has median 0.16 and maximum 0.29 (G04, calculations); flagged-question Jaccard has median 0.10 (Table 5). A and B flag 11,956 and 6,719 question–rule violations. For G09 (source qualifications), they flag 341 and 25 questions, with 8 shared. Despite shared definitions, rule decisions are less consistent than completeness checkpoints (κ = 0.71) on the same reports.

Agent-level violation profiles. To assess whether aggregation reduces disagreement, we compare each agent’s fraction of questions with any violation, separately from completeness. A flags 62–75%, versus B’s 39–63%; rates depend strongly on the judge. Flag-rate rankings give Kendall τ = 0.53 (95% CI [0.33, 0.78]; r = 0.76), versus completeness 0.87 [0.56, 0.96]. The intervals overlap, so the point-estimate difference does not establish a ranking-level contrast; the clearer difference is in individual checkpoint and rule agreement (κ 0.71 versus median 0.16). Across rules, agent-level τ has median 0.48 and range −0.04 (G09) to 0.69 (G03; Table 5). These are diagnostics, not validity scores: combining them with completeness would introduce judge dependence without validated weights.

Descriptive diagnostics. G07, G03, G08 and G04 give relatively consistent agent profiles, unlike G09 and G01. Both judges flag Doubao least often. These describe detected flaws, not validated question judgments (Appendix A).

Interpreting the gap. Checkpoints specify conditions; generic rules leave thresholds such as “overstated” to judges. Binary presence checks differ from multi-label defect detection. Checkpoints pass about half the time; most rules flag fewer than one question in six. A uses medium Target and low General effort, B high for both. These differences are plausible explanations, not separately tested causes.

8. Discussion and Limitations

Implications for evaluation. Web research reports need evaluation beyond retrieved facts. Type-level requirements and question-centred checks locate incomplete analyses, preserving profiles behind report scores. These diagnose output content, not whether omissions originate in retrieval or synthesis. Tier profiles locate agent differences on sampled requests. Keeping validity separate prevents conflating missing content with defective reasoning. Absolute scores should identify the judge; relative comparisons are most credible where judges agree.

Scope and transfer. The resource covers Chinese-language financial research (31 industries, 13 question families). Applicability screening and request-scope checks are part of construction, but their accuracy has not been independently quantified. Sampled requests do not establish applicability to every new object. New types need report pools and construction. About ten reports is an internal planning reference, not a transfer guarantee (Table 8).

Beyond financial research. A broader direction is type-level rubric induction as a reusable paradigm for document evaluation. Academic papers, literature surveys (Yifei et al., 2025), legal documents and other expert-written texts could supply genre-specific requirements, with completeness traced to evidence and argument validity assessed separately. Adapting this paradigm beyond finance will require domain-specific corpora, applicability checks and validation of both layers.

Mining versus task-generated rubrics. An LLM could instead generate a rubric for each task, including a structured rubric. Our distinction is not the presence of a hierarchy alone, but the basis for its analytical questions: model-generated expectations for one request versus analyses documented across expert reports on different objects. Mining preserves source-question records for auditing each abstraction, while the question-centred hierarchy connects detailed checks to those analyses. This is a design argument, not a head-to-head comparison.

Developing validity evaluation. Question-specific checks, such as whether growth follows from stated inputs, require separate validation. Once both layers are validated, Pareto dominance compares agents no worse on either dimension and strictly better on at least one, otherwise retaining the trade-off. Future directions include crediting checkpoints only when cited passages have no established flaw, and propagating unsupported premises through question dependencies.

Limitations. The first-author audit is not independent expert validation and does not establish task applicability or the fidelity of every Target. Source support measures recurrence, not importance; tier weights are a design choice. Source experiments restrict a frozen atlas rather than re-extract single-report or task-generated rubrics. We do not isolate the contribution of hierarchical organisation against a flat-checklist baseline. Perturbations test unrelated and duplicated text, not on-topic empty expansion. Rule effort and output format are not isolated by ablation. Shared repeat-run variance is a proxy that includes wording changes. Agents run once per task, so intervals exclude generation randomness; search has no date filter.

Ethics and access. Materials will be made publicly available after the review process. Trees omit analyst names and use opaque institution IDs. Completeness does not assess investment suitability. The audit involved only the first author, no external annotators.

9. Conclusion

Expert reports on different objects supply recurring analytical requirements, not merely reference answers. RubricAtlas organises these requirements by type and links checks to analytical questions, yielding profiles of completed and missing analyses in Web deep research reports. Reference-report selection experiments motivate cross-report aggregation; cross-judge evaluation and controlled deletions support evidence-linked completeness comparisons. A first-author audit adds a human check on sampled mappings and decisions. Argument validity remains separate: the tested generic rules require further validation before entering a combined score.

References

• Akyürek et al. (2025) A. F. Akyürek, A. Gosai, C. B. C. Zhang, V. Gupta, J. Jeong, A. Gunjal, T. Rabbani, M. Mazzone, D. Randolph, M. M. Meymand, G. Chattha, P. Rodriguez, D. Mares, P. Singh, M. Liu, S. Chawla, P. Cline, L. Ogaz, E. Hernandez, Z. Wang, P. Bhatter, M. Ayestaran, B. Liu, and Y. He PRBench: Large-Scale expert rubrics for evaluating High-Stakes professional reasoning. External Links: 2511.11562, Document, Link Cited by: §1, §2, §4.
• Chen et al. (2021) Z. Chen, W. Chen, C. Smiley, S. Shah, I. Borova, D. Langdon, R. Moussa, M. Beane, T. Huang, B. Routledge, and W. Y. Wang FinQA: a dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 3697–3711. External Links: Link, Document Cited by: §2.
• Cook et al. (2024) J. Cook, T. Rocktäschel, J. Foerster, D. Aumiller, and A. Wang TICKing all the boxes: generated checklists improve LLM evaluation and generation. External Links: 2410.03608, Document, Link Cited by: §2.
• Du et al. (2025) M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao DeepResearch bench: a comprehensive benchmark for deep research agents. External Links: 2506.11763, Document, Link Cited by: Table 1, §1, §2.
• Dubois et al. (2024) Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto Length-Controlled AlpacaEval: a simple way to debias automatic evaluators. External Links: 2404.04475, Document, Link Cited by: §2.
• Gebru et al. (2021) T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford Datasheets for datasets. Communications of the ACM 64 (12), pp. 86–92. External Links: Document, Link Cited by: §4.
• Gou et al. (2025) B. Gou, Z. Huang, Y. Ning, Y. Gu, M. Lin, W. Qi, A. Kopanev, B. Yu, B. J. Gutierrez, Y. Shu, C. H. (. Song, J. Wu, S. Chen, H. Moussa, T. Zhang, J. Xie, Y. Li, T. Xue, Z. Liao, K. Zhang, B. Zheng, Z. Cai, V. Rozgic, M. Ziyadi, H. Sun, and Y. Su Mind2Web 2: evaluating agentic search with Agent-as-a-Judge. In Advances in Neural Information Processing Systems 38, pp. 192393–192451. External Links: Document, Link Cited by: §1, §2, §2.
• Gunjal et al. (2025) A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. Hendryx Rubrics as rewards: reinforcement learning beyond verifiable domains. External Links: 2507.17746, Document, Link Cited by: §2.
• Han et al. (2025) J. Han, H. Kim, C. Lee, D. Lee, M. H. Park, H. Song, S. J. Choi, M. Lee, and H. Lee DEER: a benchmark for evaluating deep research agents on expert report generation. External Links: 2512.17776, Document, Link Cited by: §2.
• Hu et al. (2025) L. Hu, J. Jiao, J. Liu, Y. Ren, Z. Wen, K. Zhang, X. Zhang, X. Gao, T. He, F. Hu, Y. Liao, Z. Wang, C. Yang, Q. Yang, M. Yin, Z. Zeng, G. Zhang, X. Zhang, X. Zhao, Z. Zhu, H. Namkoong, W. Huang, and Y. Tang FinSearchComp: towards a realistic, Expert-Level evaluation of financial search and reasoning. External Links: 2509.13160, Document, Link Cited by: §2.
• Hu et al. (2024) X. Hu, M. Gao, S. Hu, Y. Zhang, Y. Chen, T. Xu, and X. Wan Are LLM-based evaluators confusing NLG quality criteria?. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, pp. 9530–9570. External Links: Link, Document Cited by: §2.
• Jacobs and Wallach (2021) A. Z. Jacobs and H. Wallach Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 375–385. External Links: Document, Link Cited by: §2.
• Jin et al. (2026a) J. Jin, Y. Zhang, Y. Xu, H. Qian, Y. Zhu, and Z. Dou FinSight: towards Real-World financial deep research. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5868–5894. External Links: Document, Link Cited by: §2.
• Jin et al. (2026b) S. Jin, S. Li, S. Zhang, and R. Yan FinRpt: dataset, evaluation system and LLM-based multi-agent framework for equity research report generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 507–515. External Links: Document, Link Cited by: §2.
• Kim et al. (2024) S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne, and M. Seo Prometheus: inducing fine-grained evaluation capability in language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.
• Lee et al. (2025) Y. Lee, J. Kim, J. Kim, H. Cho, J. Kang, P. Kang, and N. Kim CheckEval: a reliable LLM-as-a-judge framework for evaluating text generation using checklists. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, pp. 15771–15798. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §2.
• Li et al. (2026) R. Li, M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao DeepResearch bench II: diagnosing deep research agents via rubrics from expert reports. External Links: 2601.08536, Document, Link Cited by: Table 1, §1, §2, §2.
• Lin and Demner-Fushman (2006) J. Lin and D. Demner-Fushman Will pyramids built of nuggets topple over?. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, New York City, USA, pp. 383–390. External Links: Link Cited by: Appendix F, §2.
• Luan et al. (2026) B. Luan, R. Sun, S. Wang, Y. Gu, C. Li, Z. Xiong, J. Li, and Z. Bai FinResearchBench II: a deep research benchmark with Consensus-Derived gold rubrics for distinguishing financial report quality. External Links: 2607.12252, Document, Link Cited by: Table 1, §2.
• Mayfield et al. (2024) J. Mayfield, E. Yang, D. Lawrie, S. MacAvaney, P. McNamee, D. W. Oard, L. Soldaini, I. Soboroff, O. Weller, E. Kayi, K. Sanders, M. Mason, and N. Hibbler On the evaluation of Machine-Generated reports. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1904–1915. External Links: Document, Link Cited by: Appendix F, §2.
• Mialon et al. (2023) G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom GAIA: a benchmark for general AI assistants. External Links: 2311.12983, Document, Link Cited by: §1, §2.
• Min et al. (2023) S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi FActScore: fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 12076–12100. External Links: Link, Document Cited by: §2.
• Nakano et al. (2021) R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman WebGPT: browser-assisted question-answering with human feedback. External Links: 2112.09332, Document, Link Cited by: §1.
• Nenkova and Passonneau (2004) A. Nenkova and R. Passonneau Evaluating content selection in summarization: the pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, Boston, Massachusetts, USA, pp. 145–152. External Links: Link Cited by: §2.
• Panickssery et al. (2024) A. Panickssery, S. Bowman, and S. Feng LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems 37, pp. 68772–68802. External Links: Document, Link Cited by: §2.
• Pradeep et al. (2025) R. Pradeep, N. Thakur, S. Upadhyay, D. Campos, N. Craswell, I. Soboroff, H. T. Dang, and J. Lin The great nugget recall: automating fact extraction and RAG evaluation with large language models. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 180–190. External Links: Document, Link Cited by: Appendix F, §2.
• Sakai (2006) T. Sakai Evaluating evaluation metrics based on the bootstrap. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 525–532. External Links: Document, Link Cited by: §2.
• Shao et al. (2025) R. Shao, A. Asai, S. Z. Shen, H. Ivison, V. Kishore, J. Zhuo, X. Zhao, M. Park, S. G. Finlayson, D. Sontag, T. Murray, S. Min, P. Dasigi, L. Soldaini, F. Brahman, W. Yih, T. Wu, L. Zettlemoyer, Y. Kim, H. Hajishirzi, and P. W. Koh DR tulu: reinforcement learning with evolving rubrics for deep research. External Links: 2511.19399, Document, Link Cited by: §2.
• Sharma et al. (2025) M. Sharma, C. B. C. Zhang, C. Bandi, C. Wang, A. Aich, H. Nghiem, T. Rabbani, Y. Htet, B. Jang, S. Basu, A. Balwani, D. Peskoff, M. Ayestaran, S. M. Hendryx, B. Kenstler, and B. Liu ResearchRubrics: a benchmark of prompts and rubrics for evaluating deep research agents. External Links: 2511.07685, Document, Link Cited by: Table 1, §1, §2.
• Sun et al. (2025) R. Sun, Z. Bai, W. Zhang, Y. Zhang, L. Zhao, S. Sun, and Z. Qiu FinResearchBench: a logic tree based Agent-as-a-Judge evaluation framework for financial research agents. External Links: 2507.16248, Document, Link Cited by: Table 1, §2.
• Voorhees and Buckley (2002) E. M. Voorhees and C. Buckley The effect of topic set size on retrieval experiment error. In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 316–323. External Links: Document, Link Cited by: §2.
• Voorhees (2000) E. M. Voorhees Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management 36 (5), pp. 697–716. External Links: Document, Link Cited by: §2.
• Voorhees (2003) E. M. Voorhees Overview of the TREC 2003 question answering track. In Proceedings of the Twelfth Text REtrieval Conference (TREC 2003), External Links: Link Cited by: Appendix F, §2.
• Wang et al. (2024) P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, L. Kong, Q. Liu, T. Liu, and Z. Sui Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, pp. 9440–9450. External Links: Link, Document Cited by: §2.
• Wei et al. (2025a) J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese BrowseComp: a simple yet challenging benchmark for browsing agents. External Links: 2504.12516, Document, Link Cited by: §1, §2.
• Wei et al. (2025b) T. Wei, W. Wen, R. Qiao, X. Sun, and J. Ma RocketEval: efficient automated LLM evaluation via grading checklist. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.
• Wilson (1927) E. B. Wilson Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22 (158), pp. 209–212. External Links: Document, Link Cited by: Appendix D.
• Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §5.
• Ye et al. (2024) S. Ye, D. Kim, S. Kim, H. Hwang, S. Kim, Y. Jo, J. Thorne, J. Kim, and M. Seo FLASK: fine-grained language model evaluation based on alignment skill sets. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2.
• Yifei et al. (2025) L. S. Yifei, A. Chang, C. Malaviya, and M. Yatskar ResearchQA: evaluating scholarly question answering at scale across 75 fields with Survey-Mined questions and rubrics. External Links: 2509.00496, Document, Link Cited by: §2, §8.
• Zhao et al. (2026) J. Zhao, Z. Huan, Z. Wang, X. Zhang, J. Zhou, S. Verberne, and Z. Ren ReportLogic: evaluating logical quality in deep research reports. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8470–8502. External Links: Document, Link Cited by: §2.
• Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. Gonzalez, and I. Stoica Judging LLM-as-a-Judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems 36, pp. 46595–46623. External Links: Document, Link Cited by: §2.
• Zhu et al. (2025) F. Zhu, X. Y. Ng, Z. Liu, C. Liu, X. Zeng, C. Wang, T. Tan, X. Yao, P. Shao, M. Xu, Z. Wang, J. Wang, X. Lin, J. Li, J. Zhu, Y. Zhang, W. Wang, F. Feng, R. Hong, H. Luan, K. Huang, and T. Chua FinDeepResearch: evaluating deep research agents in rigorous financial analysis. External Links: 2510.13936, Document, Link Cited by: §2, §2.
• Zobel (1998) J. Zobel How reliable are the results of large-scale information retrieval experiments?. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 307–314. External Links: Document, Link Cited by: §2.

Appendix A Generic Argument-Flaw Rules

The judge sees report material without Web access. G01 covers errors identifiable from the text or common knowledge; G10 checks time references against the cutoff.

Descriptive diagnostics. Agent-level (τ,r) is (0.64, 0.91) for G07, (0.69, 0.73) for G03, (0.69, 0.76) for G08 and (0.60, 0.75) for G04. B flags G08 on 2.1% of questions; G04 rates are 32.9%/13.0%. Both judges flag Doubao least (61.5%/38.6%). Question-level κ≤0.29. These descriptive profiles are more consistent for report-internal features; G09/G01 can require external knowledge. They do not independently validate the rules.

Appendix B Aggregation Robustness

Table 6 contrasts reweighting both tiers with retaining one tier, using A’s fixed judgments.

For 1,607 same-task pairs within 20% in length, win rates are 60% for GLM, 54% for Doubao and 52% for Claude; their order remains unresolved. Log-length regression with agent/task fixed effects retains Claude first, followed by Mimo-V2.6-Pro under A (τ = 0.73 with main ranking) and GLM under B (0.87). Matching and regression adjustment are distinct, not a single corrected ranking.

Agents’ A-minus-B scores correlate with mean log report length (r = 0.82). Relative shifts are +0.9 [−0.4, 2.2] for Claude, the longest writer, −1.4 [−2.8, 0.0] for Kimi, the shortest, and +1.0 [−0.5, 2.5] for DeepSeek. DeepSeek’s length-adjusted shift is +0.4 [−1.4, 2.1]: no detected self-preference beyond length, but up to two points remains compatible.

Group averaging. The current scoring manifest places each checkpoint in its own group. Averaging within groups and then across groups therefore equals the checkpoint pass rate for each Target before the negative-criterion gate.

Appendix C Judge Prompt Structure

Evidence alignment. The archive prompt assigns report blocks to questions within a rubric dimension, returning IDs without evaluation.

Checkpoint evaluation. Decisions use full acceptance conditions, alternatives and exclusions. The judge cannot supply missing content from domain knowledge. It cites block IDs for yes, gives reasons for no and checks the negative criterion separately.

General evaluation. The prompt checks errors in actual claims, without Targets or scores. Findings identify question, rule, claim blocks, basis blocks and reason. Resource prompts produce JSON outputs subject to programmatic validation.

Appendix D Re-runs, Perturbations and Execution Failures

Repeat-run protocol. With a fixed seed, we sample ten research-question families without replacement, one task per family rotating across pool-size tertiles, and all ten agents (100 reports). Mean main-run T is 39.6, below the benchmark average; re-evaluation lowers scores by 1.8 points, with a shared agent shift. Perturbations use three agents on these ten tasks.

Repeat-run and perturbation experiments retain the main-run checkpoints but use condensed negative-criterion wording.

Variance proxy. Dividing the SD of paired repeat-minus-main report scores by the square root of two gives a report-level proxy of 4.1 points. Dividing this by the square root of 98 gives 0.42 points for the mean over these tasks. These steps assume independent, equal-variance run errors and independent errors across tasks. Score-interval half-widths combine the task-bootstrap half-width and 1.96 times the mean proxy by summing their squares and taking the square root. Agent-gap intervals additionally scale the mean proxy by the square root of two, assuming independent agent errors. Ranks and first-place fractions use task resampling alone. Both judges share A’s proxy, which includes wording variation; B-specific, correlated and generation errors are not separately estimated.

Perturbation results. Deletion lowers target-question T on 20/27 questions under A (95% Wilson interval (Wilson, 1927) 55–87%) and 22/27 under B (63–92%). Questions sharing deleted blocks also lose credit: 0.19 and 0.19 on average, versus 0.11 and 0.08 under unchanged re-evaluation.

Padding changes from the original run are −1.4 [−3.4, 0.5] (irrelevant) and −2.1 [−4.2, 0.1] (duplicated) under A, and +0.8 [−1.2, 2.9] and 0.0 [−1.7, 1.7] under B. A’s unchanged re-evaluation shifts scores by −2.5. Figure 3c reports paired net effects against each judge’s unchanged re-evaluation.

Reasoning settings and failures. High effort uses 8,127 tokens under A versus 8,137 at medium (Table 7). B’s high setting uses fewer tokens than A’s medium setting; labels do not imply matched budgets.

Output-validation failures (5 in the judge-B run, 20 in the repeat run and 15 in perturbations) are listed separately, not assigned zero.

Appendix E Research Types and Task Sampling

The 99 types combine 13 research-question families with 31 industries. Pools contain a median of 30 reports (range 10–60). The type list records pool size, question count, tier composition, median support and institutions per question.

Standardised screening. Fixed AI workflows and prompts check industry/question-family relevance, cross-object applicability and request scope (§4), without regenerating task rubrics or adapting standards to agent outputs.

Reports are ordered within each type by the SHA-256 hash of the seed, category ID and report ID, concatenated with vertical-bar separators. The seed is dr-bench-99-tasks-20261009-v1. The sampling record documents research-type, object-scope and readability checks, including three object adjustments and six wording changes on 2026-10-10. The incomplete task belongs to a type with nine questions, leaving 912 questions across the 98 common types.

Appendix F Further Discussion

Interpreting profiles. Figure 4 shows both tiers. Under A, GLM and Doubao are within half a point of Claude on consensus questions, but trail by about three and six on differentiating questions. B’s overall scores are 8.0 points lower. Requirement reduction affects task-level recommendations (Appendix H); comparisons aggregate 98 tasks.

Nugget evaluation. TREC nuggets define content for one input (Voorhees, 2003; Mayfield et al., 2024; Pradeep et al., 2025); our questions abstract analyses across objects. Assessor agreement (Lin and Demner-Fushman, 2006) and report support measure recurrence, but scoring uses tier weights (§3), not institution counts. A comparison can satisfy required metrics yet use inappropriate peers: its completeness and validity are distinct.

Appendix G Split-Half, Leave-One-Out, Source-Removal and k-Report Procedures

Table 9 summarises reference selection with frozen candidate questions and judgments. No rubrics are re-mined or reports re-judged.

Leave-one-out measures held-out report coverage. Split-half recomputes tiers using majority support, falling back to median support when tier counts differ by more than half the retained questions and the fallback gives a nonempty consensus tier.

Sampling k reports is without replacement (all reports for smaller pools). Controls draw equally many random questions per task. Task-source subsets use the report supplying the task (Appendix H).

Appendix H Single-Reference Rubrics and Per-Type Recommendations

Restricting evaluation to questions supported by the task’s source report changes the task-level leader on 50% of tasks and the top-three set on 76%. Size-matched random subsets change the leader on 45% of tasks (5th–95th percentiles 38–52%). Task-level rankings are therefore sensitive to both tested reductions of the requirements, motivating our focus on aggregate agent comparisons.

Appendix I First-Author Audit: Protocol and Results

The first author is a graduate student in finance and also participated in designing the framework. Annotation began from blank forms, without AI assistance or access to the model judgments. This is an author audit blind to model labels, not independent expert validation. It was conducted after the evaluated resource was fixed and does not alter the atlas, model judgments or agent scores.

Checkpoint sampling and materials. The 300 items were sampled with seed 20261023 from atlas checkpoints judged by both models. Quotas were 100 both-yes, 100 both-no, 50 A-yes/B-no and 50 A-no/B-yes. Each report–question contributed at most one item and each task at most six. Questions with no aligned passage under either judge were excluded. The author received the union of the passages aligned by both judges, not necessarily the same material either judge had evaluated. Disagreement can therefore reflect evidence selection as well as interpretation of a checkpoint.

Final labels. The author returned 177 yes and 123 no labels, with no abstentions or blanks. All 300 labels are confirmed as final judgments. Agreement is 223/300 with A and 215/300 with B, with κ = 0.487 and 0.433. Table 10 gives the confusion counts and stratified agreement. The model judges agree on 200/300 items (66.7%) in this deliberately disagreement-enriched sample, rather than the full-corpus 85.4%.

Of the items, 166 concern consensus questions and 134 differentiating questions. Author–A agreement is 74.1% and 74.6%, respectively (κ = 0.468/0.500); author–B agreement is 68.7% and 75.4% (κ = 0.359/0.514). All items are included in the primary result.

Reweighting. The stored checkpoint population contains 36,457 both-yes, 33,149 both-no, 8,814 A-yes/B-no and 3,049 A-no/B-yes items. Weighting the four sample agreement rates by these proportions gives 81.3% for A and 78.2% for B. These auxiliary estimates do not correct the report–question and task caps or the exclusion of empty evidence, and are not unbiased estimates of population accuracy.

Source-question mappings. The audit sampled 25 consensus and 25 differentiating questions, at most one per research type and six source mappings per question. Among 297 mappings, 231 were marked yes (77.8%), 45 partial (15.2%) and 21 no (7.1%). Partial mappings are reported separately: a source question may address only part of a broader canonical question. This audit examines question grouping, not every Target’s fidelity to source arguments or applicability to a task object.

Report preferences. The optional audit included 20 same-task report pairs stratified by judge-A score gaps below 3, from 3 to 8, and at least 8 points. The author’s final judgments preferred the first report in seven pairs, the second in six, and judged seven comparable. Among 13 directional preferences, eight (61.5%) matched A’s completeness ordering; the seven ties are not counted as errors.

This small author-only sample does not validate a general expert-preference ranking.

注意转载授权 · 阅读原文