Pith. sign in

REVIEW 3 major objections 4 minor 6 cited by

DEER is a benchmark that evaluates deep research reports on 101 expert-designed rubric items and verifies both cited and uncited claims against external sources, making report quality diagnosable rather than a single score.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 15:09 UTC pith:FIMZCRZX

load-bearing objection DEER is a well-designed benchmark with solid human-correlation evidence for its rubric judge; the uncited-claim verification is genuinely new but oversold and needs an error-propagation analysis. the 3 major comments →

arxiv 2512.17776 v5 pith:FIMZCRZX submitted 2025-12-19 cs.CL

DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation

Janghoon Han , Heegyu Kim , Changho Lee , Dahm Lee , Min Hyung Park , Hosung Song , Stanley Jungkyu Choi , Moontae Lee
show 1 more author
Honglak Lee
This is my paper
classification cs.CL
keywords deep researchbenchmarkLLM-as-a-judgeclaim verificationexpert rubricreport generationevaluation taxonomyevidence quality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DEER asks whether the quality of AI-generated 'deep research' reports can be evaluated rigorously, and answers yes with a reusable instrument. It combines a fixed, expert-built taxonomy—7 dimensions, 25 subdimensions, 101 concrete rubric items—with task-specific Expert Evaluation Guidance that tells an LLM judge what an expert report for that task must contain. On the evidence side, it verifies every verifiable claim, including uncited ones, by recovering the citations they implicitly depend on. The benchmark's first results show that today's deep research systems write structurally plausible, well-cited reports but consistently fall short on fulfilling expert-level requests and on analytical soundness. The value of DEER, if it works, is that it turns 'this report is bad' into specific, diagnosable deficiencies that system builders can act on.

Core claim

The paper claims that expert-level report quality can be decomposed into a shared hierarchical rubric—7 dimensions, 25 subdimensions, 101 atomic Coverage/Quality items—applied identically across tasks and supplemented with task-specific expert guidance. This decomposition, the paper argues, makes LLM judges align better with human experts (correlation 0.75 vs. 0.81 inter-human) and agree more with each other (reliability 0.55). It further claims that a report-level verification module can check far more than cited sentences: classifying claims into six types and using back-tracking to recover citations implicit claims depend on, then verifying Types A–C against retrieved evidence. Benchmark

What carries the argument

The load-bearing mechanism is a four-level rubric scaffold: dimension → subdimension → criterion → rubric item, each item scored separately for Coverage (is the content present?) and Quality (how well is it executed?). Two arms execute it. Report-quality dimensions use an LLM judge given the fixed rubric plus the task's Expert Evaluation Guidance, an expert-written list of mandatory checkable content; the paper shows this guidance is what lifts judge agreement and reliability. External-information dimensions use a module that extracts atomic claims, classifies them (types A–F), and applies semantic back-tracking to recover citations for uncited claims: an LLM names the earlier sentence the c

Load-bearing premise

The load-bearing premise sits in Section 4.4 / Appendix F.2: the verification of uncited claims depends on the LLM correctly naming the earlier sentence the claim relies on, and the paper's own test shows that pointer is right for only about 7 in 10 implicit claims—so the 'Supported' labels and the Integrity/Sufficiency dimension scores inherit that error for roughly three in ten claims.

What would settle it

Take a set of deep research reports, have experts mark the true evidence sentence for each uncited claim, then run DEER's back-tracking and verification; if the automatically generated pointers disagree with experts on a substantial share of claims and correcting those pointers flips a meaningful fraction of 'Supported' verdicts, the Integrity/Sufficiency scores would be measuring back-tracking accuracy rather than claim factuality. A cheaper variant: replace the report's sentences with a decoy sentence placed nearby and check how often the LLM still points to the original evidence.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Adding task-specific expert guidance to a fine-grained rubric raises LLM–human correlation and inter-evaluator reliability; a fixed rubric alone can actually lower both, so guidance is the active ingredient.
  • Deep research systems score high on structure, format, and ethics but low on fulfilling expert requests and on analytical soundness, so the next improvement bottleneck is intent alignment and reasoning depth, not writing quality.
  • Verifying only explicitly cited claims leaves most of a report unchecked; DEER's back-tracking implies that integrity scores from citation-only benchmarks are optimistic.
  • Integrating web search improves information-related dimensions but can degrade request fulfillment and analytical soundness, suggesting retrieval must be balanced against argument structure.
  • The fixed rubric across 50 tasks yields rubric-item-level diagnostics, so system comparisons can identify which specific criteria and domains need work rather than relying on aggregate scores.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the back-tracking pointer is wrong for roughly three in ten implicit claims, reporting Integrity and Sufficiency separately for explicitly cited versus recovered-citation claims would tell users which part of the score is trustworthy; the paper does not split these.
  • The strong effect of Expert Evaluation Guidance suggests the guidance documents, not the generic rubric, carry most of the evaluation value; a direct test would be measuring how guidance length, specificity, and coverage predict judge agreement.
  • The six-type claim scheme could be turned into a training objective for deep research systems: reduce Type F claims (claims needing evidence with no source) and improve back-tracking accuracy, and Integrity scores should rise.
  • The finding that search hurts analytical soundness could be tested by ablating the number of retrieved sources per query; DEER's metrics make that ablation straightforward, but the paper does not run it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces DEER, a benchmark for evaluating deep research agents on expert-level report generation. DEER comprises 50 report-generation tasks across 13 domains, a taxonomy of 7 dimensions and 25 subdimensions operationalized as 101 fine-grained rubric items, task-specific Expert Evaluation Guidance, and an information-verification module that verifies both cited and uncited claims. The authors validate the LLM-judge framework against expert human ratings on 45 reports, measure inter-evaluator reliability, and evaluate five deep-research/LLM systems. The central claim is that DEER provides an interpretable, diagnostic, reusable evaluation instrument rather than a single aggregate score.

Significance. If the proposed benchmark works as claimed, it is a useful contribution: the taxonomy is grounded in a broad set of professional standards, the fixed-rubric design supports cross-task comparability and diagnosis, and the human-correlation study (Pearson r=0.75 with Expert Guidance vs. 0.81 inter-human) is a genuine validation effort. The claim-verification architecture is also more ambitious than prior citation-only checkers, and the reported cost/accuracy ablations are transparent. However, the central novelty—verification of uncited claims via semantic back-tracking—has an unresolved error-propagation problem that undermines the strength of the claim in the abstract and of the Information Integrity/Sufficiency scores.

major comments (3)
  1. [§4.4, Eq. (1); Appendix F.2, Table 15] The back-tracking mechanism for uncited claims is the paper's main differentiator, but its evaluation leaves an unquantified gap. Table 15 reports Jaccard 0.707 and precision 0.711 on N=131 B/C claims; this means ~29% of recovered citation links point to the wrong evidence sentence. Since Eq. (1) uses those links to form the verification context, wrong pointers can produce incorrect Supported/Not Supported labels even when the verifier itself is perfect. The abstract's unqualified statement that DEER 'verifies both cited and uncited claims' therefore overstates what is demonstrated. I request an end-to-end error analysis: e.g., run the full pipeline on the 100 human-annotated A–C claims, report Supported-label accuracy broken down by claim type, and quantify how pointer errors affect the Integrity/Sufficiency dimension scores. At minimum, provide a conservative bound or explicit caveat.
  2. [§6.3; Appendix H] The human-correlation validation is limited to 45 reports from 15 tasks in 5 domains, and experts assign only an overall 1–5 quality score. The paper's stated contribution is fine-grained, diagnostic evaluation at the rubric-item level, but the correlation evidence is only at the aggregate report level. It is therefore not established that the 101 rubric-item scores or the 25 subdimension scores align with expert judgments. I recommend reporting human-model correlation at the subdimension or criterion level for at least a subset of reports, or explicitly scoping the validity claim to overall scores.
  3. [Appendix F.3, Table 16] The selected verification configuration, GPT-5-mini with Batch 20, Low Effort, Top-K=2, achieves 77.0% accuracy on the Original dataset and 88.5% on the Adversarial dataset. Even with a correct evidence pointer, 23% of original-dataset claims are misclassified. When the back-tracking pointer error (~29%) is also considered, the end-to-end reliability of the uncited-claim component is materially lower than the isolated verification numbers suggest. The paper should report joint performance on the human-annotated claims, or at least discuss the combined error budget. Without this, the Integrity/Sufficiency scores reported in Table 1 lack a quantified reliability statement.
minor comments (4)
  1. [Table 5] The table formatting is broken: the header row shows '%' and the rows are difficult to parse. Please provide a clear layout with column labels such as 'Group size', 'Retrieval', 'F1', and 'Cost ($/1k claims)'.
  2. [§5.1 and Appendix G] The main text says the judge is 'GPT-5.2' while Table 1 and Appendix G state 'GPT-5'. Clarify the model naming and whether the same backbone is used throughout.
  3. [Appendix F.2] The first two paragraphs of F.2 are nearly identical; one appears to be a duplication artifact. Remove the duplicate.
  4. [Appendix F.4] The normalization divisors for quantity-based metrics (D_info=15, D_cit=10, D_ref=4) are presented without justification. Please explain how these were chosen and report sensitivity of the dimension scores to the divisors.

Circularity Check

0 steps flagged

No significant circularity: DEER's rubric and verification components are validated against independent human judgments, and the reported back-tracking precision limitation is a correctness risk, not a circular reduction.

full rationale

DEER is a benchmark-construction paper rather than a derivation of predictions from fitted parameters. The report-quality taxonomy is synthesized from 80 external reporting standards (Appendix C), and the task-specific Expert Evaluation Guidance is explicitly derived from the underlying HLE items during benchmark construction (Section 3, Appendix B.4); using gold-derived criteria to score reports is normal benchmark design, not a circular prediction. The LLM-as-a-judge method is validated against independent human expert overall ratings (Section 6.3, Table 2), not against DEER's own rubric outputs, and inter-evaluator reliability is measured separately (Section 6.4). The information-verification module is evaluated against human-annotated claim labels (Appendices E and F), and the back-tracking step's limited precision (Jaccard 0.707, precision 0.7109 on N=131, Table 15) is an acknowledged accuracy floor that propagates into downstream labels; this is a correctness/robustness concern, not an equation-level circularity. Equations (1)-(5) and the metric definitions are definitions and aggregations rather than derived predictions. Self-citations (e.g., Prometheus, HLE) appear only as prior work or data sources and are not load-bearing for the paper's central claims.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The benchmark introduces no physical entities or new conserved quantities. The new constructs—taxonomy, rubrics, Expert Evaluation Guidance, and the back-tracking module—are measurement instruments, not invented entities in the sense that requires independent falsifiable evidence. The free parameters are normalization and configuration constants that affect reported scores but are not fitted to an external target.

free parameters (2)
  • Normalization divisors for quantity metrics (D_info=15, D_cit=10, D_ref=4) = 15 / 10 / 4
    In Appendix F.4, quantity-based Sufficiency metrics are converted to 0-10 with a step function using hand-chosen divisors. These divisors directly affect the Information Sufficiency dimension scores and are not derived from external benchmarks.
  • Batch size B=20 and retrieval Top-K=2 = 20 and 2
    Selected by the ablation in Table 16 as a cost-accuracy trade-off. These are configuration choices rather than fitted scientific parameters, but they change the reported claim-verification performance.
axioms (4)
  • domain assumption An LLM judge can apply 101 fixed rubric items to long reports and produce valid 1-10 scores when supplied with the rubric and Expert Evaluation Guidance.
    This powers the entire report-quality scoring pipeline. It is validated only by correlation with human overall scores on 45 reports; no item-level human rubric scores are used.
  • domain assumption Expert Evaluation Guidance derived from HLE items encodes the mandatory content of a good expert report and does not leak answers.
    Sections 3 and B.4 describe deriving guidance from the task prompt and key concepts implied by the underlying HLE item. If guidance contains answer cues, high scores can reward reproducing gold content rather than genuine analysis, and HLE contamination remains possible.
  • domain assumption Human expert overall ratings on a 1-5 scale are the correct ground truth for validating the LLM judge.
    Appendix H uses 90 ratings on 45 reports and only overall scores. Fine-grained subdimension and rubric-item scores are not validated against human item-level judgments.
  • domain assumption LLM-based semantic recall (GPT-5 as judge) is a reliable measure of claim-extraction coverage.
    In Appendix F.2, claim-extraction recall in Table 4 is measured by an LLM judge comparing ground-truth claims to extracted claims. The extraction-quality claims therefore inherit LLM-judgment limitations.

pith-pipeline@v1.3.0-alltime-deepseek · 44456 in / 12980 out tokens · 144903 ms · 2026-08-03T15:09:45.338163+00:00 · methodology

0 comments
read the original abstract

Recent advances in large language models have enabled deep research systems that generate expert-level reports through multi-step reasoning and evidence-based synthesis. However, evaluating such reports remains challenging: report quality is multifaceted, making it difficult to determine what to assess and which criteria to use; LLM-based judges may miss errors that require domain expertise to identify; and because deep research relies on retrieved evidence, report-wide claim verification is also necessary. To address these issues, we propose DEER, a benchmark for evaluating expert-level deep research reports. DEER systematizes evaluation criteria with an expert-developed taxonomy (7 dimensions, 25 subdimensions) operationalized as 101 fine-grained rubric items. We also provide task-specific Expert Evaluation Guidance to support LLM-based judging. In addition to rubric-based assessment, we propose a claim verification architecture that verifies both cited and uncited claims and quantifies evidence quality. Experiments show that current systems produce structurally plausible, evidence-citing reports, but still struggle to fully satisfy expert-level user requests and achieve logical completeness. Beyond performance comparisons, DEER makes system strengths and limitations interpretable and provides diagnostic signals for improvement.

Figures

Figures reproduced from arXiv: 2512.17776 by Changho Lee, Dahm Lee, Heegyu Kim, Honglak Lee, Hosung Song, Janghoon Han, Min Hyung Park, Moontae Lee, Stanley Jungkyu Choi.

Figure 1
Figure 1. Figure 1: Deep Research System Performance Comparison. Results for five systems on the proposed benchmark. Google, 2025; Anthropic, 2025; Yang et al., 2025; Li et al., 2025b; Huang et al., 2025; Li et al., 2025c). Unlike con￾ventional web search, these systems address complex re￾search queries by decomposing them into multiple steps and dynamically seeking additional information based on intermediate results. Throug… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the DEER evaluation framework. (a) Research question and expert guidance generation from real-world deep research queries. (b) Construction of the Deep Research Evaluation Taxonomy consisting of 7 dimensions, 25 sub-dimensions, and 101 granular rubrics. (c) The DEER evaluation pipeline, integrating expert-guided LLM-as-a-judge scoring with claim extraction and information verification to assess… view at source ↗
Figure 3
Figure 3. Figure 3: Heatmap visualizations of expert report evaluation results. (a) Criteria-wise scores across detailed evaluation categories. (b) Domain-wise scores averaged from each domain. Evaluation Method Pearson Spearman Pairwise r ρ Agr. Vanilla 0.64(0.16) 0.61(0.17) 0.66(0.14) + Dimensions 0.67(0.10) 0.65(0.14) 0.80(0.07) + Granular Rubrics 0.62(0.14) 0.59(0.17) 0.78(0.08) + Expert Guidance 0.75(0.07) 0.71(0.06) 0.8… view at source ↗
Figure 4
Figure 4. Figure 4: Topic domains extracted from real-world Deep Research service logs benchmark’s target domain distribution [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Task Query Example B.4. Construction of Expert Evaluation Guidance For each task, we constructed Expert Evaluation Guidance to specify what an expert report for the given prompt must cover. The Guidance includes only the mandatory elements required by the topic (excluding optional content or stylis￾tic preferences) and describes each element in as concrete and verifiable a form as possible so that complian… view at source ↗
Figure 6
Figure 6. Figure 6: Example of Expert Evaluation Guidance 31 [PITH_FULL_IMAGE:figures/full_fig_p031_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Simplified batch extraction prompt structure and an ex￾ample JSON output. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Abbreviated evaluator prompt template for Request Fulfillment. We omit the full rubric item list, detailed scoring bands, and the JSON output schema for space. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Full prompt for Claim Extraction and Classification. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Full prompt for Claim Verification. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?

    cs.CL 2026-05 unverdicted novelty 7.0

    REFLECT benchmark shows current LLM judges achieve below 55% accuracy detecting failures in evidence-based research agents, especially on evidence verification.

  2. ICBCBench: An Industry Consortium Benchmark for Financial Deep Research

    cs.CE 2026-06 unverdicted novelty 6.0

    ICBCBench is a new consortium-built benchmark that jointly measures retrieval-reasoning accuracy and end-to-end report quality for deep research agents in finance.

  3. Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation

    cs.AI 2026-03 unverdicted novelty 6.0

    XpertBench provides 1,346 rubric-scored expert tasks showing leading LLMs achieve a maximum ~66% success rate and ~55% mean score across domains.

  4. FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

    cs.CL 2026-07 conditional novelty 5.5

    A multi-LLM consensus pipeline turns 14,450 auto-generated candidate rubrics into 2,600 distinguishable gold rubrics that rank 10 financial deep-research systems from 58.58% to 22.23% pass rate.

  5. FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

    cs.CL 2026-07 conditional novelty 5.0

    A benchmark whose 2,600 'gold' rubrics are generated, validated, and applied entirely by LLMs — with no human in the final loop — differentiates 10 financial deep-research systems across a 36-point pass-rate spread.

  6. Mind DeepResearch Technical Report

    cs.AI 2026-04 unverdicted novelty 5.0

    MindDR combines a Planning Agent, DeepSearch Agent, and Report Agent with SFT cold-start, Search-RL, Report-RL, and preference alignment to reach competitive scores on research benchmarks using 30B-scale models.

Reference graph

Works this paper leans on

36 extracted references · 1 canonical work pages · cited by 5 Pith papers

  1. [1]

    Score= min(max(R,0),1)×10 • Quantity-based metrics(e.g., Counts): Scored via step function with divisors D (Info=15, Cit=10, Ref=4)

    Normalization (Metric Level) Each raw metric value is first converted to a 0–10 scale: • Ratio-based metrics(e.g., Factuality): Scaled linearly. Score= min(max(R,0),1)×10 • Quantity-based metrics(e.g., Counts): Scored via step function with divisors D (Info=15, Cit=10, Ref=4). Score= min max(N−1,0) D + 1,10

  2. [2]

    • Dimension Level: Average of criterion scores within each dimension

    Aggregation • Criterion Level: Average of normalized metric scores within each criterion. • Dimension Level: Average of criterion scores within each dimension. G. Baseline Model Details We use the following backbone model families in our ex- periments: Qwen3-235B, Gemini 2.5, Claude Opus 4.5, and GPT-5. For readability, we refer to the GPT family asGPT-5i...

    2025

  3. [3]

    └[3.1.1.2 (Quality)]The introduction must be sufficiently developed for a professional report, and each component specified in 3.1.1.1 must be treated with adequate depth

    Structural Coherence └3.1 Introduction 3.1.1 CriterionDoes the introduction clearly present the report’s topic, problem, and significance, avoiding excessive generalization or irrelevant topic development? Does it also provide sufficient context and motivation for the reader? └[3.1.1.1 (Coverage)]The introduction must include the report’s topic, problem, ...

  4. [4]

    Well-established terms that are standard in the field do not require formal definitions if their meaning is clear from context

    Format & Style └4.2 Writing Quality 4.2.3 CriterionAre technical terms defined when they first appear and used consistently thereafter? └ [4.2.3.1 (Coverage)]Technical terms and field-specific concepts must be defined when they are central to the argument, potentially ambiguous, or not guaranteed to be known by the intended audience. Well-established term...

  5. [5]

    Standard for professional ethics and responsibil- ity in environmental research

    URL https://www.esa.org/about/code-of-e thics/. Standard for professional ethics and responsibil- ity in environmental research. EQUATOR Network. The equator network: Enhancing the quality and transparency of health research, 2025. URLhttps://www.equator-network.org/. Umbrella authority for all health research reporting guidelines. European Mathematical S...

    Pith/arXiv arXiv 2025

  6. [8]

    Rein, D., Hou, B

    URLhttps://arxiv.org/abs/2501.14249. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. https:// arxiv.org/abs/2311.12022, 2023. Accessed: 2025-10- 14. Rhoades, S. A. The herfindahl-hirschman index.Federal Reserve Bulletin, pp. 188–189, 1993. URL ht...

    Pith/arXiv arXiv 2023

  7. [9]

    Standard for quantitative rigor and model reproducibility in economics

    URL https://www.econometricsociety.o rg/publications/econometrica/information-a uthors. Standard for quantitative rigor and model reproducibility in economics. The Journal of Organic Chemistry. Author guidelines: Stan- dard for characterization of organic compounds, 2025. URL https://pubs.acs.org/journal/joceah. The strict standard for reporting chemical ...

    2025

  8. [10]

    Standard for scientific integrity, peer review, and impartial reporting in earth sciences

    URL https://www.usgs.gov/about/organ ization/science-quality-and-integrity/funda mental-science-practices. Standard for scientific integrity, peer review, and impartial reporting in earth sciences. U.S. Securities and Exchange Commission. Regula- tion s-k: Standard instructions for filing forms, 2024. URL https://www.ecfr.gov/current/title-17/ch apter-II/...

    arXiv 2024

  9. [12]

    Concurrent work.Concurrently with DEER, several benchmarks have further advanced report-level evaluation

    evaluates multi-domain web-based deep research reports using LLM-generated evaluation criteria (RACE), citation-based fact-checking (FACT), and dimensions in- cluding coverage, depth, presentation, and citation accuracy. Concurrent work.Concurrently with DEER, several benchmarks have further advanced report-level evaluation. DeepResearchArena (Wan et al.,...

    2025

  10. [13]

    ungrounded

    provides 2,500+ expert-written rubric items spanning axes such as explicit requirements, synthesis, and reference use, with mandatory vs. optional criteria per task. A.2. Key Differences from Prior Work Alignment with expert standards.A key concern in deep-research evaluation is whether the reported score truly reflects expert notions of report quality. I...

    2025

  11. [16]

    Completeness is judged against the EG; if the explanation for any required element falls short of the EG standard, that element is considered omitted

    Request Fulfillment └1.1 Completeness 1.1.1 CriterionDoes the report include all required elements without omission and present each clearly? └[1.1.1.1 (Coverage)]The report must include all elements required by the User Query and the Expert Evaluation Guidance (EG), and each element must be presented with clear and understandable explanations. Completene...

  12. [17]

    Analytical Soundness └2.2 Reasoning 2.2.5 CriterionAll major claims follow logically from the previously presented facts, data, interpretations, and assumptions, without skipped steps or unsupported leaps. └[2.2.5.1 (Coverage)]All claims that require logical support are explicitly linked to the relevant facts, data, and interpretations (including the key ...

  13. [20]

    Multi-junction solar cells achieve efficiencies above 45% [1]

    Ethics & Compliance └5.2 Safety & Impact 5.2.1 CriterionAre the potential impacts of proposed policies, technologies, strategies, or research outcomes sufficiently considered, including key implications, possible side-effects, and interpretations from multiple perspectives (when essential)? └[5.2.1.1 (Coverage)]Potential side-effects or limitations are di...

    arXiv

  14. [21]

    Identify all required elements for this rubric item

  15. [22]

    For each required element, evaluate the relevant parts of the report as Pass/Fail (met/not met)

  16. [23]

    Classify Fails as core gaps vs minor omissions

  17. [24]

    Assign a 1–10 score based on the number and type of Fails. **Scoring Guidelines:** [Omitted for brevity; see Table 12.] **Core Principles:** * Even one core gap makes Excellent (7-8) impossible * Multiple core gaps make Good (5-6) impossible ## 5.2 Quality (Q) Evaluation This item evaluates how well the report executes the relevant written content for the...

  18. [25]

    Evaluate only the written parts relevant to this rubric item, and do not evaluate omissions

  19. [26]

    Make an overall-level judgment (academic/professional level) based on the written content, and assign a provisional 1–10 score

  20. [27]

    description

    If a core element falls short of the provisional score level, adjust the final score downward to match the level of that core element. **Scoring Guidelines:** [Omitted for brevity; see Table 12.] **Core Principles:** * If an EG core element falls short, the overall Q score should be lowered accordingly; weaknesses in non-core elements have limited impact ...

    2024

  21. [28]

    Read the Report Context to understand the global context

  22. [29]

    Target Sentences

    Process the "Target Sentences": - Break down the text into atomic claims. A single sentence may contain multiple claims (e.g., "X is Y, and Z requires W" -> Claim 1: "X is Y", Claim 2: "Z requires W"). - Extract ALL statements, including facts, opinions, structural descriptions, and summaries

  23. [30]

    Contains numbers, chemical properties, specific results? -> Likely A, B, C, or F

    For each extracted claim, analyze its relationship with the context and citations: - Step 1: Specificity Check. Contains numbers, chemical properties, specific results? -> Likely A, B, C, or F. - Step 2: Citation Check. - Citation in same sentence? -> Class A. - Citation in same paragraph? -> Class B. - Citation in previous section? -> Class C. - Step 3: ...

  24. [31]

    - Must contain an explicit citation

    Determine Evidence Position: - For Class B or C, identify the exact sentence index (e.g., "L1.S1") that contains the citation supporting this claim. - Must contain an explicit citation

  25. [32]

    Most perovskites are unstable[1], but our new material is stable

    Output Format: - Return a JSON object with a list of claims. - Each claim must include:`position`(line/sent index from input),`claim`(text),`claim_type`(A-F),`rationale`, `numeric`(bool),`citations`(list of strings),`implicit_citations`(list),`cross_references`(list). Input Format: # Report Excerpt ... # Target Sentences L1.S1: ... L1.S2: ... Extraction a...

  26. [33]

    Most perovskites are unstable

    "Most perovskites are unstable." (Class A, citations=['1'])

  27. [34]

    Our new material is stable

    "Our new material is stable." (Class E - Author's finding, or Class F if it lacks proof provided elsewhere) Examples: *Example Input:* L1.S1: Several studies[1] have shown that urban green spaces can reduce ambient air temperatures by up to 2°C. This is crucial. *Example Output (Conceptual):*

  28. [35]

    Several studies have shown that urban green spaces can reduce ambient air temperatures by up to 2°C

    Claim: "Several studies have shown that urban green spaces can reduce ambient air temperatures by up to 2°C." - Class: A - Citations: ["1"] - Position: "L1.S1"

  29. [36]

    This is crucial

    Claim: "This is crucial." - Class: B (supported by L1.S1) - Evidence Position: "L1.S1" - Position: "L1.S1" Figure 9.Full prompt for Claim Extraction and Classification. 38 DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation Full Prompt for Claim Verification You are an expert fact-checker. Verify the following claims against ...

  30. [2017]

    doi: 10.1007/978-94-024 -0881-2_11

    ISBN 9789402408799. doi: 10.1007/978-94-024 -0881-2_11. Association for Computational Linguistics. Acl rolling review author guidelines and responsible nlp research checklist, 2024. URL https://aclrollingreview.o rg. Association for Computing Machinery. Acm arti- fact review and badging policy v1.1, 2025. URL https://www.acm.org/publications/policies/ar t...

    work page doi:10.1007/978-94-024 2024

  31. [2018]

    Pri- mary ethical standard for sociological reporting and in- tegrity

    URL https://www.asanet.org/about/gover nance-and-leadership/council/code-ethics . Pri- mary ethical standard for sociological reporting and in- tegrity. American Sociological Association.ASA Style Guide. Amer- ican Sociological Association, 7th edition, 2022. Stan- dard for writing mechanics and citation in sociology. Anthropic. Meet claude. https://www.a...

    2022

  32. [2021]

    Standard for ethical prac- tice, respect, and responsibility in psychology

    URL https://www.bps.org.uk/guideline/c ode-ethics-and-conduct . Standard for ethical prac- tice, respect, and responsibility in psychology. Center for Open Science. The preregistration revolution,

  33. [2022]

    how well the system finds information,

    URL https://ies.ed.gov/ncee/wwc/Handbo oks. The governing standard for evidence validity in educational research. Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., et al. The fair guiding principles for scientific data management and stewardship.Scientific Data, 3:160018, 2016. doi: 10.1038/sdata.2016.18. World Medical Association. Wma declaration of h...

    Pith/arXiv arXiv 2016

  34. [2023]

    URLhttps://arxiv.org/abs/2311.12983. Mohr, P. J. et al. Codata recommended values of the fun- damental physical constants: 2022.Reviews of Modern Physics, 2024. URL https://physics.nist.gov/cuu /Constants/. The international standard for numeric accuracy of physical constants. NASA. Nasa systems engineering handbook, rev 2. Techni- cal Report NASA/SP-2016...

    Pith/arXiv arXiv 2022

  35. [2024]

    The standard for defining scope boundaries and hypothesis limits in behavioral science

    URL https://www.cos.io/initiatives/pre reg. The standard for defining scope boundaries and hypothesis limits in behavioral science. CFA Institute. Global investment performance standards (gips), 2020. Standard for fair representation and full disclosure of investment performance. CFA Institute. Code of ethics and standards of professional conduct, 2024. U...

    Pith/arXiv arXiv 2020

  36. [2025]

    Engineer- ing standard for verifying data reliability and system per- formance

    URL https://standards.ieee.org/. Engineer- ing standard for verifying data reliability and system per- formance. IFRS Foundation. International <ir> framework, 2021. URL https://www.integratedreporting.org/reso urce/international-ir-framework/. Standard for connecting strategy, governance, and performance in cor- porate reports. Institute of Education Sci...

    Pith/arXiv arXiv 2021