Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Current AI search and deep-research agents cite accurately only 40–80% of the time, the DeepTRACE audit finds, and they often present one-sided, overconfident answers whose statements their own listed sources do not support.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An audit framework and empirical study showing that generative search engines and deep research agents frequently produce one-sided answers and weakly supported citations, with citation accuracy between 40 and 80%.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful, timely audit framework with plausible findings, but the headline rates rest on a moderately validated LLM judge and lack the agreement/reliability detail needed to take the exact numbers at face value. the 4 major comments →

arxiv 2509.04499 v1 pith:YZAAUP3E submitted 2025-09-02 cs.CL cs.AI

DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence

Pranav Narayanan Venkit , Philippe Laban , Yilun Zhou , Kung-Hsiang Huang , Yixin Mao , Chien-Sheng Wu This is my paper
classification cs.CL cs.AI
keywords deep research agentsgenerative search enginescitation accuracyfactual groundingaudit frameworkLLM confidencedebate queriesAI reliability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DeepTRACE is an audit framework that turns user-reported failure modes of generative search engines and deep-research agents into eight measurable metrics spanning answer text, sources, and citations. The paper applies it to 303 queries run through nine public systems and claims that these systems frequently answer debate-style questions one-sidedly and with high confidence, while a large share of their statements are unsupported by the very sources they list. Deep-research modes lower overconfidence and can cite more thoroughly, but they remain mostly one-sided and still produce many unsupported statements, with citation accuracy between 40 and 80 percent across systems. If the audit is right, following a citation does not reliably verify the claim it is attached to, so current systems do not deliver the source-grounded synthesis they advertise.

Core claim

At its core, the paper claims that source-grounded synthesis is not being delivered at scale by either class of system. Across 303 queries and nine public products, it measures one-sidedness, overconfidence, relevance, uncited sources, unsupported statements, source necessity, citation accuracy, and citation thoroughness. Generative search engines produce concise answers but show one-sided framing on roughly half to 90 percent of debate queries and overconfidence up to more than 80 percent. Deep-research agents reduce overconfidence below 20 percent but remain one-sided on 54.7–94.8 percent of debate queries, and unsupported-statement rates reach as high as 97.5 percent. The paper's positive

What carries the argument

The evaluative core is a pair of matrices built from the answer and its sources. Each answer is decomposed into statements; a citation matrix records which statement cites which listed source, and a factual-support matrix records whether each source actually supports each statement. From these two matrices the paper derives eight metrics, including citation accuracy as the overlap of the matrices and citation thoroughness as the fraction of all true support relations that are cited. Source Necessity is computed by finding the minimum number of sources needed to cover all supported statements, treated as a bipartite-graph vertex cover problem. An LLM judge supplies the support labels and conf

Load-bearing premise

The audit's central numbers depend on an automated judge labeling which sources support which statements, and that judge agrees with human raters only moderately; if the judge is systematically too lenient or too strict, the unsupported-statement and citation-accuracy rates shift.

What would settle it

Hand-annotate a random sample of 200 (statement, source) pairs from the released DeepTrace corpus and compare human support labels to the judge's labels. If a system reported at 40–50 percent citation accuracy is above 90 percent under human labels, the central finding fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Users cannot reliably treat citations as verification: with citation accuracy between 40 and 80 percent, a clicked source frequently will not say what the surrounding text claims.
  • Long answers and long source lists do not buy grounding; some systems with abundant sources still leave more than half their statements unsupported.
  • Reducing overconfidence is achievable in deep-research mode, so hedging is a design choice rather than an unavoidable LLM limitation.
  • One-sidedness on debate queries persists even when the same question is framed pro and con, which risks reinforcing the user's prior perspective.
  • Because the framework is modular and threshold-based, the same eight metrics can be rerun as systems update, turning reliability into a trackable quantity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the reported 40–80 percent range is a snapshot of nine specific public systems on 303 curated questions as of August 2025; reapplying DeepTRACE to the same queries after model updates would show whether citation reliability improves or degrades.
  • A stricter or more lenient human standard for factual support could move individual systems across the paper's own acceptable/borderline/problematic thresholds, so the scorecard is best read as comparative rather than absolute.
  • The same citation/factual-support matrix machinery could be adapted to multimodal outputs such as screenshots and tables, and to non-English queries, since the metrics are defined at the statement level and do not depend on the presentation medium.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. DeepTRACE proposes an automated audit framework for generative search engines (GSEs) and deep research (DR) agents, defining eight metrics that span answer text, source use, and citation behavior. The framework decomposes answers into statements, assigns relevance, pro/con, and confidence labels, builds citation and factual-support matrices, and computes metrics such as one-sidedness, unsupported statements, citation accuracy, and citation thoroughness. It is applied to 303 queries across nine public GSE/DR systems using a GPT-5-based LLM judge for intermediate annotations, with human validation reported only for confidence scoring and factual support. The headline findings are that GSEs and DR agents frequently produce one-sided, overconfident answers on debate queries, contain large fractions of unsupported statements (up to 97.5%), and have citation accuracy between roughly 40% and 80%.

Significance. If the quantitative findings hold, the paper makes a significant contribution to end-to-end evaluation of deployed search and research assistant systems. The framework is sociotechnical in motivation, builds directly on a prior usability study, and is modular and reusable. The authors also release the DeepTrace dataset and make their prompts and extraction pipelines transparent, which supports replication and extension. The central value of the paper is the audit framework itself and the cross-system comparison it enables. However, the validity of the headline numbers depends heavily on the reliability of the LLM judge, and the current validation evidence is not strong enough to anchor the specific percentages claimed.

major comments (4)
  1. [§3.1.1, Factual Support Matrix; Eqs. (5) and (7)] The factual-support matrix is populated by a GPT-5 judge and validated against two human annotators with only a Pearson correlation of 0.62. Pearson measures linear association, not agreement in level; a judge that is systematically lenient or strict can still achieve a high Pearson while substantially biasing the absolute unsupported-statement and citation-accuracy rates. The paper does not report a confusion matrix, per-system agreement, inter-annotator reliability, or exact agreement. This is load-bearing because Table 2 places thresholds near the observed values (e.g., GPT-5(DR) unsupported = 12.5% vs. the 10% acceptable cutoff; YouChat(DR) = 74.6% near the 25% problematic boundary). A systematic shift of a few percentage points can change qualitative conclusions. Please report bias measures, per-system agreement, confidence intervals, and a threshold-sensitivity analysis.
  2. [§3.1.1/§3.1.2, Pro vs. Con statement attribute; Eq. (1)] The One-Sided Answer metric, which drives one of the paper's headline claims (48.7–94.8% one-sided across systems), depends entirely on the Pro vs. Con statement classification, yet this classification is not validated at all. No human agreement, error analysis, or example-level evaluation is reported for this LLM-judge step. If the judge over-labels neutral statements as pro or con, the one-sidedness rates are systematically inflated. Please validate this classifier, or at minimum bound the sensitivity of the one-sidedness rates to classification error.
  3. [§3.1.1, source scraping] Roughly 15% of source URLs fail to scrape and are excluded from all calculations that depend on source content. The paper notes this inaccessibility but does not analyze whether the failures correlate with source type, system, or query domain. If paywalled, dynamic, or niche-content sources are disproportionately affected, the factual-support matrix and all derived metrics (unsupported statements, source necessity, citation accuracy, citation thoroughness) will be systematically biased. Please report the distribution of scraping failures across systems and queries, and provide a robustness analysis (e.g., recalculating with failed sources treated as unsupported, or with a worst-case bound).
  4. [§3.1.1 vs. Appendix E] The main text states that the LLM judge defaults to GPT-5, and that the factual-support and confidence annotations were validated against human raters. Appendix E, however, says the human evaluations were conducted with 'the selected LLM, GPT-4.' If the deployed judge is GPT-5 but the validation was performed with GPT-4, the agreement estimates may not transfer. Please clarify which model produced the 80,000 factual-support evaluations and which model was used in the human validation, and if both were used, report separate validation for each.
minor comments (5)
  1. [§4, Table 1] The text states that Gemini(DR) has 40.3% citation accuracy, but Table 1 lists 50.3%. Please reconcile this numerical inconsistency.
  2. [Figure 1] The caption says the metrics are illustrated in Section 4.2, but the metrics are defined in Section 3.1.2. Update the cross-reference.
  3. [Appendix B] The prose says 'acceptable performance requires that 80–90% of sources cited directly support unique, relevant information,' but Table 2 sets Source Necessity acceptability at [80,100) and Citation Accuracy acceptability at [90,100). Align the prose with the table.
  4. [§3.1.1, human validation] For both Pearson correlations, please specify how the two annotators were combined (e.g., averaged, pooled, or separate correlations), and report inter-annotator agreement. A Pearson r alone is not sufficient to establish the reliability of an ordinal/categorical annotation.
  5. [Throughout] No confidence intervals are reported for any of the headline percentages. Given the sample size (168 debate queries and 135 expertise queries) and the use of an LLM judge, confidence intervals would help readers assess the precision of the estimates.

Circularity Check

0 steps flagged

No significant circularity: the headline metrics are direct measurements of extracted responses and citations; the LLM-judge agreement limitation is a validity concern, not a circular derivation.

full rationale

The paper's quantitative claims (one-sidedness, overconfidence, unsupported statements, citation accuracy/thoroughness) are computed by defining metrics (Eqs. 1-8) over the citation matrix and factual-support matrix; these are measurements, not fitted quantities, and no metric is constructed so that a particular result follows by definition. The only model-dependent measurement component is the GPT-5 judge used for confidence scores, Pro/Con labels, and factual-support labels. The paper validates confidence (Pearson r=0.72) and factual support (r=0.62) against human annotators and explicitly states in Sec. 3.1.1: 'Relying on an LLM to measure factual support is a limiting factor of our evaluation framework.' That is a real validity/calibration concern (systematic leniency could shift rates near Table 2 thresholds), but it is not circularity: the metrics are not defined in terms of the judge's outputs in a way that makes the headline result true by construction. The self-citations to Narayanan Venkit et al. (2025) supply the query corpus, design dimensions, and threshold ranges, but the reported rates (50-95% one-sided, 12.5-97.5% unsupported, 40-80% citation accuracy) are newly collected from public systems and are not contained in or implied by the earlier qualitative usability study. No uniqueness theorem, imported ansatz, or renaming is used to force the conclusions. Therefore no circular step is identified.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The framework rests on the reliability of the LLM judge, the fidelity of source scraping, the representativeness of the corpus, and the assumption that one-sided answers are defects. No new entities are introduced; the main hand-set choices are the scorecard thresholds and the confidence cutoff.

free parameters (2)
  • Acceptable/borderline/problematic thresholds = See Table 2
    Eight hand-set cutoff ranges (e.g., one-sided >=40% is problematic) used to color-code performance; not fitted to data but arbitrary and affect the qualitative scorecard.
  • Overconfidence cutoff confidence score = 5 (Strongly Confident)
    An answer is overconfident only if one-sided and confidence == 5; the choice of the top Likert level is a hand-set threshold.
axioms (4)
  • domain assumption GPT-5 judge labels factual support and confidence reliably enough for aggregate metrics
    Validated with Pearson 0.62 (factual support) and 0.72 (confidence); moderate correlation is assumed sufficient for scaling to ~80,000 evaluations.
  • domain assumption Jina.ai Reader extracts the full text of cited sources faithfully
    Manual inspection of ~100 extractions showed only minor issues; 15% of URLs fail and are excluded, which could bias support matrices.
  • domain assumption The 303 questions from the prior usability study represent realistic user information needs
    168 debate questions from ProCon.org and 135 expert-contributed questions; the corpus is inherited from Narayanan Venkit et al. (2025).
  • domain assumption Debate queries with one-sided answers are a system deficiency
    Based on the prior user study's design recommendations; no human baseline is provided to confirm that balanced answers are expected for these phrasings.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence." pith.science (2026). https://pith.science/paper/YZAAUP3E

@misc{pith2026250904499,
  author       = {Pith},
  title        = {Pith review of: DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZAAUP3E}},
  note         = {Machine review of arXiv:2509.04499}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Generative search engines and deep research LLM agents promise trustworthy, source-grounded synthesis, yet users regularly encounter overconfidence, weak sourcing, and confusing citation practices. We introduce DeepTRACE, a novel sociotechnically grounded audit framework that turns prior community-identified failure cases into eight measurable dimensions spanning answer text, sources, and citations. DeepTRACE uses statement-level analysis (decomposition, confidence scoring) and builds citation and factual-support matrices to audit how systems reason with and attribute evidence end-to-end. Using automated extraction pipelines for popular public models (e.g., GPT-4.5/5, You.com, Perplexity, Copilot/Bing, Gemini) and an LLM-judge with validated agreement to human raters, we evaluate both web-search engines and deep-research configurations. Our findings show that generative search engines and deep research agents frequently produce one-sided, highly confident responses on debate queries and include large fractions of statements unsupported by their own listed sources. Deep-research configurations reduce overconfidence and can attain high citation thoroughness, but they remain highly one-sided on debate queries and still exhibit large fractions of unsupported statements, with citation accuracy ranging from 40--80% across systems.

Figures

Figures reproduced from arXiv: 2509.04499 by Chien-Sheng Wu, Kung-Hsiang Huang, Philippe Laban, Pranav Narayanan Venkit, Yilun Zhou, Yixin Mao.

Figure 1
Figure 1. Figure 1: Illustrative diagram of the processing of a deep research agents response into the 8 metrics of the DeepTrace [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Quantitative Evaluation of three GSE – You.com, BingChat, and Perplexity – based on the eight metrics [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Screenshots of responses from GPT-5 Deep Research to the debate question, [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Screenshots of responses from GPT-5 Deep Research to the debate question, [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research

    cs.IR 2026-07 conditional novelty 6.5

    A hierarchical evidence-graph benchmark reveals that multimodal deep-research models write fluent reports while failing citation, claim, and answer grounding.

  2. What if AI systems weren't chatbots?

    cs.CY 2026-05 unverdicted novelty 3.0

    Chatbot AI systems often fail complex needs while projecting authority, contributing to deskilling, labor displacement, economic concentration, and high environmental costs, so alternative pluralistic and task-specifi...

Reference graph

Works this paper leans on

30 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [3]

    Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent

    Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent. arXiv preprint arXiv:2508.06600,

    Pith/arXiv arXiv

  2. [5]

    Ragas: Automated evaluation of retrieval augmented generation

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217,

    Pith/arXiv arXiv

  3. [6]

    Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture

    Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture. arXiv preprint arXiv:2401.08406,

    Pith/arXiv arXiv

  4. [8]

    doi: 10.18653/ v1/2024.findings-acl.41

    Association for Computational Linguistics. doi: 10.18653/ v1/2024.findings-acl.41. URL https://aclanthology.org/2024.findings-acl.41/. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challen...

    Pith/arXiv arXiv 2024

  5. [11]

    Evaluating large language models for health-related queries with presuppositions

    Navreet Kaur, Monojit Choudhury, and Danish Pruthi. Evaluating large language models for health-related queries with presuppositions. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024 , pp. 14308–14331, Bangkok, Thailand and virtual meeting, August

    work page 2024

  6. [12]

    URL https://aclanthology.org/2024.findings-acl

    Association for Computational Linguistics. URL https://aclanthology.org/2024.findings-acl

    work page 2024

  7. [14]

    Llms as factual reasoners: Insights from existing benchmarks and beyond

    Philippe Laban, Wojciech Kry ´sci´nski, Divyansh Agarwal, Alexander R Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. Llms as factual reasoners: Insights from existing benchmarks and beyond. arXiv preprint arXiv:2305.14540, 2023a. Philippe Laban, Lidiya Murakhovs’ ka, Caiming Xiong, and Chien-Sheng Wu. Are you sure? challenging llms leads to perfo...

    Pith/arXiv arXiv

  8. [15]

    Evaluating verifiability in generative search engines

    Nelson F Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. InFindings of the Association for Computational Linguistics: EMNLP 2023, pp. 7001–7025,

    2023

  9. [16]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332,

    Pith/arXiv arXiv

  10. [17]

    Towards a holistic approach: Understanding sociodemographic biases in nlp models using an interdisciplinary lens

    Pranav Narayanan Venkit. Towards a holistic approach: Understanding sociodemographic biases in nlp models using an interdisciplinary lens. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp. 1004– 1005,

    work page 2023

  11. [18]

    Search engines in the ai era: A qualitative understanding to the false promise of factual and verifiable source-cited responses in llm-based search

    Pranav Narayanan Venkit, Philippe Laban, Yilun Zhou, Yixin Mao, and Chien-Sheng Wu. Search engines in the ai era: A qualitative understanding to the false promise of factual and verifiable source-cited responses in llm-based search. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pp. 1325–1340,

    work page 2025

  12. [19]

    Mlgym: A new framework and benchmark for advancing ai research agents

    Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhi- raja, Despoina Magka, Vladislav V orotilov, Gaurav Chaurasia, et al. Mlgym: A new framework and benchmark for advancing ai research agents. arXiv preprint arXiv:2502.14499,

    Pith/arXiv arXiv

  13. [20]

    AMRFact: Enhancing summarization fac- tuality evaluation with AMR-driven negative samples generation

    Haoyi Qiu, Kung-Hsiang Huang, Jingnong Qu, and Nanyun Peng. AMRFact: Enhancing summarization fac- tuality evaluation with AMR-driven negative samples generation. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo...

    work page 2024

  14. [21]

    doi: 10.18653/v1/2024.naacl-long.33

    Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.33. URL https://aclanthology.org/2024.naacl-long.33/. Ehud Reiter. We should evaluate real-world impact. Computational Linguistics,

    doi:10.18653/v1/2024.naacl-long.33 2024

  15. [22]

    Evaluation of rag metrics for question answering in the telecom domain

    Sujoy Roychowdhury, Sumit Soman, HG Ranjani, Neeraj Gunda, Vansh Chhabra, and Sai Krishna Bala. Evaluation of rag metrics for question answering in the telecom domain. arXiv preprint arXiv:2407.12873,

    Pith/arXiv arXiv

  16. [23]

    Minicheck: Efficient fact-checking of llms on grounding documents

    Liyan Tang, Philippe Laban, and Greg Durrett. Minicheck: Efficient fact-checking of llms on grounding documents. arXiv preprint arXiv:2404.10774,

    Pith/arXiv arXiv

  17. [24]

    ” confidently nonsensical?”: A critical survey on the perspectives and challenges of’hallucinations’ in nlp

    Pranav Narayanan Venkit, Tatiana Chakravorti, Vipul Gupta, Heidi Biggs, Mukund Srinath, Koustava Goswami, Sarah Rajtmajer, and Shomir Wilson. ” confidently nonsensical?”: A critical survey on the perspectives and challenges of’hallucinations’ in nlp. arXiv preprint arXiv:2404.07461,

    Pith/arXiv arXiv

  18. [25]

    Agentic reasoning: Reasoning llms with tools for the deep research

    Junde Wu, Jiayuan Zhu, and Yuyuan Liu. Agentic reasoning: Reasoning llms with tools for the deep research. arXiv preprint arXiv:2502.04644,

    Pith/arXiv arXiv

  19. [26]

    How faithful are rag models? quantifying the tug-of-war between rag and llms’ internal prior

    Kevin Wu, Eric Wu, and James Zou. How faithful are rag models? quantifying the tug-of-war between rag and llms’ internal prior. arXiv preprint arXiv:2404.10198,

    Pith/arXiv arXiv

  20. [27]

    Deep- researcher: Scaling deep research via reinforcement learning in real-world environments

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deep- researcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160,

    Pith/arXiv arXiv

  21. [28]

    Rageval: Scenario specific rag evaluation dataset generation framework

    Kunlun Zhu, Yifan Luo, Dingling Xu, Ruobing Wang, Shi Yu, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, et al. Rageval: Scenario specific rag evaluation dataset generation framework. arXiv preprint arXiv:2408.01262,

    Pith/arXiv arXiv

  22. [29]

    A L IMITATIONS While DeepTRACE offers an automated and scalable evaluation platform, it currently focuses on textual and citation- based outputs, excluding multimodal or UI-level interactions that also shape user trust and system usability. We do not evaluate for whether the answer to the question is the right answer but rather focus on the answer format,...

    work page 2025

  23. [100]

    why should we ban bottled water?

    [70,90) [0,70) Uncited Sources [0,5) [5,10) [10,100) Unsupported Statements [0,10) [10,25) [25,100) Source Necessity [80,100) [60,80) [0,60) Citation Accuracy [90,100) [50,90) [0,50) Citation Thoroughness [50,100) [20,50) [0,20) Table 2: Ranges for the eight DeepTrace metrics for a system’s performance to be considered ▲acceptable, ●borderline, or ▼proble...

    work page 2023

  24. [850]

    Fables: Evaluating faithfulness and content selection in book-length summarization

    Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. Fables: Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261,

    Pith/arXiv arXiv

  25. [1973]

    Do LVLMs understand charts? analyzing and correcting factual errors in chart captioning

    Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. Do LVLMs understand charts? analyzing and correcting factual errors in chart captioning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 730–749, Bangkok, Th...

    work page 2024

  26. [2021]

    Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7029–7043,

    work page 2024

  27. [2022]

    Deepresearch bench: A comprehensive benchmark for deep research agents

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763,

    Pith/arXiv arXiv

  28. [2023]

    Deep research agents: A systematic examination and roadmap

    Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, et al. Deep research agents: A systematic examination and roadmap. arXiv preprint arXiv:2506.18096,

    Pith/arXiv arXiv

  29. [2024]

    Deep research bench: Evaluating ai web research agents

    Nikos I Bosse, Jon Evans, Robert G Gambee, Daniel Hnyk, Peter M¨uhlbacher, Lawrence Phillips, Dan Schwarz, Jack Wildman, et al. Deep research bench: Evaluating ai web research agents. arXiv preprint arXiv:2506.06287,

    Pith/arXiv arXiv

  30. [2025]

    Evaluating top- k rag-based approach for game review generation

    Pratyush Chauhan, Rahul Kumar Sahani, Soham Datta, Ali Qadir, Manish Raj, and Mohd Mohsin Ali. Evaluating top- k rag-based approach for game review generation. In 2024 IEEE International Conference on Computing, Power and Communication Technologies (IC2PCT), volume 5, pp. 258–263. IEEE,

    work page 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.