Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

A new discipline-aware benchmark evaluates AI survey generation across 10 fields, claiming Deep Research agents are the only paradigm robust in all of them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:52 UTC pith:B5HMNEIM

load-bearing objection A genuinely useful dataset and evaluation skeleton, but the headline ranking leans on a content rubric that barely agrees with human judgment — worth serious review, but 'Deep Research wins' needs re-grounding. the 3 major comments →

arxiv 2602.11238 v2 pith:B5HMNEIM submitted 2026-02-11 cs.CL

SurveyLens: A Discipline-Aware Benchmark for Automatic Survey Generation

Beichen Guo , Zhiyuan Wen , Jia Gu , Haochen Shi , Jian Wang , Senzhang Wang , Haoyang Li , Ruosong Yang
show 1 more author
Shuaiqi Liu
This is my paper
classification cs.CL
keywords automatic survey generationbenchmarkdiscipline-aware evaluationLLM-as-a-judgeDeep Research agentssurvey evaluation rubricsreference qualityRAMS/TAMS
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SurveyLens is a benchmark for automatic survey generation (ASG) that aims to measure whether AI systems can satisfy the writing standards of different academic fields, not just computer science. It contributes SurveyLens-1k, a curated set of 1,000 human-written surveys across 10 disciplines, and a two-part evaluation: discipline-aware rubric scoring calibrated to expert preferences, plus semantic alignment metrics against human-written surveys. Benchmarking 11 systems across three paradigms, the paper finds that Deep Research agents are the only paradigm performing well across all disciplines, specialized ASG systems lead in outline and structure quality, and all paradigms produce weak reference lists. The intended payoff is practical guidance for researchers picking a survey-generation tool for their specific field, and a reusable framework for future discipline-aware ASG evaluation.

Core claim

The paper's central claim is that ASG evaluation must be discipline-aware, and that current CS-centric, generic-criteria benchmarks miss how survey standards vary by field (e.g., evidence hierarchy in medicine, chronological flow in history, equations and tables in physics and engineering). On that premise, it introduces SurveyLens and reports the first cross-disciplinary comparison of ASG paradigms. The central empirical finding: Deep Research agents are the only paradigm robust across all 10 disciplines, ASG systems excel at structural planning but lag on content synthesis, and reference quality remains the weakest component across nearly all systems. A secondary finding is a retrieval–syn

What carries the argument

The framework's load-bearing pieces are: (1) SurveyLens-1k, a dataset of 1,000 human-written surveys, 100 per discipline, selected by citation influence and verified for quality; (2) Structured Survey Representation (SSR), a triple (outline, content, references) that lets evaluation target each component separately; (3) discipline-aware rubrics, created by expanding and merging evaluation aspects per discipline and then weighting them with a pairwise-preference model trained on simulated expert judgments to reflect field-specific priorities; (4) RAMS and TAMS, two redundancy-aware semantic matching scores: RAMS uses one-to-one optimal matching with a redundancy penalty to measure faithful co

Load-bearing premise

The whole evaluation rests on the assumption that a single frontier LLM's pairwise preferences can stand in for domain-expert judgments well enough to set the rubric weights; the paper's own validation shows content scores correlate poorly with human rankings (near chance), and the human check used only seven computer-science doctoral researchers, not experts from all ten disciplines.

What would settle it

A direct test: recruit expert reviewers from each of the ten disciplines, have them rate a sample of generated and human surveys on content quality, and compare their preference ratings against SurveyLens content scores. If Spearman correlation stays near zero or pairwise concordance stays near 50% (chance) when experts are from the matched disciplines—especially for content, where the paper already reports a correlation of 0.10—then the benchmark's discipline-aware claim is not supported for that component. Conversely, matching experts would confirm the content dimension's validity.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Researchers outside computer science now have a concrete way to compare survey-generation tools on their own field's standards, rather than relying on CS-oriented leaderboards.
  • Tool selection guidance follows directly: Deep Research agents are the safe default across disciplines, while specialized ASG pipelines can be chosen when strict outline structure matters most.
  • Reference quality is a shared bottleneck; future ASG systems should focus on accurate, pertinence-aware citation generation rather than only prose fluency.
  • The TAMS-versus-RAMS gap implies that models retrieve and write good fragments but fail at global narrative coherence; improvements should target long-range planning and synthesis, not just better retrieval.
  • The finding that data-source quality is decisive suggests that ASG performance depends as much on the retrieval corpus as on the generator.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's design—deriving rubrics from a corpus and weighting them by an LLM-simulated expert—could be adapted to other long-form generation tasks (e.g., textbooks, clinical summaries, legal briefs), where discipline-specific standards matter similarly.
  • If the content-score alignment problem persists (the paper reports near-chance concordance with human rankings for content), a next step would be to generate separate content rubrics per discipline from field-specific experts rather than from a single LLM proxy; this is a testable extension the paper leaves implicit.
  • The observed weak reference quality across all paradigms suggests that citation grounding, not generation, is the current upper-bound constraint; a targeted benchmark that isolates retrieval plus attribution subtasks could let researchers debug it directly.
  • Because vanilla LLMs did relatively better in humanities disciplines, prompt-adaptation alone might bring generic models closer to discipline-specific quality in those fields—an inexpensive experiment to run within this benchmark.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SurveyLens, a discipline-aware benchmark for automatic survey generation (ASG), consisting of SurveyLens-1k (1,000 human-written surveys across 10 disciplines) and a dual-lens evaluation framework. The first lens uses LLM-judged, discipline-specific rubrics with Bradley-Terry preference weights to score outline, content, and reference quality; the second lens uses two reference-based metrics, RAMS and TAMS, to measure alignment with human-written surveys. The authors evaluate 11 systems from three paradigms — vanilla LLMs, ASG systems, and Deep Research agents — and report that Deep Research agents are the only paradigm robust across all 10 disciplines, ASG systems lead on structural planning, and all paradigms remain weak on reference quality. Human validation is reported in Table 7, comparing rubric-derived scores with ELO ratings from seven PhD-level CS researchers.

Significance. If the validity concerns are addressed, SurveyLens would be a useful resource: it is the first benchmark covering 10 disciplines with 1,000 human-written surveys, and it decomposes evaluation into outline/content/reference components, making quality assessment more interpretable than holistic scoring. The public data/code link supports reproducibility, and the RUBRIC+BT-domain method shows strong alignment for Outline (ρ=0.77) and Reference (ρ=0.47), suggesting that component-wise, discipline-specific rubrics are a promising direction. The benchmark also introduces reference-based metrics that penalize redundancy, which is a real gap in existing ASG evaluation. However, the central cross-paradigm findings rely heavily on the Content component, for which the reported human alignment is close to chance (ρ=0.10, 50.6% concordance). As the paper itself concedes in §4.3, this dimension may require domain-specific rubrics beyond the current design. Given that the preference weights are learned from Gemini-3 Pro — the same model family as the top-ranked Gemini Deep Research — the content-driven ranking may partly reflect LLM self-preference. These issues are load-bearing for the headline

major comments (3)
  1. [§4.3, Table 7] The central claim that Deep Research agents are 'the only paradigm robust across all 10 disciplines' (Finding 1) and the 'skeleton-versus-flesh trade-off' (Finding 3) rest heavily on Content scores. Table 7 shows that the best content evaluation method (Rubric+BT-domain) achieves Spearman ρ=0.10 and pairwise concordance 50.6% against human ELO — essentially no alignment beyond chance. The paper itself notes in §4.3 that 'annotators exhibit low consensus on content aspects, suggesting this dimension may require domain-specific rubrics beyond the current design.' If the content rubric is not aligned with human judgment, then the content-driven comparisons among paradigms are not validated, and the tool-selection guidance is unsupported. The authors should either provide human validation for content from domain experts across multiple disciplines, or substantially soften the content-based f
  2. [§3.3.1, §4.2.1] The Bradley-Terry preference weights in Eq. (3) are learned from pairwise comparisons generated by Gemini-3 Pro as a 'proxy for expert judgment.' The top-ranked system in Table 4 is Gemini Deep Research, which is built on the same model family. This raises a concrete self-preference risk: the learned weights may encode Gemini-3 Pro's stylistic preferences, which could favor its own Deep Research outputs. Because the Content component is precisely where human alignment is weakest, the ranking of Gemini Deep Research against ASG systems may be an artifact of the LLM judge rather than of disciplinary standards. The authors should report a sensitivity analysis — e.g., re-learning weights with a different LLM judge, or with human pairwise preferences — and show that the main findings survive. Without this, the claim that the weights reflect 'human preference-aligned' criteria is not establish
  3. [§4.3] The human validation was conducted with only seven PhD-level researchers, all in Computer Science. Since SurveyLens is explicitly a discipline-aware benchmark, and the rubric weights are learned per discipline, validation on CS-only annotators cannot establish that the rubric weights or the resulting scores reflect the standards of Biology, Medicine, Sociology, etc. The paper's discipline-specific claims — e.g., that ASG systems favor STEM fields (Finding 4) — are therefore not validated against human judgment outside CS. A minimal fix is to run the validation with a small number of domain experts per discipline, or to explicitly frame the current validation as a CS-only pilot and restrict cross-disciplinary claims accordingly.
minor comments (4)
  1. [Table 4] The notation 'score+delta' is never explained in the text or caption. The reader must infer that the delta is relative to the 'Avg (All)' row. Please define this explicitly.
  2. [§4.2.2, Table 5] The RAMS and TAMS thresholds τ_c and penalty λ are introduced as free parameters (§3.3.2, Eq. 5–6), but no sensitivity analysis is reported. Since these metrics are new, the robustness of the conclusions to reasonable variations in τ_c and λ should be shown or at least acknowledged.
  3. [Appendix C] The scoring example in Appendix C appears to contain a typo: 'ContributeS RUB f,O' should likely be 'Score S_RUB(f,O)' or similar. Please correct.
  4. [Appendix F] The 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009' line appears to be a leftover template artifact from the ACM submission format and should be removed.

Circularity Check

1 steps flagged

Headline 'Deep Research robust across all disciplines' is driven by a content rubric whose weights are fitted to Gemini-3 Pro preferences; Table 7 shows content scores are near-chance against human ELO, so the content-based ranking is a self-referential LLM-judge artifact.

specific steps
  1. fitted input called prediction [§3.3.1 Preference-Aligned Weighting (Eq. 3), §4.2.1 Findings 1 and 3, §4.3 Table 7]
    "Since obtaining large-scale human preference labels is cost-prohibitive, we utilize a strong LLM (i.e., Gemini-3 Pro) as a proxy for expert judgment. ... Deep Research Agents consistently achieve the highest performance across all disciplines, while Vanilla LLMs rank the lowest."

    The rubric weights in Eq. (3) are optimized on pairwise preferences generated by Gemini-3 Pro, so the resulting score S_RUB = 5·Σ w_k·x̄_k is, by construction, a scalarization of Gemini-3 Pro's own preferences. Table 4 then ranks Gemini Deep Research highest on overall/content, and the paper's central claim that Deep Research agents are 'the only paradigm robust across all 10 disciplines' is read off this Gemini-calibrated score. The external check in Table 7 fails precisely for the content component driving this ranking: Rubric+BT-domain content Spearman ρ=0.10, pairwise concordance 50.6%, and the paper concedes 'annotators exhibit low consensus on content aspects, suggesting this dimension may require domain-specific rubrics beyond the current design.' The content-based prediction theref

full rationale

SurveyLens is a benchmark paper rather than a mathematical derivation, so most of its machinery is not circular: RAMS/TAMS are reference-based and independent of the LLM judge; the self-citations (SGSimEval [4], InteractiveSurvey [22]) are not load-bearing; and the outline and reference findings have partial independent support from Table 7 (outline ρ=0.77, reference ρ=0.47). The significant problem is confined to the content component of the rubric. The weights in Eq. (3) are fitted to Gemini-3 Pro's simulated expert preferences, and the same model family's Deep Research agent tops Table 4 on overall/content. The paper's own human validation shows the content dimension is essentially unvalidated (ρ=0.10, 50.6% concordance, on only 7 CS PhDs rather than experts from all 10 disciplines), and the paper itself flags low annotator consensus on content. Consequently, the headline finding that Deep Research agents are 'the only paradigm robust across all 10 disciplines' and the finding that they 'dominate content synthesis' rest on a self-referential LLM-judge loop, making the circularity partial rather than total. This yields a score of 5.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central evaluation depends on several corpus-derived constructs and LLM-simulated preferences. The main free parameters (BT weights, RAMS thresholds, λ) are not reported, and the rubrics/metrics have weak external validation outside the limited human ELO test.

free parameters (4)
  • BT preference weights w per discipline/component = not reported
    Learned from pairwise preferences simulated by Gemini-3 Pro on 100 human surveys (Eq. 3); these weights determine the final rubric scores and therefore the system rankings.
  • RAMS threshold τ_c per component = not reported
    Thresholds in Eqs. 5–6 determine which matches count as hits; no values or fitting procedure are given in the paper or Appendix B.
  • RAMS redundancy penalty λ = not reported
    Controls the redundancy penalty ω(s)=exp(-λ·max_similarity); its value is not stated.
  • Rubric expansion/merge target counts K′/K = not reported
    The number of criteria produced in the expand/merge stages (Eq. 2) is a design choice; no ablation is reported.
axioms (5)
  • domain assumption Top-cited Semantic Scholar 'Review' papers from 2020–2025 represent high-quality surveys and encode discipline norms.
    Dataset construction (Appendix D) selects by citation counts; no independent quality or norm-representativeness check.
  • domain assumption Gemini-3 Pro pairwise preferences on human surveys are a valid proxy for expert judgment across all 10 disciplines.
    Used to fit BT weights (Section 3.3.1); no human preference data from domain experts for 9 of 10 disciplines; human validation only for CS, outline/reference.
  • domain assumption LLM-generated rubrics (expand/merge) capture true discipline-specific standards.
    Rubrics are distilled by LLMs from the same corpus used for evaluation; no human verification of rubric content.
  • domain assumption The 10 chosen disciplines are coherent, disjoint categories, and 100 surveys each are sufficient to represent a discipline.
    Assumed in dataset construction and statistics (Table 1).
  • standard math Hungarian algorithm, Bradley-Terry model, and sigmoid likelihood are suitable tools.
    Standard machinery; no issue.
invented entities (2)
  • Discipline-aware rubrics (K_d,c,a) no independent evidence
    purpose: Scoring criteria per discipline/component used to judge generated surveys
    Derived by LLM expansion/merge from SurveyLens-1k itself; no external validation that these match expert standards beyond limited CS outline/reference ELO.
  • RAMS and TAMS metrics no independent evidence
    purpose: Reference-based measures of redundancy-aware coverage and semantic alignment
    New evaluation constructs; no validation against human judgments or established metrics reported.

pith-pipeline@v1.3.0-alltime-deepseek · 22154 in / 12271 out tokens · 111769 ms · 2026-08-03T00:52:44.601436+00:00 · methodology

0 comments
read the original abstract

Automatic Survey Generation (ASG) aims to produce comprehensive literature surveys by retrieving, organizing, and synthesizing academic papers. Despite rapid progress in specialized ASG frameworks and Deep Research agents, existing evaluations largely center on Computer Science or rely on generic criteria, leaving it unclear whether current systems satisfy the survey standards of diverse disciplines. We introduce SurveyLens, the first discipline-aware ASG benchmark. SurveyLens comprises SurveyLens-1k, a curated dataset of 1,000 human-written surveys across 10 disciplines, and a dual-lens framework that combines discipline-aware rubric scoring with reference-based alignment to human-written surveys. Evaluating 11 state-of-the-art systems across vanilla LLMs, ASG systems, and Deep Research agents, we find that Deep Research agents are the only paradigm robust across all 10 disciplines, ASG systems lead on structural planning, and all paradigms remain weak on reference quality, providing practical guidance for discipline-specific tool selection and future ASG design.

Figures

Figures reproduced from arXiv: 2602.11238 by Beichen Guo, Haochen Shi, Haoyang Li, Jia Gu, Jian Wang, Ruosong Yang, Senzhang Wang, Shuaiqi Liu, Zhiyuan Wen.

Figure 1
Figure 1. Figure 1: The performance comparison among ASG methods (Vanilla [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of survey papers in SurveyLens-1k. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the SurveyLens Framework. 3.3 Discipline-aware Survey Evaluation Evaluating automatically generated surveys requires assessing whether the writing follows disciplinary conventions; in light of this, we pro￾pose a discipline-aware evaluation framework centered on discipline￾aware rubrics evaluation. This component employs discipline-specific rubrics derived from expert surveys, using aspect weig… view at source ↗
Figure 4
Figure 4. Figure 4: Evaluation results of 11 ASG systems across four key dimen [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The heatmap shows the detailed aspect-wise performance of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompt used to aggregate multiple evaluation aspects into a smaller set of universal, highly aggregated aspects. Used in the [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt used to expand a single aspect into detailed criteria using survey content. Each criterion includes description and example. Defined as [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt used to merge criteria from multiple survey papers into consolidated criteria. Synthesizes common patterns across different examples. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt for per-aspect evaluation of survey outline quality. Returns individual scores for each aspect, which are then averaged. Used in [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt for per-aspect evaluation of survey content quality. Returns individual scores for each aspect, which are then averaged. Used in [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt for per-aspect evaluation of survey reference quality. Returns individual scores for each aspect, which are then averaged. Used in [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt for batch generation of systematic literature reviews. The script first issues an initial scope request, then prompts the model to [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Externalizing Research Synthesis and Validation in AI Scientists through a Research Harness

    cs.AI 2026-06 unverdicted novelty 6.0

    Xcientist externalizes research synthesis and validation in AI scientists via contract-governed artifacts to maintain traceable trajectories and avoid claim drift across three domains.

  2. Externalizing Research Synthesis and Validation in AI Scientists through a Research Harness

    cs.AI 2026-06 conditional novelty 6.0

    Xcientist is a research harness that externalizes an AI scientist's literature grounding, idea evolution, experiments, and repairs into auditable artifacts, demonstrated on memory, traffic forecasting, and PDE-solving tasks.

Reference graph

Works this paper leans on

33 extracted references · 4 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Tong Bao, Mir Tafseer Nayeem, Davood Rafiei, and Chengzhi Zhang. 2025. SurveyGen: Quality-Aware Scientific Survey Generation with Large Language Models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Suzhou, China, 2712–2736. doi:10.18653/v1/2025.emnlp-main.136

    doi:10.18653/v1/2025.emnlp-main.136 2025

  2. [2]

    Yu Chao, Siyu Lin, Xiaorong Wang, Zhu Zhang, Zihan Zhou, Haoyu Wang, Shuo Wang, Jie Zhou, Zhiyuan Liu, and Maosong Sun. 2025. LLM×MapReduce- V3: Enabling Interactive In-Depth Survey Generation through a MCP-Driven Hierarchically Modular Agent System. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstra...

    2025

  3. [3]

    Gemini. 2025. Gemini Deep Research — your personal research assistant. https: //gemini.google/overview/deep-research/

    2025

  4. [4]

    Beichen Guo, Zhiyuan Wen, Yu Yang, Peng Gao, Ruosong Yang, and Jiaxing Shen. 2026. SGSimEval: A Comprehensive Multifaceted and Similarity-Enhanced Benchmark for Automatic Survey Generation Systems. InAdvanced Data Mining and Applications, Masatoshi Yoshikawa, Xiaofeng Meng, Yang Cao, Chuan Xiao, Weitong Chen, and Yanda Wang (Eds.). Springer Nature Singapo...

    2026

  5. [5]

    Tetsu Kasanishi, Masaru Isonuma, Junichiro Mori, and Ichiro Sakata. 2023. SciReviewGen: A Large-scale Dataset for Automatic Literature Review Gen- eration. InFindings of the Association for Computational Linguistics: ACL

    2023

  6. [6]

    Yuxuan Lai, Yupeng Wu, Yidan Wang, Wenpeng Hu, and Chen Zheng. 2024. Instruct Large Language Models to Generate Scientific Literature Survey Step by Step. doi:10.48550/arXiv.2408.07884 arXiv:2408.07884 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2408.07884 2024

  7. [7]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Genera- tion for Knowledge-Intensive NLP Tasks. InAdvances in Neural Information Pro- cessing Systems, H. Larochelle, M. Ranzato, R. Hadsel...

    2020

  8. [8]

    Xun Liang, Jiawei Yang, Yezhaohui Wang, Chen Tang, Zifan Zheng, Shichao Song, Zehao Lin, Yebin Yang, Simin Niu, Hanyu Wang, Bo Tang, Feiyu Xiong, Keming Mao, and Zhiyu li. 2025. SurveyX: Academic Survey Automation via Large Language Models. doi:10.48550/arXiv.2502.14776 arXiv:2502.14776 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.14776 2025

  9. [9]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. InText summarization branches out. 74–81

    2004

  10. [10]

    Yixin Liu, Yonghui Wu, Denghui Zhang, and Lichao Sun. 2025. Agentic AutoSur- vey: Let LLMs Survey LLMs. doi:10.48550/arXiv.2509.18661 arXiv:2509.18661 [cs]

    doi:10.48550/arxiv.2509.18661 2025

  11. [11]

    T, Kieu Hai Dang, Nguyen Tien Dong, and Dung D

    Minh-Anh Nguye, Minh-Duc Nguyen, Ha Lan N. T, Kieu Hai Dang, Nguyen Tien Dong, and Dung D. Le. 2025. SurveyG: A Multi-Agent LLM Framework with Hierarchical Citation Graph for Automated Survey Generation. doi:10.48550/ arXiv.2510.07733 arXiv:2510.07733 [cs]

    doi:10.48550/arxiv.2510.07733 2025

  12. [12]

    OpenAI. 2025. Introducing deep research. https://openai.com/zh-Hans-CN/ index/introducing-deep-research/

    2025

  13. [13]

    Qwen. 2025. Qwen Chat. https://chat.qwen.ai/

    2025

  14. [14]

    Xiaofeng Shi, Qian Kou, Yuduo Li, Ning Tang, Jinxin Xie, Longbin Yu, Songjing Wang, and Hua Zhou. 2025. SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation. doi:10.48550/arXiv.2506.12689 arXiv:2506.12689 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.12689 2025

  15. [15]

    Weihang Su, Anzhe Xie, Qingyao Ai, Jianming Long, Jiaxin Mao, Ziyi Ye, and Yiqun Liu. 2026. SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation. doi:10.48550/arXiv.2508.15658 arXiv:2508.15658 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.15658 2026

  16. [16]

    Zhaojun Sun, Xuzhou Zhu, Xuanhe Zhou, Xin Tong, Shuo Wang, Jie Fu, Guoliang Li, Zhiyuan Liu, and Fan Wu. 2025. SurveyBench: Can LLM(-Agents) Write Academic Surveys that Align with Reader Needs? doi:10.48550/arXiv.2510.03120 arXiv:2510.03120 [cs]

    doi:10.48550/arxiv.2510.03120 2025

  17. [17]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and others. 2023. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805(2023)

    Pith/arXiv arXiv 2023

  18. [18]

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. 2024. MinerU: An Open-Source Solution for Precise Document Content Extraction. doi:10.48550/ arXiv.2409.18839 arXiv:2409.18839 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.18839 2024

  19. [19]

    Haoyu Wang, Yujia Fu, Zhu Zhang, Shuo Wang, Zirui Ren, Xiaorong Wang, Zhili Li, Chaoqun He, Bo An, Zhiyuan Liu, and Maosong Sun. 2025. LLM$\times$MapReduce-V2: Entropy-Driven Convolutional Test-Time Scal- ing for Generating Long-Form Articles from Extremely Long Resources. doi:10. 48550/arXiv.2504.05732 arXiv:2504.05732 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.05732 2025

  20. [20]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large Language Models are not Fair Evaluators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thaila...

    doi:10.18653/v1/ 2024

  21. [21]

    Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meis- han Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. 2024. AutoSurvey: Large Language Models Can Automatically Write Surveys. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C...

    work page doi:10.52202/079017-3655 2024

  22. [22]

    Zhiyuan Wen, Jiannong Cao, Zian Wang, Beichen Guo, Ruosong Yang, and Shuaiqi Liu. 2025. InteractiveSurvey: An LLM-based Personalized and In- teractive Survey Paper Generation System. doi:10.48550/arXiv.2504.08762 arXiv:2504.08762 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.08762 2025

  23. [23]

    Siyi Wu, Chiaxin Liang, Ziqian Bi, Leyi Zhao, Tianyang Wang, Junhao Song, Yichao Zhang, Keyu Chen, Benji Peng, and Xinyuan Song. 2025. AutoSurvey2: Empowering Researchers with Next Level Automated Literature Surveys. doi:10. 48550/arXiv.2510.26012 arXiv:2510.26012 [cs]

    doi:10.48550/arxiv.2510.26012 2025

  24. [24]

    Xiangchao Yan, Shiyang Feng, Jiakang Yuan, Renqiu Xia, Bin Wang, Lei Bai, and Bo Zhang. 2025. SURVEYFORGE : On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writing. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for C...

    work page doi:10.18653/v1/2025.acl-long.609 2025

  25. [25]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025

  26. [26]

    Hongbo Zhang, Han Cui, Yidong Wang, Yijian Tian, Qi Guo, Cunxiang Wang, Jian Wu, Chiyu Song, and Yue Zhang. 2025. Deep Literature Survey Automation with an Iterative Workflow. doi:10.48550/arXiv.2510.21900 arXiv:2510.21900 [cs]

    doi:10.48550/arxiv.2510.21900 2025

  27. [27]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi

  28. [28]

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. doi:10.48550/arXiv.2506.05176 arXiv:2506.05176 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.05176 2025

  29. [29]

    Jiahao Zhao, Shuaixing Zhang, Nan Xu, and Lei Wang. 2025. SurveyEval: Towards Comprehensive Evaluation of LLM-Generated Academic Surveys. doi:10.48550/arXiv.2512.02763 arXiv:2512.02763 [cs]

    doi:10.48550/arxiv.2512.02763 2025

  30. [30]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena. InAdvances in Neu- ral Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. ...

    2023

  31. [31]

    Review" document type for each discipline. Subsequently, we employ a hybrid filtering strategy to identify genuine surveys: papers with standard keywords in their titles (e.g.,

    Kun Zhu, Xiaocheng Feng, Xiachong Feng, Yingsheng Wu, and Bing Qin. 2023. Hierarchical Catalogue Generation for Literature Review: A Benchmark. InFind- ings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, Singapore, 6790–6804. doi:10.18653/v1/2023. findings-emnlp.453 9 B. Guo et al. A Comparison of ...

    doi:10.18653/v1/2023 2023

  32. [2020]

    doi:10.48550/arXiv

    BERTScore: Evaluating Text Generation with BERT. doi:10.48550/arXiv. 1904.09675 arXiv:1904.09675 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.1904.09675 1904

  33. [2023]

    doi:10.18653/v1/2023.findings-acl.418

    Association for Computational Linguistics, Toronto, Canada, 6695–6715. doi:10.18653/v1/2023.findings-acl.418

    work page doi:10.18653/v1/2023.findings-acl.418 2023