Pith. sign in

REVIEW 2 major objections 3 minor 7 cited by

PaSa: An LLM Agent for Comprehensive Academic Paper Search

T0 review · 2 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-agent language-model system that crawls searches and follows citations outperforms standard academic search baselines on complex queries even when trained only on synthetic data.

desk verdict PaSa is a well-engineered agent paper with a genuinely novel RL training scheme, but the real-world benchmark's gold pool is partly built from PaSa itself, so the headline recall margins are likely inflated. read the letter →

arxiv 2501.10120 v2 pith:7JXV5FMB submitted 2025-01-17 cs.IR cs.LG

Yichen He , Guanhua Huang , Peiyuan Feng , Yuan Lin , Yuchen Zhang , Hang Li , Weinan E This is my paper · ORCID
classification cs.IRcs.LG
keywords academicpapersearchLLMagentcitationnetworktraversalreinforcementlearningsynthetictrainingdatarecall@20literaturereviewqueue
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PaSa is a large-language-model-based paper search agent that tries to automate the full literature-survey behavior of a researcher: it issues search queries, reads papers, decides which sections to expand, and follows citation links until it builds a comprehensive candidate queue, then a selector judges each paper against the query. The paper claims this agentic loop, trained with reinforcement learning on purely synthetic query–paper pairs engineered from related-work sections, transfers to real academic queries. On a 50-query real-world benchmark, the agent's 7-billion-parameter version outperforms all tested baselines, including the strongest search-engine-based baseline by 37.78 percentage points in recall@20 and the prompted large-model version of the same agent by 30.36% in recall. The practical stakes are that comprehensive literature search could become an automated, reproducible step rather than a slow manual task.

What carries the argument

The load-bearing mechanism is the two-agent pipeline with a shared paper queue. The Crawler is trained as a token-level Markov decision process whose three actions—[Search], [Expand], and [Stop]—determine whether the agent issues a new query, pulls in the references of a chosen section, or resets to the next paper in the queue. Rewards are given when a newly queued paper is in the query's answer set, and because the synthetic answer sets are incomplete, the Selector is also used as an auxiliary reward model to credit papers it judges relevant. To make reinforcement learning tractable on trajectories that can stretch to hundreds of papers, the paper introduces session-level PPO, which splits each trajectory at [Stop] actions and estimates returns with Monte Carlo rollouts plus a learned value function.

What would settle it

Re-annotate RealScholarQuery's ground-truth sets using only human experts and citation snowballing from a fixed seed, without showing them PaSa's outputs, and check whether PaSa-7b still beats the strongest search-engine baseline by roughly 38 recall points.

Watch

Extended reading notes

Core claim

The central claim is that comprehensive academic search is better modeled as a sequence of interdependent decisions—what to search, what to read, which citations to expand, when to stop—than as a single retrieval call. PaSa embodies this as two cooperating agents: a Crawler that walks a growing paper queue and can invoke a search tool, expand a cited section, or stop, and a Selector that reads each queued paper and decides whether it meets the query. The paper asserts that training this loop with reinforcement learning, using a synthetic dataset whose answer papers come from citations in related-work sections, is sufficient to beat both conventional search engines and a prompted large-model version of the same architecture on fine-grained queries. In the reported numbers, PaSa-7b reaches 0.5798 recall@20 on RealScholarQuery versus 0.2020 for the strongest search-engine-based baseline, and its Selector reaches 0.85 F1 compared with 0.80 for a large general-purpose model.

Load-bearing premise

The reported recall advantage assumes the hand-built list of relevant papers for each real query is complete and unbiased, but that list was assembled by pooling results from PaSa itself and from the baselines it is compared with.

Editorial extensions

If this is right

  • Citation-graph traversal is a large part of what makes comprehensive search hard; systems that only paraphrase queries miss relevant papers found by reading and expanding references.
  • Synthetic data generated from related-work sections can substitute for expensive human-annotated search trajectories, making it feasible to train specialized search agents without manual demonstrations.
  • A 7-billion-parameter agent trained with reinforcement learning can outperform much larger prompted agents on this task, suggesting that task-specific optimization matters more than model scale in agentic retrieval.
  • The two-stage pipeline of high-recall crawling followed by high-precision selection decouples recall and precision, allowing each to be improved independently.
  • Adjusting reward coefficients and action costs gives a direct control knob over the trade-off between exploration breadth and action count in the Crawler.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same recipe—synthesize queries from related-work sections, train a crawler with citation expansion, filter with a selector—should transfer to other scientific fields, but the paper only evaluates machine-learning venues, so this is an extrapolation.
  • Editorial inference: the real-world benchmark's answer sets were assembled by pooling candidates from PaSa and from the baselines themselves, so the reported recall gap is partly a function of that pooling procedure; an independently constructed ground-truth set could shift the numbers.
  • Editorial inference: using the Selector as both the final filter and the reward model during reinforcement learning means improvements to the Selector can reinforce the Crawler's behavior, a self-referential loop that future work could isolate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. This paper introduces PaSa, an LLM-based agent for academic paper search that combines a Crawler (which performs iterative search and citation expansion) with a Selector (which judges paper relevance). The Crawler is trained with a session-level PPO variant on AutoScholarQuery, a synthetic dataset of ~35k queries derived from Related Work sections of ICLR/ICML/NeurIPS/ACL/CVPR papers, and the Selector is fine-tuned on GPT-4o-generated query-paper pairs. The authors also construct RealScholarQuery, a 50-query human-annotated benchmark, and report that PaSa-7b substantially outperforms Google, Google Scholar, Google with GPT-4o paraphrase, ChatGPT, GPT-o1, and a prompted GPT-4o version of PaSa on both benchmarks. The paper releases the model, datasets, and code.

Significance. If the reported results hold under scrutiny, PaSa is a meaningful advance: a 7B-parameter agent trained only on synthetic data outperforms much larger prompted systems and general-purpose search engines on complex, fine-grained academic queries, and the open release of model, datasets, and code will facilitate reproducibility and follow-up work. The AutoScholarQuery dataset and the RealScholarQuery benchmark are potentially useful community resources. The main caveat is that the real-world benchmark's gold standard is constructed with pooling bias, so the quantitative real-world gains require a sensitivity analysis before the headline numbers can be taken at face value.

major comments (2)
  1. [3.2, Tables 4–5] The RealScholarQuery gold standard is constructed by pooling retrieval outputs from PaSa itself and the baselines (Section 3.2, third paragraph). For recall metrics, any relevant paper that only PaSa surfaces through citation expansion is added to the gold standard, contributing to both the denominator (for all systems) and PaSa's numerator, while penalizing baselines that never retrieved it. The manuscript does not report how many gold papers were contributed exclusively by PaSa, nor does it provide a sensitivity analysis with PaSa excluded from pooling. Please quantify the PaSa-unique contributions and re-run the RealScholarQuery evaluation on a gold standard constructed without PaSa (or with PaSa's outputs removed), reporting both sets of recall figures.
  2. [Abstract, Section 5.3, Tables 4–5] The reported improvements such as '37.78% in recall@20' are absolute percentage-point differences (0.5798 - 0.2020 = 0.3778), not relative improvements over the baseline (which would be 187%). This wording overstates the gain and is inconsistent with conventional IR reporting. Please express all such improvements as 'percentage points' or provide relative improvements, and apply this consistently to the main-text claims and the ablation results in Table 7.
minor comments (3)
  1. [Section 5.3, Table 4 footnote] The claim that PaSa-7b 'outperforms all baselines on AutoScholarQuery test set' is overstated because the ChatGPT baseline was evaluated on only 100 randomly sampled instances. The text should qualify the claim and point to the fair subset comparison in Table 14.
  2. [Table 14] In Table 14, the Google and Google with GPT-4o rows report identical Recall@20/50/100 values (0.1788/0.2010/0.2101). If these are not a typo, this is surprising; please confirm and explain.
  3. [Section 5.2/5.3, Table 5] Given that RealScholarQuery contains only 50 queries, the paper should report bootstrap confidence intervals or a significance test for the main recall and precision metrics, so that the statistical reliability of the large observed margins can be assessed, especially in light of the pooling-bias concern.

Circularity Check

1 steps flagged · score 2.0 of 10

RealScholarQuery recall advantage is partly measured against a gold set that PaSa itself helped populate; no derivation-level circularity.

  1. other [Section 3.2 (RealScholarQuery construction), used in Table 5]
    "To ensure comprehensive coverage, we then applied multiple methods to retrieve additional papers, including PaSa, Google, Google Scholar, ChatGPT (search-enabled GPT-4o), and Google paired with GPT-4o for paraphrased queries. ... The results from all methods were aggregated into a pool of candidate papers. Finally, professional annotators reviewed all candidate papers for each query, selecting those that met the specific requirements of the query to create the final set of relevant papers."

    RealScholarQuery's gold set is not an independent fixed relevance set: it is the annotator-approved subset of a candidate pool that explicitly includes PaSa's own retrieval output. A relevant paper surfaced only by PaSa enters the recall denominator for all systems and can be credited only to PaSa's numerator; Google, Google Scholar, and ChatGPT cannot get credit for a paper they never produced. The headline recall gaps in Table 5 (e.g., +37.78% recall@20 over Google with GPT-4o) are therefore measured against a target set partially selected by the evaluated system.

full rationale

No circularity in the training/evaluation core. AutoScholarQuery answers are human-written references from Related Work sections; PaSa is trained with session-level PPO on those labels and evaluated on a held-out split, so test recall measures generalization to new citation-derived queries, not a re-fit. The AGILE citation (Feng et al. 2024) is a normal self-citation providing the RL framework; the paper supplies its own PPO equations, ablations, and training curves, so the central claim does not rest on unpublished authority. The only circularity-adjacent element is the RealScholarQuery pooling step described in the flagged step: including PaSa among the candidate generators can inflate its recall relative to baselines. This is a benchmark-construction caveat, not a derivation-level equivalence, hence a low score.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The main parameters are RL training hyperparameters, all tuned by hand. The axioms are domain assumptions about the representativeness of synthetic data, the effectiveness of citation expansion, and the applicability of the AGILE framework. No new physical or conceptual entities are introduced.

free parameters (7)
  • alpha (reward coefficient) = 1.5
    Scales the reward for finding new matching papers; Table 8 shows performance varies with this value.
  • c([Search]) = 0.1
    Action cost for search, tuned to balance exploration; Table 15 shows sensitivity.
  • c([Expand]) = 0.1
    Action cost for expand, tuned similarly.
  • gamma_0 (in-session discount) = 1.0
    Chosen in PPO training; not learned from data.
  • gamma_1 (across-session discount) = 0.1
    Chosen; affects how future session rewards are estimated.
  • beta (KL penalty coefficient) = 0.1
    Scales KL penalty to prevent over-optimization.
  • eta (value loss coefficient) = 10
    Weight for value loss in combined RL loss.
assumptions (3)
  • domain assumption The synthetic queries generated by GPT-4o from Related Work sections are representative of real researcher queries.
    Section 3.1; used to train and validate the agent.
  • domain assumption Citation expansion from the full text of papers is an effective way to discover additional relevant papers beyond direct search.
    Ablation in Section 5.4 shows a large recall drop without [Expand], supporting this premise.
  • domain assumption The AGILE reinforcement learning framework can be applied to the paper search task.
    Section 4; the authors adopt AGILE and extend it with session-level PPO.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PaSa: An LLM Agent for Comprehensive Academic Paper Search." pith.science (2026). https://pith.science/paper/7JXV5FMB

@misc{pith2026250110120,
  author       = {Pith},
  title        = {Pith review of: PaSa: An LLM Agent for Comprehensive Academic Paper Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7JXV5FMB}},
  note         = {Machine review of arXiv:2501.10120}
}
read the original abstract

We introduce PaSa, an advanced Paper Search agent powered by large language models. PaSa can autonomously make a series of decisions, including invoking search tools, reading papers, and selecting relevant references, to ultimately obtain comprehensive and accurate results for complex scholar queries. We optimize PaSa using reinforcement learning with a synthetic dataset, AutoScholarQuery, which includes 35k fine-grained academic queries and corresponding papers sourced from top-tier AI conference publications. Additionally, we develop RealScholarQuery, a benchmark collecting real-world academic queries to assess PaSa performance in more realistic scenarios. Despite being trained on synthetic data, PaSa significantly outperforms existing baselines on RealScholarQuery, including Google, Google Scholar, Google with GPT-4o for paraphrased queries, ChatGPT (search-enabled GPT-4o), GPT-o1, and PaSa-GPT-4o (PaSa implemented by prompting GPT-4o). Notably, PaSa-7B surpasses the best Google-based baseline, Google with GPT-4o, by 37.78% in recall@20 and 39.90% in recall@50, and exceeds PaSa-GPT-4o by 30.36% in recall and 4.25% in precision. Model, datasets, and code are available at https://github.com/bytedance/pasa.

Figures

Figures reproduced from arXiv: 2501.10120 by the authors.

Figure 1
Figure 1. Architecture of PaSa. The system consists of two LLM agents, Crawler and Selector. The Crawler [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An example of the PaSa workflow. The Crawler runs multiple [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Return and value function loss curves during [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MaskSearch: A Universal Pre-Training Framework to Enhance Agentic Search Capability

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A pre-training task called RAMP, where models practice searching to fill masked text spans, improves downstream agentic open-domain QA performance across Qwen and LLaMA models.

  2. InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation

    cs.IR 2025-05 conditional novelty 6.0 of 10

    InfoDeepSeek is a 245-question benchmark that measures how well AI agents seek information on the live web, with new metrics for answer accuracy, evidence quality, and compactness.

  3. Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents

    cs.LG 2025-09 conditional novelty 5.0 of 10

    EMPG re-weights policy-gradient updates by step-level token entropy, amplifying confident correct actions and muting uncertain ones, and adds a future-clarity bonus.

  4. Prompt-to-Paper: Agentic AI System for Bioinformatics

    cs.AI 2026-07 conditional novelty 4.5 of 10

    An agentic bioinformatics manuscript system with deterministic RAG, real experiment execution, and a quality-driven rewrite loop raises automated scores by ~18 points on five case studies at ~$0.31 per paper.

  5. AI for Auto-Research: Roadmap & User Guide

    cs.AI 2026-05 unverdicted novelty 4.0 of 10

    The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.

  6. AI4Research: A Survey of Artificial Intelligence for Scientific Research

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A survey that organizes AI-for-research work into five tasks, comprehension, survey, discovery, writing, and peer review, and compiles associated tools and benchmarks.

  7. AI Scientists Fail Without Strong Implementation Capability

    cs.AI 2025-06 conditional novelty 4.0 of 10

    AI scientist systems can propose ideas but cannot reliably implement and verify experiments, making the implementation gap, not idea generation, the current bottleneck.

Reference graph

Works this paper leans on

52 extracted references · 20 canonical work pages · cited by 7 Pith papers

  1. [1]

    Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels (2310.14122)

    arXiv

  2. [2]

    Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2306.17563)

    arXiv

  3. [3]

    Instruction Distillation Makes Large Language Models Efficient Zero-shot Rankers (2311.01555)

    arXiv

  4. [4]

    RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models (2309.15088)

    arXiv

  5. [5]

    PaRaDe: Passage Ranking using Demonstrations with Large Language Models (2310.14408)

    arXiv

  6. [6]

    A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models (2310.09497)

    arXiv

  7. [7]

    Large Language Models are Zero-Shot Rankers for Recommender Systems (2305.08845)

    arXiv

  8. [8]

    TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy (2406.11678)

    arXiv

Show all 52 references
  1. [9]

    Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2304.09542)

    arXiv

  2. [10]

    RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs (2407.02485)

    arXiv

  3. [11]

    Make Large Language Model a Better Ranker (2403.19181)

    arXiv

  4. [12]

    ExaRanker: Explanation-Augmented Neural Ranker (2301.10521)

    arXiv

  5. [13]

    Improving Zero-shot LLM Re-Ranker with Risk Minimization (2406.13331)

    arXiv

  6. [14]

    Zero-Shot Listwise Document Reranking with a Large Language Model (2305.02156)

    arXiv

  7. [15]

    LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Ranking (2406.00231)

    arXiv

  8. [16]

    Re-Ranking Step by Step: Investigating Pre-Filtering for Re-Ranking with Large Language Models (2406.18740)

    arXiv

  9. [17]

    Large Language Models for Relevance Judgment in Product Search (2406.00247)

    arXiv

  10. [18]

    Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing (2404.11791)

    arXiv

  11. [19]

    Passage-specific Prompt Tuning for Passage Reranking in Question Answering with Large Language Models (2405.20654)

    arXiv

  12. [20]

    When Search Engine Services meet Large Language Models: Visions and Challenges (2407.00128)

    arXiv

  13. [21]

    PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (2404.18424)

    arXiv

  14. [22]

    Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models (2312.02969)

    arXiv

  15. [23]

    MuGI: Enhancing Information Retrieval through Multi-Text Generation Integration with Large Language Models (2401.06311)

    arXiv

  16. [24]

    RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze! (2312.02724)

    arXiv

  17. [25]

    REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering (2402.17497)

    arXiv

  18. [26]

    Agent4Ranking: Semantic Robust Ranking via Personalized Query Rewriting Using Multi-agent LLM (2312.15450)

    arXiv

  19. [27]

    Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker (2305.13729)

    arXiv

  20. [28]

    Leveraging LLMs for Unsupervised Dense Retriever Ranking (2402.04853)

    arXiv

  21. [29]

    Unsupervised Contrast-Consistent Ranking with Language Models (2309.06991)

    arXiv

  22. [30]

    FIRST: Faster Improved Listwise Reranking with Single Token Decoding (2406.15657)

    arXiv

  23. [31]

    Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models (2310.07712)

    arXiv

  24. [32]

    Fine-Tuning LLaMA for Multi-Stage Text Retrieval (2310.08319)

    arXiv

  25. [33]

    Enhancing Legal Document Retrieval: A Multi-Phase Approach with Large Language Models (2403.18093)

    arXiv

  26. [34]

    Uncovering ChatGPT’s Capabilities in Recommender Systems (2305.02182)

    arXiv

  27. [35]

    Cognitive Personalized Search Integrating Large Language Models with an Efficient Memory Mechanism (2402.10548)

    arXiv

  28. [36]

    Zero-shot Audio Topic Reranking using Large Language Models (2309.07606)

    arXiv

  29. [37]

    Pretrained Language Model based Web Search Ranking: From Relevance to Satisfaction (2306.01599)

    arXiv

  30. [38]

    query1”, “query2

    Open-source large language models are strong zero-shot query likelihood models for document ranking (2310.13243) Table 9: Examples of queries and corresponding papers in RealScholarQuery. exaggerate the scope or introduce incorrect conditions. • A query is ambiguous if there i...

    arXiv

  31. [39]

    Towards More Relevant Product Search Ranking Via Large Language Models: An Empirical Study (2409.17460)

    arXiv

  32. [42]

    Once the consistency rate on the first pass reaches 90%, the process moves to Stage 2

    Authors review 100% of the annotations. Once the consistency rate on the first pass reaches 90%, the process moves to Stage 2. • Stage 2: Annotators work in batches of 50. Authors randomly check 40% of the annota- tions. If the consistency rate is below 90%, the entire batch i...

  33. [43]

    Which paper studied ...?

    Craft questions similar to those a researcher would pose when reviewing related works, such as “Which paper studied ...?”, “Any works about...?”, “Could you provide me some works...?”

  34. [44]

    The answer should be the cited papers in [Section from A Research Paper]

    Construct the question-answer pairs based on [Section from A Research Paper]. The answer should be the cited papers in [Section from A Research Paper]

  35. [45]

    or" or "and

    Do not ask questions including "or" or "and" that may involve more than one condition

  36. [46]

    Clarity: Formulate questions clearly and unambiguously to prevent confusion

  37. [47]

    Contextual Definitions: Include explanations or definitions for specialized terms and concepts used in the questions

  38. [48]

    paper_1": {

    Format the output as a JSON array containing five objects corresponding to the three question-answer pairs. Here are some examples: [Begin of examples] {Section from A Research Paper-1} {OUTPUT-1} {Section from A Research Paper-2} {OUTPUT-2} {Section from A Research Paper-3} {...

  39. [49]

    Yes" or

    Relevance Assessment: Decide if the paper is likely to include citations highly relevant to the given [topic]. Output "Yes" or "No" on the first line

  40. [50]

    From the list of provided sections, select only those you think may contain relevant citations

    Section Selection: If you answered "Yes" in step 1, identify which sections of the paper are likely to contain these relevant citations. From the list of provided sections, select only those you think may contain relevant citations. If no sections seem relevant even if your an...

  41. [51]

    Yes" or

    The first line: Either "Yes" or "No" based on the relevance assessment

  42. [52]

    selected_section_1

    The second line: A JSON string with selected sections, e.g., {{"selected_section_1": section_name_1, "selected_section_2": section_name_2}}. If no sections are selected, output {{}}. The prompt for Selector You are an elite researcher in the field of AI, conducting research on...

  43. [2011]

    BMC medical education, 11:1–8

    Why not just google it? an assessment of information literacy skills in a biomedical science curriculum. BMC medical education, 11:1–8. Minghan Li, Honglei Zhuang, Kai Hui, Zhen Qin, Jimmy Lin, Rolf Jagerman, Xuanhui Wang, and Michael Bendersky. 2023. Generate, filter, and fus...

    2023 arXiv

  44. [2024]

    Nature Machine Intelligence, pages 1–11

    Augmenting large language models with chem- istry tools. Nature Machine Intelligence, pages 1–11. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting for retrieval- augmented large language models. arXiv preprint arXiv:2305.14283. Lisa Messeri an...

    2022 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.