Pith. sign in

REVIEW 4 major objections 6 minor 21 references

Level-Navi Agent: A Framework and benchmark for Chinese Web Search Agents

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read The paper claims that a training-free, level-aware two-agent architecture, paired with the 481-question Web24 dataset and a four-part weighted score, can fairly rank LLMs on Chinese web search and exposes two explainable failure modes.

desk verdict Useful Chinese web search benchmark, but cross-table model rankings are compromised by different LLM judges for open- and closed-source models. read the letter →

arxiv 2502.15690 v1 pith:GAQGHFBE submitted 2024-12-20 cs.IR cs.AIcs.CL

classification cs.IRcs.AIcs.CL
keywords ChinesewebsearchagentLLMlevel-awarenavigationbenchmarkdatasetevaluationmetricchain-of-thoughtplanningfunctioncalling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to make Chinese web-search agents cheaply and fairly comparable across LLMs. Its package has three parts: Level-Navi Agent, a training-free two-agent framework in which a Planner decomposes each user question into sub-questions and a Level-Info Searcher gathers information in up to three levels; Web24, a 481-question Chinese benchmark drawn mostly from news; and a composite score that weights LLM-judged correctness, embedding similarity, response relevance, and a search-count cost term. The paper claims that on this benchmark, larger models score higher but with diminishing returns, native Chinese models beat English-centric ones, few-shot prompts improve task-completion pass rates, and token-level metrics like F1 and ROUGE misjudge open-ended answers. If the claims hold, researchers can evaluate open and closed LLMs on real Chinese web-search behaviour without retraining an agent, and they gain concrete targets for improving function calling and instruction following.

What carries the argument

The load-bearing object is the two-agent Level-Navi loop. The Planning Agent uses chain-of-thought with iterative refinement to generate only the next set of parallel sub-questions, while the Level-Info Agent searches up to three levels (internal knowledge, search-engine snippets, then opened pages) and returns as soon as it has enough information, which cuts token use and API calls. The evaluation machinery is the Web24 dataset, with five domains and four question types designed to force real searching, and the composite score $$S_{\mathrm{final}} = 60\,S_{\mathrm{co}} + 15\,S_{\mathrm{simi}} + 15\,S_{\mathrm{rele}} + 10\,$e^{{-S_c}}$,$$ where the weights encode the paper's priorities: correctness dominates, semantic similarity and relevance matter equally, and search count enters as an exponential cost incentive.

What would settle it

Take a random sample of the model outputs from the two result tables, have human annotators score them on the same 1-10 correctness scale, and compute agreement between each LLM judge and the humans; if either judge diverges from human ratings, or the two judges rank the same outputs differently, the composite-score rankings are not trustworthy.

Watch

Extended reading notes

Core claim

The central discovery claimed is that a general-purpose, training-free agent can measure LLM web-search ability in Chinese when planning is iterative rather than fixed: the Planner emits only the next batch of parallel sub-questions, and the Searcher answers at the cheapest sufficient level, from memory, search snippets, or opened pages. On the paper's 481-item Web24 dataset and its weighted score, the claimed result is that Qwen2.5-72B and Deepseek-V2.5 lead the field, parameter gains show diminishing returns, few-shot prompting raises pass rates, and two failure modes—overconfidence (not calling the search tool when unsure) and low task fidelity (not following the instruction format)—explain much of the gap between weaker and stronger models. The paper also claims that semantic, LLM-judged, and relevance metrics should replace token-level F1 and ROUGE, showing data where longer, richer answers receive lower token scores despite being better.

Load-bearing premise

The rankings depend on treating two different LLM judges as interchangeable: one scores open-source models, another scores closed-source models, and no human-agreement or judge-consistency check is reported to show their scores are on the same scale.

Editorial extensions

If this is right

  • Any open- or closed-source LLM, including small models, can be plugged into Level-Navi Agent without fine-tuning, so Chinese web-search evaluation no longer requires a dedicated trained agent.
  • The diminishing returns observed across model families imply that further score gains should come from better information filtering and source selection rather than from simply scaling parameters.
  • Few-shot prompting is a cheap, effective default for agent tasks, especially for weaker models whose pass rates otherwise drop.
  • The two diagnosed failure modes point to specific training interventions: balanced positive and negative function-calling examples to counteract overconfidence, and instruction-following data in long Chinese contexts to raise task fidelity.
  • Token-based metrics like F1 and ROUGE understate rich answers in web search, so the paper's composite metric should be used when comparing agents that summarize diverse sources.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the three-level information-acquisition pattern holds, the same Planner/Searcher split could be carried to other languages or domains by swapping the search API, without changing the metric's structure.
  • Editorial inference: because the final ranking leans on an LLM judge with no human-anchored calibration in the paper, future benchmark runs should report judge-human agreement before cross-table differences (open-source versus closed-source) are treated as real.
  • Editorial inference: the search-count term rewards agents that call the search tool less, which creates a latent incentive toward overconfidence; a testable variant would re-run the ranking with the searcher term removed to see how much of the ordering is efficiency-driven.
  • Editorial inference: Web24 is news-heavy and tied to pre-December 2024 sources, so the dataset will age; a rolling re-issue of the same pipeline on fresh Chinese news would test whether model rankings on this benchmark are stable over time.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Level-Navi Agent, a training-free web-search agent framework that decomposes user questions into sub-questions via a Planner and searches at multiple levels through a Level-Info Agent, together with a new Chinese web-search benchmark dataset (Web24, 481 questions) and a composite evaluation metric combining correctness (LLM-judged), semantic similarity, relevance, and searcher count. The authors evaluate 14 open/closed-source LLMs, report rankings, analyze failure modes (overconfidence and low task fidelity), and compare the framework with commercial products. The central claim is that the framework, dataset, and metric provide a fair, quantitative evaluation of Chinese web-search agent capabilities without fine-tuning.

Significance. If the evaluation protocol were sound, the paper would provide a useful and practical framework and benchmark for Chinese web-search agents. The strengths of the paper include the training-free design that works with any LLM, the collection of real Chinese internet sources with URLs for verification, the release of source code, and the concrete error analysis identifying overconfidence and low task fidelity. However, the significance is currently limited by the absence of validation for the LLM-judge-based correctness metric, the use of different judges for different result tables, and a metric design that may reward one of the failure modes the paper itself identifies. These issues affect the reliability of the reported rankings and the claimed suitability of the evaluation metric.

major comments (4)
  1. [§4.1, Tables 2 and 3] The correctness scores (Sco) in Table 2 are produced by GPT-4o, while those in Table 3 are produced by Qwen2.5-72B, as stated in the table captions. The paper's central rankings, including the claim that Deepseek-V2.5 performs best and the comparison of Qwen2.5-72B versus GPT-4o, require merging these tables. There is no evidence that the two LLM judges rate answers on the same scale (e.g., no calibration sample, no human-agreement correlation, no judge-consistency statistic). The 'fair settings' claim in the abstract and §4.1 is therefore unsupported. I recommend rerunning the evaluation with a single judge for all models, or restricting cross-table comparisons to within-judge results and providing a direct judge-similarity analysis.
  2. [§3.2, Eq. (1)] The searcher-count term 10 × e^{−Sc} in S_final rewards models that invoke the Searcher fewer times, but §4.4.1 identifies 'overconfidence' (refraining from web search when the answer is unknown) as a failure mode. Because a model that fails to search and gives an incorrect answer can receive a higher searcher-count sub-score than a model that searches appropriately, the metric rewards behavior that the paper itself diagnoses as error-prone. Though this term's weight is small, it is part of the claimed 'suitable evaluation metric,' so the paper should justify this design choice, remove the term, or condition it on correctness.
  3. [§3.2 and §4.3] The correctness score relies entirely on LLM-as-judge, but no evidence is given that these scores agree with human judgments. The 'effectiveness' argument in §4.3 consists of observing that the rankings 'align with empirical knowledge and common sense,' which is circular because the metric is used to define those rankings. As the benchmark and metric are contributions, the paper should include a human evaluation on a sample (e.g., at least 100 responses) to validate Sco and, ideally, the other sub-metrics. Without this, the claim of offering a 'suitable evaluation metric' is not established.
  4. [§4.2, Figure 5] The comparison with commercial products does not state which judge computed the correctness scores for the product responses or for the Level-Navi Agent response. If the judge differs from the one used for the main tables, or if the products are evaluated with a different protocol, the conclusion that the agent is at the same level as commercial products is not supported. Please specify the judge and ensure that all compared systems are evaluated under identical conditions.
minor comments (6)
  1. [Table 3 caption] The caption reads 'Open Source Model Results with Qwen2.5-72B Evaluation' while the table lists closed-source models; this is likely a typo and should be corrected.
  2. [§3.2] The text says 'We assign the highest weight to the F1 score' but the formula weights the Correctness Score (Sco); please clarify the terminology.
  3. [Table 3] The model name 'Moonshoot-v1' is misspelled; it should be 'Moonshot-v1'.
  4. [§3.1] The dataset annotation process is described only briefly; please add details on the number of annotators and inter-annotator agreement to support the 'well-annotated' claim.
  5. [§3.2] The Relevance Score description says 'another evaluation LLM' will generate inferred questions, but the identity of this LLM is not specified; please state which model is used and whether it is the same as the correctness judge.
  6. [§4.1] Closed-source models are only tested with three-shot prompting, while the few-shot analysis (zero-shot, one-shot, three-shot) is applied only to open-source models; this should be clarified to avoid overgeneralizing the prompting conclusions.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the framework, Web24 benchmark, and Eq. (1) metric are independent contributions, and the reported rankings are measurements rather than by-construction identities.

full rationale

The paper contains no claimed derivation that reduces to its own inputs. Level-Navi Agent is a prompt-based, training-free pipeline (Section 2), Web24 is independently constructed from Chinese news and knowledge with source URLs for verification (Section 3.1), and the composite metric in Eq. (1) is defined in advance with stated weights rather than fitted to the evaluated outputs. The relevance metric is imported from cited external work (RAGAs), and the LLM-as-judge correctness protocol is a cited external convention; neither is a self-citation by the authors. The Section 4.3 statement that the metrics' results 'align with empirical knowledge and common sense' is a weak external sanity check, not a reduction of the metric to the rankings it produces. The two-judge design (GPT-4o for Table 2, Qwen2.5-72B for Table 3) is a genuine validity threat to cross-table comparability, but it does not make any score definitionally equal to its own input; it is an evaluation-protocol concern, not circularity. No uniqueness theorem, self-citation chain, or fitted-then-renamed prediction appears, so the derivation chain is self-contained.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on domain assumptions about evaluation validity and dataset quality. The only hand-chosen numeric parameters are the metric weights and decay constant. No fundamentally new entities (particles, forces, dimensions) are introduced.

free parameters (2)
  • Metric weights (60/15/15/10) = 60, 15, 15, 10
    Hand-chosen weights in Eq. (1) for the final score; no sensitivity analysis or justification is provided.
  • Exponential decay constant for searcher count = base e
    Searcher count is mapped via e^{-Sc} in Eq. (1); the choice of exponential decay is ad hoc and not motivated by cost data.
assumptions (4)
  • domain assumption LLM-as-judge scores correlate with human judgments of answer correctness and relevance.
    Used in Section 3.2 for Sco and Srele; no human agreement study is reported.
  • domain assumption Web24 ground-truth answers and source URLs are correct and accurately annotated.
    Section 3.1 describes annotation by authors with cross-checking, but no inter-annotator agreement or external validation is reported.
  • domain assumption Fewer searcher invocations indicate better question decomposition and efficiency.
    Section 3.2 defines Sc as lower-is-better, but Section 4.4.1 shows low search rates can reflect overconfidence, contradicting this axiom.
  • domain assumption The search engine API returns relevant results for the sub-queries.
    The paper relies on an unnamed search engine API (Section 2.2) without evaluating retrieval quality or handling of failed queries.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Level-Navi Agent: A Framework and benchmark for Chinese Web Search Agents." pith.science (2026). https://pith.science/paper/GAQGHFBE

@misc{pith2026250215690,
  author       = {Pith},
  title        = {Pith review of: Level-Navi Agent: A Framework and benchmark for Chinese Web Search Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GAQGHFBE}},
  note         = {Machine review of arXiv:2502.15690}
}
read the original abstract

Large language models (LLMs), adopted to understand human language, drive the development of artificial intelligence (AI) web search agents. Compared to traditional search engines, LLM-powered AI search agents are capable of understanding and responding to complex queries with greater depth, enabling more accurate operations and better context recognition. However, little attention and effort has been paid to the Chinese web search, which results in that the capabilities of open-source models have not been uniformly and fairly evaluated. The difficulty lies in lacking three aspects: an unified agent framework, an accurately labeled dataset, and a suitable evaluation metric. To address these issues, we propose a general-purpose and training-free web search agent by level-aware navigation, Level-Navi Agent, accompanied by a well-annotated dataset (Web24) and a suitable evaluation metric. Level-Navi Agent can think through complex user questions and conduct searches across various levels on the internet to gather information for questions. Meanwhile, we provide a comprehensive evaluation of state-of-the-art LLMs under fair settings. To further facilitate future research, source code is available at Github.

Figures

Figures reproduced from arXiv: 2502.15690 by the authors.

Figure 1
Figure 1. Pipeline of Our Level-Navi Agent. 2 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The framework of Level-Navi Agent. 2.1 Planning Agent The Planner is a key component in our design, and its structure directly affects the performance of the entire Agent to a certain extent. Next, we will provide a detailed introduction to the design of the Planning Agent. Our Planning Agent plans the trajectory path through a process of chain of thought (Wei et al. (2022)) and iterative refinement. When user input… view at source ↗
Figure 3
Figure 3. Our Level-Navi Agent demonstrates an example of handling a user query. The Planning [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Source, domain and type of Web24 Dateset. Domain S. Cnd. Cmp. M-H All Finance 23 23 22 22 90 Gaming 28 23 23 17 91 Sports 42 18 21 19 100 Movie 29 33 24 14 100 Event 23 27 22 28 100 All 145 124 112 100 481 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparison with other products based on our metrics. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Searcher and Function call counts, the percentage at the bottom represent [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 4 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

  2. [4]

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao

    URL https://openreview.net/forum?id=tzE7VqsaJ4. Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. Mindsearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20183,

  3. [5]

    URL https://aclanthology.org/2024.eacl-demo.16

    Association for Computational Linguistics. URL https://aclanthology.org/2024.eacl-demo.16. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, and Jiajie Zhang et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools,

  4. [6]

    doi: 10.1145/3703155

    ISSN 1046-8188. doi: 10.1145/3703155. URL https://doi.org/10.1145/ 3703155. Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, et al. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv:2409.12959,

  5. [7]

    The impact of reasoning step length on large language models

    Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.04925,

  6. [8]

    Cmmlu: Measuring massive multitask language understanding in chinese

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212,

  7. [9]

    Prompting large language models with chain-of-thought for few-shot knowledge base question generation

    Yuanyuan Liang, Jianing Wang, Hanlun Zhu, Lei Wang, Weining Qian, and Yunshi Lan. Prompting large language models with chain-of-thought for few-shot knowledge base question generation. arXiv preprint arXiv:2310.08395,

  8. [10]

    doi: 10.18653/v1/2024.acl-long.624

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.624. URL https://aclanthology.org/2024.acl-long.624. Xilai Ma, Jing Li, and Min Zhang. Chain of thought with explicit evidence reasoning for few- shot relation extraction. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics:...

Show all 21 references
  1. [11]

    doi: 10.18653/v1/2023.findings-emnlp.153

    Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.153. URL https://aclanthology.org/2023.findings-emnlp.153. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, Wil...

  2. [12]

    URL https:// aclanthology.org/2023.tacl-1.75

    doi: 10.1162/tacl a 00605. URL https:// aclanthology.org/2023.tacl-1.75. Revanth Gangi Reddy, Yi Ren Fung, Qi Zeng, Manling Li, Ziqi Wang, Paul Sullivan, and Heng Ji. Smartbook: Ai-assisted situation report generation. ArXiv, abs/2303.14337,

  3. [13]

    Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani-Tur, and Heng Ji

    URL https://api.semanticscholar.org/CorpusID:257766360. Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani-Tur, and Heng Ji. Infogent: An agent-based framework for web information aggregation. arXiv preprint arXiv:2410.19054,

  4. [14]

    Comparing traditional and llm-based search for consumer choice: A randomized experiment

    Sofia Eleni Spatharioti, David M Rothschild, Daniel G Goldstein, and Jake M Hofman. Comparing traditional and llm-based search for consumer choice: A randomized experiment. arXiv preprint arXiv:2307.03744,

  5. [15]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L ´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118,

  6. [16]

    Knowledge-driven cot: Exploring faithful reasoning in llms for knowledge-intensive question answering

    Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang Xiong. Knowledge-driven cot: Exploring faithful reasoning in llms for knowledge-intensive question answering. arXiv preprint arXiv:2308.13259,

  7. [17]

    Crag–comprehensive rag benchmark

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, et al. Crag–comprehensive rag benchmark. arXiv preprint arXiv:2406.04744,

  8. [18]

    A comprehensive capability analysis of gpt-3 and gpt-3.5 series models

    Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420,

  9. [19]

    How multilingual is multilingual llm? arXiv preprint arXiv:2311.09071,

    Fei Yuan, Shuai Yuan, Zhiyong Wu, and Lei Li. How multilingual is multilingual llm? arXiv preprint arXiv:2311.09071,

  10. [20]

    Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms

    Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. ...

  11. [21]

    Agent-as-a- judge: Evaluate agents with agents

    Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agent-as-a- judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934,

  12. [2023]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, and Mei Li et al. Qwen technical report. arXiv preprint arXiv:2309.16609,

  13. [2024]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi

    URL https://github.com/meta-llama/llama3/blob/main/ MODEL_CARD.md. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Self-reflective retrieval augmented generation. In NeurIPS 2023 Workshop on Instruction T uning and Instruction Following,

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.