REVIEW 4 major objections 6 minor 21 references
Level-Navi Agent: A Framework and benchmark for Chinese Web Search Agents
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that a training-free, level-aware two-agent architecture, paired with the 481-question Web24 dataset and a four-part weighted score, can fairly rank LLMs on Chinese web search and exposes two explainable failure modes.
desk verdict Useful Chinese web search benchmark, but cross-table model rankings are compromised by different LLM judges for open- and closed-source models. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the two-agent Level-Navi loop. The Planning Agent uses chain-of-thought with iterative refinement to generate only the next set of parallel sub-questions, while the Level-Info Agent searches up to three levels (internal knowledge, search-engine snippets, then opened pages) and returns as soon as it has enough information, which cuts token use and API calls. The evaluation machinery is the Web24 dataset, with five domains and four question types designed to force real searching, and the composite score $$S_{\mathrm{final}} = 60\,S_{\mathrm{co}} + 15\,S_{\mathrm{simi}} + 15\,S_{\mathrm{rele}} + 10\,$e^{{-S_c}}$,$$ where the weights encode the paper's priorities: correctness dominates, semantic similarity and relevance matter equally, and search count enters as an exponential cost incentive.
What would settle it
Take a random sample of the model outputs from the two result tables, have human annotators score them on the same 1-10 correctness scale, and compute agreement between each LLM judge and the humans; if either judge diverges from human ratings, or the two judges rank the same outputs differently, the composite-score rankings are not trustworthy.
Extended reading notes
Core claim
The central discovery claimed is that a general-purpose, training-free agent can measure LLM web-search ability in Chinese when planning is iterative rather than fixed: the Planner emits only the next batch of parallel sub-questions, and the Searcher answers at the cheapest sufficient level, from memory, search snippets, or opened pages. On the paper's 481-item Web24 dataset and its weighted score, the claimed result is that Qwen2.5-72B and Deepseek-V2.5 lead the field, parameter gains show diminishing returns, few-shot prompting raises pass rates, and two failure modes—overconfidence (not calling the search tool when unsure) and low task fidelity (not following the instruction format)—explain much of the gap between weaker and stronger models. The paper also claims that semantic, LLM-judged, and relevance metrics should replace token-level F1 and ROUGE, showing data where longer, richer answers receive lower token scores despite being better.
Load-bearing premise
The rankings depend on treating two different LLM judges as interchangeable: one scores open-source models, another scores closed-source models, and no human-agreement or judge-consistency check is reported to show their scores are on the same scale.
Editorial extensions
If this is right
- Any open- or closed-source LLM, including small models, can be plugged into Level-Navi Agent without fine-tuning, so Chinese web-search evaluation no longer requires a dedicated trained agent.
- The diminishing returns observed across model families imply that further score gains should come from better information filtering and source selection rather than from simply scaling parameters.
- Few-shot prompting is a cheap, effective default for agent tasks, especially for weaker models whose pass rates otherwise drop.
- The two diagnosed failure modes point to specific training interventions: balanced positive and negative function-calling examples to counteract overconfidence, and instruction-following data in long Chinese contexts to raise task fidelity.
- Token-based metrics like F1 and ROUGE understate rich answers in web search, so the paper's composite metric should be used when comparing agents that summarize diverse sources.
Reading between the lines
- Editorial inference: if the three-level information-acquisition pattern holds, the same Planner/Searcher split could be carried to other languages or domains by swapping the search API, without changing the metric's structure.
- Editorial inference: because the final ranking leans on an LLM judge with no human-anchored calibration in the paper, future benchmark runs should report judge-human agreement before cross-table differences (open-source versus closed-source) are treated as real.
- Editorial inference: the search-count term rewards agents that call the search tool less, which creates a latent incentive toward overconfidence; a testable variant would re-run the ranking with the searcher term removed to see how much of the ordering is efficiency-driven.
- Editorial inference: Web24 is news-heavy and tied to pre-December 2024 sources, so the dataset will age; a rolling re-issue of the same pipeline on fresh Chinese news would test whether model rankings on this benchmark are stable over time.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Level-Navi Agent, a training-free web-search agent framework that decomposes user questions into sub-questions via a Planner and searches at multiple levels through a Level-Info Agent, together with a new Chinese web-search benchmark dataset (Web24, 481 questions) and a composite evaluation metric combining correctness (LLM-judged), semantic similarity, relevance, and searcher count. The authors evaluate 14 open/closed-source LLMs, report rankings, analyze failure modes (overconfidence and low task fidelity), and compare the framework with commercial products. The central claim is that the framework, dataset, and metric provide a fair, quantitative evaluation of Chinese web-search agent capabilities without fine-tuning.
Significance. If the evaluation protocol were sound, the paper would provide a useful and practical framework and benchmark for Chinese web-search agents. The strengths of the paper include the training-free design that works with any LLM, the collection of real Chinese internet sources with URLs for verification, the release of source code, and the concrete error analysis identifying overconfidence and low task fidelity. However, the significance is currently limited by the absence of validation for the LLM-judge-based correctness metric, the use of different judges for different result tables, and a metric design that may reward one of the failure modes the paper itself identifies. These issues affect the reliability of the reported rankings and the claimed suitability of the evaluation metric.
major comments (4)
- [§4.1, Tables 2 and 3] The correctness scores (Sco) in Table 2 are produced by GPT-4o, while those in Table 3 are produced by Qwen2.5-72B, as stated in the table captions. The paper's central rankings, including the claim that Deepseek-V2.5 performs best and the comparison of Qwen2.5-72B versus GPT-4o, require merging these tables. There is no evidence that the two LLM judges rate answers on the same scale (e.g., no calibration sample, no human-agreement correlation, no judge-consistency statistic). The 'fair settings' claim in the abstract and §4.1 is therefore unsupported. I recommend rerunning the evaluation with a single judge for all models, or restricting cross-table comparisons to within-judge results and providing a direct judge-similarity analysis.
- [§3.2, Eq. (1)] The searcher-count term 10 × e^{−Sc} in S_final rewards models that invoke the Searcher fewer times, but §4.4.1 identifies 'overconfidence' (refraining from web search when the answer is unknown) as a failure mode. Because a model that fails to search and gives an incorrect answer can receive a higher searcher-count sub-score than a model that searches appropriately, the metric rewards behavior that the paper itself diagnoses as error-prone. Though this term's weight is small, it is part of the claimed 'suitable evaluation metric,' so the paper should justify this design choice, remove the term, or condition it on correctness.
- [§3.2 and §4.3] The correctness score relies entirely on LLM-as-judge, but no evidence is given that these scores agree with human judgments. The 'effectiveness' argument in §4.3 consists of observing that the rankings 'align with empirical knowledge and common sense,' which is circular because the metric is used to define those rankings. As the benchmark and metric are contributions, the paper should include a human evaluation on a sample (e.g., at least 100 responses) to validate Sco and, ideally, the other sub-metrics. Without this, the claim of offering a 'suitable evaluation metric' is not established.
- [§4.2, Figure 5] The comparison with commercial products does not state which judge computed the correctness scores for the product responses or for the Level-Navi Agent response. If the judge differs from the one used for the main tables, or if the products are evaluated with a different protocol, the conclusion that the agent is at the same level as commercial products is not supported. Please specify the judge and ensure that all compared systems are evaluated under identical conditions.
minor comments (6)
- [Table 3 caption] The caption reads 'Open Source Model Results with Qwen2.5-72B Evaluation' while the table lists closed-source models; this is likely a typo and should be corrected.
- [§3.2] The text says 'We assign the highest weight to the F1 score' but the formula weights the Correctness Score (Sco); please clarify the terminology.
- [Table 3] The model name 'Moonshoot-v1' is misspelled; it should be 'Moonshot-v1'.
- [§3.1] The dataset annotation process is described only briefly; please add details on the number of annotators and inter-annotator agreement to support the 'well-annotated' claim.
- [§3.2] The Relevance Score description says 'another evaluation LLM' will generate inferred questions, but the identity of this LLM is not specified; please state which model is used and whether it is the same as the correctness judge.
- [§4.1] Closed-source models are only tested with three-shot prompting, while the few-shot analysis (zero-shot, one-shot, three-shot) is applied only to open-source models; this should be clarified to avoid overgeneralizing the prompting conclusions.
Circularity Check
No circularity found: the framework, Web24 benchmark, and Eq. (1) metric are independent contributions, and the reported rankings are measurements rather than by-construction identities.
full rationale
The paper contains no claimed derivation that reduces to its own inputs. Level-Navi Agent is a prompt-based, training-free pipeline (Section 2), Web24 is independently constructed from Chinese news and knowledge with source URLs for verification (Section 3.1), and the composite metric in Eq. (1) is defined in advance with stated weights rather than fitted to the evaluated outputs. The relevance metric is imported from cited external work (RAGAs), and the LLM-as-judge correctness protocol is a cited external convention; neither is a self-citation by the authors. The Section 4.3 statement that the metrics' results 'align with empirical knowledge and common sense' is a weak external sanity check, not a reduction of the metric to the rankings it produces. The two-judge design (GPT-4o for Table 2, Qwen2.5-72B for Table 3) is a genuine validity threat to cross-table comparability, but it does not make any score definitionally equal to its own input; it is an evaluation-protocol concern, not circularity. No uniqueness theorem, self-citation chain, or fitted-then-renamed prediction appears, so the derivation chain is self-contained.
Assumptions & free parameters
free parameters (2)
- Metric weights (60/15/15/10) =
60, 15, 15, 10
- Exponential decay constant for searcher count =
base e
assumptions (4)
- domain assumption LLM-as-judge scores correlate with human judgments of answer correctness and relevance.
- domain assumption Web24 ground-truth answers and source URLs are correct and accurately annotated.
- domain assumption Fewer searcher invocations indicate better question decomposition and efficiency.
- domain assumption The search engine API returns relevant results for the sub-queries.
Cite this review
Pith. "Pith review of Level-Navi Agent: A Framework and benchmark for Chinese Web Search Agents." pith.science (2026). https://pith.science/paper/GAQGHFBE
@misc{pith2026250215690,
author = {Pith},
title = {Pith review of: Level-Navi Agent: A Framework and benchmark for Chinese Web Search Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/GAQGHFBE}},
note = {Machine review of arXiv:2502.15690}
}
read the original abstract
Large language models (LLMs), adopted to understand human language, drive the development of artificial intelligence (AI) web search agents. Compared to traditional search engines, LLM-powered AI search agents are capable of understanding and responding to complex queries with greater depth, enabling more accurate operations and better context recognition. However, little attention and effort has been paid to the Chinese web search, which results in that the capabilities of open-source models have not been uniformly and fairly evaluated. The difficulty lies in lacking three aspects: an unified agent framework, an accurately labeled dataset, and a suitable evaluation metric. To address these issues, we propose a general-purpose and training-free web search agent by level-aware navigation, Level-Navi Agent, accompanied by a well-annotated dataset (Web24) and a suitable evaluation metric. Level-Navi Agent can think through complex user questions and conduct searches across various levels on the internet to gather information for questions. Meanwhile, we provide a comprehensive evaluation of state-of-the-art LLMs under fair settings. To further facilitate future research, source code is available at Github.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,
-
[4]
Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao
URL https://openreview.net/forum?id=tzE7VqsaJ4. Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. Mindsearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20183,
-
[5]
URL https://aclanthology.org/2024.eacl-demo.16
Association for Computational Linguistics. URL https://aclanthology.org/2024.eacl-demo.16. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, and Jiajie Zhang et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools,
work page 2024
-
[6]
ISSN 1046-8188. doi: 10.1145/3703155. URL https://doi.org/10.1145/ 3703155. Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, et al. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv:2409.12959,
-
[7]
The impact of reasoning step length on large language models
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.04925,
-
[8]
Cmmlu: Measuring massive multitask language understanding in chinese
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212,
-
[9]
Yuanyuan Liang, Jianing Wang, Hanlun Zhu, Lei Wang, Weining Qian, and Yunshi Lan. Prompting large language models with chain-of-thought for few-shot knowledge base question generation. arXiv preprint arXiv:2310.08395,
-
[10]
doi: 10.18653/v1/2024.acl-long.624
Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.624. URL https://aclanthology.org/2024.acl-long.624. Xilai Ma, Jing Li, and Min Zhang. Chain of thought with explicit evidence reasoning for few- shot relation extraction. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics:...
Show all 21 references
-
[11]
doi: 10.18653/v1/2023.findings-emnlp.153
Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.153. URL https://aclanthology.org/2023.findings-emnlp.153. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, Wil...
2023 arXiv
-
[12]
URL https:// aclanthology.org/2023.tacl-1.75
doi: 10.1162/tacl a 00605. URL https:// aclanthology.org/2023.tacl-1.75. Revanth Gangi Reddy, Yi Ren Fung, Qi Zeng, Manling Li, Ziqi Wang, Paul Sullivan, and Heng Ji. Smartbook: Ai-assisted situation report generation. ArXiv, abs/2303.14337,
2023 arXiv
-
[13]
Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani-Tur, and Heng Ji
URL https://api.semanticscholar.org/CorpusID:257766360. Revanth Gangi Reddy, Sagnik Mukherjee, Jeonghwan Kim, Zhenhailong Wang, Dilek Hakkani-Tur, and Heng Ji. Infogent: An agent-based framework for web information aggregation. arXiv preprint arXiv:2410.19054,
-
[14]
Comparing traditional and llm-based search for consumer choice: A randomized experiment
Sofia Eleni Spatharioti, David M Rothschild, Daniel G Goldstein, and Jake M Hofman. Comparing traditional and llm-based search for consumer choice: A randomized experiment. arXiv preprint arXiv:2307.03744,
-
[15]
Gemma 2: Improving open language models at a practical size
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L ´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118,
-
[16]
Knowledge-driven cot: Exploring faithful reasoning in llms for knowledge-intensive question answering
Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang Xiong. Knowledge-driven cot: Exploring faithful reasoning in llms for knowledge-intensive question answering. arXiv preprint arXiv:2308.13259,
-
[17]
Crag–comprehensive rag benchmark
Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, et al. Crag–comprehensive rag benchmark. arXiv preprint arXiv:2406.04744,
-
[18]
A comprehensive capability analysis of gpt-3 and gpt-3.5 series models
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420,
-
[19]
How multilingual is multilingual llm? arXiv preprint arXiv:2311.09071,
Fei Yuan, Shuai Yuan, Zhiyong Wu, and Lei Li. How multilingual is multilingual llm? arXiv preprint arXiv:2311.09071,
-
[20]
Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms
Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. ...
2023
-
[21]
Agent-as-a- judge: Evaluate agents with agents
Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agent-as-a- judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934,
-
[2023]
Qwen technical report
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, and Mei Li et al. Qwen technical report. arXiv preprint arXiv:2309.16609,
-
[2024]
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi
URL https://github.com/meta-llama/llama3/blob/main/ MODEL_CARD.md. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Self-reflective retrieval augmented generation. In NeurIPS 2023 Workshop on Instruction T uning and Instruction Following,
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.