{"id":"2c029530-874d-4125-abc9-902f1be5a421","arxiv_id":"2502.15690","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper proposes Level-Navi Agent, a training-free Chinese web search agent framework, plus the Web24 benchmark and a composite evaluation metric for comparing LLMs.","lead":"This paper introduces Level-Navi Agent, a training-free framework that helps chatbots search the Chinese web by planning sub-questions and gathering information at several levels. The authors also present a 481-question benchmark, Web24, and a new scoring method, and use them to compare 14 open and closed language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-table model rankings rely on two different LLM correctness judges (GPT-4o for open models, Qwen2.5-72B for closed models) with no demonstrated scale equivalence; the paper's 'fair settings' claim is unsupported.","rationale":"The central claim of the paper is a fair, comprehensive evaluation of Chinese web search agents. That claim depends on the comparability of scores assigned to different models. The paper's own tables show different judges for different model groups, and there is no evidence that the judges are interchangeable. The reader's weakest assumption identified exactly this; I agree. This is more load-bearing than the other issues (e.g., the searcher-count metric's exponential-decay mapping is internally inconsistent with the claimed 1-10 scale, and dataset/code are not yet public) because it directly threatens the validity of the headline rankings and the 'fair settings' statement. A concrete judge-consistency test can settle whether the concern lands: if the two judges rank a common set of responses similarly and assign similar absolute scores, the cross-table comparisons survive; if not, the central evaluation results are called into question. I therefore keep the reader's CONDITIONAL verdict; the condition should be a judge-consistency analysis or a single judge across all tables, plus human validation of at least a subset.","tokens_in":811,"tokens_out":2666,"duration_ms":47081,"concrete_test":"Sample 100 responses (50 open-source from Table 2, 50 closed-source/API from Table 3, covering the full score range). Score every response with both GPT-4o and Qwen2.5-72B using the paper's 1-10 correctness rubric and normalize to 0-1. Compute the mean paired difference, Pearson/Spearman correlation, and rank agreement. If the judges differ by more than ~0.1 in mean score or have low rank correlation (rho < 0.8), the cross-table comparisons in Tables 2 and 3 are not valid. Additionally, obtain human annotations on the same 100 responses; report the correlation of each LLM judge with the human scores to determine which judge (if either) is trustworthy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest point in the central claim is the evaluation protocol. Table 2's correctness scores (Sco) are generated by GPT-4o, while Table 3's are generated by Qwen2.5-72B. The paper's headline comparisons—'Deepseek-V2.5 achieves the best' among API-called models, and the ultimate model rankings—require merging these tables. If the two judges differ in leniency, style preference, or self-bias, the observed score gaps (e.g., Deepseek-V2.5's 73.14 vs Qwen2.5-72B's 71.30) could be artifacts of judge identity rather than model capability. Qwen2.5-72B is itself an evaluated system in Table 2, so its use as the Table 3 judge introduces a self-evaluation risk. The paper provides no human rating correlation, no judge-consistency statistic, and no calibration experiment for either judge, so there is no evidence that the two judges are interchangeable. Because the error analysis (Sections 4.4.1 and 4.4.2) and the product comparison (Section 4.2) reuse these same correctness scores, all downstream conclusions inherit this vulnerability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Level-Navi Agent, a training-free web-search agent framework that decomposes user questions into sub-questions via a Planner and searches at multiple levels through a Level-Info Agent, together with a new Chinese web-search benchmark dataset (Web24, 481 questions) and a composite evaluation metric combining correctness (LLM-judged), semantic similarity, relevance, and searcher count. The authors evaluate 14 open/closed-source LLMs, report rankings, analyze failure modes (overconfidence and low task fidelity), and compare the framework with commercial products. The central claim is that the framework, dataset, and metric provide a fair, quantitative evaluation of Chinese web-search agent capabilities without fine-tuning.","tokens_in":12389,"tokens_out":6883,"duration_ms":54001,"significance":"If the evaluation protocol were sound, the paper would provide a useful and practical framework and benchmark for Chinese web-search agents. The strengths of the paper include the training-free design that works with any LLM, the collection of real Chinese internet sources with URLs for verification, the release of source code, and the concrete error analysis identifying overconfidence and low task fidelity. However, the significance is currently limited by the absence of validation for the LLM-judge-based correctness metric, the use of different judges for different result tables, and a metric design that may reward one of the failure modes the paper itself identifies. These issues affect the reliability of the reported rankings and the claimed suitability of the evaluation metric.","major_comments":[{"comment":"The correctness scores (Sco) in Table 2 are produced by GPT-4o, while those in Table 3 are produced by Qwen2.5-72B, as stated in the table captions. The paper's central rankings, including the claim that Deepseek-V2.5 performs best and the comparison of Qwen2.5-72B versus GPT-4o, require merging these tables. There is no evidence that the two LLM judges rate answers on the same scale (e.g., no calibration sample, no human-agreement correlation, no judge-consistency statistic). The 'fair settings' claim in the abstract and §4.1 is therefore unsupported. I recommend rerunning the evaluation with a single judge for all models, or restricting cross-table comparisons to within-judge results and providing a direct judge-similarity analysis.","section":"§4.1, Tables 2 and 3"},{"comment":"The searcher-count term 10 × e^{−Sc} in S_final rewards models that invoke the Searcher fewer times, but §4.4.1 identifies 'overconfidence' (refraining from web search when the answer is unknown) as a failure mode. Because a model that fails to search and gives an incorrect answer can receive a higher searcher-count sub-score than a model that searches appropriately, the metric rewards behavior that the paper itself diagnoses as error-prone. Though this term's weight is small, it is part of the claimed 'suitable evaluation metric,' so the paper should justify this design choice, remove the term, or condition it on correctness.","section":"§3.2, Eq. (1)"},{"comment":"The correctness score relies entirely on LLM-as-judge, but no evidence is given that these scores agree with human judgments. The 'effectiveness' argument in §4.3 consists of observing that the rankings 'align with empirical knowledge and common sense,' which is circular because the metric is used to define those rankings. As the benchmark and metric are contributions, the paper should include a human evaluation on a sample (e.g., at least 100 responses) to validate Sco and, ideally, the other sub-metrics. Without this, the claim of offering a 'suitable evaluation metric' is not established.","section":"§3.2 and §4.3"},{"comment":"The comparison with commercial products does not state which judge computed the correctness scores for the product responses or for the Level-Navi Agent response. If the judge differs from the one used for the main tables, or if the products are evaluated with a different protocol, the conclusion that the agent is at the same level as commercial products is not supported. Please specify the judge and ensure that all compared systems are evaluated under identical conditions.","section":"§4.2, Figure 5"}],"minor_comments":[{"comment":"The caption reads 'Open Source Model Results with Qwen2.5-72B Evaluation' while the table lists closed-source models; this is likely a typo and should be corrected.","section":"Table 3 caption"},{"comment":"The text says 'We assign the highest weight to the F1 score' but the formula weights the Correctness Score (Sco); please clarify the terminology.","section":"§3.2"},{"comment":"The model name 'Moonshoot-v1' is misspelled; it should be 'Moonshot-v1'.","section":"Table 3"},{"comment":"The dataset annotation process is described only briefly; please add details on the number of annotators and inter-annotator agreement to support the 'well-annotated' claim.","section":"§3.1"},{"comment":"The Relevance Score description says 'another evaluation LLM' will generate inferred questions, but the identity of this LLM is not specified; please state which model is used and whether it is the same as the correctness judge.","section":"§3.2"},{"comment":"Closed-source models are only tested with three-shot prompting, while the few-shot analysis (zero-shot, one-shot, three-shot) is applied only to open-source models; this should be clarified to avoid overgeneralizing the prompting conclusions.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a technical report with a useful framework and dataset, but the evaluation-protocol problems are substantial. The manuscript would benefit from a more rigorous validation of the metric, including human evaluation and a single-judge protocol. The writing also has several typos and inconsistencies that should be cleaned up. The scope fits cs.IR, but the current form is not acceptable without the proposed revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper gives the field something it didn't have: a Chinese web search agent benchmark (Web24, 481 items) with domain/type labels, a training-free agent framework, and a composite metric. The error analysis in Section 4.4 (overconfidence, task fidelity) is genuinely useful and matches what you'd expect from real deployments. If you just need a dataset to test Chinese search agents, this is a reasonable starting point.\n\nThe soft spot is the evaluation protocol. Table 2 uses GPT-4o as the correctness judge for open-source models; Table 3 uses Qwen2.5-72B for closed-source models. The paper then compares across tables (e.g., Deepseek-V2.5 as best among API-called models). That is only valid if the two judges rate on the same scale, and there's no evidence they do. Qwen2.5-72B is itself an evaluated model in Table 2, so using it as the Table 3 judge also invites self-evaluation bias. This is the load-bearing flaw for the headline ranking. The other issues are real but smaller: the searcher-count metric rewards fewer invocations, but Section 4.4.1 shows that failing to call the search function is a failure mode—so a lazy model can score well on Sc; the dataset and code are claimed to be available but no URL appears in the text; the dataset is only 481 items, which is small for stable conclusions; and there's no human agreement study for the LLM judge.\n\nThe metric design itself is honest: weighting correctness at 60%, using semantic similarity and relevance, and showing that F1/ROUGE behave counterintuitively (Table 4). I'd trust the relative ordering within Table 2, and within Table 3, but not the cross-table comparisons.\n\nVerdict: This deserves a serious referee, but with a request for major revision. The authors should release data/code, use a single judge (or show judge consistency), and fix the Sc metric or explicitly report function-call rates alongside it.\n\nI'd bring it to a reading group; I probably wouldn't cite it until the artifacts are out and the judge issue is addressed.","headline":"Useful Chinese web search benchmark, but cross-table model rankings are compromised by different LLM judges for open- and closed-source models.","tokens_in":12963,"tokens_out":3099,"would_cite":false,"duration_ms":24310,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a training-free, level-aware two-agent architecture, paired with the 481-question Web24 dataset and a four-part weighted score, can fairly rank LLMs on Chinese web search and exposes two explainable failure modes.","keywords":["Chinese web search","web search agent","LLM agent","level-aware navigation","benchmark dataset","evaluation metric","chain-of-thought planning","function calling"],"falsifier":"Take a random sample of the model outputs from the two result tables, have human annotators score them on the same 1-10 correctness scale, and compute agreement between each LLM judge and the humans; if either judge diverges from human ratings, or the two judges rank the same outputs differently, the composite-score rankings are not trustworthy.","tokens_in":11937,"feed_emoji":"🔍","tokens_out":9100,"duration_ms":70835,"temperature":0.7,"pith_summary":"The paper sets out to make Chinese web-search agents cheaply and fairly comparable across LLMs. Its package has three parts: Level-Navi Agent, a training-free two-agent framework in which a Planner decomposes each user question into sub-questions and a Level-Info Searcher gathers information in up to three levels; Web24, a 481-question Chinese benchmark drawn mostly from news; and a composite score that weights LLM-judged correctness, embedding similarity, response relevance, and a search-count cost term. The paper claims that on this benchmark, larger models score higher but with diminishing returns, native Chinese models beat English-centric ones, few-shot prompts improve task-completion pass rates, and token-level metrics like F1 and ROUGE misjudge open-ended answers. If the claims hold, researchers can evaluate open and closed LLMs on real Chinese web-search behaviour without retraining an agent, and they gain concrete targets for improving function calling and instruction following.","feed_headline":"Training-free agent framework puts Chinese web search LLMs to the test","feed_subtitle":"A 100-point composite score and a 481-question dataset claim to reveal which LLMs best search the Chinese web.","key_machinery":"The load-bearing object is the two-agent Level-Navi loop. The Planning Agent uses chain-of-thought with iterative refinement to generate only the next set of parallel sub-questions, while the Level-Info Agent searches up to three levels (internal knowledge, search-engine snippets, then opened pages) and returns as soon as it has enough information, which cuts token use and API calls. The evaluation machinery is the Web24 dataset, with five domains and four question types designed to force real searching, and the composite score $$S_{\\mathrm{final}} = 60\\,S_{\\mathrm{co}} + 15\\,S_{\\mathrm{simi}} + 15\\,S_{\\mathrm{rele}} + 10\\,$e^{{-S_c}}$,$$ where the weights encode the paper's priorities: correctness dominates, semantic similarity and relevance matter equally, and search count enters as an exponential cost incentive.","core_discovery":"The central discovery claimed is that a general-purpose, training-free agent can measure LLM web-search ability in Chinese when planning is iterative rather than fixed: the Planner emits only the next batch of parallel sub-questions, and the Searcher answers at the cheapest sufficient level, from memory, search snippets, or opened pages. On the paper's 481-item Web24 dataset and its weighted score, the claimed result is that Qwen2.5-72B and Deepseek-V2.5 lead the field, parameter gains show diminishing returns, few-shot prompting raises pass rates, and two failure modes—overconfidence (not calling the search tool when unsure) and low task fidelity (not following the instruction format)—explain much of the gap between weaker and stronger models. The paper also claims that semantic, LLM-judged, and relevance metrics should replace token-level F1 and ROUGE, showing data where longer, richer answers receive lower token scores despite being better.","pith_inferences":["Editorial inference: if the three-level information-acquisition pattern holds, the same Planner/Searcher split could be carried to other languages or domains by swapping the search API, without changing the metric's structure.","Editorial inference: because the final ranking leans on an LLM judge with no human-anchored calibration in the paper, future benchmark runs should report judge-human agreement before cross-table differences (open-source versus closed-source) are treated as real.","Editorial inference: the search-count term rewards agents that call the search tool less, which creates a latent incentive toward overconfidence; a testable variant would re-run the ranking with the searcher term removed to see how much of the ordering is efficiency-driven.","Editorial inference: Web24 is news-heavy and tied to pre-December 2024 sources, so the dataset will age; a rolling re-issue of the same pipeline on fresh Chinese news would test whether model rankings on this benchmark are stable over time."],"forward_implications":["Any open- or closed-source LLM, including small models, can be plugged into Level-Navi Agent without fine-tuning, so Chinese web-search evaluation no longer requires a dedicated trained agent.","The diminishing returns observed across model families imply that further score gains should come from better information filtering and source selection rather than from simply scaling parameters.","Few-shot prompting is a cheap, effective default for agent tasks, especially for weaker models whose pass rates otherwise drop.","The two diagnosed failure modes point to specific training interventions: balanced positive and negative function-calling examples to counteract overconfidence, and instruction-following data in long Chinese contexts to raise task fidelity.","Token-based metrics like F1 and ROUGE understate rich answers in web search, so the paper's composite metric should be used when comparing agents that summarize diverse sources."],"supporting_citations":[{"why":"It is a nearest prior agent framework whose planning approach Level-Navi Agent is designed to improve on.","marker":"Chen et al. (2024)"},{"why":"It supplies the information-aggregation agent approach that the paper builds on for its Searcher.","marker":"Reddy et al. (2024)"},{"why":"It provides the chain-of-thought prompting technique at the core of the Planner.","marker":"Wei et al. (2022)"},{"why":"It establishes the few-shot prompting paradigm used in the paper's prompt comparisons.","marker":"Brown et al. (2020)"},{"why":"It is the ROUGE metric the paper argues cannot judge open-ended web-search answers.","marker":"Lin (2004)"},{"why":"It motivates the LLM-as-judge correctness scoring used in the benchmark.","marker":"Yang et al. (2024)"},{"why":"It supplies the response-to-question inference idea behind the relevance score.","marker":"Es et al. (2024)"},{"why":"It provides the Chinese embeddings used to compute semantic similarity.","marker":"Xiao et al. (2024)"},{"why":"It frames the hallucination and overconfidence literature behind the paper's main failure-mode analysis.","marker":"Huang et al. (2024)"}],"fun_headline_variants":["Training-free agent benchmarks Chinese web search LLMs","Level-Navi: training-free agent ranks Chinese web searchers","Chinese web search: training-free agent benchmarks LLMs","Training-free agent reveals top LLMs for Chinese web search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings depend on treating two different LLM judges as interchangeable: one scores open-source models, another scores closed-source models, and no human-agreement or judge-consistency check is reported to show their scores are on the same scale.","fun_headline_variants_meta":{"raw":{"variants":["Training-free agent benchmarks Chinese web search LLMs","Level-Navi: training-free agent ranks Chinese web searchers","Chinese web search: training-free agent benchmarks LLMs","Training-free agent reveals top LLMs for Chinese web search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001195,"raw_usage":{"total_tokens":4920,"prompt_tokens":929,"completion_tokens":3991,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":3925}},"tokens_in":545,"tokens_out":3991,"duration_ms":25402,"temperature":1.0,"reasoning_tokens":3925,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:13:34.073348+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the model outputs from the two result tables, have human annotators score them on the same 1-10 correctness scale, and compute agreement between each LLM judge and the humans; if either judge diverges from human ratings, or the two judges rank the same outputs differently, the composite-score rankings are not trustworthy.","supporting_citations":[],"review_version":1}