{"id":"ec8db5c6-f163-4d01-a232-6edabc7df0a7","arxiv_id":"2607.20510","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A reproducible bilingual benchmark of 100 multi-hop telecom tasks evaluates AI agents over a frozen website, a synthetic SQL database, and web archives; the strongest tested model solves 71%.","lead":"Telco-GAIA is a new bilingual benchmark that tests AI agents on realistic telecom customer-service tasks, using a frozen snapshot of a real operator's website, a synthetic billing database, and live web pages. It offers companies a reproducible way to measure how well agents handle multi-step questions that combine text, images, PDFs, and tables.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Live Wikipedia/arXiv access undercuts the 'reproducible over time' claim for 16/100 tasks.","rationale":"Telco-GAIA is a thoughtfully designed benchmark with clear construction principles, honest limitations, and a reproducible scoring script. I evaluated the central claim that it is a rigorous, reproducible testbed. The strongest internal tension is the live-web dependency of Web Archives tasks, which the reader also identified. This is not a matter of disagreement with prior work; it is an internal inconsistency: the paper claims a semi-closed, time-stable environment (Contributions 2 and Abstract) while Appendix A confirms that Wikipedia and ArXiv are fetched directly from the internet. Because the gold answers for 16 tasks depend on content the authors do not control and have not snapshotted, the reproducibility claim is overbroad. I agree with the reader's CONDITIONAL verdict: the issue is real but fixable (e.g., snapshot the web pages or explicitly exclude Web Archives from the reproducibility claim). No additional fatal flaw emerged; the missing error bars and absent human baseline noted by the reader are secondary. The proposed API-based stability check would settle whether this concern actually lands. Given the concern is already reflected in the reader's verdict, I recommend no change to the CONDITIONAL outcome.","tokens_in":16243,"tokens_out":8437,"duration_ms":93511,"concrete_test":"For each Web Archives task, use the Wikipedia/arXiv APIs to retrieve the exact pages and facts referenced in the gold steps. Verify that the gold answer is still present and correct in the current revision, and query the revision history to see whether the fact has changed since the benchmark freeze date (February 2026). If any answer is altered or absent, the benchmark's 'reproducible over time' claim fails for that task; if all 16 remain stable, the claim is supported for the observed period. A complementary check: re-run the 16 tasks twice, once against live pages and once against a locally cached copy of those pages from the freeze date, and compare scores; any divergence shows material dependence on third-party content.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'objective, deterministic, and reproducible over time' (Abstract) with the environment 'semi-closed' because 'web archives are virtually immutable' (§1)—is not supported for the 16 Web Archives tasks. Appendix A states that 'Wikipedia and ArXiv are accessed directly on the internet' (A.1), so these tasks depend on live, third-party content that is neither frozen nor immutable. Wikipedia pages are continuously edited; a fact in an infobox can be changed, reverted, or removed, and a page can be renamed or deleted. ArXiv pages, while more stable, can also be updated. If such a change occurs, the gold answer may no longer be present or correct, so the same run conducted at different times can have different ground truth. The paper's stated mitigation, 'we prefer update-resilient targets such as infobox fields and ISO codes' (§3.2), is an unverified design preference: no data are provided showing these 16 answers have been stable, and the pages themselves are not snapshotted. This is an internal inconsistency between the reproducibility claim (Contribution 2) and the live-web component, not a matter of outside consensus. The concern is load-bearing because reproducibility is a headline property and the proposed template for 'closed-domain benchmarks' depends on a fully sandboxed environment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Telco-GAIA presents a 100-task bilingual benchmark for tool-using agents in telecommunications. Each task is a multi-hop QA problem requiring reasoning over a locally served snapshot of a telecom operator's website (HTML, images, PDFs), a synthetic relational SQLite database exposed via REST, and, for 16 Web Archives tasks, live Wikipedia/ArXiv pages. The paper describes human-verification and anti-shortcut design principles, adversarial database traps, and an objective exact-match scorer. The authors evaluate a purpose-built reference agent across twelve commercial/open LLMs, reporting best accuracy 71%, with lowest scores on image/PDF-visual categories.","tokens_in":16502,"tokens_out":7354,"duration_ms":85227,"significance":"Assuming the design and release are as described, Telco-GAIA is a useful resource and one of the few benchmarks combining multimodal retrieval, relational-database querying, bilingual content, and deterministic, LLM-judge-free scoring. The paper ships concrete infrastructure (Docker services, evaluate.py, gated ground truth), uses human-verified golden steps, and provides a broad multi-model sweep. It is a solid contribution to enterprise-agent evaluation. Two issues need to be addressed before the central claims are fully supported: reproducibility over time for the 16 live-web tasks, and the reproducibility/comparability of the reference-agent protocol given hidden per-task tool restrictions. Both are fixable within the manuscript's scope.","major_comments":[{"comment":"The reproducibility claim is not supported for the 16 Web Archives tasks. The abstract and §1 state that the environment is 'semi-closed' and runs are 'reproducible over time' because 'web archives are virtually immutable'; however §A.1 states that 'Wikipedia and ArXiv are accessed directly on the internet,' and §3.2 merely expresses a preference for 'update-resilient targets such as infobox fields and ISO codes.' Wikipedia pages are continuously edited, so an infobox value, ISO code, or article identity can change, making the gold answer absent or wrong on a later run. No stability data or snapshots are provided. Since reproducibility is Contribution 2, this is load-bearing. The fix is straightforward: snapshot and serve the specific web-archive pages inside the container, or explicitly restrict the reproducibility claim to local components and mark Web Archives tasks as time-dependent.","section":"§1, §3.2, §A.1"},{"comment":"The reference-agent baseline uses privileged per-task tool gating that external users cannot replicate. §A.2 states that 'only task_id and question are exposed to the agent,' while the full ground-truth record includes the tools field and 'the harness additionally restricts the reference agent to each task's permitted tool subset.' Thus a third party running the public benchmark cannot know which tools are allowed per task and cannot reproduce the reference-agent conditions. This weakens the comparability of the reported 12-model sweep and the 'golden steps' anti-shortcut design. Please release the per-task tool lists as non-answer metadata (or as part of questions.json) so that any agent can be run under the same gating, or explicitly state that the reference-agent numbers use privileged metadata.","section":"§A.2, §C"}],"minor_comments":[{"comment":"Accuracies are from a single run. Several comparisons (e.g., 68% vs 67%) are within sampling noise for 100 binary tasks. Report confidence intervals or multi-run averages for at least the headline models.","section":"Table 2"},{"comment":"No human baseline is reported. A human accuracy estimate (or a reference to one) would help calibrate the 'challenging' claim.","section":"General"},{"comment":"For the 17 textual answers, exact string matching can be brittle even with normalization; consider documenting the set of accepted alternate phrasings or the normalization rules in more detail.","section":"§A.4"},{"comment":"The '‡' footnote marker is used both in the table and in the text body; this is confusing and should be cleaned up.","section":"Table 3"},{"comment":"Section D is quite long relative to the rest of the paper; condensing would improve readability, though the content is relevant.","section":"Section D"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about live web archives is valid and is the main reason I cannot recommend acceptance as-is. The second issue about hidden tool gating is also real but easily fixed by releasing per-task tool metadata. I would be comfortable with acceptance after a major revision addressing these; no need to re-run the full sweep if the local web tasks are unchanged."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: Telco-GAIA is a solid, reusable benchmark, and the paper is worth a serious referee. The unique combination — bilingual (English/Arabic), multi-modal (website, PDFs, images, SQL database), telecom domain, golden-step annotations, and LLM-judge-free exact-match scoring — is genuinely new relative to GAIA, WebArena, WixQA, and TelAgentBench. The construction is careful: strictly causal chains, anti-spoiling, and the injected data-quality traps in the database are a thoughtful touch that evidently raise the difficulty above naive SQL queries. The evaluation across twelve commercial and open models gives a useful cost–accuracy picture, even if the absolute numbers will age quickly.\n\nThe main soft spot is the reproducibility claim. The abstract and Section 1 say the environment is \"semi-closed\" and \"reproducible across time\" because the website snapshot and database are frozen and \"web archives are virtually immutable.\" But Appendix A states plainly that Wikipedia and arXiv are \"accessed directly on the internet.\" Those 16 Web Archives tasks therefore depend on live, third-party content. Wikipedia pages get edited; infoboxes change; pages get renamed or deleted. No snapshot is provided, so a future run can face different ground truth. The authors' stated preference for \"update-resilient targets\" is a reasonable design heuristic, but it is not evidence of stability. This needs to be fixed either by snapshotting the web pages or by softening the reproducibility claim to apply only to the local corpus. It's not fatal — 84 of 100 tasks do run on frozen content — but it is a genuine inconsistency.\n\nTwo smaller issues: the main model sweep (Table 2) is a single run, so the accuracy differences between models in the 67–71% band are within noise; only gpt-5.2 gets five-run variance. And there is no human baseline, so \"challenging\" is not calibrated, though 71% top accuracy already implies the tasks are non-trivial.\n\nOverall, this is a well-executed resource paper. It will be valuable to anyone building closed-domain enterprise agent benchmarks, and the category-level results — especially the weak visual understanding — give the agent community a concrete target. I'd send it to peer review with a request to address the live-web reproducibility issue and, if easy, to add error bars or a second run for at least a few top models. The core contribution holds up.","headline":"A carefully built bilingual telecom agent benchmark that mostly delivers on its stated design goals, except the 'reproducible over time' claim is overstated for the 16 live-web tasks.","tokens_in":17018,"tokens_out":2056,"would_cite":true,"duration_ms":25897,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Telco-GAIA introduces 100 bilingual, multi-hop telecom tasks scored by deterministic exact matching, with the best tested model solving only 71 percent.","keywords":["telecom agents","bilingual benchmark","multi-hop reasoning","tool use","exact string matching","relational database","visual document understanding","enterprise AI evaluation"],"falsifier":"Take any Web Archives task, edit its target third-party page so that the previously correct value changes, then rerun the unchanged evaluation script against the released ground truth. A correct environment should still pass; if the task now scores zero, the benchmark's reproducibility claim fails for that task.","tokens_in":16145,"feed_emoji":"📡","tokens_out":3583,"duration_ms":42902,"temperature":0.7,"pith_summary":"The paper introduces Telco-GAIA, a benchmark for AI agents that answer realistic telecom customer questions over a single operator's public data. Each of its 100 tasks is a human-verified multi-hop chain, averaging 4.2 hops, that forces an agent to navigate a frozen copy of the operator's website, read linked PDFs and images, query a synthetic relational database, and sometimes consult live third-party web archives. The authors claim this is the first benchmark to combine all these properties, and that scoring by normalized exact string matching makes evaluation objective, deterministic, and reproducible over time without an LLM judge. On a reference agent tested with twelve commercial and open models, the strongest reaches 71 percent, while the visually grounded categories remain the clearest bottleneck. The paper positions Telco-GAIA both as a difficulty probe for enterprise agents and as a reusable template for building closed-domain benchmarks.","feed_headline":"Tool-driven telecom benchmark caps top LLM at 71 percent","feed_subtitle":"100 bilingual tasks over a frozen site, database, and web archives make agent scores deterministic and repeatable.","key_machinery":"The load-bearing mechanism is the task-construction rule of strict causal chains: each task must form a single linear chain where the output of hop N is required for hop N+1, with no dead ends, no redundant facts, and no spoiling of intermediate answers. This is paired with a Docker-served frozen website and relational database, a normalized exact-match scorer with no LLM judge, and controlled data-quality traps in the database so that a naive SELECT SUM or COUNT returns the wrong answer and only agents that inspect and filter the data succeed. The environment is semi-closed: the operator website and database are frozen locally, while sixteen Web Archives tasks reach live third-party encyclo","core_discovery":"Telco-GAIA's central claim is that a closed enterprise benchmark can keep the discipline of exact-match evaluation while covering heterogeneous, realistic sources: a static website snapshot, linked PDFs and images, a synthetic SQLite customer database exposed through a REST API, and external web archives. The 100 tasks span seven categories in English and Arabic, with 83 numeric and 17 textual answers, all human-verified and scored by normalized exact string matching. The reference agent experiments show a clean accuracy spread from 13 to 71 percent across twelve backends, with the strongest model solving 71 percent of tasks; under a moderate cost budget accuracy falls to about 38 percent. C","pith_inferences":["The semi-closed design has a self-hardening property: as the live operator site drifts from the frozen snapshot, parametric-memory shortcuts decay over time and later runs become harder to pass without genuine retrieval, which is an unusual direction for a benchmark.","The same recipe — a frozen site snapshot, a synthetic database, and exact-match scoring — could be lifted to other enterprise or regulated domains, with the catalogue of data-quality traps as a reusable component.","The sixteen tasks that depend on live third-party pages are the weak point of the reproducibility guarantee; freezing those pages as snapshots as well would close the remaining leak.","Because task difficulty is reported alongside cost and latency, the benchmark can double as a cost-calibration instrument for deployment decisions, not only as a quality probe."],"forward_implications":["Model scores on Telco-GAIA are directly comparable across time and across labs, because the served corpus is frozen and scoring involves no judge model.","A moderate-cost backend can reach near-frontier accuracy at roughly half the cost of the top model, so budget, latency, and accuracy should be treated as largely independent axes when choosing an agent backend.","The visual categories (Images, PDF, PDF Visual) lag far behind text and database categories, pointing to document and image understanding as the current binding constraint for enterprise agents.","The Arabic subset is close in difficulty to the English subset, so bilingual evaluation can be carried out without one language becoming an easy out.","The controlled database traps make naive SQL fail, so agents must inspect and filter data rather than pattern-match, rewarding genuine tool use and data-quality awareness."],"fun_headline_variants":["Bilingual telecom agent benchmark: best LLM hits 71%","Telco-GAIA benchmark: top agent 71%, budget 40%","Exact-match telecom benchmark: LLM ceiling 71%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reproducibility guarantee assumes that the live third-party encyclopedia and preprint pages used by the Web Archives tasks will not change their task-relevant content; if such a page is edited, the gold answer can become stale even though the served environment is unchanged.","fun_headline_variants_meta":{"raw":{"variants":["Bilingual telecom agent benchmark: best LLM hits 71%","Telco-GAIA benchmark: top agent 71%, budget 40%","Exact-match telecom benchmark: LLM ceiling 71%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001382,"raw_usage":{"total_tokens":5432,"prompt_tokens":739,"completion_tokens":4693,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":4632}},"tokens_in":483,"tokens_out":4693,"duration_ms":33136,"temperature":1.0,"reasoning_tokens":4632,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:44:22.093665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any Web Archives task, edit its target third-party page so that the previously correct value changes, then rerun the unchanged evaluation script against the released ground truth. A correct environment should still pass; if the task now scores zero, the benchmark's reproducibility claim fails for that task.","supporting_citations":[],"review_version":1}