{"id":"34b501d9-541e-4172-8a07-4f5d4b8f9fa0","arxiv_id":"2505.03735","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors present SoccerBench (10K multimodal soccer QA pairs), SoccerWiki (a soccer knowledge base), and SoccerAgent, a multi-agent system that outperforms general multimodal LLMs on the benchmark.","lead":"This paper builds SoccerWiki, a soccer knowledge base, SoccerBench, a 10,000-question multimodal test covering 13 soccer tasks, and SoccerAgent, a multi-agent program that answers the questions by chaining specialized tools. General-purpose AI models score about 57% on the test, while the agent scores about 61%, showing the value of combining knowledge retrieval with perception tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark QA label quality is load-bearing: LLM-generated answers and distractors are manually filtered but never quantitatively validated, so both SoccerBench rankings and SoccerAgent's claimed superiority could shift under independent re-annotation.","rationale":"The reader's weakest assumption is the same one I land on: benchmark label and distractor quality is the gate for every downstream claim. Scale claims about SoccerBench ('largest', 'most comprehensive') depend on counting, but the scientific value depends on the 10K answers being right and the four options being non-trivially distinguishable. SoccerAgent's performance is measured directly against those same options, so if the audit reveals systematic label errors or easy distractors, the headline comparison is invalid even if the multi-agent architecture is sound. I considered alternative concerns: (i) SoccerAgent receives task descriptions and recommended tool chains while baselines do not; (ii) Table 2 contradicts 'leading across all tasks' for Q5/Q8/Q12/Q13. Both are real but secondary. (i) is a design asymmetry for a benchmark whose purpose is knowledge-grounded QA, and (ii) is an overstatement in the text rather than a flaw that changes the aggregate conclusion. Credit where due: the benchmark reuses public SoccerNet/SoccerReplay annotations and open-source tools, so the audit is feasible and the sources are checkable. The absence of measured curation quality is precisely why CONDITIONAL is the right verdict: not an accusation, but a missing number that the authors can supply. If the audit comes back clean, the verdict should move to ACCEPT; if the error rates cross the stated thresholds, Table 2 must be recomputed on the cleaned subset. Thus I recommend UNCHANGED relative to the reader's CONDITIONAL.","tokens_in":29160,"tokens_out":8648,"duration_ms":91715,"concrete_test":"Run an independent re-annotation audit on a stratified random sample of 600 SoccerBench items (~50 per task, preserving the text/image/video split). Two soccer-knowledgeable annotators who are not authors independently verify each item: (1) the designated answer matches the cited source (SoccerWiki page or source-dataset annotation); (2) every distractor is plausible enough that a competent fan cannot eliminate it from the question text alone; (3) for Q3-Q13, the image/video is genuinely required. Pre-commit to a rule: if verified label error exceeds 5%, or if more than 10% of visual items are answerable without the media, or if more than 20% of items have at least one obviously eliminable distractor, recompute Table 2 on the cleaned subset and re-evaluate the superiority claim. Report Cohen's kappa and per-task error rates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 2 scores every model against the 10K QA labels of SoccerBench. Those labels were produced by an automated DeepSeek-v3 pipeline and author-side manual selection (Sec. 3.3.2; App. C.3), with no inter-annotator agreement, no per-task error rate, no distractor-plausibility measurement, and no external audit. App. C.3 says that ImageQA/VideoQA items answerable without visual context were filtered, but it reports no number for how many were removed or how reliably the criterion was applied. For the LLM-generated tasks (Q1, Q2, Q4, Q6, Q11, Q12), a hallucinated answer or an easily eliminated distractor propagates directly into every accuracy number. Since the paper's 'challenging benchmark' claim and the superiority of SoccerAgent are both defined relative to these labels, a nontrivial rate of wrong answers, ambiguous questions, or trivially distinguishable options would change the rankings in Table 2. This is a missing-validation concern, not an allegation of fraud; the fix is a measured audit. A secondary overstatement is that SoccerAgent loses to GPT-4o on Q5/Q13 and to Gemini on Q8/Q12, so 'leading results across TextQA, ImageQA, and VideoQA' holds only on aggregate, not per task.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-part framework for soccer understanding: SoccerWiki, a multimodal knowledge base of players, teams, referees, and venues; SoccerBench, a 10K-sample multiple-choice QA benchmark spanning 13 tasks over text, image, and video; and SoccerAgent, a multi-agent system with 18 tools that plans and executes tool chains to answer soccer questions. The authors evaluate SoccerAgent against commercial and open-source MLLMs, reporting the highest aggregate accuracy on TextQA, ImageQA, and VideoQA, and they provide ablations over task descriptions and execution examples, cost analysis, and a failure analysis.","tokens_in":29397,"tokens_out":5812,"duration_ms":59287,"significance":"If validated, SoccerBench would be a useful unifying evaluation resource for soccer understanding, and SoccerAgent would be a strong tool-based baseline. The paper has several concrete strengths: it integrates multiple external soccer datasets into a standardized QA format, releases 17 of 18 tools as open-source, provides a transparent failure analysis in Appendix B.3, and reports cost and deployment details. The aggregate results in Table 2 are broadly consistent with the claim that an agent with retrieval and specialized tools outperforms generic MLLMs on this benchmark. The main risks are that the benchmark labels themselves are not independently validated, the ablation does not isolate the multi-agent contribution, and the reported comparisons lack uncertainty quantification; these issues are fixable but are load-bearing for the paper's headline claims.","major_comments":[{"comment":"The benchmark labels are the reference standard for every accuracy number in Table 2, yet their correctness and ambiguity are never measured. The pipeline synthesizes 100K QA pairs with DeepSeek-v3 and manually selects 10K, but the paper reports no inter-annotator agreement, no per-task label error rate, no distractor-plausibility measurement, and no external audit. Appendix C.3 states that ImageQA and VideoQA items answerable without visual context were filtered, but it gives no count of removed items and no reliability estimate for that filter. For the LLM-generated tasks (Q1, Q2, Q4, Q6, Q11, Q12), a hallucinated answer or an easily eliminated distractor directly changes the rankings. I request a measured audit: independent re-annotation of a stratified sample (e.g., 200 items per task), per-task agreement and label-accuracy numbers, a distractor-plausibility check (e.g., how often human annotators select the correct option at above chance), and a report of how many items were removed by each manual filter. Without this, the 'challenging benchmark' and 'superiority' claims are not yet established.","section":"Section 3.3.2 / Appendix C.3"},{"comment":"The ablation does not isolate the multi-agent contribution. Table 3 varies only whether the planning agent receives task descriptions and whether the execution agent receives 20 examples; every variant still uses the full tool-based multi-agent architecture. There is no single-agent control that calls the same tools without decomposition, no 'no-tools' control that asks the agent core to answer directly, and no control that removes the retrieval tools. Consequently, the large gains on Q1 and Q4 could be driven by the SoccerWiki retrieval tools themselves rather than by the multi-agent decomposition and collaboration. I request ablations that separate (a) the agent core alone, (b) the agent with tools but no decomposition, and (c) the full system, so that the specific contribution of the multi-agent design is identifiable.","section":"Section 5.3, Table 3"},{"comment":"No confidence intervals or significance tests are reported, so the headline comparisons are not yet statistically grounded. Several aggregate differences are small relative to the per-task sample sizes: for example, SoccerAgent (MCQ) achieves 60.9% VideoQA versus GPT-4o's 57.5% over approximately 4,600 video questions, while individual tasks have only 200-1,000 items. I request bootstrap confidence intervals for all cells and per-task paired tests (e.g., McNemar) for SoccerAgent versus the strongest baseline. This is particularly important because the paper makes comparative claims such as 'leading results across TextQA, ImageQA, and VideoQA' that require more than point estimates.","section":"Section 5.2, Table 2"},{"comment":"The paper's superiority claim is stronger than the per-task data support. In Table 2, SoccerAgent (MCQ) is below GPT-4o on Q5 (85.7 vs. 89.6), Q8 (35.7 vs. 40.0), and Q13 (55.5 vs. 59.7), and below Gemini 2.0 Flash on Q12 (46.0 vs. 52.4). The claim 'leading results across TextQA, ImageQA, and VideoQA' is accurate only at the aggregate-category level, and the abstract's 'superiority' should be qualified accordingly. The authors should either report the specific tasks where the agent wins and loses or soften the claim to 'highest aggregate accuracy.'","section":"Section 5.2, Table 2 (per-task rows)"},{"comment":"The evaluation may favor the in-house pipeline because the agent's tools and the benchmark draw on the same research-group datasets: UniSoccer tools are used for Q9/Q10, and SoccerReplay-1988 and MatchTime are both benchmark sources and agent tool sources. Appendix C.1 states that only test sets of the respective datasets are used to avoid leakage, but the paper does not demonstrate that the tool models have no access to those test splits or that the benchmark templates do not directly echo the tool outputs. I request a concrete contamination check: report the train/test split provenance for every source dataset used by the tools, and measure how SoccerAgent performs when the same-group datasets are excluded from the benchmark or when the involved tools are replaced by external ones. This is a correctness-risk concern, not an allegation of misconduct.","section":"Section 4.2 / Appendix C.1 / Table 1"}],"minor_comments":[{"comment":"In the task-coverage table, the SoccerNet-v2 row appears to mark 'Jersey Number' as covered, but jersey-number annotations are introduced by SoccerNet-JN; please verify this entry against the cited dataset.","section":"Appendix Table 4"},{"comment":"The MatchTime and SoccerReplay-1988 references are used inconsistently: Appendix C.1.2 cites MatchTime as [43] and SoccerReplay-1988 as [44], while the main text and Table 1 use [43] for SoccerReplay-1988 and [44] for MatchTime. Please align all citations.","section":"References and Appendix C.1"},{"comment":"The task-description prompt lists 14 tasks, including 'Task3: Match Events and Statistical QA,' but Table 1 defines only 13 benchmark tasks. Please reconcile the numbering or explain the extra task.","section":"Appendix D.1"},{"comment":"Q10 is described as commentary generation, yet it is scored as multiple-choice accuracy in Table 2. Please clarify what the four options are for this task and how template-generated commentary options are selected, since the underlying annotations are free-form captions.","section":"Table 2 and Section 5.1.3"},{"comment":"The cost statement 'each inference costs within $0.1' should specify whether this is per question, per tool call, or per full SoccerAgent trajectory, and whether it includes the local GPU cost or only the DeepSeek API fee.","section":"Appendix B.2"},{"comment":"There is a small typo in the figure: the final step result is written as '</Anwer>' instead of '</Answer>'.","section":"Figure 4"},{"comment":"The paper does not state where the dataset, code, and SoccerWiki will be released, despite the 'open-source' emphasis for the tools. Please add a public-release plan or an explicit statement of availability.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's benchmark and agent tools substantially reuse the authors' own prior datasets (SoccerReplay-1988, MatchTime, UniSoccer), so the same-group provenance deserves prominent disclosure and a concrete split-disjointness demonstration. The 'first' and 'largest' claims should also be checked against SPORTU and Sports-QA in the related-work comparison. The aggregate result is plausible, but the missing label audit and missing multi-agent ablation are substantial enough that I cannot recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look, and worth a serious referee, but the headline claim needs one more round of validation before I would trust any ranking in Table 2.\n\nWhat is actually new: SoccerBench is the first unified multimodal QA set over 13 soccer tasks, and SoccerWiki is a genuinely useful knowledge resource. The paper assembles these from existing datasets plus a lot of new curation, and the multi-agent system is a reasonable recipe for tool-use QA in a specialized domain. The transparency about tool bottlenecks (Appendix B.3) is a real credit: they say where the failures come from, and they quantify that Entity Recognition at 63.4% player ID accuracy directly caps Q4 at 65.3%. That is the kind of honest reporting that makes a systems paper useful.\n\nThe stress-test concern lands. The whole benchmark evaluation is built on 10K QA pairs synthesized by DeepSeek-v3 and manually filtered, with no inter-annotator agreement, no per-task error rate, no distractor plausibility measurement, and no external audit. Appendix C.3 mentions filtering out image/video questions answerable without visual context but gives no count and no reliability check. If even a few percent of labels are wrong or distractors are trivially disambiguable, the accuracy numbers in Table 2 shift, and the claimed superiority of SoccerAgent over chat-only MLLMs could change. That is a missing-validation problem, not a fraud allegation, but it is load-bearing.\n\nThere are softer issues too. The paper reports aggregate accuracy without confidence intervals, and the ablation never isolates the multi-agent contribution from the tools: the agent loses to GPT-4o on Q5 and Q13, and to Gemini on Q8 and Q12, so the \"leading results across TextQA, ImageQA, and VideoQA\" is only true in aggregate. Since several tools (UniSoccer, MatchTime, SoccerReplay-1988, SoccerWiki) come from the same group, the evaluation could favor the in-house pipeline even absent any intent. That is not disqualifying, but it deserves an external re-run. And the authors say the benchmark and code are not released, which limits verification.\n\nWho is this for: anyone working on sports QA, tool-augmented VLMs, or domain-specific benchmarks. It deserves a serious referee, not a desk reject. My recommendation: engage with it, but require a measured QA audit — a random sample of N questions independently re-annotated, per-task agreement, and a release of labels plus code. If the audit comes back clean, this becomes a standard resource. If not, the rankings need to be recomputed.","headline":"A useful, well-scoped soccer QA benchmark and agent toolkit whose central comparison is only as trustworthy as its QA labels, and those labels are currently unvalidated.","tokens_in":29959,"tokens_out":1136,"would_cite":false,"duration_ms":14276,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a 10,000-question benchmark and an 18-tool multi-agent system bring comprehensive, knowledge-driven soccer understanding within reach.","keywords":["soccer understanding","multimodal question answering","multi-agent system","knowledge base","benchmark","video understanding","tool use","sports analytics"],"falsifier":"Take a random sample of 200 to 300 SoccerBench items, have independent soccer-knowledgeable annotators answer them without seeing the published options' origin, and compare their answers and confidence to the dataset's labels. If a material fraction of items has a wrong key or a distractor that is obviously false to an informed reader, then the benchmark rankings and SoccerAgent's margin rest on shaky ground. A complementary check is to feed the image and video questions to a strong text-only model; any substantial accuracy above chance would mean the visual material is not load-bearing for those items.","tokens_in":28931,"feed_emoji":"⚽","tokens_out":8362,"duration_ms":79564,"temperature":0.7,"pith_summary":"The paper tries to establish that holistic soccer understanding can be turned into a standard, knowledge-driven benchmark and solved by an agentic system rather than a single monolithic model. To do so it assembles SoccerWiki, a multimodal knowledge base of players, teams, referees, venues, and match records; builds SoccerBench, roughly 10,000 multiple-choice questions across 13 text, image, and video tasks; and introduces SoccerAgent, a multi-agent system that plans an ordered chain of specialized tools and executes them to answer questions. The paper's core empirical claim is that SoccerAgent outperforms representative commercial and open-source vision-language models on SoccerBench, with the largest gains on knowledge-intensive tasks. A sympathetic reader would care because a unified benchmark plus a reusable agent baseline could move sports AI from isolated perception tasks toward open-ended, knowledge-grounded reasoning about matches.","feed_headline":"New agent system tops single models on 10,000 soccer questions","feed_subtitle":"A soccer-specific knowledge base and 18-tool agent claims the lead on a new 13-task benchmark.","key_machinery":"The load-bearing mechanism is the plan-then-execute agent loop. Given a question, a planning module first decomposes it into an ordered tool chain, chosen from 18 tools that include face recognition, match search, action classification, camera detection, jersey number recognition, score/time recognition, replay grounding, foul recognition, and general-purpose operations such as frame selection and textual retrieval. An execution module then calls each tool in sequence, feeding each call a structured instruction with tool name, query, file paths, and purpose, while accumulating an execution history that later steps can consult. The chain ends with an answer-selection step that maps the gathered evidence to a multiple-choice option. SoccerBench itself is the other half of the machinery: for each task, questions are produced either by templates or by prompting a large language model, distractors are sampled or generated, and roughly 10,000 of 100,000 synthesized pairs are kept after manual filtering. The benchmark supplies the measurement that makes the agent's superiority visible.","core_discovery":"The central discovery the paper argues for is that SoccerBench, with about 10,000 multimodal multiple-choice QA pairs spanning 13 tasks, is the largest and most comprehensive soccer understanding benchmark currently available, and that SoccerAgent, a multi-agent system combining a planning module, an execution module, and 18 specialized tools over SoccerWiki, achieves robust performance on it. The authors report that SoccerAgent's best variant reaches 85.0 percent on TextQA, 73.3 percent on ImageQA, and 60.9 percent on VideoQA, the top scores among the models compared, while the strongest baseline commercial model trails on most knowledge-heavy tasks. They further argue that the system works in an open-ended mode, generating answers without seeing the candidate options, and that swapping in a stronger vision tool improves results, evidence that the architecture is scalable. On the paper's own account, the benchmark differentiates models, with accuracy ranges wide enough to separate generic perception from soccer-specific knowledge.","pith_inferences":["The curation pipeline of templates plus language-model generation with manual filtering is not soccer-specific, so a similar benchmark could be built for other sports or for other knowledge-heavy video domains without changing the core machinery.","Because SoccerWiki can refresh from live match data, SoccerBench could evolve into a living benchmark that tracks how quickly models and agents absorb new seasons; the paper hints at updatability but does not measure it.","A text-only probe of the image and video questions would quantify how much of SoccerBench actually requires pixels; the authors manually filter obvious cases, but an automatic leakage statistic would harden the benchmark."],"forward_implications":["SoccerBench gives the field a single standardized yardstick for comparing models across 13 soccer tasks, from background knowledge to multi-view foul recognition, replacing fragmented specialist evaluations.","SoccerAgent's open-ended variant answers without seeing the candidate options and remains competitive, so the benchmark can measure genuine reasoning rather than option-matching behavior.","Because SoccerAgent's components are mostly open-source and interchangeable, replacing one vision tool with a stronger model raises overall accuracy, indicating the framework can absorb future advances without redesign.","The paper's failure analysis attributes most errors to specific tools such as match search, action classification, and entity recognition, making bottlenecks explicit and giving future work a clear target."],"supporting_citations":[{"why":"Supplies broadcast video, camera status, replay, and action annotations used for several SoccerBench tasks.","marker":"[7]"},{"why":"Provides dense video captions and commentary annotations for the commentary-related QA tasks.","marker":"[38]"},{"why":"Supplies action and commentary annotations, match records, and the two off-the-shelf soccer tools used inside SoccerAgent.","marker":"[43]"},{"why":"Provides event labels, game files, and commentary material for text and video QA tasks.","marker":"[44]"},{"why":"Supplies player images and jersey-number ground truth for the jersey number recognition task.","marker":"[5]"},{"why":"Supplies multi-view foul video labels for the foul recognition task.","marker":"[22]"},{"why":"A large language model that generates open-ended QA pairs, creates distractors, and serves as the agent core for planning and execution.","marker":"[32]"},{"why":"A vision-language model that powers the image and video understanding tools, including camera detection, jersey number recognition, and replay grounding.","marker":"[3]"}],"fun_headline_variants":["SoccerAgent: 10K-question benchmark tops single models","Multi-agent soccer AI beats single models on 10K bench","Holistic soccer understanding: new agent, 13-task benchmark","Agent + knowledge base: soccer QA across text, image, video","SoccerBench: 10K questions, 13 tasks, agent system wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the roughly 10,000 multiple-choice answers in SoccerBench are correct, unambiguous, and paired with three genuinely plausible wrong answers, even though the filtering that produced them was manual and no inter-annotator agreement or external validation is reported.","fun_headline_variants_meta":{"raw":{"variants":["SoccerAgent: 10K-question benchmark tops single models","Multi-agent soccer AI beats single models on 10K bench","Holistic soccer understanding: new agent, 13-task benchmark","Agent + knowledge base: soccer QA across text, image, video","SoccerBench: 10K questions, 13 tasks, agent system wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1621,"prompt_tokens":889,"completion_tokens":732,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":640}},"tokens_in":505,"tokens_out":732,"duration_ms":6224,"temperature":1.0,"reasoning_tokens":640,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:43:49.315500+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 200 to 300 SoccerBench items, have independent soccer-knowledgeable annotators answer them without seeing the published options' origin, and compare their answers and confidence to the dataset's labels. If a material fraction of items has a wrong key or a distractor that is obviously false to an informed reader, then the benchmark rankings and SoccerAgent's margin rest on shaky ground. A complementary check is to feed the image and video questions to a strong text-only model; any substantial accuracy above chance would mean the visual material is not load-bearing for those items.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides dense video captions and commentary annotations for the commentary-related QA tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies multi-view foul video labels for the foul recognition task."}],"review_version":1}