{"id":"e3017c8c-4768-4035-9489-2a2f8ba39c75","arxiv_id":"2510.27568","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SIGMA uses four specialized retrieval agents plus a moderator to improve math reasoning, reporting up to 7.4 absolute MATH500 points over Search-o1 at 1.5B scale.","lead":"SIGMA is a multi-agent framework that splits math questions among four specialist LLM agents, each able to search for facts on demand and report back to a moderator. On MATH500 the 1.5B version gains 7.4 points over Search-o1, but the paper's broader 'consistent outperformance' and efficiency claims are not supported by its own numbers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 contradicts the abstract's 'consistently outperforms' claim: SIGMA loses to Search-o1 on AMC23 and GPQA at 1.5B, and to direct reasoning on AMC23 at 3B.","rationale":"The reader's weakest assumption focused on unverified comparability of baselines. That is real, but the more decisive issue is internal: the paper's own Table 1 contradicts the abstract's 'consistently outperforms' claim on multiple cells. This is not a matter of missing protocol details or cherry-picking; it is visible in the reported numbers. The efficiency claim is additionally disowned by the text itself ('detailed analysis of latency and token usage remains future work'), and the ablation exclusion removes the ability to attribute gains to the multi-agent mechanism. These are load-bearing gaps for the central claim. I still credit the paper for a coherent framework, a concrete search-budget example, and reproducible benchmark choices, and the MATH500 gains are plausible, but the abstract overstates the evidence. The verdict should remain rejection, as the reader concluded; my concern is a different (stronger) route to the same outcome. The proposed sign-matrix check is a minimal computational test that settles the consistency question without needing access to code or reruns.","tokens_in":9569,"tokens_out":7357,"duration_ms":59761,"concrete_test":"Parse Table 1 and compute, for every benchmark column, whether SIGMA's score at each released scale is greater than or equal to every comparable baseline entry in that column. Require zero losses to sustain 'consistently outperforms.' This check, applied to the published table, already returns losses at AMC23 (1.5B and 3B) and GPQA (1.5B), so the central claim fails without additional experiments.","verdict_should_be":"REJECT","load_bearing_attack":"The abstract's central claim—'SIGMA consistently outperforms both open- and closed-source systems' with a 7.4% absolute improvement—cannot survive the paper's own Table 1. At Qwen2.5-1.5B, SIGMA scores 20.0 vs Search-o1's 25.0 on AMC23 and 19.19 vs 20.2 on GPQA overall; at Qwen2.5-3B, SIGMA scores 30.0 on AMC23 versus 50.0 for direct Qwen2.5-3B and ties Search-o1 on AIME24. Thus the reported advantage is not consistent across benchmarks, scales, or baselines. The companion efficiency claim is also explicitly unmeasured: the 'Inference Cost and Efficiency' section says 'detailed analysis of latency and token usage remains future work,' so the abstract's 'significantly enhances both reasoning accuracy and efficiency' is unsupported. Finally, the paper states that single-agent ablations 'were excluded' because they would be misleading; without ablations, the attribution to multi-agent on-demand knowledge integration rather than to prompt engineering or retrieval itself is not established. The weaker claim that SIGMA improves MATH500 at several scales may be true, but the manuscript as submitted does not support its headline assertions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SIGMA, a multi-agent retrieval-augmented reasoning framework for mathematical and scientific question answering. SIGMA orchestrates four specialized agents (FACTUAL, LOGICAL, COMPUTATIONAL, COMPLETENESS) that each perform reasoning-search cycles using HyDE-based retrieval, followed by a deterministic moderator that synthesizes their outputs into a final answer. The authors evaluate SIGMA on MATH500, AMC23, AIME24, and GPQA using Qwen2.5 backbones at 1.5B, 3B, and 7B scales, comparing against direct reasoning, Search-o1, and several larger closed-source models. The paper claims that SIGMA consistently outperforms open- and closed-source systems with a 7.4% absolute improvement and that it significantly enhances both reasoning accuracy and efficiency.","tokens_in":9944,"tokens_out":3197,"duration_ms":29918,"significance":"If the claims were substantiated, SIGMA would represent a practical recipe for boosting small open-weight models on knowledge-intensive math and science benchmarks through multi-agent orchestration and on-demand retrieval, without retraining. The core idea of combining perspective-specific agents with a lightweight heuristic moderator is interesting and the choice of relatively weak backbones makes the comparison relevant. However, the central claims are undermined by the paper's own results: Table 1 shows several benchmarks where SIGMA is below baselines, the efficiency claim is explicitly deferred to future work, and the absence of ablations prevents attribution of the observed gains to the proposed mechanism. The paper does provide a clean framework description and a clear evaluation protocol, but the evidence as reported is insufficient for the headline contributions.","major_comments":[{"comment":"The abstract's claim that SIGMA 'consistently outperforms both open- and closed-source systems' is contradicted by Table 1: SIGMA 1.5B is below Search-o1 1.5B on AMC23 (20.0 vs 25.0) and GPQA overall (19.19 vs 20.2); SIGMA 3B is below direct Qwen2.5-3B on AMC23 (30.0 vs 50.0). No variance, seeds, or significance tests are reported, so even the MATH500 gains could be within noise. The 'absolute improvement of 7.4%' is a single selected comparison (SIGMA 1.5B vs Search-o1 1.5B on MATH500) and does not generalize.","section":"Abstract / Table 1"},{"comment":"The abstract and introduction claim SIGMA 'significantly enhances both reasoning accuracy and efficiency,' but the section 'Inference Cost and Efficiency' states that 'a detailed analysis of latency and token usage remains future work.' No token counts, wall-clock times, search counts, or cost measurements are provided anywhere. Since SIGMA runs up to four reasoning trajectories plus a synthesis step, the computational overhead relative to direct reasoning is nontrivial and unquantified. This efficiency claim must be removed or substantiated with measurements.","section":"Inference Cost and Efficiency"},{"comment":"The paper attributes the observed gains to multi-agent orchestration and on-demand knowledge integration, but provides no ablations isolating these components. The statement that single-agent ablations were excluded because 'removing a single agent changes how the entire system functions' and would yield 'misleading results' is not a valid substitute for experimental evidence. Without comparisons to a single-agent system using the same search budget and HyDE retrieval, the gains could be due to prompt templates, search strategy, or retrieval implementation rather than the multi-agent design.","section":"Methodology / Experiments"},{"comment":"The setup asserts that all baselines were evaluated 'under identical search budgets, retrieval protocols, and decoding parameters (temperature, seed, pass@k),' but no concrete values are reported: per-agent search budget, retrieval corpus, embedding model, number of runs, or decoding seeds. Also, only GPT-4o is marked as taken from official releases; the provenance of other baseline numbers is unclear. Without this information the comparability premise cannot be verified, and the claimed consistent outperformance becomes unsubstantiated. Please report all protocol parameters and release configuration files.","section":"Experimental Setup"}],"minor_comments":[{"comment":"AMC2023 and AIME2024 are described as 'middle school–level competitions'; AIME is an advanced high-school contest, not middle school. Please correct the characterization.","section":"Experimental Setup"},{"comment":"The caption promises 'a qualitative analysis of how the agents interact during reasoning,' but the table only contains quantitative results. Either add the qualitative analysis or remove that clause.","section":"Table 1 caption"},{"comment":"There are several spacing and formatting issues, e.g., 'we introduceSIGMA' should be 'we introduce SIGMA' and similar missing spaces throughout. A careful proofread is needed.","section":"Introduction / Methodology"},{"comment":"The methodology section references 'Appendices A and B' for search token instructions and prompts, but the submitted manuscript does not include these appendices. Please include them or remove the references.","section":"Appendices"},{"comment":"The Results section reports gains such as '5.0% and 3.3% improvements' without specifying the exact comparison baselines and scales. Please make each reported improvement concrete and consistent with Table 1.","section":"Results"},{"comment":"Figure 1 places all closed-source models at '>100B' on the x-axis, but exact sizes are known for some (e.g., GPT-4o). Clarify whether these are actual parameter counts or illustrative positions, and label the axes accordingly.","section":"Figure 1"}],"recommendation":"reject","confidential_remarks":"The paper's headline claims are not supported by its own evidence: the 'consistently outperforms' statement fails on multiple rows of Table 1, the efficiency claim is explicitly deferred, and the design attribution is untestable without ablations. These are load-bearing issues that would require extensive rework and new experiments to address. The framework idea has some merit, but as submitted the manuscript does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: the one genuinely new piece is the four-perspective decomposition (factual/logical/computational/completeness) plus a cheap deterministic moderator; everything else is a competent recombination of Search-o1-style triggered search, HyDE, and multi-agent prompting. And the MATH500 gains look real: SIGMA beats Search-o1 at 1.5B, 3B, and 7B by 7.4, 2.8, and 3.6 points respectively. That is worth something for small-model engineering.\n\nThe soft spot is load-bearing: the abstract says 'consistently outperforms both open- and closed-source systems' and claims efficiency gains. Table 1 does not support it. At 1.5B, SIGMA loses to Search-o1 on AMC23 (20 vs 25) and GPQA overall (19.19 vs 20.2). At 3B, it loses to direct Qwen2.5-3B on AMC23 (30 vs 50). The 7.4% is precisely the MATH500 gain at 1.5B, not an absolute improvement across benchmarks. And the efficiency sentence is followed in the paper itself by 'detailed analysis of latency and token usage remains future work.' So the central claims are overclaims.\n\nThe evidence is also thinner than it should be: no error bars or seeds, no protocol details, code withheld. The decision to exclude ablations because removing an agent would be 'misleading' is a dodge; that is what ablations exist for. Without them you cannot attribute the gains to the multi-agent integration rather than to extra retrieval calls or prompt engineering.\n\nWhat is good: the system description is clear, the equations are correctly used (though mostly decorative), the citations to Search-o1, HyDE, and the rest are fair, and there is no fitted-parameter circularity. The framework is salvageable as a modest empirical contribution if the claims are corrected and the code and protocol are released.\n\nThis paper deserves a serious referee because the framework is testable and the MATH500 gains could be reproducible. My own take: skeptical until we see code and variance, but not reject-because-useless. For a reading group it is a useful case study in abstract-vs-table discipline.","headline":"A coherent multi-agent retrieval recipe with real MATH500 gains at small scale, but the abstract's 'consistently outperforms' and efficiency claims are contradicted by its own Table 1.","tokens_in":10424,"tokens_out":3329,"would_cite":false,"duration_ms":28202,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a framework of four specialized agents that search only when uncertain and merge via a fixed moderator lets small open models beat far larger closed-source models on mathematical and PhD-level science reasoning.","keywords":["multi-agent reasoning","retrieval-augmented generation","mathematical reasoning","hypothetical document enhancement","on-demand search","agentic search","knowledge integration","large language models"],"falsifier":"Re-run SIGMA and each baseline under a published fixed protocol (same search budget, temperature, seed, pass@k, retrieval index) and check whether the 7.4% gap persists; separately measure total tokens and latency per problem and check whether SIGMA is actually cheaper than a single-agent ReAct loop.","tokens_in":9415,"feed_emoji":"🔍","tokens_out":5467,"duration_ms":43521,"temperature":0.7,"pith_summary":"SIGMA is a framework for doing math and science reasoning with a small language model. It runs four agents — FACTUAL, LOGICAL, COMPUTATIONAL, and COMPLETENESS — that each think aloud and only trigger a web search when they hit uncertainty. Before searching, each agent writes a hypothetical passage describing what the ideal answer chunk would look like, then uses that passage to rank retrieved text. A deterministic moderator collates and prioritizes the agents' conclusions. The paper's central claim is that this multi-perspective, on-demand retrieval gives models of 1.5B to 7B parameters consistent accuracy gains, including outperforming several much larger closed-source systems on MATH500, AIME, and GPQA, with modest compute overhead.","feed_headline":"Four agents with on-demand search lift small models past GPT-4o","feed_subtitle":"A 7.4% absolute boost on math and science QA, letting 1.5B-7B open models beat far bigger closed ones.","key_machinery":"The load-bearing object is the four-agent-plus-moderator loop. Each agent is a distinct instruction prompt running in the same model; it walks a reasoning trajectory, emits a special token to request a search only when uncertain, writes a hypothetical ideal passage to rank candidate retrieved chunks by cosine similarity, and updates its internal state. The moderator then merges the terminal states with a fixed prioritization scheme, such as giving verified computational results higher weight than speculative logical hypotheses. This design makes retrieval perspective-specific and on-demand, and the moderator's determinism keeps the synthesis reproducible.","core_discovery":"SIGMA claims that a unified model with four specialized agents — FACTUAL, LOGICAL, COMPUTATIONAL, COMPLETENESS — can solve knowledge-intensive math problems better than any single-strategy retrieval or chain-of-thought baseline. Each agent maintains its own state, chooses between reasoning internally, searching, or synthesizing, and generates a hypothetical document (a fake ideal search result) to steer retrieval toward its analytic needs. The moderator is deliberately simple and non-learnable: it deduplicates propositions and applies a fixed priority order (verified computational results over speculative logical ones). The authors argue this design yields broad reasoning coverage without th","pith_inferences":["If the efficiency claim holds, this pattern could make high-accuracy reasoning feasible on-device or in latency-sensitive settings where big closed models are impractical.","The moderator's fixed priority order (computational over logical) may be a double-edged sword: it prevents conflicts but could discard correct logical insights when the computational agent errs; a learnable or probabilistic moderator might extend gains.","A direct comparison of HyDE-guided queries against naive keyword queries with the same search budget would test whether hypothetical-document retrieval is what drives the gains rather than the multi-agent decomposition alone.","The authors avoided a clean agent-removal ablation because removing an agent changes the whole system; a fairer test would compare SIGMA against the same budget of independent single-agent searches to isolate the value of cross-agent coupling."],"forward_implications":["Small open models (1.5B–7B) can reach or exceed the MATH500/AIME/GPQA scores of much larger closed-source models when equipped with multi-agent on-demand search.","Perspective-specific hypothetical-document retrieval yields bigger gains than generic retrieval, because each agent retrieves what it needs for its own reasoning step.","Multi-agent orchestration in a shared model costs little extra latency and tokens, because searches are triggered only under uncertainty and communication is implicit.","SIGMA generalizes beyond math to PhD-level physics, chemistry, and biology QA.","The framework is model-agnostic, so improvements should carry over to newer base models."],"fun_headline_variants":["Four-agent search framework lifts math scores 7.4%","On-demand search and multi-agent reasoning beat GPT-4o","SIGMA's agentic search improves math reasoning by 7.4%","Small models defeat GPT-4o with agentic search integration","Search-agents on demand: +7.4% on math and science QA"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The baselines were all run under identical search budgets, retrieval protocols, and decoding settings, and the efficiency advantage is real despite the paper's own statement that detailed latency and token analysis is future work.","fun_headline_variants_meta":{"raw":{"variants":["Four-agent search framework lifts math scores 7.4%","On-demand search and multi-agent reasoning beat GPT-4o","SIGMA's agentic search improves math reasoning by 7.4%","Small models defeat GPT-4o with agentic search integration","Search-agents on demand: +7.4% on math and science QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1306,"prompt_tokens":706,"completion_tokens":600,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":520}},"tokens_in":450,"tokens_out":600,"duration_ms":6006,"temperature":1.0,"reasoning_tokens":520,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:54:14.199311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run SIGMA and each baseline under a published fixed protocol (same search budget, temperature, seed, pass@k, retrieval index) and check whether the 7.4% gap persists; separately measure total tokens and latency per problem and check whether SIGMA is actually cheaper than a single-agent ReAct loop.","supporting_citations":[],"review_version":1}