{"id":"4ab3aec6-3839-48f0-9c2c-18ea6a2da20c","arxiv_id":"2607.28595","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Beacon improves agentic visual reasoning by teaching models when tools are necessary and how to use them for net gains, via necessity-aware rewards and hint-guided RL.","lead":"Beacon is an agentic vision-language model trained to call visual tools only when needed and to gain real accuracy from those calls. It introduces metrics and RL rewards that fix a common failure: tools help on hard cases but hurt on easy ones, canceling out.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"HCE may inflate TE via teacher-distilled trajectories rather than pure policy discovery of tool skill.","rationale":"The reader correctly flags online group labels and HCE transfer as the weakest assumption for both MA and TE. I sharpen the same point: the load-bearing risk is not merely over-calling tools or label noise from 8 rollouts, but that measured Tool-Gain may be teacher-distilled rather than discovered. NAAR (§4.4.1) is a reasonable soft preference and the five-rollout MA/TE protocol is more robust than single-shot priors; overall open-source leadership on 13 benchmarks is real under the reported setup. No internal contradiction appears. The verdict stays CONDITIONAL—release code/data and a teacher-free or self-hint HCE control—rather than REJECT, because the diagnosis (gain≈harm, poor MA) and metrics are still useful even if part of Beacon’s TE is distillation. Confidence remains moderate for the same reasons the reader gave (Gemini-in-the-loop, author-aligned metrics, tool-skewed MA_text).","tokens_in":29612,"tokens_out":624,"duration_ms":11916,"concrete_test":"Train an HCE-ablated twin that replaces Gemini hints with either (i) no re-rollout or (ii) self-generated hints from the SFT policy only (no Gemini). Recompute Table 3 Tool-Gain, Tool-Harm, ΔTE, and the 13-benchmark average. If ΔTE falls below ~+1.5% or average accuracy drops >1.5 pts toward Metis, the TE claim depends on teacher strategies rather than policy-side capability expansion.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that Beacon shows genuine tool-induced capability extension (ΔTE ≈ +3.14% vs ~0 for priors) rests on Hint-Guided Capability Expansion (§4.4.2): all-wrong groups are re-rolled under Gemini-3.1-Pro answer-free hints, then optimized with the hint stripped from the new-policy importance ratio (Eq. 6) while kept in the old-policy context. This is intended to expand the hint-free policy. But the paper never isolates how much of Tool-Gain and overall gains come from (a) the policy internalizing reusable tool strategies versus (b) offline imitation of teacher-chosen crops/rotations/pixel ops that the 8B model would not discover under standard GRPO. Ablation Table 4 shows HCE helps ΔTE (+2.96 vs GRPO+NAAR +2.54), yet every hard trajectory still originates from the same proprietary teacher used for SFT synthesis (§4.2). If TE gains are mostly teacher distillation under a GRPO wrapper, the claim of RL-driven genuine extension beyond text-only reasoning is overstated, and MA/TE improvements are less attributable to NAAR+HCE as general mechanisms.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"This paper argues that agentic visual reasoning should be judged by Mode Adaptiveness (MA)—whether an MLLM invokes tools only when text-only reasoning is unreliable—and Tool Effect (TE)—whether tools raise success on text-hard items without harming text-easy ones. After multi-run analysis of Thyme, CodeV, DeepEyes-V2, and Metis showing weak MA and near-zero net TE, the authors train Beacon (Qwen3-VL-8B) via SFT on Gemini-synthesized code trajectories plus GRPO with Necessity-Aware Adaptive Reward (NAAR; Eq. 1) and Hint-Guided Capability Expansion (HCE; §4.4.2, Eqs. 5–6). On 13 benchmarks Beacon leads open-source averages and reports higher MA_mean and ΔTE≈+3.14% versus near-zero for priors (Table 3, Fig. 2).","tokens_in":29948,"tokens_out":1768,"duration_ms":32132,"significance":"If the MA/TE framing and Beacon results hold, the work usefully reframes agentic VLM evaluation away from aggregate accuracy and indiscriminate tool calling toward necessity-aware, net-positive tool use. Strengths include a clearer multi-rollout solvability protocol than single-shot labels, explicit gain/harm decomposition, ablations of NAAR and HCE (Tables 4, 6), training-dynamics evidence of mode-alignment rather than pure mode collapse (Figs. 5–7), and broad benchmark coverage. The NAAR design also addresses concrete flaws the authors identify in CodeDance, AdaTooler-V, and Metis. The contribution is primarily empirical/methodological rather than theoretical, but it is timely for the agentic multimodal literature.","major_comments":[{"comment":"§4.4.2 and Eqs. (5)–(6): HCE re-rolls all-wrong groups under Gemini-3.1-Pro answer-free hints, then optimizes with the hint stripped only from the new-policy ratio while retained in the old-policy context. Table 4 attributes part of ΔTE to HCE (+2.96 GRPO+HCE vs +1.40 GRPO; full model +3.14), yet every hard trajectory lineage still originates from the same proprietary teacher used for SFT (§4.2). The manuscript does not isolate how much Tool-Gain is reusable policy skill versus distillation of teacher-chosen crops/rotations/pixel ops that standard GRPO would not discover. A load-bearing control is needed: e.g., HCE with a weaker/open teacher, frozen-hint imitation without GRPO, or TE measured only on items never seen by the teacher. Without this, the claim of “genuine tool-induced capability extension” via RL (§1, §5.3 Obs. 3, conclusion) is only partially supported.","section":"§4.4.2, Eqs. (5)–(6), Table 4"},{"comment":"§3.1 definitions of Tool-Gain and Tool-Harm normalize by 5N over the full test set, while MA_text/MA_tool average only over E and H. Table 3 shows Beacon’s MA_text is often low (e.g., 22.91% average; 10.77% on BabyVision) with very high tool-call rates under p_tool (70–95%), so much of ΔAcc and Tool-Gain may reflect frequent tool use plus stronger base competence rather than selective necessity detection. The paper acknowledges a code preference but still markets “improved Mode Adaptiveness.” Please report MA conditioned on difficulty bins more finely (Fig. 2a-style curves for all models on the full 13-benchmark suite), tool-call rate vs. text-only accuracy calibration error, and ΔTE restricted to samples where the model actually switches mode relative to its text-only success rate—so that adaptiveness is not conflated with a high baseline tool prior.","section":"§3.1, Table 3, Fig. 2(a)"},{"comment":"§4.4.1 Eq. (1): I_text(G) labels necessity from whether any of N=8 on-policy rollouts is correct text-only. Appendix E.1 correctly notes that forced extra text rollouts biased labels toward text; the dual risk remains that small-N groups mislabel “text-sufficient” problems as code-necessary (or vice versa), and that the same group structure used for reward is reused in the MA/TE story. Sensitivity of final MA_mean, ΔTE, and overall accuracy to N, to the 0.25 partial-credit constant, and to the ≥4 / ≤1 easy/hard thresholds should be reported. Absent that, it is unclear whether NAAR’s gains are robust or tuned to these free parameters listed implicitly throughout §3–4.","section":"§4.4.1 Eq. (1), §3.1, Appendix E.1"},{"comment":"Tables 1–2 vs. Table 3: Main headline gains (+6.07 avg over Qwen3-VL-8B) mix perception-heavy sets where cropping helps with math sets where ΔAcc is small or negative for several models (MathVista Beacon ΔAcc +0.62, ΔTE −0.38). The abstract’s “genuine tool-induced performance gains” should be qualified by task family. Please break out MA/TE and tool-available vs tool-free accuracy for all 13 benchmarks (not only five in Table 3), and state clearly on which families tools extend capability versus where gains are mostly SFT/RL general improvement visible already in Tool-Free Acc (51.57% vs base ~49.75% on the five-set slice).","section":"Tables 1–3, §5.2–5.3"}],"minor_comments":[{"comment":"Figure 2(c) and related text use “DeepEyes V2” / “DeepEyes-V2” inconsistently; unify naming with the body (DeepEyesV2).","section":"Figure 2"},{"comment":"§3.2 “Issue attribution” invokes a general limitation of RLVR without citing quantitative evidence inside this paper beyond the later HCE motivation; a forward pointer to §4.4.2 and Fig. 7 would tighten the narrative.","section":"§3.2"},{"comment":"Prompt boxes and appendix refer to both <tool_response> and <observation> conventions; a short inference-time state machine diagram would help reproducibility alongside the promised code release.","section":"Appendix C.3–C.4"},{"comment":"Table 1 lists PyVision-7B with many dashes; either complete the re-evaluation under the same protocol or mark averages as non-comparable more explicitly in the caption.","section":"Table 1"},{"comment":"Typos/style: “T ool Effect” spacing artifacts in several figure captions; “amost zero” → “almost zero” (§5.3); arXiv ID/date in footer is fine for preprint but ensure camera-ready cleans “Preprint. Work in progress.” headers.","section":"§5.3, captions"}],"recommendation":"major_revision","confidential_remarks":"The work is a solid systems/empirical paper for a CV/ML venue, but novelty relative to concurrent adaptive-tool RL (Metis, CodeDance, AdaTooler-V) hinges on NAAR+HCE and the MA/TE audit. The heaviest risk for the editor is over-crediting “RL expands genuine tool capability” when SFT and HCE both depend on Gemini 3.1 Pro. I would accept after major revision if the authors add a teacher-isolation or open-teacher HCE control and full-suite MA/TE breakdowns; without those, the central TE claim remains only partially substantiated. Fit is appropriate for cs.CV / multimodal reasoning tracks."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is one of the cleaner agentic-VLM papers lately. They name a real failure mode—models either spam tools or barely use them, and tool gain roughly cancels tool harm—and they measure it with multi-rollout text-easy/hard bins instead of a single noisy pass. That alone is worth having on the shelf.\n\nWhat is new is less a new tool API than the dual frame (MA/TE) plus two RL knobs that match the diagnosis. NAAR is mode-conditioned and online: full credit for correct text when the group can solve text-only, partial credit for correct code in that case, full credit for code when text fails. That is a real fix relative to pure “use tools less” or teacher-labeled need-tools flags. HCE recycles all-wrong groups with answer-free Gemini hints, then strips the hint from the new-policy ratio so the hint-free policy is what gets updated. Ablations and training curves are coherent: alignment to adaptive labels rises without a pure mode collapse, ~40% of all-wrong groups become informative, and average open-source numbers on 13 benchmarks lead, with ΔTE around +3% versus near zero for priors.\n\nSoft spots, in proportion. MA_text is still weak—Beacon prefers code—so “adaptiveness” is more “calls tools on hard stuff” than “stays text on easy stuff.” The stress-test point on HCE is fair: hard trajectories still originate from the same proprietary teacher used in SFT, so some of Tool-Gain is distilled strategy under a GRPO wrapper, not pure on-policy discovery. They do not fully isolate that. Reward mix, easy/hard cutoffs, and filtering thresholds are free parameters; results depend on Gemini in the loop for data and judging. None of that breaks the central empirical story, but it caps how much you should treat NAAR+HCE as general mechanisms versus a strong teacher-assisted recipe.\n\nMath is standard GRPO with a sensible importance-ratio tweak; citations cover the right prior agentic and adaptive-tool lines. For people building or evaluating tool-using VLMs, this is useful. I would send it to referees; I would bring it to reading group if we are in that lane this month; I would cite the MA/TE operationalization and the NAAR design even if I re-implement without their full stack.","headline":"Solid systems paper: clearer MA/TE diagnostics plus a usable NAAR+HCE recipe that actually moves open 8B agentic VLMs, with the usual teacher-distillation caveat on how “genuine” the tool gains are.","tokens_in":30646,"tokens_out":610,"would_cite":true,"duration_ms":17039,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Agentic visual models should call tools only when text fails—and those calls should extend capability, not cancel it out.","keywords":["agentic visual reasoning","multimodal large language models","tool use","mode adaptiveness","tool effect","reinforcement learning","code execution","visual tools"],"falsifier":"Re-run the five-rollout MA/TE protocol on the same hard/easy splits: if Beacon’s average ΔTE collapses toward zero or MAmean falls to the always-tool / never-tool 50% baseline while accuracy stays flat versus strong text-only ablations, the central claim fails.","tokens_in":30464,"feed_emoji":"🔦","tokens_out":897,"duration_ms":20762,"temperature":0.7,"pith_summary":"Most agentic visual reasoners look powerful because they can crop, annotate, and run code, yet on careful measurement they barely beat plain text reasoning. Gains on hard images are largely wiped out by mistakes tools introduce on easy ones, and the models do not reliably know when tools are needed. This paper defines two yardsticks—Mode Adaptiveness (call tools when text fails, skip them when text works) and Tool Effect (net gain after subtracting tool-caused harm)—and shows prior systems sit near chance on both. Beacon is trained so that rewards favor text when any text-only rollout in the group is already correct, and so that expert answer-free hints rescue all-wrong groups so the policy can learn real tool skill. Across 13 benchmarks it leads open-source averages and posts a clearly positive tool-gain-minus-harm gap, arguing that progress should be scored by when and whether tools truly help, not by how often they are invoked.","feed_headline":"Visual agents that know when not to call tools win more","feed_subtitle":"Beacon’s rewards and hints turn tool use into net gains instead of canceled errors","key_machinery":"Necessity-Aware Adaptive Reward (NAAR): in each GRPO group, if any correct text-only answer exists, correct text gets full reward and correct code gets partial credit (0.25); if no text answer is correct, correct code gets full reward. Paired with Hint-Guided Capability Expansion, which injects expert answer-free hints only on all-wrong groups, then optimizes the trajectories under the original hint-free prompt.","core_discovery":"Existing agentic visual reasoning models have limited Mode Adaptiveness and near-zero net Tool Effect: tool gains on text-hard items are largely offset by tool harm on text-easy items. Beacon, trained with a Necessity-Aware Adaptive Reward and Hint-Guided Capability Expansion, improves overall accuracy while raising adaptiveness and producing a genuine positive gap between tool gain and tool harm.","pith_inferences":["The same MA/TE split could audit tool-using agents outside vision (code interpreters, web search) where “always tool” is equally common.","Partial credit for correct-but-unnecessary tool use may generalize as a soft efficiency prior in any multi-mode RL setting.","If hint transfer works mainly by teaching crop/rotate/count patterns, lighter self-generated curricula might eventually replace external expert hints."],"forward_implications":["Benchmarking agentic VLMs should report Mode Adaptiveness and Tool-Gain vs Tool-Harm, not only aggregate accuracy.","RL for tool use should condition rewards on whether text-only already solves the item, rather than always encouraging or always discouraging tools.","Hard all-wrong groups can be recycled via answer-free hints instead of being discarded as zero-advantage noise.","Open-source 8B agentic models can close more of the gap to strong closed models when tool use is necessity-aware rather than habitual."],"fun_headline_variants":["Beacon makes visual agents skip tools when they hurt more than help","Tool gains stop canceling out when models know when not to call","Mode-adaptive rewards turn tool use into net wins for MLLMs","Beacon lifts accuracy by cutting needless tool calls on easy cases","Hint-guided training gives real tool effect, not offset errors"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That a handful of on-policy rollouts (plus optional expert hints on total failures) correctly label when tools are necessary and transfer tool skill without making the model over-call tools or copy the teacher.","fun_headline_variants_meta":{"raw":{"variants":["Beacon makes visual agents skip tools when they hurt more than help","Tool gains stop canceling out when models know when not to call","Mode-adaptive rewards turn tool use into net wins for MLLMs","Beacon lifts accuracy by cutting needless tool calls on easy cases","Hint-guided training gives real tool effect, not offset errors"]},"model":"grok-4.5","effort":"low","cost_usd":0.004566,"raw_usage":{"total_tokens":1352,"prompt_tokens":832,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":45664000,"prompt_tokens_details":{"text_tokens":832,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":448,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":832,"tokens_out":72,"duration_ms":9637,"temperature":1.0,"reasoning_tokens":448,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T02:43:33.528163+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the five-rollout MA/TE protocol on the same hard/easy splits: if Beacon’s average ΔTE collapses toward zero or MAmean falls to the always-tool / never-tool 50% baseline while accuracy stays flat versus strong text-only ablations, the central claim fails.","supporting_citations":[],"review_version":1}