{"id":"d4da2eef-90a7-4d89-aa2a-7e21632fb327","arxiv_id":"2607.09438","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":5.5,"correctness_risk":"low","formal_verification":"none","parameter_count":6,"one_line_summary":"On EXAMS-V, token budget and parseability dominate chain count, PRM-guided search, and selectors for small VLMs; the policy model itself yields the largest gain.","lead":"Small open vision-language models improve more from longer per-chain token budgets and parseable answer formats than from PRM search or post-hoc selectors on multilingual visual exams. The result shows how to extract large accuracy gains under a single-GPU limit without elaborate machinery.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified","rationale":"The reader correctly isolates the Qwen-family restriction as the weakest assumption while judging the internal evidence strong enough for ACCEPT. That assessment holds: the paper’s own Limitations section already flags the same point, the ablations are run at full validation scale (n=4 651) with proper significance testing, code is released, and the first-place ImageCLEF result is consistent with the validation diagnosis. No stronger load-bearing flaw (e.g., uncontrolled parse failures after the budget fix, selector ceiling effects misinterpreted, or PRM saturation left undiagnosed) appears on close reading. Therefore the verdict and confidence remain unchanged; the suggested concrete_test is simply the natural next verification that would further stress the already-owned generalization caveat.","tokens_in":16402,"tokens_out":525,"duration_ms":20242,"concrete_test":"Re-run the exact SC-N=8 (1k vs 2k) and SC-N=16 (2k) configurations of Table 3 on one non-Qwen ≤7B open VLM (e.g., InternVL2-4B or LLaVA-OneVision-7B) under identical max_pixels, vLLM settings, and MMMU closer; if the token-budget delta remains ≥3 pp while the chain-count delta stays <0.5 pp, the dominance claim generalizes beyond the Qwen family.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that parseability and per-chain token budget dominate search strategy, chain count, and post-hoc selectors for small open VLMs on EXAMS-V—is tightly supported by the full-validation ablations (Tables 2–4, Figure 1). The 1k→2k budget recovers +3.7 pp while N=8→16 adds only +0.15 pp; PRM-BAS trails SC by 0.39 pp at ~8× cost; both the generative critic and Qwen-VL-PRM fail to beat majority vote on two pools of different strength. The Qwen-only policy scope (explicitly owned in §7) is a real external-validity caveat but does not undermine the internal ranking of factors or the reported McNemar-tested deltas. No hidden circularity, extraction artifact left uncontrolled, or statistical overclaim is present once the MMMU closer and guided repair are applied.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies whether test-time scaling (TTS) techniques that help large language models transfer to small open vision–language models under a single-GPU, ≤7B constraint on the multilingual visual MCQ benchmark EXAMS-V (ImageCLEF 2026). Across Qwen2.5-VL-7B-Instruct and Qwen3.5-4B, the authors compare flat self-consistency, describe-then-reason with PRM-guided beam annealing (PRM-BAS), and two post-hoc selectors (a training-free generative critic and Qwen-VL-PRM-7B). The central empirical claim is that the conditions under which TTS runs—especially answer parseability and per-chain token budget—dominate search strategy, chain count, and verification machinery: fixing the answer cue and raising max_new_tokens from 1k to 2k recovers +3.7 pp, while N=8→16 adds only +0.15 pp; once chains finish, PRM-BAS trails majority-vote SC by 0.39 pp at ~8× cost, and neither selector beats majority vote on two pools of different strength. The largest single gain is the policy switch itself (+11.4 pp). The best configuration (Qwen3.5-4B, SC-N=16, 2k tokens, guided parse repair) reaches 81.6% on full validation and 84.1% on the held-out test split, ranking first on the Visual MCQ leaderboard.","tokens_in":16663,"tokens_out":1495,"duration_ms":28007,"significance":"If the result holds, it is a practically important corrective for TTS work on small multimodal models: under realistic compute envelopes, engineering substrate (prompt format that forces answer commitment, sufficient decoding budget) and policy quality matter more than elaborate search or trained verifiers. The paper supports this with full-validation ablations (n=4,651), Wilson CIs, McNemar tests with Bonferroni correction, agreement-tier and per-language/subject stratification, and a held-out leaderboard ranking. Code is released. The transparent diagnosis of parse-fail as truncation-plus-extraction rather than reasoning failure, and the controlled comparison of selectors on both weak and strong pools, are particularly useful for the community. The Qwen-only scope is a real external-validity limit, but it is stated clearly in §7 and does not erase the internal ranking of factors on the models and task studied.","major_comments":[{"comment":"Abstract, Introduction, and Conclusion frame the contribution as applying to “small open vision-language models” in general, while §7 correctly states that only two Qwen-family policies were tested and that parseability in particular may be family-specific. The internal ranking (token budget ≫ chain count; SC ≳ PRM-BAS; majority ≳ selectors) is well-supported for these policies and is the paper’s load-bearing claim. Please tighten the abstract/conclusion wording so the scope of generalization matches the evidence (e.g., “on two Qwen-family small VLMs” or “under the policies and budget studied”), leaving broader transfer as an explicit open question rather than an implied result. This is a framing fix, not a request for new models.","section":null},{"comment":"§5.2 / PRM-BAS val-scale comparison: the diagnosis that DTR removes image access at the reasoning stage and that PRM P(+) saturates (0.962 / 0.849) is important and well-argued. The net −0.39 pp vs SC is reported on n=4,319 with correction/regression counts. To make the “elaborate methods contribute little” claim fully load-bearing, please state more explicitly in the main text (not only diagnosis) that the comparison is against a DTR+PRM-BAS pipeline whose structural image-loss bottleneck is partly independent of the PRM, so the result is best read as “this structured pipeline does not beat flat SC on this task,” not as a universal indictment of all PRM-guided search. A one-sentence scope sentence would suffice.","section":null}],"minor_comments":[{"comment":"Table 2 and §5.1: the note that Q3.5 CoT/SC(1k) parse-fail rates (16–18%) cause the CoT-to-SC(1k) margin to conflate truncation absorption with diversity is excellent; consider elevating a short parenthetical or footnote so readers do not over-interpret the 1k SC number as pure sampling gain.","section":null},{"comment":"Figure 1a: the dual arrows for +3.7 pp (budget) vs +0.15 pp (chain count) are clear; adding the absolute call counts or a brief legend note that DTR/PRM-BAS points are Q2.5/dev-200 would reduce any risk of cross-policy misreading.","section":null},{"comment":"Table 4: the Q2.5 pool is pre-MMMU closer (anchor 65.60% vs post-closer 66.42% in Table 2). The caption already notes this; a matching one-line reminder in the main §5.6 text would help readers comparing across tables.","section":null},{"comment":"§3 / Guided parse repair: the method is clean and eliminates all-unparseable questions by construction. Briefly note whether repaired single-token answers are included in the majority with equal weight to free-form chains (they appear to be) so that replication is unambiguous.","section":null},{"comment":"Appendix D / Table 8: several cells show SC regressions vs CoT or ZS under Q2.5 (e.g., Islamic Studies, Agriculture). A short pointer in §5.7 that majority vote can amplify correlated errors in text-heavy subjects would strengthen the stratified discussion without new experiments.","section":null},{"comment":"Title line “Nika at ImageCLEF 2026…” is fine for working notes but may confuse journal readers; if the venue expects a standalone title, consider moving the team handle to the author block or a footnote.","section":null},{"comment":"Minor consistency: abstract reports 84.1% test; §5.8 and Table 12 report 84.06%. Prefer a single rounded form throughout.","section":null}],"recommendation":"minor_revision","confidential_remarks":"This is strong, carefully executed empirical work with honest limitations and a clear practical message. The reader’s ACCEPT and the skeptic’s null objection are well-founded on the internal evidence. I recommend minor_revision only to align abstract/conclusion scope language with §7 and to add two short clarifying sentences on the PRM-BAS comparison; either could be handled quickly. Fit is good for a methods/empirical multimodal reasoning venue; the ImageCLEF working-notes origin does not reduce the technical quality of the ablations."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is practical and well-supported: under a single-GPU, ≤7B open-VLM constraint on EXAMS-V, what matters is whether chains finish and emit a letter, not fancy search or verification. Raising max tokens from 1k to 2k recovers 3.7 pp; going from N=8 to 16 adds 0.15 pp. Once that is fixed, PRM-BAS trails plain self-consistency by 0.39 pp at ~8× cost, and both a generative critic and Qwen-VL-PRM fail to beat majority vote on two pool strengths. Policy switch alone is +11.4 pp. They land first on the ImageCLEF 2026 Visual MCQ test set at 84.1%.\n\nWhat is new is the clean ranking of factors, not a new algorithm. They run the full validation set (n=4,651) with Wilson CIs and McNemar+Bonferroni, report parse-fail rates and agreement tiers, own the early prompt artifact that left correct reasoning uncommitted, and ship guided repair plus code. The two-axis scaling plot and the selector nulls on both Q2.5 and Q3.5 pools are the useful parts. Stratified language/subject cuts show where SC helps (mid-accuracy languages, STEM) and where correlated errors or missing knowledge make it useless.\n\nSoft spots are real but owned. Everything is Qwen-family; they say so in Limitations and cannot claim the ranking generalizes to other small VLMs. Dev-200 ablations have wide CIs, so they re-ran the load-bearing contrasts at full val. PRM P(+) saturates and answer diversity collapses under beam search—that is a diagnosis, not a flaw in the experiment. No circularity; labels are external. Citations are appropriate.\n\nThis is for people doing efficient multimodal reasoning or shared-task baselines who need to know where to spend the token budget. It deserves a serious referee. I would cite the scaling and selector results when discussing TTS for small VLMs, and I would bring the tables to reading group.","headline":"Solid shared-task working notes: for small open VLMs on multilingual visual MCQ, parseability and per-chain token budget dominate PRM search, chain count, and selectors; first-place 84.1% with code.","tokens_in":17309,"tokens_out":549,"would_cite":true,"duration_ms":6680,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"For small open vision-language models on multilingual visual MCQ, the conditions that let reasoning chains finish and commit an answer matter more than elaborate search or verifiers.","keywords":["test-time scaling","vision-language models","multimodal reasoning","self-consistency","process reward models","multilingual visual MCQ","parseability","token budget"],"falsifier":"Re-run the matched self-consistency versus PRM-BAS and selector comparisons on a third small open VLM family at a fixed 2k-token budget on the full validation set, and check whether PRM search or a critic then beats majority vote by a clear, significant margin.","tokens_in":17279,"feed_emoji":"🖼️","tokens_out":1127,"duration_ms":25850,"temperature":0.7,"pith_summary":"This paper tests whether test-time scaling techniques that help large language models also help small open vision-language models on hard multilingual visual multiple-choice exams. Under a single-GPU and at-most-7B constraint on EXAMS-V, the authors compare plain self-consistency against describe-then-reason with process-reward-model beam search and against two post-hoc answer selectors. They find that parseability and per-chain token budget dominate: an answer cue plus guided repair largely removes chains that reason correctly but never emit a letter, and raising the limit from 1k to 2k tokens recovers 3.7 percentage points while doubling chains from 8 to 16 adds only 0.15. Once chains can finish, PRM-guided beam search trails majority-vote self-consistency by 0.39 points at over eight times the cost, and neither a generative critic nor a trained multimodal PRM beats majority vote. A sympathetic reader should care because, under tight budgets, the practical priority is finishing and extracting answers, not building more sophisticated search or verification machinery; the largest single gain still comes from switching the policy model itself.","feed_headline":"More tokens beat bigger search trees for small VLMs","feed_subtitle":"On multilingual visual exams, 2k tokens recover 3.7 points; PRM beam search trails majority vote at 8× cost.","key_machinery":"Self-consistency with a large decoding budget and guided parse repair: sample parallel chains under a standard answer cue, extract the letter (or force one token under a guided choice when every chain fails to commit), then take majority vote with log-probability tie-breaking. This simple object is what PRM-guided beam search and post-hoc selectors fail to beat once chains are allowed to finish.","core_discovery":"What matters for test-time scaling of small open VLMs is the conditions under which sampling runs—especially answer parseability and a large enough per-chain token budget—not the search or verification machinery. Raising the token limit from 1k to 2k recovers 3.7 pp while going from 8 to 16 chains adds 0.15 pp; PRM-guided beam search trails plain self-consistency by 0.39 pp at over eight times the cost; neither a training-free generative critic nor a trained multimodal PRM beats majority vote across both policies; and switching the policy yields the largest gain (+11.4 pp). The best setup reaches 84.1% on the held-out ImageCLEF 2026 test split.","pith_inferences":["The same finish-and-parse priority may apply to other small multimodal models and answer formats beyond letter MCQ, not only the two Qwen policies studied here.","If process reward models stay poorly calibrated on multilingual non-math content, future work may need either broader PRM training data or to abandon per-step search for flat parallel sampling.","Decoupling perception from reasoning, as the authors flag, is a natural next place to re-test whether structured search becomes useful once truncation and parse failures are removed.","Compute-optimal TTS recipes derived on large text models need re-derivation when the bottleneck is chain truncation rather than verifier quality."],"forward_implications":["Under single-GPU and ≤7B constraints, allocate tokens first to longer completions and reliable answer formatting before investing in process reward models or beam search.","TTS studies that do not control parse-failure rates risk attributing extraction failures to reasoning or scaling method failures.","Parallel sampling helps most where chains err independently (mid-accuracy languages, STEM); it cannot fix high-agreement shared mistakes or missing domain knowledge.","Policy-model quality remains the largest lever; once the substrate is fixed, elaborate inference-time machinery adds little.","The reported best configuration (newer 4B policy, N=16, 2k tokens, guided repair) ranks first on the Visual MCQ leaderboard at 84.1% on the held-out test split."],"fun_headline_variants":["Token budget beats search trees for small VLMs on visual MCQ","Parseability and 2k tokens drive TTS gains in small VLMs","Self-consistency tops PRM beams; token limit recovers 3.7pp","TTS for small VLMs: conditions matter more than machinery","2k tokens add 3.7pp; more chains add 0.15pp on EXAMS-V"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The ranking of token budget and parseability over PRM search and selectors is assumed to hold beyond the two Qwen-family models tested; other small open VLMs may emit answers or respond to search differently.","fun_headline_variants_meta":{"raw":{"variants":["Token budget beats search trees for small VLMs on visual MCQ","Parseability and 2k tokens drive TTS gains in small VLMs","Self-consistency tops PRM beams; token limit recovers 3.7pp","TTS for small VLMs: conditions matter more than machinery","2k tokens add 3.7pp; more chains add 0.15pp on EXAMS-V"]},"model":"grok-4.5","effort":"low","cost_usd":0.005552,"raw_usage":{"total_tokens":1598,"prompt_tokens":913,"num_sources_used":0,"completion_tokens":86,"cost_in_usd_ticks":55520000,"prompt_tokens_details":{"text_tokens":913,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":599,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":913,"tokens_out":86,"duration_ms":5914,"temperature":1.0,"reasoning_tokens":599,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T02:59:13.536465+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the matched self-consistency versus PRM-BAS and selector comparisons on a third small open VLM family at a fixed 2k-token budget on the full validation set, and check whether PRM search or a critic then beats majority vote by a clear, significant margin.","supporting_citations":[],"review_version":1}