{"id":"45299fab-a81b-4632-bc81-d1564d8df77d","arxiv_id":"2412.14835","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AR-MCTS retrieves per-step multimodal knowledge during tree search and trains a process reward model, improving MLLM reasoning accuracy on three benchmarks by a few points.","lead":"This paper combines retrieval of step-by-step hints with Monte Carlo tree search and a trained process reward model to improve multimodal language models on math and exam reasoning. The authors report gains of several percentage points on MathVista, We-Math, and the Chinese GAOKAO-MM benchmark across multiple model sizes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may be inflated by cross-benchmark contamination: WE-MATH testmini is placed in the MATHVISTA knowledge base and vice versa, and Table 7 shows 4.2% 13-gram overlap for MATHVISTA-testmini retrieved by WE-MATH queries.","rationale":"The reader's weakest_assumption pinpoints exactly the same fragility: the retrieval corpus and PRM training data may not be sufficiently separated from the evaluation benchmarks. My reading sharpens this in three ways. First, the cross-inclusion is explicit in supplementary §B.4 and is not merely a hypothetical risk: each of the two main benchmarks has the other's testmini placed inside its retrieval knowledge base. Second, the paper's own contamination table (Table 7) reports a 4.2% 13-gram overlap for MATHVISTA-testmini retrieved for WE-MATH queries; while the authors describe this as 'less than 5%' and conclude 'no overlap,' a nonzero percentage of near-duplicate solved problems is exactly the kind of material that can be exploited by a retriever, and the 13-gram test is too coarse to catch paraphrase or image-based reuse. Third, the magnitude of the reported improvements (roughly 2-5 points over Self-Consistency on MATHVISTA and WE-MATH S3/S2) is comparable to the observed contamination rate, so even a small amount of leaked solved examples could account for a substantial share of the gains. I do not claim the authors acted improperly; the concern is purely about whether the evaluation protocol supports the central empirical claim. Independent support exists: the GAOKAO-MM temporal split is methodologically cleaner and shows improvements, and the PRM/DPO training details are standard and reproducible in principle. But the paper's headline quantitative evidence comes from the two cross-contaminated benchmarks, so the central claim is not secure until the leakage question is settled. The concrete test — rerunning with a strictly cleaned corpus and an embedding-based deduplication — would settle whether the effect survives. Therefore I keep the reader's CONDITIONAL verdict unchanged: the paper should not be fully accepted until this cleanup is performed and reported.","tokens_in":34021,"tokens_out":3460,"duration_ms":32621,"concrete_test":"Re-run the MATHVISTA and WE-MATH evaluations with a retrieval corpus that excludes MATHVISTA-testmini, WE-MATH-testmini, and the LLaVA-OneVision pseudo-answers for the MATHVISTA test split, then additionally deduplicate the remaining corpus against both test sets using embedding similarity (e.g., cosine ≥ 0.8) instead of 13-gram overlap. Compare the AR-MCTS minus Self-Consistency delta before and after this cleanup; if the delta shrinks to near zero or changes sign, the reported gains are contamination-driven rather than reasoning improvements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AR-MCTS improves multimodal reasoning rests on the evaluation being contamination-free. Supplementary §B.4 states that 'we incorporate the testmini set of WE-MATH into the knowledge base of MATHVISTA and include the testmini set of MATHVISTA into the knowledge base of WE-MATH.' This means, for each benchmark, near-duplicate solved problems from the other benchmark's testmini can be retrieved as 'knowledge' and their answers copied. The paper's own contamination analysis (Table 7) reports a 4.2% 13-gram overlap between MATHVISTA-testmini and top-50 retrieved items for WE-MATH queries; the reported AR-MCTS gains over Self-Consistency are on the order of 2-5 points on these benchmarks. The 13-gram exact-overlap test is a weak proxy for leakage: paraphrased, translated, or visually re-rendered versions of the same problem will evade it, and the regex filtering in §4.1 is not described in enough detail to verify it removes all test-influencing material. If even a few percent of queries retrieve a solved near-duplicate, the measured improvement could reflect memorization rather than reasoning. The GAOKAO-MM temporal split (2010-2021 train, 2022-2023 test) is a cleaner design and partially supports the claim, but the two main benchmarks carry the quantitative weight of the paper, and their evaluation is not demonstrably clean.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AR-MCTS, a framework that augments Monte Carlo Tree Search (MCTS) decoding for multimodal large language models (MLLMs) with active retrieval: at each expansion step, a hybrid-modal retrieval module supplies problem-solving insights that are filtered by knowledge-concept similarity, and a process reward model (PRM) is trained from automatically generated step-level annotations via step-wise DPO and point-wise fine-tuning. The method is evaluated on MATHVISTA and WE-MATH testmini, with an additional cross-domain evaluation on GAOKAO-MM using a temporal split (2010-2021 train, 2022-2023 test), across GPT-4o, GPT-4V, LLaVA-OneVision-72B, InternVL2-8B, Qwen2-VL-7B, and LLaVA-NeXT. The authors report consistent improvements over Self-Consistency, Self-Correction, and ORM baselines, and present ablations, scaling analyses, diversity visualizations, and a contamination analysis of the retrieval corpus.","tokens_in":34373,"tokens_out":5850,"duration_ms":41546,"significance":"If the reported gains are trustworthy, the core idea is valuable: dynamically retrieving step-specific knowledge during MCTS expansion, rather than relying on the model's internal knowledge or static beam search, is a sensible and timely direction for multimodal reasoning, and the automated generation of step-level supervision without human annotation is an important practical contribution. The paper tests the framework on several open and closed MLLMs, which strengthens the generality claim. The GAOKAO temporal split is a genuinely cleaner evaluation design than the two primary benchmarks. However, the significance is conditional: the two benchmark results that carry most of the quantitative weight are compromised by the retrieval-corpus construction, and the DPO objective as written is mathematically vacuous. The framework's promise is therefore not yet demonstrated by the evidence in the manuscript.","major_comments":[{"comment":"The retrieval corpus construction directly undermines the central claim on MATHVISTA and WE-MATH. Supplementary §B.4 states that 'we incorporate the testmini set of WE-MATH into the knowledge base of MATHVISTA and include the testmini set of MATHVISTA into the knowledge base of WE-MATH.' This means that for each benchmark, solved near-duplicate problems from the other benchmark's testmini can be retrieved as 'knowledge' and their answers copied during inference. Table 7 reports a 4.2% 13-gram overlap for MATHVISTA-testmini retrieved by WE-MATH queries, yet the text concludes 'there is no overlap between our retrieval library and the test sets.' The 13-gram exact-overlap test is a weak contamination proxy: paraphrased, translated, or visually re-rendered versions of the same problem evade it, and the regex filtering described in §4.1 is not specified in enough detail to verify that it removes all test-influencing material. Because the reported AR-MCTS gains over the ORM/SC baselines in Table 1 are often only 1-5 points, even a few percent of retrieved near-duplicates could explain the apparent improvements. The evaluation must be redone with the cross-included testmini sets removed (or with a properly held-out corpus), and the contamination analysis should use a stronger method such as embedding-based similarity on both text and images.","section":"Supplementary §B.4 and Table 7"},{"comment":"The step-wise DPO objective as written is a constant. The equation reads log[πθ(y+|Qm)/πθ(y+|Qm)] for the preferred term and log[πref(y−|Qm)/πref(y−|Qm)] for the dispreferred term; in both ratios the same distribution appears in numerator and denominator, so each log term is zero and the entire loss reduces to log σ(0) = log 2, independent of the model. The standard DPO ratio should be πθ(y+|Qm)/πref(y+|Qm) and πθ(y−|Qm)/πref(y−|Qm). This is a load-bearing error because the paper claims that the 'Step-wise DPO Pre-alignment' phase aligns the PRM's reasoning discernment capability; as written, that phase performs no learning. The equation must be corrected and the experiments rerun or clearly reported with the corrected objective.","section":"§4.5, LSDPO equation"},{"comment":"The PRM training data are not separated from the evaluation benchmarks. In §5.1 the authors state that they use 'four multimodal and two text-only datasets for process annotation, excluding any sources currently under evaluation,' but the four multimodal datasets include MATHVISTA and WE-MATH, the evaluation benchmarks, and Supplementary §B.4 explicitly places the testmini sets of these benchmarks into the retrieval corpus used during the MCTS annotation process. The value function in §4.4 uses ground-truth answers to score rollout paths, so the automatically generated step labels encode the correct answers for problems that are near-duplicates of the test questions. The GAOKAO-MM evaluation with its temporal split is a cleaner design and partially mitigates this concern, but the main quantitative claims rest on MATHVISTA and WE-MATH, where the circularity is present. The authors should either re-run the PRM training with a corpus that strictly excludes the evaluation test sets (and any dataset containing their answers), or present the GAOKAO temporal split as the primary evidence and clearly label the MATHVISTA/WE-MATH results as preliminary.","section":"§5.1, §4.4, Supplementary §B.4"},{"comment":"The paper repeatedly uses the word 'significantly' for improvements that are within the range of sampling noise, and it provides no error bars, confidence intervals, or significance tests. For example, on MATHVISTA (ALL), GPT-4o AR-MCTS is 62.6 versus Self-Consistency 61.8 and ORM 61.9, a 0.7-0.8 point difference on a 1,000-example testmini set; LLaVA-OneVision-72B shows 66.3 versus 66.0 for Self-Consistency. Several WE-MATH differences are similarly around 1 point. Without repeated runs or statistical testing, these small differences cannot support the claim of a 'significant improvement' or the stronger comparative conclusions (§5.2). The authors should report variance across multiple seeds or bootstrap confidence intervals, and temper the language accordingly.","section":"Table 1"}],"minor_comments":[{"comment":"Equation (1) is presented as a theoretical model of MCTS expansion and simulation, but no derivation or justification is given for the factorization p(y|x) = max_i pθ(y|r_i,x)·pφ(r_i|x), and the variables i, r_i, and k are not defined precisely enough to make the equation meaningful. The subsequent text does not use this equation in any technical way, so it should be either derived properly or labeled as an intuitive motivation rather than a theoretical model.","section":"§1, Eq. (1)"},{"comment":"The point-wise fine-tuning loss uses π_SDPO(r_i) without defining what this distribution represents. If r_i is a sigmoid score produced by the PRM, the notation should be a scalar probability rather than a model distribution, and the loss should be written as a standard binary cross-entropy over the score. Please clarify.","section":"§4.5, L_PFT"},{"comment":"The text refers to 'AC-MCTS' in two places in the scaling analysis; this should be 'AR-MCTS'.","section":"§5.4 and Figure 4"},{"comment":"The diversity analysis reports cluster counts (38 vs. 46) and a visualization, but no quantitative measure of dispersion or reproducibility. The claim that AR-MCTS 'efficiently covers the problem-solving space' needs a metric such as average pairwise distance or cluster purity, ideally with multiple samples.","section":"§5.5, Figure 5"},{"comment":"The contamination analysis is self-contradictory: it reports a 4.2% overlap for the cross-included testmini set and then concludes that no overlap exists. At minimum, the authors should acknowledge this overlap and explain why they believe it does not affect the evaluation.","section":"Table 7 and §C.3"},{"comment":"The phrase 'filtered out any overlapping portions with our testing benchmark using regular expressions' is too vague to be checked. The authors should provide the actual regexes or a concrete description of what was removed, especially since the testmini sets are later intentionally included in the corpus in §B.4.","section":"§4.1"},{"comment":"There are numerous typos and inconsistent notations (e.g., 'accuarcy' in §4, 'fomulated' in the back-propagation description, 'Seld-Correction' in Table 6, 'MATHVISTA' versus 'MATHVISTA-testmini' in Table 7). A careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The cross-inclusion of testmini sets in the retrieval corpus (Supplementary §B.4) and the mathematically degenerate DPO equation in §4.5 are substantial technical flaws that should have been caught in internal review. The GAOKAO temporal split shows the authors are aware of leakage concerns, which makes the choice to cross-include testmini in the two main benchmarks particularly puzzling. If the corrected DPO objective and a clean re-evaluation on MATHVISTA/WE-MATH still show gains, the paper could become a solid contribution, but the current version's central quantitative claims are not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look if you care about retrieval-augmented reasoning for MLLMs: it puts retrieval inside the MCTS loop, per step, rather than once per question. That configuration is genuinely new relative to the cited prior work, and the experiments are broad: five or six base models, three benchmarks, ablations, scaling, and a diversity analysis. The GAOKAO-MM temporal split (2010-2021 for building the knowledge base, 2022-2023 for test) is a clean design and shows gains of several points for both GPT-4o and Qwen2-VL.\n\nThe central idea has real promise. Replacing beam-search expansion with actively retrieved insights at each MCTS step is a sensible fix for the misalignment problem in MLLMs, and the automatic PRM annotation via MCTS is a practical way to get process supervision without human labels. The paper is honest enough to include a limitations section and a contamination analysis.\n\nNow the soft spots, and they matter. The two main benchmarks carry the quantitative weight, and the evaluation is not demonstrably clean. Supplementary B.4 states that the We-Math testmini is put into the MathVista knowledge base and vice versa. That means a query from one benchmark can retrieve a near-duplicate solved problem from the other benchmark's testmini and copy the answer. The paper's own Table 7 reports 4.2% 13-gram overlap for MathVista-testmini retrieved by We-Math queries; the reported gains are 2-5 points on these benchmarks. The 13-gram exact-match test is a weak proxy for leakage: paraphrased or visually re-rendered versions evade it. So the contribution of memorized answers to the measured gains is unknown. The GAOKAO split is cleaner and partially supports the claim, but it is one benchmark.\n\nAlso, Equation (1) is presented as a theoretical model but is a factorization of conditional probabilities; that is overclaiming. There are no error bars or significance tests, and no code release is mentioned. Those are fixable.\n\nBottom line: this is a solid, incremental systems paper with a novel configuration and a claim that likely has some truth to it, but the headline numbers on MathVista and We-Math are not trustworthy until the retrieval corpus is cleaned and the evaluation is repeated. I'd send it to a serious referee, because the idea deserves scrutiny and the GAOKAO result deserves a closer look, but I would not cite it in its current form.","headline":"Useful new configuration of per-step retrieval inside MCTS for MLLMs, with a clean GAOKAO split, but the main benchmark results are clouded by cross-contamination in the retrieval corpus.","tokens_in":34889,"tokens_out":2372,"would_cite":false,"duration_ms":17669,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AR-MCTS shows that retrieving key insights at each reasoning step, rather than beam-search sampling, improves multimodal reasoning.","keywords":["active retrieval","multimodal reasoning","Monte Carlo tree search","process reward model","retrieval-augmented generation","multimodal large language models","reasoning verification"],"falsifier":"Run AR-MCTS with the retrieval corpus restricted to sources with zero n-gram and image overlap with the test sets, or on a newly released multimodal reasoning benchmark created after the corpus was built; if the gains over beam search vanish or shrink to the level of contamination baselines, the central claim fails. A simpler check: for each solved test item, inspect the top retrieved insight and test whether deleting near-duplicate documents removes the improvement.","tokens_in":33841,"feed_emoji":"🔍","tokens_out":5521,"duration_ms":38193,"temperature":0.7,"pith_summary":"This paper tries to establish that the weak point of Monte Carlo tree search for multimodal reasoning is not the verifier but the way candidate steps are generated: multimodal models expand reasoning paths from their own internal knowledge, which is often misaligned across image and text. The authors replace beam-search expansion with active retrieval of hybrid-modal insights at every step and use the search itself to produce step-level annotations that train a process reward model. On MATHVISTA, WE-MATH, and the Chinese GAOKAO-MM benchmark, the resulting AR-MCTS framework improves accuracy over self-consistency, self-correction, and outcome reward models across several open and closed multimodal models. The claim matters because it offers a path to reliable stepwise verification in multimodal reasoning without human annotations.","feed_headline":"Active retrieval beats beam search for multimodal reasoning","feed_subtitle":"AR-MCTS retrieves insights at each reasoning step and trains a process reward model with no human step labels.","key_machinery":"The carrying object is the AR-MCTS loop built on the factorization in Equation (1), which splits reasoning success into an expansion term (how candidate next steps are generated) and a simulation term (how paths are scored). Expansion uses a unified retrieval module: Contriever for text retrieval and CLIP for cross-modal retrieval, fused through FAISS, followed by a knowledge-concept filter that keeps only retrieved items similar to both the query and its fine-grained concept label. Each MCTS node re-retrieves insights conditioned on the accumulated reasoning steps, and a one-step rollout values each node by whether sampled answers match the ground truth, with back-propagation updating node values. A curriculum process reward model, pre-aligned with step-wise DPO and then fine-tuned with point-wise logistic labels, supplies the step scores used at inference.","core_discovery":"The central claim is that reasoning-path expansion, not just simulation, decides success in multimodal MCTS reasoning: $p(y|x) = \\max_i p_\\theta(y|r_i,x)\\,p_\\phi(r_i|x)$. Since MLLMs' internal knowledge is unreliable for multimodal steps, the paper augments each expansion with actively retrieved insights from a hybrid-modal corpus, refreshed step by step, and filters them by knowledge-concept consistency. The search then generates positive and negative step pairs automatically, and a two-stage curriculum, step-wise DPO pre-alignment followed by point-wise fine-tuning, yields a process reward model that scores each step at inference. The paper reports consistent gains across models such as GPT-4o, Qwen2-VL-7B, and InternVL2-8B, with the largest improvements on the multi-step WE-MATH S3 metric and on smaller models.","pith_inferences":["A natural extension the paper does not pursue: because retrieval, not search depth, appears to carry much of the gain, stronger retrievers or rerankers should yield further improvements at the same sampling budget.","The contamination controls (regex filtering, a 13-gram overlap threshold, and the temporal split for GAOKAO-MM) are the main trust boundary; an independent evaluation on a brand-new multimodal reasoning set would settle whether gains are reasoning gains or retrieval of near-duplicate solutions.","The per-step re-retrieval mechanism could transfer to other multimodal agent tasks such as science-diagram understanding or medical imaging reasoning, where stepwise knowledge needs are heterogeneous.","Since the PRM is trained on a text backbone while scoring multimodal steps, an obvious test is whether jointly aligning the PRM inside the vision-language model itself, which the paper lists as future work, further improves verification."],"forward_implications":["If correct, inference-time scaling for multimodal models should spend compute on retrieving per-step knowledge rather than only on sampling more beams.","Process reward models for multimodal reasoning can be built automatically from MCTS rollouts, removing the manual step-annotation bottleneck.","The framework's largest gains on smaller MLLMs imply active retrieval can compensate for internal knowledge gaps without retraining the base model.","Knowledge-concept filtering is a load-bearing component: removing it costs accuracy, indicating that retrieved noise, not just missing knowledge, hurts multimodal reasoning.","Generalization to the Chinese GAOKAO-MM benchmark suggests the recipe transfers across languages and disciplines beyond mathematics."],"supporting_citations":[{"why":"Baseline that samples many reasoning paths and votes on the final answer; AR-MCTS is compared against it.","marker":"[107]"},{"why":"Defines outcome reward models, the comparison verifier ORM.","marker":"[17]"},{"why":"Supplies automated step-level supervision and the MCTS-style annotation approach that AR-MCTS adapts to multimodal retrieval.","marker":"[106]"},{"why":"Process supervision data used to pre-align the PRM before multimodal annotation.","marker":"[49]"},{"why":"One of the two main math benchmarks and a source of multimodal corpus data.","marker":"[58]"},{"why":"The other main math benchmark, with step-level S1-S3 evaluation used to show PRM superiority.","marker":"[75]"},{"why":"Cross-domain Chinese examination benchmark whose earlier years form the retrieval knowledge base and later years form the test set.","marker":"[127]"},{"why":"Dense text retriever (Contriever) used for text-to-text retrieval in the unified retrieval module.","marker":"[34]"},{"why":"CLIP provides the cross-modal encoders used for image-text retrieval and knowledge-concept filtering.","marker":"[77]"},{"why":"Direct Preference Optimization, the step-level preference objective used to pre-align the PRM.","marker":"[78]"}],"fun_headline_variants":["AR-MCTS: Active retrieval beats beam search","Step-wise retrieval outperforms beam search","AR-MCTS: Progressive reasoning via active retrieval","Active retrieval improves MLLM reasoning step by step"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the retrieval corpus and the PRM's training data are truly separate from the evaluation benchmarks: the regex filtering, the 13-gram contamination threshold, and the time-based split are assumed to remove any test-influencing material, while cross-including WE-MATH testmini in MATHVISTA's knowledge base and vice versa is assumed not to leak solutions.","fun_headline_variants_meta":{"raw":{"variants":["AR-MCTS: Active retrieval beats beam search","Step-wise retrieval outperforms beam search","AR-MCTS: Progressive reasoning via active retrieval","Active retrieval improves MLLM reasoning step by step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000978,"raw_usage":{"total_tokens":4139,"prompt_tokens":916,"completion_tokens":3223,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":3165}},"tokens_in":532,"tokens_out":3223,"duration_ms":16667,"temperature":1.0,"reasoning_tokens":3165,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:51:41.246953+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AR-MCTS with the retrieval corpus restricted to sources with zero n-gram and image overlap with the test sets, or on a newly released multimodal reasoning benchmark created after the corpus was built; if the gains over beam search vanish or shrink to the level of contamination baselines, the central claim fails. A simpler check: for each solved test item, inspect the top retrieved insight and test whether deleting near-duplicate documents removes the improvement.","supporting_citations":[{"cited_title":"Le, Ed H","cited_arxiv_id":null,"evidence_quote":"Baseline that samples many reasoning paths and votes on the final answer; AR-MCTS is compared against it."},{"cited_title":"Advanced embedding techniques in multimodal retrieval augmented generation a comprehensive study on cross modal ai applications","cited_arxiv_id":null,"evidence_quote":"Cross-domain Chinese examination benchmark whose earlier years form the retrieval knowledge base and later years form the test set."}],"review_version":1}