{"id":"890c0e2b-2091-447d-a9e4-56a84ee37056","arxiv_id":"2506.12364","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MM-R5, a 7B multimodal re-ranker trained with SFT and GRPO, achieves state-of-the-art page-level recall on MMDocIR by generating per-page reasoning chains.","lead":"Researchers trained a multimodal re-ranker, MM-R5, that reasons about each candidate page before ranking it, using a two-stage recipe of supervised fine-tuning and reinforcement learning. On the MMDocIR benchmark it beats larger models and improves top-1 recall by over four points, suggesting reasoning chains help multimodal retrieval.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim unverified because MonoQwen, a directly relevant specialized reranker cited in §2.2, is absent from Table 1 and §4.1.4; if MonoQwen matches or exceeds 0.6951 macro Recall@1, the headline superiority fails.","rationale":"The reader's formal weakest assumption is the quality of the GPT-4o-generated SFT reasoning data. That concern is plausible but less load-bearing than the missing MonoQwen baseline. The RL stage is trained on ground-truth labels rather than GPT-4o judgments, and the SFT-only ablation already improves over the base model, so the empirical results provide some evidence that the constructed data carry useful signal. The missing baseline, by contrast, directly conditions the truth of the paper's central claim: 'state-of-the-art on MMDocIR.' A SOTA claim cannot be evaluated when a directly relevant prior reranker is cited in related work but omitted from all experiments. The reader did mention the MonoQwen omission in the rationale, so my concern is partially aligned with their assessment, even though it was not listed as the weakest assumption. My recommended verdict remains CONDITIONAL, unchanged from the reader: the method may be strong and the pipeline plausible, but the SOTA assertion needs the missing comparison before it can be accepted. The concrete test is straightforward and should be part of the revision if the authors wish to retain the SOTA claim.","tokens_in":14646,"tokens_out":7855,"duration_ms":94953,"concrete_test":"Obtain or reproduce MonoQwen results on MMDocIR under the exact Table 1 protocol: use ColQwen top-10 candidate pages for each query, the same prompt template described in §3.3.3, and the same macro/micro Recall@1/3/5 evaluation script. If MonoQwen's macro Recall@1 reaches or exceeds 0.6951, the central SOTA claim is contradicted. If MonoQwen's paper already reports MMDocIR numbers, verify that the candidate sets and metrics match the authors' protocol; if they differ, re-run MonoQwen with the released checkpoint before drawing conclusions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MM-R5 achieves state-of-the-art performance on MMDocIR. To establish that, the comparison set must include existing reranking methods for the same task. The paper itself cites MonoQwen (Chaffin and Lac 2024) in §2.2 as one of the few training-based multimodal rerankers, yet MonoQwen appears nowhere in the experimental comparison: §4.1.4 lists RAG-VL and general-purpose VLMs (Qwen2.5-VL, Gemma3), and Table 1 contains no MonoQwen row. This is not a matter of model size or availability: MonoQwen is a Qwen-based visual document reranker, the most directly comparable prior system to MM-R5. Without it, the 0.6951 macro Recall@1 figure may still be strong, but the label 'state-of-the-art' is unsupported. If MonoQwen outperforms MM-R5 on the same candidate sets, the paper's headline quantitative claim is false as stated. The omission is especially notable because the paper reports second place in the WWW 2025 Multimodal RAG Challenge, implying at least one competing system was stronger; the reader cannot verify whether that system is MonoQwen or another excluded reranker. This is a correctness-of-claim concern, not a stylistic preference: the SOTA assertion is unverified as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MM-R5, a multimodal reranker built on Qwen2.5-VL-7B, trained in two stages: supervised fine-tuning on GPT-4o-generated reasoning chains and reinforcement learning with GRPO using a result reward (normalized inverse-cube DCG) plus a format reward. Evaluation on MMDocIR reports macro Recall@1 of 0.6951 when reranking ColQwen candidates, an absolute gain of 4.7 points over ColQwen, and consistent gains when applied to four other retrievers. The paper claims state-of-the-art performance on MMDocIR and attributes the gains to the reasoning-enhanced two-stage pipeline.","tokens_in":14925,"tokens_out":5497,"duration_ms":60019,"significance":"The work is potentially useful: it provides a concrete recipe for injecting explicit CoT reasoning into a multimodal reranker, with ablations showing that both SFT and RL contribute, and a generalization study across five retrievers. The public code and use of a public benchmark are strengths, and the reward design based on ground-truth relevance labels avoids circularity. The main quantitative claims, however, rest on a comparison set that omits MonoQwen, the most directly comparable prior training-based reranker, and on differences of 1–3 points that are reported without variance or significance. The core idea is interesting and the experiments are extensive, but the SOTA label is not yet established.","major_comments":[{"comment":"The claim that MM-R5 achieves state-of-the-art performance on MMDocIR is not verifiable from the reported comparisons because MonoQwen (Chaffin and Lac 2024), cited in §2.2 as a training-based multimodal reranker, is not included in Table 1 or §4.1.4. Since MonoQwen is a Qwen-based visual document reranker specifically designed for this task, it is the most directly comparable prior system; without its numbers on the same ColQwen top-10 candidates, the SOTA assertion for macro Recall@1 (0.6951) is unsupported. Please add MonoQwen and, if possible, other reranking systems from the WWW 2025 Multimodal RAG Challenge leaderboard, since the Introduction reports second place without identifying the winner.","section":"§4.1.4, Table 1"},{"comment":"The reported margins over the strongest baselines are small (e.g., 0.6951 vs. 0.6768 for Qwen2.5-VL-32B-cot and 0.6951 vs. 0.6673 for the SFT-only variant), and no variance estimates, multiple runs, or significance tests are provided. Without these, it is difficult to judge whether the claimed improvements are stable or within noise, particularly for the macro Recall@1 differences of about 1–3 points. Please report standard deviations over at least three seeds or provide a significance analysis for the main comparisons.","section":"Table 1, §4.2.1"},{"comment":"The SFT stage depends entirely on 7,200 automatically generated reasoning chains produced by GPT-4o via single-image sub-tasks, followed by a GPT-4o-based refinement. The paper asserts that these are high-quality but provides no human evaluation, no inter-annotator agreement, and no automatic quality metrics for the reasoning chains. Since the quality of this data is the foundation for the claimed contribution of the SFT stage, the paper should include at least a small-scale human validation or a comparison of the generated reasoning against an alternative construction method, such as full multi-image GPT-4o reasoning.","section":"§3.2.1"}],"minor_comments":[{"comment":"The sentence defining the length accuracy reward uses \"donate\" where \"denote\" is intended; please correct the typo.","section":"§3.3.2"},{"comment":"The text describes the three format-reward components as \"independent and differentiable,\" but the structure validity reward is a hard binary indicator and is not differentiable; please rephrase to avoid the inaccurate claim.","section":"§3.3.2, Eq. (4)"},{"comment":"The figure uses closing tags written as \"<\\think>\" and \"<\\answer>\" while the text uses \"</think>\" and \"</answer>\"; please unify the notation.","section":"Figure 1"},{"comment":"The baseline is labeled \"RagVL\" in the table but \"RAG-VL\" in the text; please make the naming consistent.","section":"Table 1"},{"comment":"Please clarify the initialization of the RL-only variant (Qwen2.5-VL-7B-rl): does it start from the base Qwen2.5-VL-7B or from the SFT checkpoint? This affects the interpretation of the ablation.","section":"§4.1.3, Table 1 caption"},{"comment":"The reference to MonoQwen (Chaffin and Lac 2024) gives only a title and no venue or technical report identifier; since the paper discusses it in §2.2 and it is directly relevant to the experiments, please provide a complete citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main reason for major revision is the missing MonoQwen comparison, which directly affects the SOTA claim. The paper would also benefit from variance reporting and a small human validation of the generated SFT reasoning data. The 'second place in the WWW 2025 Multimodal RAG Challenge' statement is vague without identifying the winning system; this should be clarified in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper deserves a serious referee, and the main thing to fix before acceptance is the comparison set. The authors report 0.6951 macro Recall@1 on MMDocIR, beating ColQwen and larger VLMs, and the gains look real. But they cite MonoQwen, the most directly comparable prior multimodal reranker, and don't evaluate it. Without a MonoQwen row in Table 1, 'state-of-the-art' is an unsupported label.\n\nWhat is genuinely new: the single-image reasoning data construction (pairing each candidate with the query, asking GPT-4o for per-image relevance judgments, then stitching them into a <think> chain) is a nice way to dodge weak multi-image reasoning. Combining that SFT stage with GRPO using a rank-based result reward and a format reward is a new application of RL to multimodal reranking. The ablations are clean: SFT-only and RL-only both fall short of the full pipeline, which supports the two-stage story.\n\nThe experiments are more thorough than most: five different retrievers, consistent gains across all of them, and the improvements on weak retrievers like CLIP are huge. The generalization table is the strongest part of the paper.\n\nSoft spots, in order. First, the missing MonoQwen comparison is a real problem, not a stylistic one. The paper itself says MonoQwen is one of the few training-based multimodal rerankers, so it has to be in the table or explicitly excluded with a reason. Second, no error bars or significance tests anywhere. The reported margins over Qwen2.5-VL-32B-cot (1.83 macro Recall@1) could be noise. Third, the SFT reasoning data from GPT-4o is unverified; the authors don't report any human evaluation or quality checks on those chains. That doesn't sink the method, but it's a real limitation on reproducibility. Fourth, everything rests on a single benchmark (MMDocIR). That's acceptable for a first paper, but it does cap the strength of the SOTA claim.\n\nI don't think there's a circularity problem: the reward uses ground-truth labels, the SFT data is externally generated, and evaluation is on a held-out split.\n\nBottom line: the core contribution holds up, the writing is clear, and the code/data release is promised. It deserves peer review—I'd send it out and ask the authors to add MonoQwen and at least some variance estimates. I'd probably cite this for the data construction idea if I worked in this area.","headline":"Solid empirical paper on RL+CoT for multimodal reranking, but the SOTA claim is undercut by the missing MonoQwen baseline.","tokens_in":15509,"tokens_out":2281,"would_cite":true,"duration_ms":24232,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7B reranker explicitly trained to reason about page relevance reaches the top MMDocIR results, beating retrieval-only baselines and larger 32B models.","keywords":["multimodal reranking","chain-of-thought reasoning","reinforcement learning","GRPO","vision-language models","document retrieval","MMDocIR","retrieval-augmented generation"],"falsifier":"Train the same two-stage pipeline but replace the GPT-4o reasoning traces with (a) labels-only SFT without any reasoning chains or (b) reasoning chains whose per-page relevance judgments are randomly shuffled, keeping the RL stage identical; if macro Recall@1 on MMDocIR does not drop substantially from 0.6951, the reasoning data is not the carrier of the gain. A more direct check is to compare GPT-4o's per-image relevance judgments with the page-level ground-truth labels on a held-out MMDocIR subset: low agreement would predict the SFT stage cannot teach correct relevance.","tokens_in":14435,"feed_emoji":"🧠","tokens_out":11446,"duration_ms":118921,"temperature":0.7,"pith_summary":"MM-R5 is a reranker for multimodal document retrieval: given a query and a set of candidate pages, it looks at every page, writes a short chain-of-thought about each page's relevance, and outputs a ranked list. The paper's central claim is that this reasoning behavior can be trained in two stages—supervised fine-tuning on per-image reasoning traces, then reinforcement learning with a reward that favors correct pages near the top—and that this training is what makes the reranker accurate. On the MMDocIR benchmark, MM-R5 reaches macro Recall@1 of 0.6951 and micro Recall@1 of 0.6759, improving on the best retriever-only baseline by 4.70 and 4.05 absolute points and beating much larger 32B vision-language models. The paper concludes that a relatively small model can outperform larger general-purpose models on reranking when it is explicitly taught to reason, and that the visible reasoning chains make the reranker's decisions interpretable.","feed_headline":"Trained reasoning lifts a 7B document reranker past 32B models","feed_subtitle":"Trained on reasoning traces plus reinforcement learning, MM-R5 beats larger models and every baseline on MMDocIR.","key_machinery":"The load-bearing mechanism is the two-stage training recipe plus the reward shape. For SFT, the data construction decomposes each multi-page ranking question into single-image sub-tasks; a strong vision-language model (GPT-4o) explains why each page is or is not relevant, those explanations are concatenated into a <think>...</think> reasoning chain, and the ground-truth label order is wrapped in <answer>...</answer>. This sidesteps the multi-image reasoning weakness of existing VLMs and yields 7,200 training instances. For RL, the paper applies GRPO with two rewards: a result reward $R_{\\text{result}} = \\frac{\\sum_{j} s_j / j^3}{\\sum_{j=1}^{|G|} 1/j^3}$, where $s_j=1$ if the item predicted at rank $j$ is in the golden set, which weights correct pages by the inverse cube of their predicted rank and so emphasizes the top position far more than logarithmic discounting; and a format reward $R_{\\text{format}} = R_{\\text{valid}} \\times R_{\\text{len}} \\times R_{\\text{range}}$, which hard-gates on the <think>/<answer> structure and smoothly penalizes wrong list lengths and out-of-range indices. The RL data is drawn with a resolution-balanced sampling strategy so that documents with very different image sizes contribute evenly.","core_discovery":"On the paper's own terms, the discovery is that a multimodal reranker can be made both more accurate and more interpretable by treating chain-of-thought reasoning as a trainable behavior rather than a prompted style. The evidence is on MMDocIR: the full MM-R5 model reaches macro Recall@1 of 0.6951, versus 0.6481 for ColQwen retriever-only, 0.6768 for Qwen2.5-VL-32B-cot, 0.6673 for the SFT-only variant, and 0.6586 for the RL-only variant. The same model reranking five different retrievers improves Recall@1 in every configuration, with the largest relative gains on the weakest retrievers (CLIP macro Recall@1 rises from 0.3334 to 0.5942). From these results the paper concludes that the two stages are complementary, that CoT improves ranking, and that the gains are not tied to a particular retrieval backbone.","pith_inferences":["One testable consequence the paper does not run: swapping GPT-4o for a cheaper or open-weight teacher in the SFT data pipeline would isolate how much of the gain depends on teacher quality; the paper's design implies teacher quality matters but never varies it.","The inverse-cube reward is motivated as stronger top-rank emphasis than NDCG's logarithmic discount, but the paper does not ablate reward shape; comparing $j^{-3}$ against $1/\\log_2(j+1)$ on the same RL pipeline would show whether the cube itself is load-bearing.","The format-reward hard gate on <think>/<answer> tags suggests the pipeline is also a structured-output trainer; the same SFT+GRPO recipe may transfer to other list-generation tasks with strict output schemas, such as layout ranking or multi-step tool selection, though the paper only tests document pages.","The resolution-balanced sampling addresses a distributional shift that likely affects any VLM reranker trained on mixed-resolution documents; applying this sampling to other multimodal training regimes could reduce RL instability beyond reranking."],"forward_implications":["On MMDocIR, reranking with MM-R5 raises macro Recall@1 by 4.70 absolute points over ColQwen alone, so reasoning-trained reranking can be a drop-in improvement on top of a strong retriever.","The full two-stage model beats both single-stage variants (0.6951 vs 0.6673 SFT-only and 0.6586 RL-only), so the paper's recipe says SFT and RL contribute complementary gains rather than being redundant.","MM-R5 improves Recall@1 for all five tested retrievers, with the largest relative gains for the weakest ones, meaning the method can partially compensate for retriever weakness.","A 7B model with task-specific reasoning training surpasses the 32B general-purpose VLM with CoT prompting (macro Recall@1 0.6951 vs 0.6768), suggesting task-specific training can substitute for model scale in reranking.","Because the model emits explicit per-page reasoning inside <think> tags, reranker outputs become inspectable, which the paper presents as a step toward controllable and trustworthy multimodal retrieval."],"supporting_citations":[{"why":"Supplies MMDocIR, the benchmark whose page-level ground-truth labels are used for SFT data construction, RL sampling, and all reported evaluation.","marker":"Dong et al. 2025"},{"why":"Provides ColQwen, the strongest retriever-only baseline and the retriever whose top-10 candidates are reranked in the main comparison.","marker":"Faysse et al. 2024"},{"why":"Defines Qwen2.5-VL-7B, the base VLM that MM-R5 fine-tunes, and also the larger Qwen2.5-VL variants used as reranking baselines.","marker":"Bai et al. 2025"},{"why":"Introduces the DeepSeek-R1 reasoning-through-RL approach that motivates using GRPO to train reasoning in the reranker.","marker":"Guo et al. 2025"},{"why":"Supplies the GRPO algorithm itself, the RL optimizer used in the second training stage.","marker":"Shao et al. 2024"},{"why":"Establishes Rank-R1, the LLM-based reasoning reranker trained with RL that this paper extends to multimodal candidates.","marker":"Zhuang et al. 2025"},{"why":"Provides the open-source RL training implementation used for the GRPO stage with LoRA.","marker":"Shen et al. 2025"},{"why":"Supplies RAG-VL, the prior multimodal reranking method included as a baseline in the main results.","marker":"Chen et al. 2024"}],"fun_headline_variants":["Reasoning-trained reranker beats 32B models at 7B scale","MM-R5: RL and reasoning lift reranking past larger models","Small reranker learns to reason, outperforming 32B counterparts","CoT + RL reranker surpasses larger models on MMDocIR","Reranker with reasoning and RL gains 4% recall over retrievers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated reasoning traces from GPT-4o are accurate and complete enough to teach correct page-level relevance; no human verification or quality metric is reported for those chains, so any bias or noise in them could propagate through both training stages into the final rankings.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning-trained reranker beats 32B models at 7B scale","MM-R5: RL and reasoning lift reranking past larger models","Small reranker learns to reason, outperforming 32B counterparts","CoT + RL reranker surpasses larger models on MMDocIR","Reranker with reasoning and RL gains 4% recall over retrievers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000344,"raw_usage":{"total_tokens":1947,"prompt_tokens":1063,"completion_tokens":884,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":787}},"tokens_in":679,"tokens_out":884,"duration_ms":10543,"temperature":1.0,"reasoning_tokens":787,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:51:04.586717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same two-stage pipeline but replace the GPT-4o reasoning traces with (a) labels-only SFT without any reasoning chains or (b) reasoning chains whose per-page relevance judgments are randomly shuffled, keeping the RL stage identical; if macro Recall@1 on MMDocIR does not drop substantially from 0.6951, the reasoning data is not the carrier of the gain. A more direct check is to compare GPT-4o's per-image relevance judgments with the page-level ground-truth labels on a held-out MMDocIR subset: low agreement would predict the SFT stage cannot teach correct relevance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ColQwen, the strongest retriever-only baseline and the retriever whose top-10 candidates are reranked in the main comparison."}],"review_version":1}