Pith. sign in

REVIEW 10 cited by

Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.19702 v1 pith:YVOGZY6Q submitted 2025-05-26 cs.CV

Minheng Ni , Zhengyuan Yang , Linjie Li , Chung-Ching Lin , Kevin Lin , Wangmeng Zuo , Lijuan Wang This is my paper
classification cs.CV
keywords reasoningvisualfinetuninggroundedmultimodalreinforcementdocumentpoint-rft
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advances in large language models have significantly improved textual reasoning through the effective use of Chain-of-Thought (CoT) and reinforcement learning. However, extending these successes to vision-language tasks remains challenging due to inherent limitations in text-only CoT, such as visual hallucinations and insufficient multimodal integration. In this paper, we introduce Point-RFT, a multimodal reasoning framework explicitly designed to leverage visually grounded CoT reasoning for visual document understanding. Our approach consists of two stages: First, we conduct format finetuning using a curated dataset of 71K diverse visual reasoning problems, each annotated with detailed, step-by-step rationales explicitly grounded to corresponding visual elements. Second, we employ reinforcement finetuning targeting visual document understanding. On ChartQA, our approach improves accuracy from 70.88% (format-finetuned baseline) to 90.04%, surpassing the 83.92% accuracy achieved by reinforcement finetuning relying solely on text-based CoT. The result shows that our grounded CoT is more effective for multimodal reasoning compared with the text-only CoT. Moreover, Point-RFT exhibits superior generalization capability across several out-of-domain visual document reasoning benchmarks, including CharXiv, PlotQA, IconQA, TabMWP, etc., and highlights its potential in complex real-world scenarios.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. V-Reflection: Transforming MLLMs from Passive Observers to Active Interrogators

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    V-Reflection introduces a think-then-look mechanism where MLLM latent states actively interrogate visual features via two-stage distillation from a box-guided teacher to a dynamic autoregressive student, narrowing the...

  2. Latent Visual Reasoning

    cs.CV 2025-09 unverdicted novelty 7.0 of 10

    Latent Visual Reasoning enables autoregressive generation of latent visual states that reconstruct critical image tokens, yielding gains on perception-heavy VQA benchmarks such as 71.67% on MMVP.

  3. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  4. CARE: Competence-Aware Reward Shaping for Adaptive Reasoning Length in Video-MLLMs

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    CARE uses exponential moving average competence estimates to progressively shift RL rewards from exploration-oriented long reasoning to efficiency-oriented concise reasoning in video-MLLMs, with batch normalization an...

  5. From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    MedTPE compresses EHR token sequences by up to 31% via merging common medical token pairs, reducing LLM inference latency 34-63% while maintaining or improving performance on mortality and phenotyping tasks.

  6. CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    CharTide decouples chart-to-code data into three perspectives and uses inquiry-driven RL with atomic QA verification to let smaller VLMs surpass GPT-4o on chart-to-code tasks.

  7. CharTide: Data-Centric Chart-to-Code Generation via Tri-Perspective Tuning and Inquiry-Driven Evolution

    cs.CV 2026-04 conditional novelty 6.0 of 10

    A 7B/8B model trained with decoupled tri-perspective SFT and QA-verified RL matches GPT-4o and approaches GPT-5 on chart-to-code generation benchmarks.

  8. HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Framework

    cs.CV 2026-02 conditional novelty 6.0 of 10

    HART uses a closed-loop 'crop-and-answer' training scheme plus a dynamic-weight GRPO variant to improve LMM grounding and high-resolution reasoning without bounding-box annotations.

  9. Visual Programmability: A Guide for Code-as-Thought in Chart Understanding

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A vision-language model learns to dynamically switch between code-based and visual reasoning for chart questions, improving average accuracy by about one point over fixed strategies.

  10. SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs

    cs.CV 2026-02 conditional novelty 5.0 of 10

    Separating visual search from reasoning, then answering from the located crops, lifts VLM accuracy on high-resolution visual QA while cutting token use versus monolithic 'thinking with images'.

Pith tools