{"id":"93dffa3f-66ed-42f4-9ec9-5da7cec5af54","arxiv_id":"2508.06125","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An RL framework that trains vision-language models to self-correct captions via a scene-graph-based reward outperforms SFT and DPO on caption quality.","lead":"Image captioning models can invent objects that aren't there or leave out real ones. The authors train vision-language models to self-correct their first draft using reinforcement learning, and their method beats standard preference tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SC-Captioner's reported advantage over DPO may be inflated because the RL reward and the headline evaluation metrics share the same FACTUAL scene-graph pipeline; independent metrics should be used to verify.","rationale":"I read the paper in good faith: the method is coherent, the multi-turn RL setup is plausible, and the authors include useful ablations plus a DOCCI-training generalization check. The reader's CONDITIONAL verdict is reasonable. However, I think the most load-bearing concern is not primarily the completeness of RefinedCaps references, but the tight coupling between the reward used for training and the metrics used for evaluation. The authors' own reward and their refined metrics both rely on FACTUAL scene-graph parsing and set-difference matching, so SC-Captioner is explicitly optimized for the same kind of signal that the paper uses to demonstrate superiority. That makes the headline claim vulnerable to metric overfitting, especially because the improvements are largest on the parser-based F1/QA metrics and smaller or inconsistent on BLEU-4/METEOR. The human-correlation experiment is a step in the right direction but is too small to establish that the metric remains unbiased after the model has been trained against it. The reference-completeness issue flagged by the reader is real and acknowledged by the authors, but it is a secondary threat: it would cause the reward to penalize some correct additions, while the parser-overlap issue could systematically inflate the measured gains of the proposed method over DPO. My proposed test—independent evaluation with a metric that does not share the parser, or retraining with a different parser—would directly settle whether the reported advantage is genuine. I therefore keep the reader's CONDITIONAL verdict unchanged rather than moving to ACCEPT or REJECT; the concern is substantial but empirically testable, and the paper already contains partial evidence (human correlation, DOCCI training) that mitigates but does not eliminate it.","tokens_in":20151,"tokens_out":4418,"duration_ms":53834,"concrete_test":"Run a blind side-by-side human preference evaluation or an independent LLM-judge metric (e.g., DC-Score or GPT-4o-based comparison) on a fresh sample of at least 200 images from DOCCI500 and COCO-LN500, comparing SFT+DPO* vs SFT+Ours* from Tables 1-2. Compute the preference gap and a 95% confidence interval. If the gap is not positive or is within noise, the central claim is not supported. Alternatively/additionally, retrain SC-Captioner with the same reward but an independently implemented scene-graph parser, and check whether the evaluation gains on the proposed metrics persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SC-Captioner significantly outperforms DPO for self-correction in image captioning. The main evidence is Tables 1-3 on DOCCI500 and COCO-LN500, using CAPTURE and the authors' refined metrics. The reward in Sec. 3.3 decomposes captions into object/attribute/relation sets with the FACTUAL parser and Sentence Transformer similarity; the evaluation metrics in Sec. 4.2 decompose captions with the same FACTUAL parser and similar set-matching logic, with precision expanded by GPT-4o/human annotations. The model is therefore trained to maximize a score that is nearly identical to the measured outcome. This is not merely philosophical: on BLEU-4/METEOR, which do not share the parser, the SC-Captioner advantage over SFT+DPO is inconsistent (e.g., LLaVA DOCCI500: BLEU-4 43.04 vs 42.67, METEOR 23.88 vs 23.47; Qwen COCO-LN500: BLEU-4 35.05 vs 30.34 but METEOR only 27.34 vs 26.77), while the headline parser-based F1/QA gaps are consistently positive. The human-consistency check in Sec. 9 uses only 100 images and ranks four model outputs, which is too weak to rule out reward hacking of FACTUAL's error patterns. The completeness-of-reference caveat in Sec. 3.3 is secondary: even if RefinedCaps references are complete, a model can improve parser-based scores by learning parser-friendly surface forms rather than better captions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SC-Captioner, a reinforcement-learning framework for training image-captioning LVLMs to self-correct their own outputs. The reward function parses initial and self-corrected captions into object, attribute, and relation sets using the FACTUAL scene-graph parser, then gives bonuses for additions/removals that match the reference and penalties for those that do not. The paper also introduces RefinedCaps, a 6.5K-image human-refined training set derived from COCO, and proposes refinements to the CAPTURE evaluation metric, including object/relation matching changes and a question-answering relation score. Experiments on LLaVA-1.5-7B and Qwen2-VL-7B compare SFT, SFT+DPO, and SFT+SC-Captioner on DOCCI500 and COCO-LN500, reporting improvements in CAPTURE and the proposed metrics. The central claim is that SC-Captioner significantly outperforms DPO for self-correction and improves caption quality on both initial and self-corrected outputs.","tokens_in":20579,"tokens_out":3258,"duration_ms":32730,"significance":"If the central claim holds, the paper makes a useful contribution: it demonstrates a training signal for self-correction in image captioning, introduces a new human-refined dataset, and proposes a more detailed evaluation protocol. The release of code and the human annotation effort are concrete strengths. However, the headline conclusion depends on the validity of the proposed metrics, and the reward function and those metrics are built from the same scene-graph parsing pipeline. Independent evaluation is therefore essential before the significance of the claimed improvement over DPO can be assessed.","major_comments":[{"comment":"The reward in §3.3 and the headline metrics in §4.2 share the same FACTUAL scene-graph parser, the same object/attribute/relation decomposition, and similar set-matching logic. Tables 1 and 2 show that SC-Captioner's advantage over SFT+DPO is positive across parser-based columns, but inconsistent on parser-independent BLEU-4/METEOR (e.g., LLaVA DOCCI500: BLEU-4 43.04 vs 42.67, METEOR 23.88 vs 23.47; Qwen COCO-LN500: BLEU-4 35.05 vs 30.34 but METEOR 27.34 vs 26.77). Consequently, the reported improvements may partly reflect the model learning to match FACTUAL's error patterns rather than producing better captions. Please evaluate with reference-free metrics that do not share the parser, and with an independent human preference study, before claiming a general advantage over DPO.","section":"§3.3 vs §4.2; Tables 1-2"},{"comment":"The human-consistency check uses only 100 images and 4 model outputs, and reports Kendall's tau without confidence intervals, inter-annotator agreement, or significance testing. This is too weak to rule out reward hacking of FACTUAL's specific failure modes. A larger evaluation (e.g., 300+ images, pairwise preferences, multiple annotators) is needed to establish that the proposed metric, and the improvements measured with it, align with human judgments of caption quality.","section":"§9, Table 4"},{"comment":"All experimental results are reported as single runs without error bars or significance tests. The test sets are small (500 images each), and COCO-LN500 is filtered to captions containing human-related words and at least 60 words. The abstract's 'significantly outperforming' claim is not supported by statistical evidence. Please provide standard deviations over multiple seeds and paired significance tests for the main SFT+DPO vs SFT+Ours comparisons.","section":"§5.1, Tables 1-2"},{"comment":"The reward formulation relies on the premise that reference captions are complete enough that penalizing additions absent from the reference is safe. The paper acknowledges in §3.3 that this may punish correct descriptions, and states that the phenomenon is 'less common' when GT is complete, but no quantitative evidence is provided. The paper's own discussion in §5.2 shows that shorter GT captions in COCO-LN500 cause CAPTURE to penalize correctly identified objects, so the concern is not hypothetical. Please quantify how often the reward punishes image-true but reference-absent additions, or modify the reward to avoid this asymmetry.","section":"§3.3, §5.2"}],"minor_comments":[{"comment":"The test set name is given as COCO-LR500 in the Introduction but COCO-LN500 in Tables 1-2; please make the naming consistent.","section":"Abstract/§1"},{"comment":"There are several typos and garbled formulas, e.g., 'challanges' in the Introduction and the corrupted math rendering in Eq. (1) and Eq. (4). Please proofread the equations and text.","section":"§1, §3.1"},{"comment":"The dataset construction states that 'at least 80% of the objects in the image should appear in the caption' but does not report how this was measured or verified. Clarify the procedure and annotator instructions.","section":"§4.1"},{"comment":"The comparison between DOCCI-trained and RefinedCaps-trained models is informative, but the discussion should avoid over-claiming generalization from a single cross-domain test set; the same statistical caveats as the main experiments apply.","section":"§11.1, Table 6"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the central issue is not that the method is wrong but that the evaluation and reward share a parser, making the headline advantage over DPO hard to interpret. Independent metrics and a stronger human study would address the main risk. If the authors can provide those experiments, the paper could become suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know about this paper because it is one of the first attempts to train LVLMs to self-correct image captions via multi-turn RL, and it does so with a reward that is actually tied to the semantics of the correction. The trick is to parse both the initial and corrected captions into scene graph elements, compute the set difference, then reward additions/removals that match the reference and punish the ones that do not. That is a sensible and novel reward design, and the ablations show each component earns its keep. The paper also introduces a human-refined dataset, RefinedCaps, and a set of proposed metrics, and the authors ship code.\n\nThe soft spot is real and it is in the evaluation loop. The reward and the headline metrics (the refined CAPTURE-style ones) both use the same FACTUAL parser and similar set-matching logic. So part of the measured gain could be the model learning to say things that please FACTUAL, not better captions. The stress-test note captures this well. The authors try to claim \"significantly outperforming DPO,\" but on parser-independent metrics like BLEU-4 and METEOR the edges are small and inconsistent across models and datasets. They do show a few positive numbers there, so the claim is not baseless, but it is not the robust story the abstract implies. The test sets are also just 500 images each, one filtered by word length and human words, and there are no error bars or significance tests. The completeness-of-reference assumption is acknowledged, but it remains a load-bearing premise.\n\nThat said, the paper is more honest than many in this area. It flags the GT dependence, the training cost, and the caveat on punishing correct additions. The supplementary experiments with DOCCI training and the human-in-the-loop metric check are useful, even if the human check is small. I would like to see the authors verify on independent metrics (like DC-Score or pure QA-based evaluation), add error bars, and ideally release RefinedCaps. Until then, treat the quantitative claims as promising but provisional.\n\nThis deserves a real peer review, not a desk reject. If I were an editor, I would send it out with a request for the circularity issue to be addressed. I would bring it to a reading group; the reward formulation is worth discussing even if the evaluation needs work. I would probably cite it for the method, not the numbers.\n\nRecommendation: engage with it, but hold the evaluation to a higher bar.","headline":"A genuinely new RL self-correction recipe for captioning with a sensible reward, but the headline gains may be partly an artifact of sharing the same scene-graph parser between reward and metrics.","tokens_in":20996,"tokens_out":2513,"would_cite":true,"duration_ms":27381,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SC-Captioner teaches vision-language models to fix their own captions with a scene-graph-based reward, beating DPO.","keywords":["image captioning","self-correction","reinforcement learning","reward design","scene-graph parsing","large vision-language models","direct preference optimization","refined evaluation metrics"],"falsifier":"Take a dataset with sparse or short reference captions, train with SC-Captioner, and have human annotators mark which model-added elements are actually visible in the image; if a large fraction of the punished additions are correct, the reward is miscalibrated. A simpler check: compute the correlation between the reward and human judgment of individual edits on a held-out set.","tokens_in":20094,"feed_emoji":"🖼️","tokens_out":6013,"duration_ms":63382,"temperature":0.7,"pith_summary":"This paper tries to establish that image-caption models can learn to improve their own first-pass captions if a reinforcement-learning reward is designed around the actual corrections the model makes. The reward decomposes captions into objects, attributes, and relations, then rewards additions and removals that move the caption closer to a reference while punishing those that move it away. Trained this way on a new 6.5K-image human-refined dataset, two 7B vision-language models outperform the same models trained with direct preference optimization on detailed caption metrics, both in natural and self-corrected captions. The paper also proposes a more consistent evaluation metric for detailed captions and a question-answering based relation score.","feed_headline":"A reward that scores caption edits beats DPO at self-correction","feed_subtitle":"Scene-graph grading of added and removed words beats direct preference training on caption quality.","key_machinery":"The machinery is a correction-based reward computed from scene-graph-parsed set differences. Objects, attributes, and relations are extracted from the initial and self-corrected captions; added and removed elements are identified by set difference and matched against the reference sets. Correct additions and removals earn a bonus; wrong ones incur a punishment. This reward is plugged into a multi-turn policy-gradient objective, with a policy loss on the second-turn self-corrected caption plus a KL term preserving the initial caption, so the model learns when and what to change.","core_discovery":"The central claim is that self-correction is a learnable behavior and that the right training signal is not a preference over whole captions but a fine-grained reward over caption edits. Using a scene-graph parser, the method represents initial, corrected, and ground-truth captions as sets of objects, attributes, and relations. Set differences isolate what the model added or removed; each such element is matched against the reference sets to give a correctness bonus for accurate refinements and a mistake punishment for wrong insertions and deletions. On DOCCI500 and COCO-LN500, models trained with this reward show higher object and attribute recall and relation QA accuracy than DPO, which te","pith_inferences":["Because the reward is defined over edit differences rather than full-caption quality, the same formulation could be adapted to other editing tasks (e.g., text-to-image correction or summarization revision) where a reference is available; the paper does not investigate this extension.","The dependence on scene-graph parser output means false parses directly corrupt rewards; using a higher-quality parser or combining multiple parses could improve stability further.","A direct test of the completeness assumption would be to apply the same training to a dataset with deliberately sparse references and check whether the model's correct additions get penalized; the paper's own caveat in Sec. 3.3 suggests this is the main failure mode.","The method could be combined with data augmentation that synthesizes incomplete initial captions, potentially reducing the cost of collecting paired initial/corrected data."],"forward_implications":["Applied as a post-training step after supervised fine-tuning, the method lifts CAPTURE, object and attribute recall, and relation QA scores over SFT alone on two 7B vision-language models.","The self-corrected captions produced by the trained models score higher than their initial captions, while untrained or SFT-only models' corrections do not.","Training on a public dataset such as DOCCI instead of the new RefinedCaps still shows the same advantage over SFT and DPO in same-domain and cross-domain tests.","The refined evaluation metrics, including a question-answering based relation score, correlate better with human judgment than the original CAPTURE metric, offering a more reliable reading of detailed caption quality."],"supporting_citations":[{"why":"Supplies the FACTUAL scene-graph parser that decomposes captions into objects, attributes, and relations, the basis of the reward and the evaluation metrics.","marker":"[25]"},{"why":"Defines direct preference optimization, the main comparison baseline for self-correction training.","marker":"[40]"},{"why":"Gives the multi-turn reinforcement learning objective with policy-gradient and KL regularization that SC-Captioner adapts to image captioning.","marker":"[20]"},{"why":"Introduces CAPTURE, the reference-based metric whose object/attribute/relation scoring is refined and whose problems motivate the new metrics.","marker":"[8]"},{"why":"Provides the COCO 2017 train split from which the RefinedCaps 6.5K images are sampled.","marker":"[4]"},{"why":"DOCCI is the source of the DOCCI500 test set and of the public training data used in cross-domain checks.","marker":"[33]"},{"why":"Localized Narratives is the source of the COCO-LN500 test set used for the human-centric evaluation scenario.","marker":"[36]"},{"why":"Qwen2-VL-7B is one of the two base vision-language models trained and evaluated.","marker":"[50]"},{"why":"LLaVA-1.5-7B is the other base model trained and evaluated.","marker":"[29]"}],"fun_headline_variants":["Edit-level rewards make caption models self-correct better than DPO","Reward the right edits: caption self-correction beats DPO","Scene-graph-graded edit rewards outperform DPO for captions","Learn to fix captions: edit-level rewards beat DPO"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"Training assumes the reference captions are complete enough that an element absent from the reference but present in the image is rare, so punishing such additions is safe.","fun_headline_variants_meta":{"raw":{"variants":["Edit-level rewards make caption models self-correct better than DPO","Reward the right edits: caption self-correction beats DPO","Scene-graph-graded edit rewards outperform DPO for captions","Learn to fix captions: edit-level rewards beat DPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0006,"raw_usage":{"total_tokens":2623,"prompt_tokens":711,"completion_tokens":1912,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":1838}},"tokens_in":455,"tokens_out":1912,"duration_ms":15973,"temperature":1.0,"reasoning_tokens":1838,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:54:04.682466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset with sparse or short reference captions, train with SC-Captioner, and have human annotators mark which model-added elements are actually visible in the image; if a large fraction of the punished additions are correct, the reward is miscalibrated. A simpler check: compute the correlation between the reward and human judgment of individual edits on a held-out set.","supporting_citations":[{"cited_title":"Direct preference optimization: Your language model is secretly a reward model","cited_arxiv_id":null,"evidence_quote":"Defines direct preference optimization, the main comparison baseline for self-correction training."},{"cited_title":"Connecting vision and lan- guage with localized narratives","cited_arxiv_id":null,"evidence_quote":"Localized Narratives is the source of the COCO-LN500 test set used for the human-centric evaluation scenario."},{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"LLaVA-1.5-7B is the other base model trained and evaluated."}],"review_version":1}