{"id":"5e4db691-7856-4322-8edf-92cedc1a86d2","arxiv_id":"2507.07340","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A contrastive reinforcement learning method with synthetic negative stories improves cross-frame entity grounding and re-identification for a 7B visual storyteller, evaluated only on the authors' own dataset.","lead":"The authors fine-tune a Qwen2.5-VL based storytelling model with a contrastive DPO scheme that adds synthetic negative stories, teaching the model when not to link entities across frames. The method shows improved grounding and cross-frame entity persistence scores on the authors' own Story Reasoning dataset, though the evaluation shares the same annotations and metrics used to compute the training reward.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 3 rewards persistent entity labels in the generated CoT, not matches to true visual identity; the reported persistence and grounding gains may be label reuse and tag inflation rather than genuine re-identification.","rationale":"The reader's weakest assumption correctly targets evaluation validity through annotation reliability, but the deeper problem is internal: the reward and the headline metric do not measure identity at all. Rreid is a label-persistence count over the model's own CoT, and Rgrounding is a tag-coverage ratio; both admit straightforward reward hacking. The model can increase F1 and persistence by emitting more entity tags and reusing IDs, and the precision/recall pattern in Table 1 is consistent with that. This does not require the Story Reasoning annotations to be wrong, making it a more load-bearing concern than the dataset-quality issue alone. I grant the paper's positive points: the equations are internally consistent, the synthetic-negative idea is plausible, and the authors release models and data. Those do not overcome the metric-alignment problem. The absence of error bars, the missing full-fine-tuning results, and the lack of any human verification of entity identity further weaken the empirical support. The central empirical claim is therefore not established, so rejection remains the right verdict.","tokens_in":10379,"tokens_out":6888,"duration_ms":82928,"concrete_test":"Sample 100 real test stories from the Story Reasoning split; for each, have two annotators label whether each model-assigned entity ID (e.g., 'char1', 'obj2') in the generated CoT refers to the same physical person/object across every frame in which it appears, using full-image context. Compare the verified correct-persistence rate (fraction of entity tracks that are truly one visual entity) for the baseline and the LoRA R=2048 DPO model. If raw persistence rises while verified correct-persistence is flat or declines, the reported re-identification gain is label reuse, not genuine entity re-identification.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that contrastive DPO improves cross-frame entity re-identification and grounding. The weakest load-bearing step is the definition of the re-identification reward in Eq. 3. Rreid is computed from the model's generated CoT entities: it is the average number of frames per detected character/object label, normalized by the frame count. It never checks that two occurrences of the same label correspond to the same physical entity, nor that distinct labels are distinct people/objects. A model that emits one 'char1' in every frame of every real story receives Rreid = 1 without any visual evidence. The headline persistence metric in Fig. 1 is the same kind of label persistence in the generated text, so the abstract's 'cross-frame persistence' gains can be manufactured by reusing entity IDs. The grounding reward in Eq. 5 similarly counts whether pronouns/proper nouns are tagged with entity IDs, so tagging every referential expression improves it; this is consistent with the observed precision drop (0.57 to 0.45) and recall rise (0.40 to 0.48). Thus the reported improvements may be an artifact of the optimized reward, not of better identity tracking. The reader's annotation-reliability concern is valid and compounds the problem, but even with perfect annotations the current reward and evaluation would not establish the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a contrastive reinforcement learning framework to improve entity re-identification and grounding in visual storytelling. It extends the Story Reasoning dataset with synthetic negative stories, designs a dual-component rule-based reward function (re-identification and grounding), and fine-tunes the Qwen Storyteller model (Qwen2.5-VL 7B) using Direct Preference Optimization. The authors report improvements over a supervised fine-tuned baseline, including grounding mAP from 0.27 to 0.31 (+14.8%), F1 from 0.35 to 0.41 (+17.1%), and increased cross-frame entity persistence. The central claim is that contrastive RL with synthetic negatives teaches the model when to establish entity connections across frames, improving genuine re-identification and grounding.","tokens_in":10635,"tokens_out":5825,"duration_ms":57931,"significance":"The problem of cross-frame entity consistency in visual storytelling is timely and important, and the idea of using synthetic negative examples with DPO is a plausible and interesting direction. The paper is clearly written and the authors release the trained models and synthetic dataset, which supports reproducibility. However, the evaluation is largely circular with the reward objective, and the reward itself measures label persistence rather than verified visual identity. As a result, the reported improvements do not convincingly establish the central claim. The lack of statistical validation and the omission of the promised full fine-tuning experiment further weaken the evidence. If the core issues were addressed, the contrastive framework could be a useful contribution, but in its current form the results are not reliable.","major_comments":[{"comment":"The re-identification reward R_reid is computed as the average number of frames per entity label in the model's generated chain-of-thought, normalized by the total frame count. It never verifies that two occurrences of the same label correspond to the same physical entity, nor that different labels refer to distinct identities. A degenerate policy that reuses a single 'char1' label in every frame scores R_reid = 1 on real stories with no visual evidence. Consequently, the persistence gains reported in Fig. 1 and Table 2 can be manufactured by label reuse rather than by genuine cross-frame re-identification, which is the central claim of the paper.","section":"Section 3.2.2, Eqs. (3)-(4)"},{"comment":"The evaluation metrics (mAP, precision, recall, F1, persistence) are computed against the Story Reasoning annotations, which are the same annotations used to define the reward function in Eqs. (1)-(5). The reward is computed from the model's own generated CoT and story using these annotations, and the evaluation measures the same persistence and grounding quantities against the same ground-truth source. This circularity means the reported improvements may simply reflect the model learning to reproduce the annotation conventions rather than learning to track visual entities. The paper itself acknowledges in Section 5 that the underlying bounding boxes are not validated, further compromising both the reward signal and the evaluation.","section":"Section 4.1 and Section 3.2"},{"comment":"The text states that a second experiment uses full fine-tuning to assess the impact of training all model parameters, but Table 1 reports only the baseline and LoRA variants (R=512, 1024, 2048). The full fine-tuning results are absent from the table and from the rest of the paper. This omission makes it impossible to verify the authors' claim that the approach is effective beyond parameter-efficient fine-tuning, and it leaves a described experiment unreported.","section":"Section 3.3 and Table 1"},{"comment":"No error bars, confidence intervals, or statistical significance tests are reported anywhere in the evaluation. All metrics are single-run point estimates, and several differences are very small (e.g., BLEU-4 0.054 vs. 0.057, ROUGE-L 0.16 vs. 0.18). Without any measure of variance, the robustness of the claimed improvements cannot be assessed, especially given the multiple free parameters (alpha, beta, gamma, delta, DPO temperature, real-to-synthetic ratio, preference threshold, etc.) that are not varied in a sensitivity analysis.","section":"Section 4"}],"minor_comments":[{"comment":"The paper states a 2:1 ratio of real to synthetic stories, but later says the synthetic construction creates 4,178 synthetic stories alongside 4,178 real stories, which is a 1:1 ratio. Please clarify the intended ratio.","section":"Section 3.1"},{"comment":"The text refers to 'LoRA rank 2028' when comparing with the baseline, but Table 1 lists the LoRA rank as 2048. This appears to be a typo.","section":"Section 4.1"},{"comment":"The sentence 'Both experiments employee the temperature parameter' should read 'employ' instead of 'employee'.","section":"Section 3.3"},{"comment":"The caption states that best and worst values are highlighted, but the table as rendered in the manuscript shows no highlighting. Please add highlighting or remove the statement.","section":"Table 1 caption"},{"comment":"The discussion of Table 2 is ambiguous: for synthetic stories, a lower R_reid is the desired outcome (since the reward is 1.0 - (alpha*R_char + beta*R_obj)), yet the authors say the Rank 2048 model's decrease from 0.72 to 0.67 indicates the model 'would benefit from more training on negative stories.' Clarify the desired direction and interpretation.","section":"Section 4.4"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim is undermined by the circularity between the reward design and the evaluation, and by the re-identification reward's inability to distinguish true identity tracking from label reuse. These are fundamental issues that cannot be resolved by superficial editing; the evaluation would need to be rebuilt around independently validated ground truth and a reward that checks visual identity. The missing full fine-tuning results and lack of statistical testing add to the concern. I recommend rejection, though the authors could return with a substantially revised evaluation and reward."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one new thing here is a contrastive DPO setup for cross-frame entity re-identification in visual storytelling: synthetic negative stories, a rule-based reward with re-id and grounding components, and DPO fine-tuning of a Qwen2.5-VL storyteller. That's a sensible combination and the authors ship models and data, which is real credit. The writing is clear and the method section is easy to follow.\n\nThe problem is that the evaluation doesn't actually measure the thing the title promises. Equation 3 computes Rreid as the average number of frames per detected character/object label, normalized by frame count. It never checks that two occurrences of the same label are the same physical entity, or that distinct labels are distinct people/objects. A model that emits 'char1' in every frame scores full marks with zero visual evidence. Fig. 1's persistence curves are built the same way, so the abstract's 'cross-frame persistence' gains can be manufactured by reusing IDs. The grounding reward in Eq. 5 rewards tagging pronouns/proper nouns with entity IDs, so the observed precision drop (0.57 to 0.45) and recall rise (0.40 to 0.48) are consistent with tag inflation, not better grounding.\n\nThe reader's circularity concern is compounded: training reward and evaluation both consume the same Story Reasoning annotations, which the paper admits rely on visual similarity within cropped boxes and are not validated for box-to-object correspondence. With perfect annotations the current metrics still wouldn't establish re-identification, because Rreid never uses the boxes at all.\n\nOther soft spots are proportionally minor but add up: no error bars or significance tests; the full fine-tuning experiment mentioned in Sec. 3.3 never appears in Table 1; Table 2 shows the model's synthetic-story control actually worsened (Rreid 0.72 to 0.67, lower is better), undercutting the contrastive story; and no comparison with TARN-VIST or CharGrid is attempted. The language metric gains are also tiny, though at least they aren't the load-bearing claim.\n\nThis is not a takedown. The idea is plausible and the paper is honest about some limitations, but as it stands the reported improvements are not trustworthy. Give it a serious referee: a good reviewer could demand identity-aware metrics, error bars, the missing full fine-tuning numbers, and a comparison to at least one existing method. I wouldn't cite the numbers until that's done, but the framework is worth engaging with.","headline":"Plausible contrastive-DPO idea for visual storytelling, but the reward and evaluation both measure label persistence rather than true re-identification, so the headline gains aren't established.","tokens_in":11202,"tokens_out":1581,"would_cite":false,"duration_ms":19599,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a visual storyteller on deliberately incoherent image sequences improves cross-frame entity re-identification and grounding.","keywords":["visual storytelling","entity re-identification","cross-frame consistency","contrastive reinforcement learning","direct preference optimization","synthetic negative stories","entity grounding","pronoun grounding"],"falsifier":"Evaluate the same contrastive RL model on a subset of Story Reasoning where entity IDs and bounding boxes have been manually corrected or where re-identification uses whole-image context: if the reported mAP and F1 gains shrink or disappear, the improvement is an artifact of noisy automatic annotations rather than genuine entity re-identification. A second decisive test is to replace the cross-movie synthetic negatives with shuffled-frame sequences from the same movie; if the gains vanish, the model learned a movie-level incoherence cue rather than entity-level discrimination.","tokens_in":10127,"feed_emoji":"🎬","tokens_out":12536,"duration_ms":111964,"temperature":0.7,"pith_summary":"The paper claims that a visual storytelling model can learn when to connect characters and objects across frames if it is trained not only on coherent stories but also on deliberately incoherent ones. It constructs synthetic negative stories by sampling frames from unrelated movies, then fine-tunes a Qwen2.5-VL 7B-based storyteller with Direct Preference Optimization—a training method that optimizes a policy directly from preference pairs—under a dual reward that rewards grounding and cross-frame re-identification on real stories and penalizes both on synthetic ones. Against a supervised fine-tuned baseline, grounding mAP rises from 0.27 to 0.31, grounding F1 from 0.35 to 0.41, pronoun grounding improves for nearly every pronoun type except \"its\", and the share of entities appearing in five or more frames rises from 29.3% to 33.3%. The authors take this as evidence that explicit positive/negative training on entity connectivity reduces referential inconsistency and hallucination in visual storytelling.","feed_headline":"Unrelated movie frames teach a storyteller to keep characters straight","feed_subtitle":"Contrastive preference training lifts grounding F1 by 17 percent and pronoun accuracy in a 7B storyteller.","key_machinery":"The load-bearing mechanism is contrastive Direct Preference Optimization over synthetic negative stories. The paper extends the Story Reasoning dataset with 4,178 synthetic stories built by deterministically sampling 5 to 15 frames from far-apart real stories, giving a 2:1 ratio of real to synthetic stories. The reward function combines an entity re-identification component, which measures how many frames each detected character and object persists across (characters weighted 0.6, objects 0.4), with a grounding component, which measures the fraction of pronouns and proper nouns placed inside entity tags; the re-identification term is inverted for synthetic stories so that connecting entities is penalized, and structurally invalid outputs receive $-1.0$. Direct Preference Optimization converts this reward into offline preference pairs, requiring the chosen response to beat the rejected one by at least 0.05, and optimizes the policy directly.","core_discovery":"The central claim is that cross-frame entity re-identification in visual storytelling is a trainable behavior, not an emergent byproduct of scale or language modeling. A model that is explicitly rewarded for linking entities in coherent sequences and penalized for linking them in incoherent sequences learns to persist characters and objects across frames: the authors report grounding mAP rising from 0.27 to 0.31, grounding F1 from 0.35 to 0.41, pronoun grounding gains for every pronoun type except \"its\", and cross-frame entity persistence increasing across all frame counts, with entities appearing in five or more frames rising from 29.3% to 33.3%. The authors also report that well-structured stories containing both chain-of-thought and grounded story increased from 79.1% to 97.5%, and that standard language metrics did not degrade.","pith_inferences":["The authors do not test same-movie shuffled frames as negatives; because their synthetic stories come from different movies, the model may be learning a coarse \"different movie means no connection\" heuristic rather than fine-grained entity discrimination. Shuffled same-movie frames would separate those two explanations.","The reward and evaluation both rely on Story Reasoning annotations whose re-identification was itself based on visual similarity within cropped bounding boxes; training on that signal may entrench the context-blindness the authors identify, such as two same-colored cars from different scenes. A human-verified subset of boxes and IDs would reveal whether the gains survive cleaner supervision.","Table 2 shows the best model still scores 0.67 on synthetic-story re-identification versus 0.72 for the baseline, so it continues to connect entities in incoherent sequences; the reward analysis suggests additional training on negatives or harder negatives could push this discrimination further.","The framework of synthetic negatives plus a dual rule-based reward is directly portable to video captioning and visual question answering, which the authors list as future work, though the format tags and story-specific structure validation would need to be redefined for those tasks."],"forward_implications":["Adding synthetic negative stories and contrastive DPO to a 7B visual storyteller improves grounding mAP by 14.8% and grounding F1 by 17.1% over the supervised fine-tuned baseline.","Cross-frame entity persistence improves at every frame count, with the share of entities appearing in five or more frames rising from 29.3% to 33.3%, indicating more reliable re-identification.","Pronoun grounding improves across nearly all pronoun types, most sharply for gender-specific \"he\" (to 99.1%) and \"she\" (to 98.6%), reducing ambiguous references.","Well-structured outputs containing both chain-of-thought and grounded story rise from 79.1% to 97.5%, showing the reward's structural validation also shapes output compliance.","The contrastive framework teaches the model when not to establish cross-frame connections, targeting the false connections that arise when visually similar entities appear in unrelated images."],"supporting_citations":[{"why":"Supplies the Story Reasoning dataset with entity IDs, bounding boxes, chain-of-thought tables, and the adapted mAP metric used both as the training reward signal and as evaluation ground truth.","marker":"Oliveira and de Matos, 2025"},{"why":"Provides the ID-based grounding tag system (<gdi>, <gdo>, <gda>, <gdl>) that Story Reasoning and this paper's grounding reward reuse.","marker":"Oliveira et al., 2025"},{"why":"Direct Preference Optimization is the training objective that turns the contrastive reward into preference pairs and updates the policy without a separate reward model.","marker":"Rafailov et al., 2023"},{"why":"Motivates the rule-based reward design used here to avoid reward hacking by neural reward models.","marker":"DeepSeek-AI et al., 2025"},{"why":"LoRA is the parameter-efficient fine-tuning method used to build Qwen Storyteller and to run the contrastive RL experiments at rank 2048.","marker":"Hu et al., 2022"}],"fun_headline_variants":["Contrastive RL stops storytellers from losing track of who's who","Rewarding entity re-ID boosts visual story consistency","Contrastive training teaches models to re-identify characters","Contrastive RL lifts grounding F1 by 17% in storytellers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Story Reasoning annotations—entity identifiers, bounding boxes, and the adapted mAP computation—are accurate enough to serve both as the reward that trains the model and as the ground truth that measures improvement; the paper itself notes that re-identification was based on visual similarity within cropped boxes and that bounding boxes were not validated.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive RL stops storytellers from losing track of who's who","Rewarding entity re-ID boosts visual story consistency","Contrastive training teaches models to re-identify characters","Contrastive RL lifts grounding F1 by 17% in storytellers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001064,"raw_usage":{"total_tokens":4475,"prompt_tokens":976,"completion_tokens":3499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":3426}},"tokens_in":592,"tokens_out":3499,"duration_ms":26490,"temperature":1.0,"reasoning_tokens":3426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:43:19.967290+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the same contrastive RL model on a subset of Story Reasoning where entity IDs and bounding boxes have been manually corrected or where re-identification uses whole-image context: if the reported mAP and F1 gains shrink or disappear, the improvement is an artifact of noisy automatic annotations rather than genuine entity re-identification. A second decisive test is to replace the cross-movie synthetic negatives with shuffled-frame sequences from the same movie; if the gains vanish, the model learned a movie-level incoherence cue rather than entity-level discrimination.","supporting_citations":[{"cited_title":"D., and Finn, C","cited_arxiv_id":null,"evidence_quote":"Direct Preference Optimization is the training objective that turns the contrastive reward into preference pairs and updates the policy without a separate reward model."}],"review_version":1}