{"id":"023441d0-298e-4c84-87c1-3f11368b0263","arxiv_id":"2505.10292","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new multi-frame visual storytelling dataset with explicit entity grounding, plus a fine-tuned Qwen2.5-VL baseline that reduces measured hallucinations by 12.3%.","lead":"The paper introduces StoryReasoning, a dataset of 4,178 movie-image stories with entity-level grounding, and a fine-tuned Qwen2.5-VL baseline called Qwen Storyteller. The authors report 12.3% fewer hallucinations and 31% more creative stories than the base model, though the evaluation relies on LLM judges with low agreement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline hallucination and creativity gains rest entirely on LLM-judge scores with Krippendorff alpha as low as 0.14; no significance test, confidence interval, or human validation is provided, so the 12.3% and 31% deltas are not yet established.","rationale":"The reader identified exactly the same load-bearing weakness: the headline numbers come from LLM judges with low inter-evaluator agreement and no statistical validation. I read the full manuscript in good faith and find the dataset release, the fine-tuned model release, the documented prompts, and the interactive visualization to be real contributions. The automatic grounding metrics in Table 1 are interesting, but they do not compare against the base model, so they cannot support the headline claim. The paper's own Section 6 acknowledges that LLM-based assessments may contain biases, which further underlines that the evaluation is not secure. A reviewer should not reject the paper on this basis alone, because the concern is addressable with additional analysis or a small human study, and the dataset itself is likely useful to the community. However, the central empirical claim should not be treated as established until the suggested test is run. Since the reader's CONDITIONAL verdict already captures this, no verdict change is needed.","tokens_in":20358,"tokens_out":4772,"duration_ms":48593,"concrete_test":"Run a blinded, paired evaluation on a random subset of at least 100 of the 626 image sequences: have the same four LLM judges score both the Qwen2.5-VL 7B and Qwen Storyteller outputs with model identity hidden and order randomized, and have at least 3 human annotators apply the same hallucination taxonomy to the same subset. Report per-story paired differences, Krippendorff's alpha for humans and for each LLM, and a two-sided Wilcoxon signed-rank test with a 95% bootstrap confidence interval for the mean difference in total hallucinations and creativity. If the human-annotated difference is not statistically significant or the confidence interval includes zero, the headline claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Sections 5.2-5.3, Tables 2-3) is that fine-tuning reduces hallucinations from 4.06 to 3.56 (-12.3%) per story and raises creativity from 2.58 to 3.38 (+31.0%) over the non-fine-tuned baseline. These are averages over four LLM judges (Claude 3.7 Sonnet, ChatGPT-4o, Gemini Flash 2.5, Qwen 2.5-VL 72B), but the paper's own inter-evaluator reliability metrics are low: Krippendorff's alpha is 0.14 for description accuracy, 0.20 for creativity, 0.34 for hallucination rate, and 0.24-0.26 for grounding recall/precision (Table 3). An alpha of 0.14 for description accuracy means the judges' ratings are nearly unrelated across raters; the reported mean differences could reflect systematic judge bias or noise rather than true quality differences. No confidence intervals, paired significance tests, effect-size distributions, or human annotation are reported. Appendix B shows only aggregate histograms over N=626 stories per model, not per-judge or per-category breakdowns, so it is impossible to tell whether the reduction is consistent across stories and judges or driven by a few outliers. Additionally, the relationship-hallucination category actually increases after fine-tuning (0.60 to 0.63), so the total reduction depends entirely on object and attribute categories for which no inter-rater agreement is reported. Since the paper's headline claim is the only quantitative evidence that the fine-tuned model improves on the base model, this measurement gap is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces StoryReasoning, a dataset of 4,178 grounded visual stories derived from 52,016 movie frames, with structured scene analysis (chain-of-thought) and XML-based grounded narratives. The authors also present Qwen Storyteller, a Qwen2.5-VL 7B model fine-tuned on this dataset. The central empirical claim is that fine-tuning reduces average hallucinations per story from 4.06 to 3.56 (-12.3%) and improves creativity from 2.58 to 3.38 (+31.0%) relative to the non-fine-tuned Qwen2.5-VL 7B model, based on ratings from four LLM judges.","tokens_in":20875,"tokens_out":5625,"duration_ms":55895,"significance":"If the empirical claims are established, the dataset and model release would be a useful contribution: the dataset is publicly available, contains multi-frame entity tracking with grounding tags, and the fine-tuned model is also released. The structured tabular CoT representation and the explicit grounding scheme are sensible design choices, and the paper's automatic-metric results across several training configurations provide useful baseline evidence. The main quantitative evidence for the headline improvement, however, rests entirely on LLM-judge scores with reported inter-rater reliability as low as alpha = 0.14, no human validation, and no significance testing. The central claim is therefore plausible but not yet established to the standard required for publication.","major_comments":[{"comment":"The headline deltas (-12.3% hallucinations, +31.0% creativity) are averages over four LLM judges whose inter-rater reliability is very low: Krippendorff's alpha is 0.14 for description accuracy, 0.20 for creativity, and 0.34 for hallucination rate. An alpha of 0.14 indicates agreement barely above chance, and the reported model differences are within the range one would expect from judge noise. The paper gives no confidence intervals, paired significance tests, effect-size distributions, or human validation. Without these, the claim that fine-tuning reduces hallucinations and improves creativity is not established.","section":"§5.2, Tables 2 and 3"},{"comment":"The reduction in total hallucinations from 4.06 to 3.56 is not uniform across categories: relationship hallucinations increase from 0.60 to 0.63 after fine-tuning, while object and attribute hallucinations decrease. Since the paper motivates the work by referential hallucinations in Section 1, the increase in the relationship category, which includes incorrect actions or interactions between entities, undercuts that motivation. Moreover, no inter-rater reliability is reported for the object and attribute categories that actually drive the decrease, so the reliability of the category-level decomposition is unknown.","section":"Table 2, hallucination categories"},{"comment":"The hallucination histograms in Appendix B are based on N=626 stories per model, but the main text never states that the LLM evaluation was performed on a subset of the 4,178 stories, nor how that subset was selected. If the subset is not a random sample, the average counts in Table 2 may not generalize to the full dataset. The subset size, sampling procedure, and per-judge score distributions should be reported in the main text or in the appendix.","section":"Appendix B, evaluation subset"},{"comment":"Table 1 reports automatic grounding metrics (Precision, Recall, mAP, METEOR, ROUGE-L, BLEU-4) only for the four fine-tuned configurations; there is no row for the non-fine-tuned Qwen2.5-VL 7B baseline. Consequently, the claim that the fine-tuned model improves grounding over the base model is not supported by the automatic metrics. The only comparison against the base model is the low-reliability LLM evaluation in Table 2. Adding the base model to Table 1, or explaining why it cannot be evaluated with the same automatic pipeline, is necessary.","section":"§5.1, Table 1"},{"comment":"The evaluation protocol for the LLM judges is not fully specified. The paper does not provide the exact prompts or rubrics used for description accuracy, creativity, and the four hallucination categories, nor does it give per-judge score distributions. Without these, the reader cannot reproduce the numbers or assess whether the judges were instructed to count hallucination types consistently. This is a reproducibility gap for the paper's only quantitative comparison against the base model.","section":"§5.2, evaluation protocol"}],"minor_comments":[{"comment":"The text says 'agreement between the three LLM evaluators,' but Section 5.2 lists four evaluators. Please clarify how many judges were used and over which judges the alpha coefficients were computed.","section":"§5.3, Table 3"},{"comment":"The comparison in Table 2 does not report decoding hyperparameters (temperature, top-p, max tokens) for either model. Sampling temperature affects creativity and hallucination counts, so these settings should be specified for reproducibility.","section":"§5.2, decoding settings"},{"comment":"The figure showing 100% of stories containing all five narrative phases reflects the dataset construction rule rather than an emergent property; the text should state explicitly that this is a design constraint, not an empirical finding.","section":"Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The dataset and model release are potentially valuable, but the paper's central quantitative claim is supported only by LLM-judge scores with low inter-rater reliability and no statistical validation. The paper also builds heavily on the authors' own prior GroundCap dataset; the novelty is the multi-frame extension and the trained model. For a journal in this area, a human-validated evaluation or at least a properly powered machine evaluation with significance tests and a full baseline comparison is required before the headline claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my read of the StoryReasoning paper. The dataset is the real contribution; the headline empirical claims are not yet established.\n\nWhat is new: StoryReasoning is, to my knowledge, the first public multi-frame storytelling dataset with entity-level grounding—4,178 stories from 52,016 movie frames, with cross-frame character/object IDs, structured scene tables, and stories carrying XML grounding tags. The construction pipeline (Mask2Former detections, SigLIP + ArcFace re-identification with adaptive thresholds, landmark fallback, validation loop for tag/box consistency) is clearly documented, and the dataset statistics—especially the pronoun-grounding analysis showing that first/second-person pronouns are mostly ungrounded because they sit in dialogue—are genuinely informative. The Qwen Storyteller baseline is also shipped, so the work is reproducible.\n\nWhere it wobbles: the paper's central claim that fine-tuning reduces hallucinations by 12.3% and raises creativity by 31% rests on LLM-judge scores with low inter-evaluator agreement (Krippendorff's alpha 0.14 for description accuracy, 0.20 for creativity, 0.34 for hallucination rate). Those numbers are averages over four judges; there are no confidence intervals, no paired significance tests, and no human validation. Appendix B only gives aggregate histograms, not per-judge breakdowns. And the relationship-hallucination category actually worsens after fine-tuning (0.60 to 0.63), so the total reduction depends on object and attribute categories that carry the lowest reliability. That makes the 12.3% figure load-bearing and unsupported as reported. The comparison is also only against the non-fine-tuned base model, so we do not learn whether the gains come from the CoT structure, the grounding tags, or simply from more training. The automatic precision/recall numbers are computed against reference stories, which conflates grounding with lexical similarity.\n\nThat said, the paper is honest about its limitations, and the dataset artifact itself is solid and will be useful to anyone working on visual storytelling or multimodal grounding. The evaluation section needs major work before the performance claims can be believed, but the resource deserves a serious referee. I would send it to review, with the expectation that a revised version needs human evaluation, per-judge statistics, and ideally an ablation of the CoT and grounding components.","headline":"A genuinely useful grounded-story dataset and reproducible baseline, undermined by an evaluation whose headline hallucination and creativity numbers come from unreliable LLM judges and need proper statistics before being believed.","tokens_in":21240,"tokens_out":2586,"would_cite":false,"duration_ms":25485,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a vision-language model on a dataset of 4,178 grounded movie-image stories reduces hallucinations per story by 12.3% and raises creativity by 31% compared with the base model.","keywords":["visual storytelling","chain-of-thought reasoning","grounded story generation","cross-frame object re-identification","hallucination reduction","vision-language models","dataset","scene understanding"],"falsifier":"Have human annotators count hallucinations and rate creativity on the same 626 generated stories per model; if the human-rated reduction in hallucinations or increase in creativity is absent or much smaller than 12.3% and 31%, the central claim fails. Alternatively, recompute the four LLM judges' per-story scores into a paired confidence interval: if the interval for the mean difference includes zero, the claimed improvement is not statistically supported.","tokens_in":20163,"feed_emoji":"🎬","tokens_out":9620,"duration_ms":82737,"temperature":0.7,"pith_summary":"This paper tries to establish that visual storytelling can become both less hallucinated and more creative when every narrative reference is explicitly anchored to a visual entity and the model is forced to reason about the scene in structured steps first. It introduces StoryReasoning, a dataset of 4,178 stories built from 52,016 movie frames, in which characters and objects carry consistent IDs across frames and each story is paired with a structured scene analysis (character, object, setting, and narrative-phase tables). The paper then fine-tunes Qwen2.5-VL 7B on this data to produce Qwen Storyteller, a single model that does detection, re-identification, and grounded story generation end-to-end. The headline result is a reduction from 4.06 to 3.56 hallucinations per story (-12.3%) and a rise in creativity from 2.58 to 3.38 (+31.0%) relative to the non-fine-tuned model. A reader should care because referential hallucination—describing people, objects, or places that are not actually in the images—remains a known failure mode of large vision-language models, and the paper offers a concrete training recipe aimed directly at it.","feed_headline":"Fine-tuned vision model cuts story hallucinations by 12%","feed_subtitle":"Grounded chain-of-thought training also lifts creativity scores 31 percent over the base.","key_machinery":"The load-bearing mechanism is a three-stage grounding pipeline. First, cross-frame object re-identification assigns persistent global IDs (char1, obj1, etc.) by combining masked-object visual embeddings from SigLIP with ArcFace face embeddings for people, using adaptive thresholds rather than motion prediction. Second, a chain-of-thought scene analysis renders each story as hierarchical tables—characters, objects, settings, and a five-phase narrative structure—so the model reasons about relations across frames before writing. Third, grounded story generation wraps every textual reference in XML tags (<gdo> for entities, <gda> for actions, <gdl> for locations and landmarks, <gdi> for image boundaries) that map back to those IDs, so even pronouns are traceable to a specific visual entity. The dataset supplies the ground-truth entity correspondences, and the fine-tuned model learns to reproduce the entire chain end-to-end.","core_discovery":"The central claim is that fine-tuning a 7-billion-parameter vision-language model on grounded, multi-frame stories yields measurably better visual storytelling: average hallucinations per story fall from 4.06 to 3.56 (-12.3%), creativity rises from 2.58 to 3.38 (+31.0%), and description accuracy edges up from 2.69 to 2.76, all relative to the same model without fine-tuning. The improvement is attributed to the training signal in StoryReasoning, where each story is decomposed into a chain-of-thought scene analysis and then generated with XML grounding tags that tie every character, object, action, and location mention—including pronouns—to entity IDs that persist across frames. The fine-tuned model performs object detection, cross-frame re-identification, and landmark recognition end-to-end, and it is evaluated by four large language model judges using a four-way hallucination taxonomy (object, attribute, relationship, environment) as well as automatic grounding metrics.","pith_inferences":["If the recipe transfers, the same entity-ID plus tag-grounding scheme could be applied to video captioning or embodied instruction tasks where referential consistency is scored; a cheap first test is to prompt a general vision-language model with the StoryReasoning format on an existing video-description benchmark.","Because the dataset is assembled from cinematic scenes with a fixed five-phase narrative arc, the creativity gains may partly encode movie conventions rather than general narrative skill; on candid, unstructured image sequences the gap might shrink.","The near-flat relationship-hallucination numbers point to the next bottleneck: relational grounding. Explicitly tagging interactions (subject, verb, object) rather than only entities could be a testable extension of the scheme.","Since no human evaluation is reported anywhere in the paper, the headline gains rest entirely on LLM judges; a human rating of the same 626 stories per model would settle whether the 31% creativity increase is perceptible to people."],"forward_implications":["A single model can detect, re-identify, and describe objects and characters across discontinuous movie frames without a separate detector or tracker, since Qwen Storyteller performs all three end-to-end.","The grounded output format makes stories machine-readable: every entity mention can be traced to a specific frame and bounding box, which supports automated consistency checks and interactive visualization.","Parameter-efficient training on this recipe outperforms full fine-tuning: the LoRA rank-2048 variant reaches 0.62 character recall and 0.27 mAP, while full fine-tuning reaches 0.37 and 0.16, so the recommended regimen is LoRA rather than full-model training.","The hallucination reduction is carried almost entirely by object and attribute hallucinations (1.84 to 1.54 and 1.47 to 1.28), while relationship hallucinations stay essentially flat (0.60 to 0.63), so the method fixes what is present in the scene better than how entities relate."],"supporting_citations":[{"why":"Supplies the Qwen2.5-VL 7B architecture that Qwen Storyteller fine-tunes, plus the model used as the LLM fallback in landmark detection.","marker":"[4]"},{"why":"Mask2Former provides the segmentation-based object detections that ground entity crops and drive re-identification.","marker":"[5]"},{"why":"Defines the four-way hallucination taxonomy (object, attribute, relationship, environment) used in the LLM evaluation.","marker":"[8]"},{"why":"GroundCap is the source dataset of 52,016 movie images and the ID-based grounding scheme that the paper extends to multi-frame stories.","marker":"[9]"},{"why":"Chain-of-thought prompting supplies the reasoning paradigm behind the structured scene-analysis tables.","marker":"[10]"},{"why":"ChatGPT-4o is one of the four LLM judges whose average scores produce the hallucination and creativity numbers.","marker":"[14]"},{"why":"Claude 3.7 Sonnet is another of the four LLM judges contributing to the reported evaluation scores.","marker":"[15]"},{"why":"SigLIP is the visual embedding model selected for cross-frame object re-identification after empirical comparison with alternatives.","marker":"[29]"},{"why":"ArcFace provides the face embeddings that prioritize identity matching for person detections.","marker":"[31]"},{"why":"Provides the inter-evaluator agreement coefficient used to quantify reliability of the LLM judges.","marker":"[39]"}],"fun_headline_variants":["Fine-tuned model grounds stories, cuts hallucinations 12%","StoryReasoning: grounded stories reduce hallucinations by 12%","Grounded chain-of-thought lifts story creativity 31%","Visual storytelling gets grounded: 12% fewer hallucinations","Qwen Storyteller: grounded fine-tuning cuts errors, boosts creativity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers come from four AI judges that often disagree with each other (their agreement score is 0.14 for description accuracy and 0.34 for hallucination rate), and no human check or statistical test is provided.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned model grounds stories, cuts hallucinations 12%","StoryReasoning: grounded stories reduce hallucinations by 12%","Grounded chain-of-thought lifts story creativity 31%","Visual storytelling gets grounded: 12% fewer hallucinations","Qwen Storyteller: grounded fine-tuning cuts errors, boosts creativity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000959,"raw_usage":{"total_tokens":4091,"prompt_tokens":959,"completion_tokens":3132,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":3046}},"tokens_in":575,"tokens_out":3132,"duration_ms":25568,"temperature":1.0,"reasoning_tokens":3046,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:11:30.143397+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators count hallucinations and rate creativity on the same 626 generated stories per model; if the human-rated reduction in hallucinations or increase in creativity is absent or much smaller than 12.3% and 31%, the central claim fails. Alternatively, recompute the four LLM judges' per-story scores into a paired confidence interval: if the interval for the mean difference includes zero, the claimed improvement is not statistically supported.","supporting_citations":[{"cited_title":"Cheng, I","cited_arxiv_id":null,"evidence_quote":"Mask2Former provides the segmentation-based object detections that ground entity crops and drive re-identification."},{"cited_title":"Huang, W","cited_arxiv_id":null,"evidence_quote":"Defines the four-way hallucination taxonomy (object, attribute, relationship, environment) used in the LLM evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting supplies the reasoning paradigm behind the structured scene-analysis tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ChatGPT-4o is one of the four LLM judges whose average scores produce the hallucination and creativity numbers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Claude 3.7 Sonnet is another of the four LLM judges contributing to the reported evaluation scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SigLIP is the visual embedding model selected for cross-frame object re-identification after empirical comparison with alternatives."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ArcFace provides the face embeddings that prioritize identity matching for person detections."},{"cited_title":"Krippendorff, Estimating the reliability, systematic error and random error of interval data, Educational and Psychological Measurement 30 (1970) 61 – 70","cited_arxiv_id":null,"evidence_quote":"Provides the inter-evaluator agreement coefficient used to quantify reliability of the LLM judges."}],"review_version":1}