{"id":"524a41a1-4ef4-46b9-81d0-b90f895208ef","arxiv_id":"2608.10908","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LVLM judges are near chance when asked to pick the correctly ordered version of an image sequence, and this temporal blindness persists after fine-tuning and at larger scale.","lead":"This paper tests whether vision-language AI models can tell whether a sequence of images tells a story in the correct order, and finds that most cannot. It introduces two benchmarks, shows that fine-tuning does not fix the problem, and argues that the failure is built into the model architecture.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central architectural claim is untested: no ablation isolates causal masking or RoPE, and larger-model results contradict the 'collapse to chance' framing.","rationale":"The reader's weakest-assumption identification is exactly the point that matters: the paper's headline contribution is not merely that LVLMs struggle with temporal order—that empirical claim is well supported by the PRISM/MIRAGE benchmarks, human validation, and the per-position asymmetry tables. The decisive additional claim is that this difficulty is structural, caused specifically by causal masking and RoPE, and that scaling data or parameters cannot fix it. That claim is load-bearing for the conclusion that 'architectures, not datasets, are the bottleneck' and that the community should pursue architectural redesign. The paper does not test this mechanism; it explicitly defers to prior literature. Therefore the central argument is underdetermined by the evidence presented. A careful reader should not reject the empirical contribution, which is substantial: the pairwise-versus-pointwise contrast, the controlled perturbation taxonomy, and the scale checks are genuinely informative. But the strongest verbal conclusion goes beyond what the experiments establish. The appropriate verdict remains CONDITIONAL, requiring either a mechanistic ablation or a substantive softening of the architectural claim. I agree with the reader's conditional assessment and do not see grounds to move the verdict further in either direction based on this concern alone.","tokens_in":43296,"tokens_out":6630,"duration_ms":73424,"concrete_test":"Fix LLaVA-OneVision and evaluate on PRISM-Temporal (P6) while varying one architectural factor at a time: (i) replace RoPE in the language decoder with learned absolute position embeddings; (ii) switch the decoder's causal mask to a prefix-causal or bidirectional scoring setup (e.g., a reranker-style encoder head); (iii) a text-only control with shuffled step captions but identical swap positions, to separate visual from positional effects. Report per-swap-position accuracy, sample-weighted Wavg, and all-inference accuracy including NULL answers. If the recency gradient and near-chance Wavg persist under (i) and (ii), the RoPE/causal-masking attribution is unsupported; if the gradient flattens and Wavg rises meaningfully above 0.6, the architectural mechanism is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's conclusion—that judges collapse toward chance on temporal order 'not for lack of data, but because of primacy and recency biases rooted in architectural constraints'—rests on an attribution that is not directly tested. The authors state in Supplementary Section F that 'the attribution of primacy and recency effects to causal masking and RoPE rests on prior literature rather than on direct measurement.' No experiment varies positional encoding, attention masking, or RoPE; the per-position profiles in Tables S17 and S19 could equally arise from task-difficulty gradients, prompt-layout differences (P0–P6 reorder text and images), or from limited visual grounding rather than from the two named mechanisms. The conclusion is also internally strained by the paper's own scale study: Gemma-4-31B reaches 0.91 aggregate accuracy on PRISM and about 0.77 across temporal swap positions (Table S19), so 'collapse toward chance' is not a general statement even on the temporal subset. Scale does not eliminate the remaining positional gradient, but it shifts the operating point so much that 'not for lack of data' and 'scale does not resolve the bottleneck' are not supported as stated. Without a mechanistic ablation, the prescriptive claim that architectures rather than datasets are the bottleneck, and that 'architectural redesign' is necessary, does not follow from the experiments, even though the empirical difficulty of temporal judgment remains credible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether large vision-language models (LVLMs) can act as reliable judges of temporal order in multi-image sequences. It introduces two benchmarks—PRISM, built from controlled semantic and temporal perturbations of cooking procedures, and MIRAGE, built from human-annotated generated image sequences—and evaluates LLaVA-OneVision and LLaVA-Critic in zero-shot and LoRA fine-tuned settings, using Gemini-2.5-Flash rationales for CoT supervision. The reported experiments show that at 7B scale, pointwise scoring appears reasonable while pairwise temporal-swap detection is near chance, with positional profiles suggesting primacy effects for semantic outliers and recency effects for temporal swaps. The paper concludes that current transformer-based LVLM judges are structurally ill-equipped for temporal reasoning because of causal masking and rotary position embeddings, that SFT consolidates rather than mitigates these biases, and that scaling does not resolve the bottleneck.","tokens_in":43497,"tokens_out":5665,"duration_ms":59080,"significance":"If the central claim were fully supported, this would be an influential result: it would show that off-the-shelf LVLM judges cannot be trusted for sequential visual evaluation, that fine-tuning on current architectures does not fix the problem, and that architectural changes are required. The benchmark construction is a genuine contribution: the perturbations are objectively grounded, the MIRAGE data come from real generative pipelines, and the human validity checks strengthen the claim that the observed failures are not annotation artifacts. The 7B-scale finding that temporal-swap discrimination is near chance is credible, and the observation that SFT improves semantic outlier detection while leaving temporal swap detection largely unchanged is a useful diagnostic. However, the paper's strongest interpretive claims—the architectural origin in causal masking and RoPE, the 'collapse to chance' framing, and the claim that scaling does not help—are not supported by the evidence as currently presented, and some are contradicted by the paper's own supplementary results.","major_comments":[{"comment":"The concluding claim that \"judges collapse toward chance when discriminating temporal order\" is contradicted by the paper's own scaling results. Gemma-4-31B reaches 0.91 aggregate accuracy on PRISM (Table S18), and on the temporal-swap subset the position-weighted accuracy computed from Table S19 is approximately 0.77, well above the 0.50 chance level. The weaker statement that scaling does not remove the positional gradient may be supportable, but the categorical \"collapse toward chance\" claim and the inference that \"scale does not resolve the bottleneck\" for current models generally do not follow. The claim should be restricted to the 7B LLaVA family, or a principled threshold for \"collapse\" should be defined.","section":"Section G (Conclusion); Supplementary §E, Tables S18–S19"},{"comment":"The load-bearing attribution of the observed primacy and recency profiles to causal masking and RoPE is not directly tested. Supplementary Section F states that the attribution \"rests on prior literature rather than on direct measurement.\" The per-position gradients in Tables S4, S17, and S19 could equally arise from task-difficulty gradients across swap positions, from prompt-layout differences (P0–P6 reorder text and images), or from limited visual grounding. An ablation that varies the positional encoding or attention masking—for example, a bidirectional-attention variant or a judge without RoPE—is required before the conclusion that \"architectural redesign rather than data scaling\" is necessary can be drawn from these experiments.","section":"Section C (Eq. 3); Supplementary §F"},{"comment":"The abstract's \"catastrophic collapse\" framing is an overstatement relative to the reported numbers. LLaVA-Critic reaches 0.94 accuracy on MIRAGE at P6 (Table S2), and on PRISM-Semantic at P6 with reasoning it reaches 0.77 accuracy (Table S13). The dichotomy is real for the PRISM-Temporal subset at 7B scale, but the paper itself reports high absolute performance on other subsets and at larger scale. Please calibrate the wording so that \"collapse\" refers to the specific temporal-swap subset at 7B, rather than to LVLM judges generally.","section":"Abstract; §F.2 MIRAGE results"},{"comment":"Explanation quality is measured against the same Gemini-2.5-Flash rationales that are used as SFT supervision (Eq. 4), so the METEOR and cosine-similarity scores are measures of fidelity to that teacher rather than independent assessments of explanation quality. The claim that \"explanation fluency and judgment reliability are orthogonal axes\" is therefore only as strong as the assumption that Gemini rationales are valid references. This circularity should be stated explicitly where the explanation-quality results are interpreted, and the corresponding conclusions should be softened.","section":"§F.1 Evaluation Metrics; Eq. (4)"}],"minor_comments":[{"comment":"The sentence \"There are some to text-to-video evaluation suites such as T2V-CompBench...\" is ungrammatical and appears to be missing a phrase; please revise.","section":"Section B (Related Work)"},{"comment":"Table S1 lists two rows with the identical condition label \"Pw_R_P5\" but different metric values; either the labels are incomplete or one row is duplicated. Please disambiguate.","section":"Table S1 (main text)"},{"comment":"There is a typo in \"eqals\" (should be \"equals\") in the Allen Interval Algebra paragraph.","section":"Supplementary §G.4"},{"comment":"The chronological ordering probe contains only 10 sequences, so the differences between 7/10, 3/10, 1/10, and 0/10 are not statistically strong; adding more items or confidence intervals would make the qualitative claim more robust.","section":"Supplementary §G.3, Table S21"},{"comment":"Equation (2) defines a symmetry condition relating two different comparisons rather than an invariance of a single probability; consider rephrasing the definition of position invariance to avoid confusion.","section":"Section C, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of the paper is valuable, and the PRISM/MIRAGE benchmarks are a useful resource. The main risk is overclaiming: the paper's title and abstract promise a structural, architecture-level finding, but the manuscript's own supplementary material concedes that the causal-masking/RoPE attribution is not directly measured, and the scaling results contradict the 'collapse to chance' slogan. A revision that narrows the claims, integrates the scaling study into the main text, and either adds a mechanistic ablation or explicitly defers the architectural attribution to future work would make the contribution sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical core is worth your time: the PRISM and MIRAGE benchmarks are cleanly constructed, the perturbation design is objectively grounded, and the finding that 7B LVLM judges sit near chance on pairwise temporal-swap detection, while SFT barely moves the needle, looks robust. The pairwise-versus-pointwise comparison and the per-position diagnostic tables are the real contributions, and the human validation on MIRAGE gives the negative results some teeth. On the evidence, this is the most direct demonstration I have seen that order-blindness is not just a data problem for interleaved visual narratives.\n\nBut the headline claim overshoots the evidence. The abstract's \"catastrophic collapse\" is contradicted by the paper's own tables: Critic hits 0.94 on MIRAGE, and Gemma-4-31B reaches 0.91 aggregate on PRISM, with around 0.77 on the temporal subset. That is not collapse to chance. The paper even acknowledges in Supplementary Section F that the attribution to causal masking and RoPE \"rests on prior literature rather than on direct measurement.\" No ablation varies positional encoding or attention masking, so the central prescriptive conclusion—that architectures, not datasets, are the bottleneck and that architectural redesign is required—does not follow from these experiments. The empirical difficulty is credible; the mechanistic explanation is not tested.\n\nTwo smaller issues. First, the monotonic primacy claim in F.3 is not supported by Table S17, where OneVision shows elevated accuracy at the final position (p5) on the semantic subset; the profile is U-shaped, not monotonic. Second, the larger-model accuracy is computed over valid responses only, and Gemma-4-31B produced no extractable answer in 38.6% of PRISM inferences. Reported as all-inference accuracy, Gemma's 0.91 becomes something near 0.56, which changes the scale story substantially. These are fixable with reporting changes.\n\nThe paper is honest about its own limits in the supplementary, and the benchmarks are a real resource for the evaluation community. With the abstract softened, the mechanistic claims hedged or backed by ablations, and valid-response filtering reported transparently, this would be a solid contribution. As it stands, it deserves serious refereeing but not acceptance in its current form. I would send it to review and push hard for either a positional-encoding ablation or a rewritten conclusion that stays close to the empirical findings.","headline":"Solid empirical probe of temporal-order blindness in LVLM judges, with an architectural explanation that the paper itself admits it never tests.","tokens_in":44106,"tokens_out":1812,"would_cite":true,"duration_ms":22079,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that transformer-based LVLM judges are structurally unable to discriminate temporal order in image sequences, and that the failure persists after fine-tuning and scaling because primacy and recency biases rooted in…","keywords":["Multimodal Large Language Models","Visual Sequence Evaluation","Temporal Reasoning","Logical Consistency","LLM-as-a-judge","Vision-Language Models","Positional Bias","Primacy and recency effects"],"falsifier":"Run the same pairwise temporal-swap benchmark with only the positional scheme varied—for instance, bidirectional attention with no causal mask, or learned absolute position embeddings instead of rotary embeddings—and measure per-position swap-detection accuracy; if accuracy stays near chance in all conditions, the causal-masking/RoPE attribution is wrong, while a large jump would confirm it.","tokens_in":43014,"feed_emoji":"🕰️","tokens_out":8391,"duration_ms":71245,"temperature":0.7,"pith_summary":"This paper tries to establish that large vision-language models used as automated judges cannot tell whether a sequence of images tells a story in the correct order. The authors show a sharp split in performance: the same models assign plausible pointwise scores to individual sequences, but when asked to choose which of two sequences is temporally correct they fall to near-chance accuracy. They argue this failure is structural, not a data problem, and locate it in positional asymmetries, namely primacy and recency effects that make a judge overweigh the first or last frame and ignore the logical middle. The evidence comes from two new benchmarks, one built from controlled perturbations of procedural image sequences and one from real generative-model failures, plus fine-tuning experiments and a chronological ordering probe. If the claim is right, off-the-shelf LVLM evaluation of sequential visual narratives is unreliable, and fixing it requires architectural changes rather than more data.","feed_headline":"LVLM judges collapse on temporal order even after fine-tuning","feed_subtitle":"Pairwise tests expose primacy/recency biases rooted in attention architecture; pointwise scores hide the failure.","key_machinery":"The central object is the pairwise discrimination protocol built into the paper's Sequence-Judge framework, together with the per-position sensitivity analysis over its two perturbation families. The protocol casts evaluation as a relative choice between a gold sequence and a perturbed variant, with randomized presentation order to neutralize a first-option bias that the paper quantifies at up to 15% via a position-invariance test. The positional analysis then measures accuracy for each frame position, producing the primacy/recency profiles that carry the argument: a bright cell at position 0 for semantic anomalies and a gradient toward the end for temporal swaps. The formal scaffolding is the Bradley–Terry model of pairwise preference, with position invariance as the property a fair judge should satisfy.","core_discovery":"The paper's central claim is that LVLM judges exhibit a 'reasoning chasm': they appear competent in pointwise scoring yet collapse when required to discriminate temporal order in pairwise comparisons. On the temporal subset where frames are swapped, all tested 7B judges remain at or below chance (weighted-mean accuracy around 0.47–0.49, macro F1 around 0.45), and fine-tuning with chain-of-thought rationales yields only marginal gains, never crossing the 0.40 F1 threshold. Diagnostic probes locate the failure in positional asymmetries: detection of a semantic outlier peaks when it sits in the opening frame (primacy) and degrades monotonically afterward, while detection of a temporal swap improves toward the end of the sequence (recency), leaving middle-sequence violations effectively invisible. The authors argue these asymmetries are structural, rooted in causal masking and rotary position embeddings, and show that scale alone does not remove them: a roughly 31B model reaches 0.91 aggregate accuracy on the synthetic benchmark yet still loses a third of its accuracy as a function of violation position, and a separate probe requiring chronological ordering of four shuffled historical images gets only 7 out of 10 right. The conclusion is that temporal illiteracy is an architectural bottleneck, and that fine-tuning consolidates rather than mitigates the bias.","pith_inferences":["Editorial inference: the same positional blindness should affect any task where an LVLM must compare two multi-image documents, such as story continuation ranking or visual instruction following; the paper's pairwise protocol could be adapted to those settings without changing its core design.","Editorial inference: if the causal-masking and rotary-embedding attribution is right, replacing the position scheme with bidirectional attention or learned absolute positions should substantially improve temporal swap detection on the synthetic benchmark; the paper does not run this ablation, but it is a direct testable prediction.","Editorial inference: because the bias is positional, simple input-order augmentation or averaging over several permutations of the two sequences may recover some temporal judgment accuracy at inference time, a cheap mitigation the paper leaves untested."],"forward_implications":["Pointwise LVLM scores for multi-image narratives should not be read as evidence of temporal coherence: the same judge that assigns plausible 1–5 ratings is at chance when asked to pick the correctly ordered sequence.","Supervised fine-tuning with chain-of-thought rationales improves score calibration and semantic anomaly detection but does not fix temporal swap detection; on the temporal subset all variants remain below F1 = 0.40, so reasoning supervision is not a cure.","Scaling model size shifts the operating point without changing the failure structure: larger models improve aggregate pairwise accuracy but retain the same positional signature and still struggle with middle-of-sequence violations.","Reconstructing chronological order outright is much harder than pairwise discrimination: in the visual ordering probe, 7B judges answer essentially by position (0/10 and 1/10 correct) and even a large model reaches only 7/10 on four-image ordering.","Evaluation benchmarks for generative multimedia should treat time as a first-class logical dimension and measure per-position sensitivity, not just aggregate accuracy or fidelity scores."],"supporting_citations":[{"why":"supplies the pointwise judge evaluation paradigm and the binary-collapse critique that the paper extends to sequential inputs.","marker":"[4]"},{"why":"documents serial position (primacy/recency) effects in language models, which the paper predicts will appear in visual judges.","marker":"[18]"},{"why":"supports the claim that instruction tuning amplifies rather than suppresses biases, grounding the SFT-consolidation result.","marker":"[24]"},{"why":"provides the lost-in-the-middle effect used to explain why middle-frame swaps are invisible.","marker":"[36]"},{"why":"provides the rotary position embedding mechanism cited as the source of recency bias.","marker":"[55]"},{"why":"gives the theoretical result that causal masking makes representations converge to the first token, sourcing the primacy bias.","marker":"[62]"},{"why":"supplies the single-image evaluation-specialized judge baseline that the paper adapts to interleaved sequences.","marker":"[65]"},{"why":"provides the pairwise versus pointwise judge evaluation methodology that the paper transfers to visual sequences.","marker":"[72]"}],"fun_headline_variants":["LVLM judges: pointwise fine, pairwise collapse on temporal order","Shuffled frames expose LVLM temporal blindness despite scale","Primacy and recency: why LVLM judges can't rank story frames","Temporal reasoning chasm: LVLMs fail pairwise order tests","Fine-tuning consolidates LVLM temporal bias, doesn't cure it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the observed position-dependent errors come from causal masking and rotary position embeddings inside the judge architecture, rather than from the task design, the prompt layout, or limited visual grounding; the paper itself notes that this attribution rests on prior literature rather than direct measurement.","fun_headline_variants_meta":{"raw":{"variants":["LVLM judges: pointwise fine, pairwise collapse on temporal order","Shuffled frames expose LVLM temporal blindness despite scale","Primacy and recency: why LVLM judges can't rank story frames","Temporal reasoning chasm: LVLMs fail pairwise order tests","Fine-tuning consolidates LVLM temporal bias, doesn't cure it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1596,"prompt_tokens":1061,"completion_tokens":535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":444}},"tokens_in":677,"tokens_out":535,"duration_ms":5657,"temperature":1.0,"reasoning_tokens":444,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:32:34.250307+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pairwise temporal-swap benchmark with only the positional scheme varied—for instance, bidirectional attention with no causal mask, or learned absolute position embeddings instead of rotary embeddings—and measure per-position swap-detection accuracy; if accuracy stays near chance in all conditions, the causal-masking/RoPE attribution is wrong, while a large jump would confirm it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"documents serial position (primacy/recency) effects in language models, which the paper predicts will appear in visual judges."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supports the claim that instruction tuning amplifies rather than suppresses biases, grounding the SFT-consolidation result."}],"review_version":1}