Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences

Guilherme Fernandes; Joao Magalhaes; Martina Ianaro; Maurizio Gabbrielli

REVIEW 4 major objections 5 minor 81 references

Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that transformer-based LVLM judges are structurally unable to discriminate temporal order in image sequences, and that the failure persists after fine-tuning and scaling because primacy and recency biases rooted in…

desk verdict Solid empirical probe of temporal-order blindness in LVLM judges, with an architectural explanation that the paper itself admits it never tests. read the letter →

arxiv 2608.10908 v1 pith:CLKDGE5O submitted 2026-08-11 cs.CV cs.CL

Martina Ianaro , Guilherme Fernandes , Maurizio Gabbrielli , Joao Magalhaes This is my paper · ORCID

classification cs.CVcs.CL

keywords MultimodalLargeLanguageModelsVisualSequenceEvaluationTemporalReasoningLogicalConsistencyLLM-as-a-judgeVision-LanguagePositionalBiasPrimacyandrecencyeffects

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large vision-language models used as automated judges cannot tell whether a sequence of images tells a story in the correct order. The authors show a sharp split in performance: the same models assign plausible pointwise scores to individual sequences, but when asked to choose which of two sequences is temporally correct they fall to near-chance accuracy. They argue this failure is structural, not a data problem, and locate it in positional asymmetries, namely primacy and recency effects that make a judge overweigh the first or last frame and ignore the logical middle. The evidence comes from two new benchmarks, one built from controlled perturbations of procedural image sequences and one from real generative-model failures, plus fine-tuning experiments and a chronological ordering probe. If the claim is right, off-the-shelf LVLM evaluation of sequential visual narratives is unreliable, and fixing it requires architectural changes rather than more data.

What carries the argument

The central object is the pairwise discrimination protocol built into the paper's Sequence-Judge framework, together with the per-position sensitivity analysis over its two perturbation families. The protocol casts evaluation as a relative choice between a gold sequence and a perturbed variant, with randomized presentation order to neutralize a first-option bias that the paper quantifies at up to 15% via a position-invariance test. The positional analysis then measures accuracy for each frame position, producing the primacy/recency profiles that carry the argument: a bright cell at position 0 for semantic anomalies and a gradient toward the end for temporal swaps. The formal scaffolding is the Bradley–Terry model of pairwise preference, with position invariance as the property a fair judge should satisfy.

What would settle it

Run the same pairwise temporal-swap benchmark with only the positional scheme varied—for instance, bidirectional attention with no causal mask, or learned absolute position embeddings instead of rotary embeddings—and measure per-position swap-detection accuracy; if accuracy stays near chance in all conditions, the causal-masking/RoPE attribution is wrong, while a large jump would confirm it.

Watch

Extended reading notes

Core claim

The paper's central claim is that LVLM judges exhibit a 'reasoning chasm': they appear competent in pointwise scoring yet collapse when required to discriminate temporal order in pairwise comparisons. On the temporal subset where frames are swapped, all tested 7B judges remain at or below chance (weighted-mean accuracy around 0.47–0.49, macro F1 around 0.45), and fine-tuning with chain-of-thought rationales yields only marginal gains, never crossing the 0.40 F1 threshold. Diagnostic probes locate the failure in positional asymmetries: detection of a semantic outlier peaks when it sits in the opening frame (primacy) and degrades monotonically afterward, while detection of a temporal swap improves toward the end of the sequence (recency), leaving middle-sequence violations effectively invisible. The authors argue these asymmetries are structural, rooted in causal masking and rotary position embeddings, and show that scale alone does not remove them: a roughly 31B model reaches 0.91 aggregate accuracy on the synthetic benchmark yet still loses a third of its accuracy as a function of violation position, and a separate probe requiring chronological ordering of four shuffled historical images gets only 7 out of 10 right. The conclusion is that temporal illiteracy is an architectural bottleneck, and that fine-tuning consolidates rather than mitigates the bias.

Load-bearing premise

The load-bearing premise is that the observed position-dependent errors come from causal masking and rotary position embeddings inside the judge architecture, rather than from the task design, the prompt layout, or limited visual grounding; the paper itself notes that this attribution rests on prior literature rather than direct measurement.

Editorial extensions

If this is right

Pointwise LVLM scores for multi-image narratives should not be read as evidence of temporal coherence: the same judge that assigns plausible 1–5 ratings is at chance when asked to pick the correctly ordered sequence.
Supervised fine-tuning with chain-of-thought rationales improves score calibration and semantic anomaly detection but does not fix temporal swap detection; on the temporal subset all variants remain below F1 = 0.40, so reasoning supervision is not a cure.
Scaling model size shifts the operating point without changing the failure structure: larger models improve aggregate pairwise accuracy but retain the same positional signature and still struggle with middle-of-sequence violations.
Reconstructing chronological order outright is much harder than pairwise discrimination: in the visual ordering probe, 7B judges answer essentially by position (0/10 and 1/10 correct) and even a large model reaches only 7/10 on four-image ordering.
Evaluation benchmarks for generative multimedia should treat time as a first-class logical dimension and measure per-position sensitivity, not just aggregate accuracy or fidelity scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same positional blindness should affect any task where an LVLM must compare two multi-image documents, such as story continuation ranking or visual instruction following; the paper's pairwise protocol could be adapted to those settings without changing its core design.
Editorial inference: if the causal-masking and rotary-embedding attribution is right, replacing the position scheme with bidirectional attention or learned absolute positions should substantially improve temporal swap detection on the synthetic benchmark; the paper does not run this ablation, but it is a direct testable prediction.
Editorial inference: because the bias is positional, simple input-order augmentation or averaging over several permutations of the two sequences may recover some temporal judgment accuracy at inference time, a cheap mitigation the paper leaves untested.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The empirical core is worth your time: the PRISM and MIRAGE benchmarks are cleanly constructed, the perturbation design is objectively grounded, and the finding that 7B LVLM judges sit near chance on pairwise temporal-swap detection, while SFT barely moves the needle, looks robust. The pairwise-versus-pointwise comparison and the per-position diagnostic tables are the real contributions, and the human validation on MIRAGE gives the negative results some teeth. On the evidence, this is the most direct demonstration I have seen that order-blindness is not just a data problem for interleaved visual narratives.

But the headline claim overshoots the evidence. The abstract's "catastrophic collapse" is contradicted by the paper's own tables: Critic hits 0.94 on MIRAGE, and Gemma-4-31B reaches 0.91 aggregate on PRISM, with around 0.77 on the temporal subset. That is not collapse to chance. The paper even acknowledges in Supplementary Section F that the attribution to causal masking and RoPE "rests on prior literature rather than on direct measurement." No ablation varies positional encoding or attention masking, so the central prescriptive conclusion—that architectures, not datasets, are the bottleneck and that architectural redesign is required—does not follow from these experiments. The empirical difficulty is credible; the mechanistic explanation is not tested.

Two smaller issues. First, the monotonic primacy claim in F.3 is not supported by Table S17, where OneVision shows elevated accuracy at the final position (p5) on the semantic subset; the profile is U-shaped, not monotonic. Second, the larger-model accuracy is computed over valid responses only, and Gemma-4-31B produced no extractable answer in 38.6% of PRISM inferences. Reported as all-inference accuracy, Gemma's 0.91 becomes something near 0.56, which changes the scale story substantially. These are fixable with reporting changes.

The paper is honest about its own limits in the supplementary, and the benchmarks are a real resource for the evaluation community. With the abstract softened, the mechanistic claims hedged or backed by ablations, and valid-response filtering reported transparently, this would be a solid contribution. As it stands, it deserves serious refereeing but not acceptance in its current form. I would send it to review and push hard for either a positional-encoding ablation or a rewritten conclusion that stays close to the empirical findings.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether large vision-language models (LVLMs) can act as reliable judges of temporal order in multi-image sequences. It introduces two benchmarks—PRISM, built from controlled semantic and temporal perturbations of cooking procedures, and MIRAGE, built from human-annotated generated image sequences—and evaluates LLaVA-OneVision and LLaVA-Critic in zero-shot and LoRA fine-tuned settings, using Gemini-2.5-Flash rationales for CoT supervision. The reported experiments show that at 7B scale, pointwise scoring appears reasonable while pairwise temporal-swap detection is near chance, with positional profiles suggesting primacy effects for semantic outliers and recency effects for temporal swaps. The paper concludes that current transformer-based LVLM judges are structurally ill-equipped for temporal reasoning because of causal masking and rotary position embeddings, that SFT consolidates rather than mitigates these biases, and that scaling does not resolve the bottleneck.

Significance. If the central claim were fully supported, this would be an influential result: it would show that off-the-shelf LVLM judges cannot be trusted for sequential visual evaluation, that fine-tuning on current architectures does not fix the problem, and that architectural changes are required. The benchmark construction is a genuine contribution: the perturbations are objectively grounded, the MIRAGE data come from real generative pipelines, and the human validity checks strengthen the claim that the observed failures are not annotation artifacts. The 7B-scale finding that temporal-swap discrimination is near chance is credible, and the observation that SFT improves semantic outlier detection while leaving temporal swap detection largely unchanged is a useful diagnostic. However, the paper's strongest interpretive claims—the architectural origin in causal masking and RoPE, the 'collapse to chance' framing, and the claim that scaling does not help—are not supported by the evidence as currently presented, and some are contradicted by the paper's own supplementary results.

major comments (4)

[Section G (Conclusion); Supplementary §E, Tables S18–S19] The concluding claim that "judges collapse toward chance when discriminating temporal order" is contradicted by the paper's own scaling results. Gemma-4-31B reaches 0.91 aggregate accuracy on PRISM (Table S18), and on the temporal-swap subset the position-weighted accuracy computed from Table S19 is approximately 0.77, well above the 0.50 chance level. The weaker statement that scaling does not remove the positional gradient may be supportable, but the categorical "collapse toward chance" claim and the inference that "scale does not resolve the bottleneck" for current models generally do not follow. The claim should be restricted to the 7B LLaVA family, or a principled threshold for "collapse" should be defined.
[Section C (Eq. 3); Supplementary §F] The load-bearing attribution of the observed primacy and recency profiles to causal masking and RoPE is not directly tested. Supplementary Section F states that the attribution "rests on prior literature rather than on direct measurement." The per-position gradients in Tables S4, S17, and S19 could equally arise from task-difficulty gradients across swap positions, from prompt-layout differences (P0–P6 reorder text and images), or from limited visual grounding. An ablation that varies the positional encoding or attention masking—for example, a bidirectional-attention variant or a judge without RoPE—is required before the conclusion that "architectural redesign rather than data scaling" is necessary can be drawn from these experiments.
[Abstract; §F.2 MIRAGE results] The abstract's "catastrophic collapse" framing is an overstatement relative to the reported numbers. LLaVA-Critic reaches 0.94 accuracy on MIRAGE at P6 (Table S2), and on PRISM-Semantic at P6 with reasoning it reaches 0.77 accuracy (Table S13). The dichotomy is real for the PRISM-Temporal subset at 7B scale, but the paper itself reports high absolute performance on other subsets and at larger scale. Please calibrate the wording so that "collapse" refers to the specific temporal-swap subset at 7B, rather than to LVLM judges generally.
[§F.1 Evaluation Metrics; Eq. (4)] Explanation quality is measured against the same Gemini-2.5-Flash rationales that are used as SFT supervision (Eq. 4), so the METEOR and cosine-similarity scores are measures of fidelity to that teacher rather than independent assessments of explanation quality. The claim that "explanation fluency and judgment reliability are orthogonal axes" is therefore only as strong as the assumption that Gemini rationales are valid references. This circularity should be stated explicitly where the explanation-quality results are interpreted, and the corresponding conclusions should be softened.

minor comments (5)

[Section B (Related Work)] The sentence "There are some to text-to-video evaluation suites such as T2V-CompBench..." is ungrammatical and appears to be missing a phrase; please revise.
[Table S1 (main text)] Table S1 lists two rows with the identical condition label "Pw_R_P5" but different metric values; either the labels are incomplete or one row is duplicated. Please disambiguate.
[Supplementary §G.4] There is a typo in "eqals" (should be "equals") in the Allen Interval Algebra paragraph.
[Supplementary §G.3, Table S21] The chronological ordering probe contains only 10 sequences, so the differences between 7/10, 3/10, 1/10, and 0/10 are not statistically strong; adding more items or confidence intervals would make the qualitative claim more robust.
[Section C, Eq. (2)] Equation (2) defines a symmetry condition relating two different comparisons rather than an invariance of a single probability; consider rephrasing the definition of position invariance to avoid confusion.

Circularity Check

1 steps flagged · score 3.0 of 10

Central temporal-order result is independently constructed; the only self-referential element is the Gemini-supervised explanation-quality metric.

fitted input called prediction [Section E, 'Temporal-Visual CoT Explanations'; Section F.1, 'Evaluation Metrics'; Section F.2, 'Explanation Quality and the Cost of Rationale Supervision']
"Both PRISM perturbations and MIRAGE model outputs are augmented with CoT rationales automatically generated by Gemini-2.5-Flash. These explanations serve as a supervision signal for Supervised Fine-Tuning (SFT) and as a reference for assessing reasoning quality. ... Rationales are evaluated against Gemini references via METEOR and Cosine Similarity over all-MiniLM-L6-v2 embeddings, computed on valid non-empty explanations only."

The same Gemini-2.5-Flash outputs are simultaneously the SFT target (L_total = alpha*L_score + beta*L_CoT, Eq. 4) and the reference for METEOR and cosine-similarity explanation scores. Consequently, the reported 'explanation quality' measures the fine-tuned judge's fidelity to its own training oracle; the observation that SFT leaves explanation quality unchanged, and the derived claim that verbal reasoning quality and judgment reliability are orthogonal, are not independent of the supervision signal. This is a secondary circularity: the main pairwise temporal-order results rest on objective swap/outlier labels and human-validated MIRAGE negatives, not on the Gemini oracle.

full rationale

The paper's central claim that LVLM judges collapse on temporal-order discrimination is built from objectively constructed perturbations (consecutive and non-consecutive swaps, semantic outliers), human-validated MIRAGE negatives, and per-position accuracy profiles; none of these reduce to a fitted parameter or to the paper's own prior work. The only self-citation, reference [12], is used as one of several MIRAGE generation pipelines and is not load-bearing. Supplementary Section F candidly admits that attributing primacy/recency to causal masking and RoPE 'rests on prior literature rather than on direct measurement'; that is an untested mechanistic assumption, not a circular reduction, and it weakens rather than tautologizes the architectural conclusion. The scale study also explicitly bounds the claim, noting that Gemma-4-31B reaches 0.91 aggregate accuracy. The genuine circular step is confined to the explanation-quality pipeline: Gemini-2.5-Flash rationales are used as SFT supervision and then as the METEOR/cosine-similarity reference, so those scores partly measure agreement with the training oracle. Because this circularity does not support the main temporal-order finding, the overall score remains low.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claim rests on, first, an empirical benchmark construction that is mostly self-contained, and second, a causal explanation borrowed from prior literature. The benchmark perturbations are objective, swaps and substitutions, and do not require fitting. The free parameters are training hyperparameters and the valid-response filtering choice, neither of which is fitted to produce the main negative result. The axioms list the borrowed mechanism, causal masking bound, RoPE suppression, instruction-tuning amplification, the Gemini oracle assumption, and the human majority-vote assumption. No invented entities are introduced; PRISM and MIRAGE are datasets, not postulates.

free parameters (2)

LoRA fine-tuning hyperparameters = r=128, alpha=256, lr=2e-5, global batch 128, 1 epoch; Optuna-selected variants per run
Selected via Bayesian search targeting macro-F1 on PRISM-Temporal. The conclusion that SFT does not mitigate temporal blindness is conditional on this single training configuration, so the negative result is only as general as this setup.
Valid-response filtering for accuracy = none; accuracy computed over valid responses only
For Gemma-4-31B on PRISM, 38.6% of inferences yield no extractable A/B answer and are excluded before computing accuracy. This choice materially affects headline accuracy, from 0.91 over valid responses to about 0.56 over all inferences.

assumptions (6)

standard math Pairwise preference follows the Bradley-Terry model (Eq. 1).
Used to define judge preference probabilities and position invariance in Eq. (2); standard but unproved in the paper.
domain assumption Hidden representations converge exponentially to the first token with depth (Eq. 3), per Wu et al. [62].
This bound is the theoretical basis for the claimed primacy bias. The paper does not measure this convergence in the evaluated judges; Supplementary F acknowledges the attribution rests on prior literature.
domain assumption RoPE suppresses interactions between distant tokens, producing recency and lost-in-the-middle effects [36, 55].
Used to predict recency bias in temporal swap detection. No ablation varies the positional encoding, so the mechanism is assumed from cited work.
ad hoc to paper Gemini-2.5-Flash CoT rationales are a valid supervision signal and reference for explanation quality.
Gemini rationales are used in Eq. (4) for SFT and as the gold reference for METEOR and cosine similarity. The paper reports Gemini's own scores collapse, 91% in the {1,2} range, so the oracle is itself biased and this assumption is load-bearing for the reasoning-augmentation results.
domain assumption Instruction tuning and RLHF amplify, rather than suppress, primacy and recency biases [24, 59].
Used in Sections C and F.2 to interpret SFT results as consolidating structural bias. The paper only runs LoRA SFT, not full instruction tuning or RLHF, so the extrapolation rests on cited work.
domain assumption Human majority-vote consensus is a reliable gold standard for pointwise coherence despite low inter-rater correlation (rH = 0.34).
The paper argues majority voting is less noisy than individual annotators. This assumption underlies the claim that models can exceed the ceiling and that SFT recovers rH = 0.34.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences." pith.science (2026). https://pith.science/paper/CLKDGE5O

@misc{pith2026260810908,
  author       = {Pith},
  title        = {Pith review of: Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CLKDGE5O}},
  note         = {Machine review of arXiv:2608.10908}
}

read the original abstract

As generative multimedia evolves from static image synthesis to complex, interleaved visual narratives, a foundational bottleneck has emerged: the judgment crisis. While human perception naturally synthesizes the temporal and logical flow of a story, automated evaluation systems remain largely "blind" to sequential continuity, often failing to distinguish between a coherent narrative and a semantically shuffled or contradictory sequence. This work identifies a critical structural gap in current multimodal evaluation paradigms, arguing that the reliance on Large Vision-Language Models (LVLMs) as judges is fundamentally limited by architectural biases. Our analysis reveals a profound performance dichotomy: while models may appear competent in isolated pointwise scoring, they suffer a catastrophic collapse when required to perform pairwise discrimination of temporal order. We demonstrate that this is not merely a data-scarcity issue but a structural one. Through a series of diagnostic probes, we uncover systematic positional asymmetries, specifically primacy and recency effects, where a model's judgment of a story is significantly influenced by the placement of a frame, often more than by its semantic consistency. These biases, potentially rooted in causal masking and rotary embeddings, suggest that current transformer-based judges are inherently ill-equipped for long-form visual reasoning. By exposing these blind spots, we challenge the multimedia community to move beyond snapshot-centric metrics and instead pioneer Temporally-Aware Evaluation paradigms that treat visual sequences as unified logical structures rather than unordered collections of frames.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 26 canonical work pages

[1]

Tosin Adewumi, Lama Alkhaled, Namrata Gurung, Goya van Boven, and Irene Pagliai. 2024. Fairness and Bias in Multimodal AI: A Survey. arXiv:2406.19097 [cs.CL] https://arxiv.org/abs/2406.19097

arXiv 2024
[2]

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. InProceedings of the 25th ACM SIGKDD International Confer- ence on Knowledge Discovery & Data Mining(Anchorage, AK, USA)(KDD ’19). Association for Computing Machinery, New York, NY, USA, 2623–2631. doi:10.1...

arXiv 2019
[3]

James F. Allen. 1983. Maintaining knowledge about temporal intervals.Commun. ACM26, 11 (1983), 832–843

1983
[4]

Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. Mllm-as-a- judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning

2024
[5]

Luyu Chen, Zeyu Zhang, Haoran Tan, Quanyu Dai, Hao Yang, Zhenhua Dong, and Xu Chen. 2025. Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge.arXiv preprint arXiv:2505.12301(2025)

work page Pith review arXiv 2025
[6]

Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming- Hsuan Yang, and Sergey Tulyakov. 2024. Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 13320–13331

work page 2024
[7]

Wei Chen, Lin Li, Yongqi Yang, Bin Wen, Fan Yang, Tingting Gao, Yu Wu, and Long Chen. 2025. CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8073–8082

work page 2025
[8]

Muzhi Dai, Jiashuo Sun, Zhiyuan Zhao, Shixuan Liu, Rui Li, Junyu Gao, and Xue- long Li. 2025. From Captions to Rewards (CAREVL): Leveraging Large Language Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Model Experts for Enhanced Reward Modeling in Large Vision-Language Models. ...

arXiv 2025

Show all 81 references

[9]

Hongkun Dou, Junzhe Lu, Jinyang Du, Chengwei Fu, Wen Yao, Hongjue Li, and Yue Deng. 2025. Towards a Unified Framework for Consistency Generative Modeling.IEEE Transactions on Artificial Intelligence(2025), 1–13. doi:10.1109/ TAI.2025.3624330

2025
[10]

Mounia Drissi. 2024. More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Models. arXiv:2412.17505 [stat.ML] https://arxiv.org/abs/2412.17505

2024 arXiv
[11]

Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. 2024. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding.Advances in Neural Information Processing Systems 37 (2024), 89098–89124

2024
[12]

Guilherme Fernandes, Vasco Ramos, Regev Cohen, Idan Szpektor, and João Ma- galhães. 2025. Latent Beam Diffusion Models for Decoding Image Sequences. arXiv:2503.20429 [cs.CV] https://arxiv.org/abs/2503.20429

2025
[13]

Junyu Gao, Xuan Yao, Yong Rui, and Changsheng Xu. 2025. Building embod- ied evoagent: A brain-inspired paradigm for bridging multimodal large models and world models. InProceedings of the 33rd ACM International Conference on Multimedia. 3280–3289

2025
[14]

Gemini Team. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reason- ing, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv:2507.06261 [cs.CL] https://arxiv.org/abs/2507.06261

2025 arXiv
[15]

Gemma Team. 2026. Gemma 4 Technical Report. arXiv:2607.02770 [cs.CL] https://arxiv.org/abs/2607.02770

2026 arXiv
[16]

Brian Gordon, Yonatan Bitton, Andreea Marzoca, Yasumasa Onoe, Xiao Wang, Daniel Cohen-Or, and Idan Szpektor. 2025. Unblocking Fine-Grained Evaluation of Detailed Captions: An Explaining AutoRater and Critic-and-Revise Pipeline. arXiv:2506.07631 [cs.CL] https://arxiv.org/abs/2506.07631

2025 arXiv
[17]

Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. 2024. When attention sink emerges in language models: An empirical view.arXiv preprint arXiv:2410.10781(2024)

2024 arXiv
[18]

Xiaobo Guo and Soroush Vosoughi. 2025. Serial position effects of large language models. InFindings of the Association for Computational Linguistics: ACL 2025. 927–953

2025
[19]

Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al. 2024. Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos.arXiv preprint arXiv:2406.08407(2024)

2024 arXiv
[20]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 [cs.CL] https://arxiv.org/abs/2106.09685

2021 arXiv
[21]

Hui Huang, Xingyuan Bu, Hongli Zhou, Yingqi Qu, Jing Liu, Muyun Yang, Bing Xu, and Tiejun Zhao. 2025. An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4. In Findings of the Association for Computational Linguis...

2025
[22]

Jiaxing Huang and Jingyi Zhang. 2024. A Survey on Evaluation of Multimodal Large Language Models. arXiv:2408.15769 [cs.CV] https://arxiv.org/abs/2408. 15769

2024 arXiv
[23]

Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. 2024. VBench: Comprehensive Benchmark Suite for Video Generative Mod...

2024
[24]

Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, and Yonatan Belinkov. 2024. In- structed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias.Transactions of the Association for Computational Linguistics12 (2024), 771–

2024
[25]

Yuxin Jiang, Bo Huang, Yufei Wang, Xingshan Zeng, Liangyou Li, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, and Wei Wang. 2025. Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization. arXiv:2408.07471 [cs.CL] https://arxiv.org/abs/2408.07471

2025 arXiv
[26]

Bin Kang, Bin Chen, Junjie Wang, Yulin Li, Junzhi Zhao, Junle Wang, and Zhuotao Tian. 2025. Calibclip: Contextual calibration of dominant semantics for text- driven image retrieval. InProceedings of the 33rd ACM International Conference on Multimedia. 5140–5149

2025
[27]

Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. 2023. Generating Images with Multimodal Language Models. arXiv:2305.17216 [cs.CL] https://arxiv.org/ abs/2305.17216

2023 arXiv
[28]

Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, and Minjoon Seo
[29]

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. LLaVA-OneVision: Easy Visual Task Transfer.arXiv preprint arXiv:2408.03326(2024)

2024 arXiv
[30]

Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu
[31]

Xiaochuan Li, Ke Wang, Girija Gouda, Shubham Choudhary, Yaqun Wang, Linwei Hu, Joel Vaughan, and Freddy Lecue. 2025. Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation Systems. arXiv:2512.01786 [cs.AI] https://arxiv.org/abs/2512.01786

2025
[32]

Yijie Li and Yuan Sun. 2026. A fine-grained evaluation framework for language models: Combining pointwise grading and pairwise comparison.Information Processing & Management63, 1 (2026), 104270. doi:10.1016/j.ipm.2025.104270

2026
[33]

Jianxin Liang, Xiaojun Meng, Huishuai Zhang, Yueqian Wang, Jiansheng Wei, and Dongyan Zhao. 2025. ReasVQA: Advancing VideoQA with imperfect reasoning process. InProceedings of the 2025 Conference of the Nations of the Americas Chap- ter of the Association for Computational Lin...

2025
[34]

Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. 2024. Evaluating Text-to-Visual Genera- tion with Image-to-Text Generation.arXiv preprint arXiv:2404.01291(2024)

2024 arXiv
[35]

Fangxin Liu, Junjie Wang, Ning Yang, Zongwu Wang, Junping Zhao, Li Jiang, and Haibing Guan. 2025. Aster: Adaptive dynamic layer-skipping for efficient transformer inference via markov decision process. InProceedings of the 33rd ACM International Conference on Multimedia. 11853–11861

2025
[36]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172 [cs.CL] https://arxiv.org/abs/2307.03172

2023 arXiv
[37]

Tao Liu, Kai Wang, Senmao Li, Joost van de Weijer, Fahad Shahbaz Khan, Shiqi Yang, Yaxing Wang, Jian Yang, and Ming-Ming Cheng. 2025. One-Prompt-One- Story: Free-Lunch Consistent Text-to-Image Generation Using a Single Prompt. arXiv:2501.13554 [cs.CV] https://arxiv.org/abs/2501.13554

2025 arXiv
[38]

Xu Liu, Guikun Chen, and Wenguan Wang. 2026. SinkTrack: Attention Sink based Context Anchoring for Large Language Models. InThe Fourteenth International Conference on Learning Representations

2026
[39]

Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024. Tempcompass: Do video llms really understand videos?. InFindings of the Association for Computational Linguistics: ACL 2024. 8731–8772

2024
[40]

Olga Loginova and Sofía Ortega Loguinova. 2025. Deep Temporal Reasoning in Video Language Models: A Cross-Linguistic Evaluation of Action Duration and Completion through Perfect Times. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V...

2025
[41]

Andrew Lu, Wentinn Liao, Liuhui Wang, Huzheng Yang, and Jianbo Shi. 2025. Artifacts and Attention Sinks: Structured Approximations for Efficient Vision Transformers. arXiv:2507.16018 [cs.CV] https://arxiv.org/abs/2507.16018

2025 arXiv
[42]

Sachit Menon, Ishan Misra, and Rohit Girdhar. 2024. Generating Illustrated Instructions. arXiv:2312.04552 [cs.CV] https://arxiv.org/abs/2312.04552

2024 arXiv
[43]

2023.GPT-4V(ision) System Card

OpenAI. 2023.GPT-4V(ision) System Card. Technical Report. https://cdn.openai. com/papers/GPTV_System_Card.pdf

2023
[44]

Rishab Parthasarathy, Jasmine Collins, and Cory Stephenson. 2025. What Makes a Good Generated Image? Investigating Human and Multimodal LLM Image Preference Alignment. arXiv:2509.12750 [cs.CV] https://arxiv.org/abs/2509.12750

2025
[45]

Pouya Pezeshkpour and Estevam Hruschka. 2024. Large Language Models Sen- sitivity to The Order of Options in Multiple-Choice Questions. InFindings of the Association for Computational Linguistics: NAACL 2024, Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for ...

2024 doi
[46]

Yukun Qi, Yiming Zhao, Yu Zeng, Xikun Bao, Wenxuan Huang, Lin Chen, Zehui Chen, Jie Zhao, Zhongang Qi, and Feng Zhao. 2025. Vcr-bench: A comprehensive evaluation framework for video chain-of-thought reasoning.arXiv preprint arXiv:2504.07956(2025)

2025 arXiv
[47]

Zelu Qi, Ping Shi, Chaoyang Zhang, Shuqi Wang, Fei Zhao, Da Pan, and Ze- feng Ying. 2025. Towards Holistic Visual Quality Assessment of AI-Generated Videos: A LLM-Based Multi-Dimensional Evaluation Model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

2025
[48]

Haiyi Qiu, Minghe Gao, Long Qian, Kaihang Pan, Qifan Yu, Juncheng Li, Wenjie Wang, Siliang Tang, Yueting Zhuang, and Tat-Seng Chua. 2025. STEP: Enhancing Video-LLMs’ Compositional Reasoning by Spatio-Temporal Graph-guided Self- Training. InProceedings of the IEEE/CVF Conferenc...

2025
[49]

Qwen Team. 2026. Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model. https://qwen.ai/blog?id=qwen3.6-27b

2026
[50]

Benjamin M Rottman and Yiwen Zhang. 2025. Learning about causal relations that change over time: primacy and recency over long timeframes in causal judgments and memory.Cognitive Research: Principles and Implications10, 1 (2025), 9. MM ’26, November 10–14, 2026, Rio de Janeiro...

2025
[51]

Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, and Alessio Devoto. 2025. Attention sinks in diffusion language models.arXiv preprint arXiv:2510.15731(2025)

2025
[52]

Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. 2022. Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities. arXiv:2203.14712 [cs.CV] https://arxiv.org/abs/2203.14712

2022 arXiv
[53]

Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. 2025. Judging the Judges: A Systematic Study of Position Bias in LLM- as-a-Judge. InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference o...

2025
[54]

Alyssa H Sinclair, Yuxi C Wang, and R Alison Adcock. 2024. First impressions or good endings? Preferences depend on when you ask.Journal of Experimental Psychology: General(2024)

2024
[55]

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864 [cs.CL] https://arxiv.org/abs/2104.09864

2023 arXiv
[56]

Hao Sun, Yunyi Shen, and Jean-Francois Ton. 2025. Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alter- natives. arXiv:2411.04991 [cs.AI] https://arxiv.org/abs/2411.04991

2025 arXiv
[57]

Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. 2025. T2v-compbench: A comprehensive benchmark for compositional text- to-video generation. InProceedings of the Computer Vision and Pattern Recognition Conference. 8406–8416

2025
[58]

Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019. COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis. arXiv:1903.02874 [cs.CV] https://arxiv.org/abs/ 1903.02874

2019 arXiv
[59]

Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design.Transactions of the Association for Computational Linguistics12 (2024), 1011–1026. doi:10.1162/tacl_a_00685

2024 doi
[60]

Yibin Wang, Zhimin Li, Yuhang Zang, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. 2025. Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning. arXiv:2505.03318 [cs.CV] https://arxiv.org/abs/2505. 03318

2025
[61]

Kakade, Hao Peng, and Heng Ji

Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, and Heng Ji. 2025. Eliminating Position Bias of Language Models: A Mechanistic Approach. arXiv:2407.01100 [cs.CL] https: //arxiv.org/abs/2407.01100

2025 arXiv
[62]

Xinyi Wu, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. 2025. On the Emergence of Position Bias in Transformers. arXiv:2502.01951 [cs.LG] https: //arxiv.org/abs/2502.01951

2025 arXiv
[63]

Zhixuan Wu, Bo Cheng, Jiale Han, Jiabao Ma, Shuhao Zhang, Yuli Chen, and Changbo Li. 2025. VideoQA-TA: Temporal-Aware Multi-Modal Video Question Answering. InProceedings of the 31st International Conference on Computational Linguistics. 7239–7252

2025
[64]

Zheng Xing and Weibing Zhao. 2026. Temporal Visual Semantics-Induced Human Motion Understanding With Large Language Models.IEEE Transactions on Image Processing35 (2026), 2182–2197. doi:10.1109/TIP.2026.3663857

2026
[65]

Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. 2025. LLaVA-Critic: Learning to Evaluate Multi- modal Models. arXiv:2410.02712 [cs.CV] https://arxiv.org/abs/2410.02712

2025 arXiv
[66]

Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2024. Justice or Prejudice? Quantifying Biases in LLM-as-a- Judge. arXiv:2410.02736 [cs.CL] https://arxiv.org/abs/2410.02736

2024 arXiv
[67]

Hong-Tao Yu, Xiu-Shen Wei, Yuxin Peng, and Serge Belongie. 2025. Benchmark- ing Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehen- sive Evaluation. arXiv:2504.14988 [cs.CV] https://arxiv.org/abs/2504.14988

2025 arXiv
[68]

Tao Yu, Yi-Fan Zhang, Chaoyou Fu, Junkang Wu, Jinda Lu, Kun Wang, Xingyu Lu, Yunhang Shen, Guibin Zhang, Dingjie Song, et al. 2025. Aligning multimodal llm with human preference: A survey.arXiv preprint arXiv:2503.14504(2025)

2025 arXiv
[69]

Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, and Jiaqi Wang. 2025. InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi- Modal Reward Model. arXiv:2501.12368 [cs.CV] https:...

2025 arXiv
[70]

Ziyi Zhang, Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, and Dacheng Tao
[71]

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang
[72]

Xing, Hao Zhang, Joseph E

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL] https://ar...

2023 arXiv
[73]

Xu Zheng, Zihao Dongfang, Lutao Jiang, Boyuan Zheng, Yulong Guo, Zhenquan Zhang, Giuliano Albanese, Runyi Yang, Mengjiao Ma, Zixin Zhang, Chenfei Liao, Dingcheng Zhen, Yuanhuiyi Lyu, Yuqian Fu, Bin Ren, Linfeng Zhang, Danda Pani Paudel, Nicu Sebe, Luc Van Gool, and Xuming Hu. ...

2025
[74]

Confronting reward overoptimization for diffusion models: A perspective of inductive and primacy biases.arXiv preprint arXiv:2402.08552(2024)

2024
[75]

binary collapse

Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. 2019. Cross-task weakly supervised learning from instructional videos. arXiv:1903.08225 [cs.CV] https://arxiv.org/abs/1903.08225 Order Matters: LVLMs as Judges for Tempor...

1957 arXiv
[76]

arXiv:2309.03882 [cs.CL] https://arxiv.org/abs/2309.03882

Large Language Models Are Not Robust Multiple Choice Selectors. arXiv:2309.03882 [cs.CL] https://arxiv.org/abs/2309.03882

arXiv
[79]

Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou
[80]

arXiv:2405.01434 [cs.CV] https://arxiv.org/abs/2405.01434

StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation. arXiv:2405.01434 [cs.CV] https://arxiv.org/abs/2405.01434

arXiv
[785]

doi:10.1162/tacl_a_00673

doi
[2023]

arXiv:2310.05470 [cs.CL] https://arxiv.org/abs/2310.05470

Generative Judge for Evaluating Alignment. arXiv:2310.05470 [cs.CL] https://arxiv.org/abs/2310.05470

arXiv
[2024]

arXiv:2401.06591 [cs.CL] https://arxiv.org/abs/2401.06591

Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation. arXiv:2401.06591 [cs.CL] https://arxiv.org/abs/2401.06591

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Tosin Adewumi, Lama Alkhaled, Namrata Gurung, Goya van Boven, and Irene Pagliai. 2024. Fairness and Bias in Multimodal AI: A Survey. arXiv:2406.19097 [cs.CL] https://arxiv.org/abs/2406.19097

arXiv 2024

[2] [2]

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. InProceedings of the 25th ACM SIGKDD International Confer- ence on Knowledge Discovery & Data Mining(Anchorage, AK, USA)(KDD ’19). Association for Computing Machinery, New York, NY, USA, 2623–2631. doi:10.1...

arXiv 2019

[3] [3]

James F. Allen. 1983. Maintaining knowledge about temporal intervals.Commun. ACM26, 11 (1983), 832–843

1983

[4] [4]

Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. Mllm-as-a- judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning

2024

[5] [5]

Luyu Chen, Zeyu Zhang, Haoran Tan, Quanyu Dai, Hao Yang, Zhenhua Dong, and Xu Chen. 2025. Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge.arXiv preprint arXiv:2505.12301(2025)

work page Pith review arXiv 2025

[6] [6]

Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming- Hsuan Yang, and Sergey Tulyakov. 2024. Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 13320–13331

work page 2024

[7] [7]

Wei Chen, Lin Li, Yongqi Yang, Bin Wen, Fan Yang, Tingting Gao, Yu Wu, and Long Chen. 2025. CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8073–8082

work page 2025

[8] [8]

Muzhi Dai, Jiashuo Sun, Zhiyuan Zhao, Shixuan Liu, Rui Li, Junyu Gao, and Xue- long Li. 2025. From Captions to Rewards (CAREVL): Leveraging Large Language Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Model Experts for Enhanced Reward Modeling in Large Vision-Language Models. ...

arXiv 2025