{"id":"56b5aac0-d165-4e3b-92a6-ba7832a276fe","arxiv_id":"2411.18499","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A large new benchmark and an offline judge model for open-ended interleaved image-text generation, with IntJudge matching human agreement better than GPT-4o.","lead":"This paper introduces OpenING, a new benchmark with 5,400 human-annotated examples for evaluating models that generate mixed text and images. It also presents IntJudge, a trained judge model that agrees with human preferences 82.42% of the time, beating GPT-4o as a judge by about 11 percentage points.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on human pairwise preferences that are never validated for inter-annotator reliability; without such statistics, the 82.42% agreement rate and the benchmark rankings are not interpretable.","rationale":"The reader's weakest assumption identifies both missing inter-annotator agreement and synthetic RAG labels. I agree that the missing IAA is the most load-bearing issue; the RAG labels are a training-data concern, but they would be less damaging if the human ground truth were demonstrated reliable. The IAA check also addresses generalizability beyond the 28 annotators. The paper is otherwise carefully described and the experiments are extensive, so the appropriate action is to request this validation before treating OpenING and IntJudge as definitive evaluation tools, which matches the CONDITIONAL verdict.","tokens_in":38768,"tokens_out":5047,"duration_ms":47166,"concrete_test":"Sample 200 Test Set battle pairs stratified across meta-topics. Have each pair independently judged by at least 3 annotators from the original pool and by a separate group of 3-5 fresh annotators recruited from outside the benchmark team. Compute pairwise inter-annotator agreement (Cohen's kappa or Krippendorff's alpha) within the original pool and between original and fresh judgments. Also compare IntJudge's predictions to the fresh annotators' majority vote. If original-pool agreement is below 0.6, or if IntJudge's agreement with the fresh majority is materially below the reported 82.42%, the ground truth is too unstable to support the central claim and the agreement numbers need to be revised or re-estimated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"OpenING's ground truth and IntJudge's evaluation target are the pairwise judgments of 28 annotators described in Sec. 4.1-4.2 and Sec. A.5. The paper never reports how many annotators judged each battle pair, whether any pairs were double-annotated, or any inter-annotator agreement statistic (Cohen's kappa, Krippendorff's alpha). Table 3 reports IntJudge's 82.42% FDT agreement with 'human judgments' as if that target were a single stable gold label, but if the annotators themselves disagree on a substantial fraction of open-ended comparisons, the target is noisy and the agreement number is inflated or at best uninterpretable. The same annotator pool also produced the gold answers used both as benchmark ground truth and as the 'winner' in RAG training data (Sec. 4.3), creating a shared preference for gold-answer-augmented outputs that may inflate IntJudge's apparent alignment. The paper's own Sec. F admits potential biases in crowdsourced aesthetic judgments, but no reliability analysis is offered. This is load-bearing because both the benchmark's 'high-quality' claim and IntJudge's headline improvement over GPT-4o are measured against this unvalidated signal.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OpenING, a benchmark of 5,400 human-annotated interleaved image-text instances spanning 56 tasks across 23 meta-topics, and IntJudge, a Qwen2-VL-7B-based judge model trained on pairwise preferences from a Dev Set plus Reference-Augmented Generation (RAG) pairs. The authors report that IntJudge agrees with human judgments at 82.42% (FDT metric), an 11.34% improvement over GPT-4o, and they use the benchmark to rank ten to fourteen interleaved generation methods. The benchmark, judge model, and leaderboard are positioned as the paper's main contributions.","tokens_in":39135,"tokens_out":5635,"duration_ms":52903,"significance":"If the human ground truth is reliable, OpenING is a valuable resource: it is substantially larger and broader than OpenLEAF and InterleavedBench, it provides a reproducible open-source offline judge, and the RAG-based data pipeline for scaling judge training is a useful practical contribution. The paper also offers a detailed leaderboard, per-criterion analyses, and error analyses that can inform future model development. The key caveat is that the human-preference signal is treated as a stable gold label without any reliability or uncertainty analysis, so the headline agreement numbers and rankings are currently not fully interpretable.","major_comments":[{"comment":"The benchmark's ground truth is a set of pairwise human judgments, but the paper never reports how many annotators judged each battle pair, whether any pairs were double-annotated, or any inter-annotator agreement statistic such as Cohen's kappa or Krippendorff's alpha. Without such a reliability analysis, the 82.42% agreement figure in Table 3 and the win-rate rankings in Table 2 have an unvalidated target. Please add a reliability study on a double-annotated subset, describe the per-pair annotation workflow, and state how disagreements among annotators were resolved.","section":"Sec. 4.1, 4.2, A.5; Table 3"},{"comment":"The central quantitative claims are reported as point estimates without uncertainty quantification. The 82.42% vs. 71.08% agreement difference and the mid-table win-rate differences (e.g., SEED-X at 51.98% vs. Anole at 51.90% in human FDT) could easily be within noise. Please provide bootstrap confidence intervals for the agreement rates and win rates, and a paired significance test for the IntJudge versus GPT-4o comparison and for the key ranking differences.","section":"Table 3, Sec. 5.2, Fig. 7"},{"comment":"The RAG training pairs are labeled by construction: the gold-answer-augmented output is always assigned as the winner, and the gold answers were produced by the same annotator pool that created the human-preference labels used for evaluation. The 7.8% FDT improvement on unseen models from RAG could therefore partly reflect a preference for those annotators' style rather than a general quality signal. Please provide evidence that the improvement transfers to independent human judgments, for example by evaluating on a held-out annotator group or by showing that the RAG-trained judge's agreement is stable across annotator subpopulations.","section":"Sec. 4.3, Fig. 8"}],"minor_comments":[{"comment":"The main text says 10 representative interleaved methods were evaluated, but the supplementary Table 12 reports 14 models; please harmonize the count and clarify which models the '10' refers to.","section":"Sec. 5.1 vs. Table 12"},{"comment":"The symbol g in 'A bag of models, including g seen interleaved generation methods' is introduced but never defined; please define it or rephrase.","section":"Sec. 4.3"},{"comment":"Equation (5) is typeset in a way that makes the coverage-time formula hard to parse; please rewrite with explicit floor/ceiling notation and define every symbol.","section":"Sec. 4.1, Eq. (5)"},{"comment":"The phrase 'tested in a zero-shot setting on both unseen and seen models' is misleading for seen models, since IntJudge was trained on Dev Set outputs from those models; please clarify that the evaluation is on held-out instances rather than zero-shot in the usual sense.","section":"Sec. 4.3"},{"comment":"The claim of 'reducing data contamination risks' is not fully supported, because several OpenING tasks reuse public sources such as VIST and WikiHow that also appear in prior interleaved benchmarks; please document explicit overlap checks with evaluated models' training corpora and with prior benchmarks.","section":"Sec. 2, Table 1"},{"comment":"There are several typos and formatting issues, including 'favarable' in Section 5.2 and 'promopts' in the Table 11 header; a careful proofreading pass is recommended.","section":"Various"},{"comment":"The limitation statement in Section F acknowledges potential biases from crowdsourced aesthetic judgments, but the main text around Table 3 does not reference this limitation; please cross-link the two so that readers see the caveat where the headline numbers are presented.","section":"Sec. F"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid contribution if the reliability gap can be closed. The editor may want to ask the authors to disclose whether the same annotators who created the gold answers also served as human judges, and to provide a reliability analysis before the benchmark claims are accepted. The RAG training-data labeling rule should also be documented in the main text, not only in the supplementary material."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"OpenING is a real step up in scale and diversity for interleaved image-text generation benchmarks: 5,400 instances across 56 tasks and 23 meta-topics, versus roughly 10 tasks and under a thousand instances in prior work. The annotation pipeline is concrete—an in-house tool, 28 annotators under expert supervision, explicit filtering protocols. IntJudge, an offline judge trained on Dev Set preferences plus RAG-augmented data, is a useful contribution because it removes API dependence and reports 82% agreement with human pairwise judgments, beating GPT-4o by a wide margin. The experiments are thorough: win rates under four tie-handling metrics, ablations on sampling size and RAG data, and evaluation on unseen models.\n\nThe main soft spot is the one the stress-test flags. The human pairwise preferences are both the benchmark's ground truth and IntJudge's training/evaluation target, yet the paper never reports inter-annotator agreement—no kappa or alpha, no statement of how many annotators judged each battle or whether any pairs were double-annotated. Without that, the 82.42% agreement rate is hard to interpret; it may be inflated relative to a noisy target, and the model rankings may be less stable than they look. The RAG training data also always marks the gold-augmented output as the winner, which is a synthetic rule rather than independent human judgment—fine as data augmentation, but not a human label. The paper also admits using AI-generated content in the \"human-annotated\" set in tricky cases, which further blurs the gold standard.\n\nThese are fixable, not fatal. The benchmark itself is still a useful resource, and the judge model is plausible. But the headline agreement and the rankings should be treated cautiously until the authors provide reliability statistics and confidence intervals. I recommend sending this to peer review; it deserves referee time. With a revision reporting IAA and clarifying the construction of synthetic labels, this could become a standard evaluation platform for interleaved generation.","headline":"A genuinely larger and more diverse benchmark for interleaved generation, but IntJudge's headline agreement rests on human preferences that are never validated for reliability.","tokens_in":39594,"tokens_out":2424,"would_cite":true,"duration_ms":24142,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenING claims a 5,400-instance benchmark for mixed image-text generation and a judge model, IntJudge, that matches human pairwise preferences 82.42% of the time, outperforming GPT-4o-based evaluation by 11.34 percentage points.","keywords":["interleaved image-text generation","multimodal evaluation","benchmark","judge model","human preference alignment","reference-augmented generation","pairwise comparison","open-ended generation"],"falsifier":"Take a random subset of 200 OpenING battle pairs, have two independent annotation teams judge them, and compute the agreement between teams; if inter-annotator agreement is markedly below IntJudge's 82.42% agreement with the original labels, then IntJudge is matching one noisy label set rather than measuring true quality. Separately, run a human preference test on the RAG training pairs: if humans do not prefer the gold-augmented output at near 100%, the RAG 'winner' labels embed a bias that inflates judge accuracy.","tokens_in":38588,"feed_emoji":"🖼️","tokens_out":4582,"duration_ms":38478,"temperature":0.7,"pith_summary":"The paper sets out to fill a gap in how researchers measure models that produce interleaved image-and-text answers, such as a travel itinerary with a picture at each step. It introduces OpenING, a benchmark of 5,400 human-annotated examples spanning 56 everyday tasks, and IntJudge, a compact open judge model trained to compare two such outputs and pick the better one. The central claim is that IntJudge agrees with human preference 82.42% of the time, beating GPT-4o-based judging by 11.34 percentage points. If true, open-ended multimodal generation can be evaluated offline, cheaply, and reproducibly instead of relying on proprietary APIs or slow human panels.","feed_headline":"Open judge model beats GPT-4o at scoring image-text answers","feed_subtitle":"New 5,400-example benchmark lets models be evaluated offline; IntJudge agrees with human preferences 82.42% of the time.","key_machinery":"The load-bearing object is IntJudge, a Qwen2-VL-7B model fine-tuned with a four-term loss (cross-entropy, contrastive, MSE, and pairwise ranking) on 31,996 training samples: 6,014 arena pairwise judgments plus 25,982 reference-augmented pairs. The RAG pipeline generates contrastive pairs by prompting seen generators with gold answers and always labeling the gold-augmented output the winner; the paper reports that adding these pairs raises force-dividing-tie agreement on unseen models by 7.8%. Evaluation runs in an Interleaved Arena where two anonymous outputs are compared on seven ordered criteria—correctness, image-text coherency, multi-step consistency, content quality, human preference alignment, completeness, and content richness—with ties resolved by forced leaning judgments.","core_discovery":"The paper claims that open-ended interleaved generation can be judged reliably by a compact model trained on pairwise human preferences plus reference-augmented data. On 4,320 battle pairs sampled from the OpenING Test Set, IntJudge matches human decisions at 82.42% agreement under the force-dividing-tie metric, outperforming GPT-4o (71.08%) and random guessing (49.83%), with stronger agreement on seen models (84.05%) than unseen models (80.75%). The paper also reports that all current generators trail human-annotated content, that integrated pipelines such as GPT-4o+DALL·E-3 and Gemini1.5+Flux lead the leaderboard, and that end-to-end models such as Anole, MiniGPT-5, GILL, and NExT-GPT lag substantially.","pith_inferences":["The RAG training signal labels gold-augmented outputs as winners by construction, so part of IntJudge's agreement gain may come from learning that rule rather than from genuine human preference; a human study on RAG pairs would reveal how much of the 82.42% is preference alignment versus shortcut learning.","The human ground truth itself lacks reported inter-annotator agreement, so the 82.42% figure is an agreement with one label set; measuring pairwise agreement between independent annotation teams on a subset would calibrate the ceiling of judge performance.","If IntJudge is adopted as a reward model for reinforcement learning, as the paper hints, its bias toward complete reference-augmented answers and against generated images would shape model behavior; a testable extension is training a generator with IntJudge rewards and checking whether human preference actually improves.","The benchmark's 23-meta-topic structure invites per-topic diagnostic leaderboards: models that win overall may lose on GUI navigation or embodied-AI tasks, which could guide where next-generation training data should come from."],"forward_implications":["IntJudge can replace GPT-based scoring on OpenING with higher human agreement, enabling reproducible offline leaderboards for interleaved generation.","The 56-task Test Set provides a zero-shot evaluation suite for future interleaved models, including models never seen during judge training.","Integrated pipelines currently outperform end-to-end generators in image-text coherence and visual quality, while unified two-stage and end-to-end architectures retain substantial headroom.","Image-generator quality is a dominant factor: swapping in Flux-dev for Flux-schnell or SEED-X images changes win rates dramatically, so progress in image generators will directly lift interleaved generation.","Fine-tuning a small generator on the OpenING Dev Set improves its interleaved output, as shown by MiniGPT-5OpenING outperforming the MiniGPT-5 baselines across all tie-handling metrics."],"supporting_citations":[{"why":"OpenLEAF is the prior open-domain interleaved benchmark with only two meta-topics, against which OpenING positions its broader coverage.","marker":"[4]"},{"why":"InterleavedBench supplies the main prior interleaved benchmark with 815 instances, the comparison baseline showing OpenING's larger scale and diversity.","marker":"[43]"},{"why":"GPT-4o is used both as the text generator in the integrated pipeline and as the GPT-based judge baseline that IntJudge is measured against.","marker":"[48]"},{"why":"Qwen2-VL-7B is the foundation model selected for IntJudge after exploring InternLM-XComposer2.5 and Qwen2-VL variants.","marker":"[69]"},{"why":"MT-Bench and Chatbot Arena motivate the arena-style pairwise comparison methodology and the tie-handling instructions used in the Interleaved Arena.","marker":"[93]"},{"why":"MLLM-as-a-Judge provides evidence that pairwise comparison is more stable than subjective scoring and that judge models carry inherent bias.","marker":"[14]"},{"why":"K-sort arena informs the k-wise preference sampling and roulette matching algorithm used to form battle pairs.","marker":"[38]"},{"why":"Chameleon is the autoregressive mixed-modal model family that Anole fine-tunes from, anchoring the end-to-end generator category in the evaluation.","marker":"[63]"},{"why":"Anole serves as the representative end-to-end interleaved generator evaluated on OpenING and found to lag integrated pipelines.","marker":"[16]"}],"fun_headline_variants":["IntJudge beats GPT-4o by 11% in image-text answer scoring","New benchmark tests interleaved image-text generation with 5,400 tasks","IntJudge matches human preference 82% for scoring generated image-text","OpenING: 5,400 human-annotated tasks expose gaps in interleaved generation","Judge model for interleaved generation scores 82% human agreement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on the assumption that the pairwise preferences collected from 28 annotators are a reliable and unbiased ground truth for open-ended interleaved quality; the paper reports no inter-annotator agreement, and part of IntJudge's training signal is constructed by a rule that always favors gold-augmented outputs.","fun_headline_variants_meta":{"raw":{"variants":["IntJudge beats GPT-4o by 11% in image-text answer scoring","New benchmark tests interleaved image-text generation with 5,400 tasks","IntJudge matches human preference 82% for scoring generated image-text","OpenING: 5,400 human-annotated tasks expose gaps in interleaved generation","Judge model for interleaved generation scores 82% human agreement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000752,"raw_usage":{"total_tokens":3339,"prompt_tokens":928,"completion_tokens":2411,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2319}},"tokens_in":544,"tokens_out":2411,"duration_ms":14461,"temperature":1.0,"reasoning_tokens":2319,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:07:15.702408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subset of 200 OpenING battle pairs, have two independent annotation teams judge them, and compute the agreement between teams; if inter-annotator agreement is markedly below IntJudge's 82.42% agreement with the original labels, then IntJudge is matching one noisy label set rather than measuring true quality. Separately, run a human preference test on the RAG training pairs: if humans do not prefer the gold-augmented output at near 100%, the RAG 'winner' labels embed a bias that inflates judge accuracy.","supporting_citations":[{"cited_title":"Hello GPT-4o","cited_arxiv_id":null,"evidence_quote":"GPT-4o is used both as the text generator in the integrated pipeline and as the GPT-based judge baseline that IntJudge is measured against."},{"cited_title":"Judging LLM-as-a-judge with MT-Bench and Chatbot Arena","cited_arxiv_id":null,"evidence_quote":"MT-Bench and Chatbot Arena motivate the arena-style pairwise comparison methodology and the tie-handling instructions used in the Interleaved Arena."}],"review_version":1}