{"id":"dd0d691a-e81e-4e38-bb02-b1d0b332f33a","arxiv_id":"2411.17188","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ISG evaluates interleaved text-and-image outputs with four question-answering levels and a 1,150-sample benchmark, showing unified models lag compositional agents by a wide margin.","lead":"This paper introduces ISG, a system that automatically checks AI answers that mix text and images by asking hundreds of small yes/no and scoring questions about structure, blocks, and image contents. It also releases a 1,150-sample benchmark where unified models score far below a plan-then-execute agent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1 drops structurally mismatched responses from block/image scoring, so Table 6 compares different per-model subsets (pass rates 29.5%, 32.3%, 38.5%, 87.1%); the claimed fine-grained rankings may be an artifact of sample selection.","rationale":"The central claim is that ISG is a human-aligned automatic evaluator and that ISG-Bench yields meaningful model rankings. The reader identified the structural gate as fragile; my pass sharpens that concern: the gate does not merely risk miscalibration, it produces a measurable selection asymmetry in the headline tables. Because Algorithm 1 returns early on structure mismatch, all fine-grained scores are conditional on a per-model event with very different probabilities. Table 6 does not disclose this conditioning, so direct comparisons of block and image averages across models are not apples-to-apples. Recomputing on a common subset is a direct and inexpensive test. This does not defeat the framework—holistic results and structural accuracies are still informative—but it makes the block/image ranking claim conditional until addressed. Therefore the reader's CONDITIONAL verdict remains appropriate, and no change to the overall verdict is needed.","tokens_in":43801,"tokens_out":4710,"duration_ms":45830,"concrete_test":"Reproduce Table 6 block and image rows on the intersection of samples for which Show-o, Claude+SD3, Gemini+SD3, and ISG-Agent all pass structural matching, and recompute the averages; if ISG-Agent's lead persists on this common subset, the selection concern is mitigated. Also report per-cell sample counts and bootstrap 95% confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 and Algorithm 1 (lines 3-5) route any response failing StructureMatch to holistic-only evaluation, so block and image scores in Table 6 can only be computed on the subset of samples that matches the predicted structure. With structural accuracies of Show-o 0.295, Claude+SD3 0.323, Gemini+SD3 0.385, and ISG-Agent 0.871 (Table 5), the block/image averages in Table 6 are not over a common set of samples. ISG-Agent is scored on nearly the whole benchmark while compositional baselines are scored on roughly a third; unified models with 0.000 structural accuracy are absent from those rows entirely. If structure-match probability correlates with task difficulty or with text-image consistency, the headline comparison that ISG-Agent leads all levels, and the unified-versus-compositional gap, are contaminated by selection. The paper presents these as unconditional model-level scores without reporting per-model sample counts, pass rates, confidence intervals, or any statement that the scores are conditional on structural matching.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ISG, an automatic multi-granularity evaluation framework for interleaved text-and-image generation. ISG parses a user query into a scene-graph-like structure, then evaluates generated responses at four levels: holistic, structural, block, and image. The authors introduce ISG-Bench, a benchmark of 1,150 manually collected samples across 8 categories and 21 subcategories with golden answers, and report that ISG aligns with human judgments (block-level Pearson 0.718, image-level accuracy 0.907, holistic agreement 0.730). They evaluate several unified and compositional generation systems, reporting that unified models underperform compositional pipelines, and present ISG-Agent, a plan-execute-refine agent that achieves the highest scores on the benchmark. The paper also provides extensive appendices with prompts, task definitions, case studies, cost analysis, and safety checks.","tokens_in":25,"tokens_out":4698,"duration_ms":96193,"significance":"If the claims hold, this is a useful contribution to multimodal generation evaluation: it provides a fine-grained, interpretable alternative to holistic LLM-as-a-judge, a benchmark with golden answers and vision-centric tasks, and transparent evaluation prompts. The paper is also commendable for including ablation studies on vision input and few-shot prompting, cost estimates, safety filtering, and case studies. However, the headline claim that ISG outperforms previous evaluation methods is not directly supported by the experiments as reported, and the fine-grained model rankings in Table 6 are vulnerable to a sample-selection artifact. The moderate human-correlation numbers and the heavy reliance on GPT-4o and author annotations further temper the significance until these points are addressed.","major_comments":[{"comment":"The structural gate introduces a selection bias in Table 6. Algorithm 1 routes any response that fails StructureMatch to holistic-only evaluation, so block-level and image-level scores are computed only on the subset of responses whose structure matches the predicted structure. According to Table 5, the structural pass rates are 0.295 for Show-o, 0.323 for Claude+SD3, 0.385 for Gemini+SD3, and 0.871 for ISG-Agent, while all unified models except Show-o have 0.000 pass rates and are absent from the block/image rows entirely. The block and image averages in Table 6 are therefore not computed over a common set of samples, and if structural compliance correlates with task difficulty or with text-image consistency, the claimed rankings and the unified-versus-compositional gap are confounded. The paper should report per-model sample counts and pass rates alongside Table 6, explicitly state that the fine-grained scores are conditional on structural matching, or provide a matched-subset analysis as a robustness check.","section":"Section 3.1 and Algorithm 1 (Appendix D.1, lines 3-5)"},{"comment":"The central claim that ISG 'outperforms previous evaluation methods in alignment with humans' is not supported by the reported experiments. Table 3 reports Pearson correlations and accuracies for ISG's own modules, but there is no head-to-head comparison with previous evaluation methods (e.g., TIFA, VQAScore, CLIPScore, or GPT-4V-as-a-judge) on the same human-annotated samples. Without such a comparison, the claim of superiority cannot be verified. The authors should either add direct comparisons on the same annotations or revise the claim to state that ISG achieves these alignment numbers without claiming superiority over prior methods.","section":"Section 1 and Table 3"},{"comment":"The structural parser is reported to achieve perfect accuracy on golden answers, yet this component is load-bearing because it gates all fine-grained evaluation in Algorithm 1. The paper does not provide error analysis, confidence scores, or per-category breakdowns for this perfect number, and a small parser miscalibration could either exclude valid responses or admit invalid ones, changing the block and image results in Table 6. The authors should report structural prediction accuracy on held-out or adversarial queries, show sensitivity to tolerance in block order and image count, and analyze how parser errors propagate to the final scores.","section":"Section 3.1 and Table 3, 'Structural Direct Match Accuracy 1,150 1.000'"},{"comment":"The human-alignment validation is weakened by the fact that the annotations were produced by six of the paper's authors, while GPT-4o was used both to generate benchmark questions and golden answers and to serve as the VQA judge. This creates a risk of systematic agreement between the evaluator and the benchmark construction process rather than independent evidence of validity. The authors should report inter-annotator agreement statistics, clarify the independence of the human annotators, and provide an analysis of how much of the reported alignment depends on using the same model family for question generation and answer judging.","section":"Appendix B.3 and Section 4.1"}],"minor_comments":[{"comment":"The title in the arXiv metadata ('Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment') differs from the running header in the PDF ('Interleaved Scene Graphs for Text-and-Image Generation Evaluation'); the two should be unified.","section":"Title and running header"},{"comment":"The figure caption and Table 1 contain unrendered LaTeX commands such as '/ctre/fnt' and 'MGHolistic', making the dominance classification and column labels hard to read.","section":"Figure 1 caption and Table 1"},{"comment":"The example query 'How do to make egg fried rice?' contains a grammatical error; it should read 'How do I make egg fried rice?'","section":"Abstract"},{"comment":"The Q-Gen accuracy uses a BERTScore threshold of 0.8, but the paper does not report sensitivity to this threshold. Since it is a free parameter, a small sensitivity analysis would strengthen the claim that the question generation module is accurate.","section":"Section 4.1"},{"comment":"Model names are inconsistent: the paper alternates between 'Seed-Llama-14b', 'SEED-LLaMA', and 'Seed-Llama-14B'; a single naming convention should be used throughout.","section":"Section 4.2 and Appendix D.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid benchmark and framework contribution, but the selection-bias issue in Table 6 and the lack of a direct head-to-head comparison with prior metrics are load-bearing for the main claims. I would ask the authors to address these points before acceptance, and to strengthen the independence of the human annotation protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious, useful piece of work that deserves referee time, but the fine-grained model rankings have a selection problem that the paper does not address.\n\nThe stress-test note is right. Algorithm 1 routes any response that fails structural matching to holistic-only evaluation. Table 6 then reports block and image averages over very different subsets: Show-o is scored on roughly 29.5% of samples, Claude+SD3 on 32.3%, Gemini+SD3 on 38.5%, and ISG-Agent on 87.1%. If structural-match probability correlates with task difficulty or text-image consistency, which it plausibly does, the claimed agent superiority and the unified-versus-compositional gap are at least partly artifacts. The paper never reports per-model sample counts, confidence intervals, or even a sentence saying the scores are conditional on structure matching. That is the load-bearing flaw for the model comparison.\n\nWhat is actually new: ISG transfers scene-graph QA evaluation to interleaved generation, with four levels and golden answers. ISG-Bench is not huge, but it covers vision-centric tasks that prior interleaved benchmarks mostly miss, like style transfer and 3D scene transformation. ISG-Agent is a reasonable compositional baseline with a clean plan-execute-refine design and ablations. The human-alignment numbers are moderate (block Pearson 0.718, image accuracy 0.907, holistic agreement 0.730 with golden answer), but the framework is described in enough detail that someone could reimplement it. The paper is also honest about limitations, includes a cost analysis and safety filtering, and makes a real effort to avoid benchmark contamination.\n\nSoft spots beyond the structural gate: there is no head-to-head comparison against InterleavedBench, MMIE, or GATE OpenING as evaluators on the same annotations; human annotations were done only by the authors, with no inter-annotator agreement reported; and GPT-4o is used both to build the benchmark and to judge it, a circularity concern that is mitigated but not eliminated by the independent human-alignment study. The benchmark data also do not appear to be shipped, which hurts reproducibility.\n\nBottom line: someone working on interleaved generation should read and cite this paper. The metric is plausible and the benchmark fills a real gap. But Table 6 needs re-analysis conditional on structure match, ideally with per-subset sample counts and confidence intervals. A serious referee would engage with this and ask for major revisions rather than desk-rejecting it.","headline":"A genuinely useful four-level evaluation framework and benchmark for interleaved text-and-image generation, but the fine-grained model rankings in Table 6 are contaminated by the structural-matching gate and should not be taken at face value.","tokens_in":44563,"tokens_out":1907,"would_cite":true,"duration_ms":19977,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that interleaved text-and-image generation can be judged automatically at four levels of detail, and that this automated judgment tracks human judgment closely.","keywords":["interleaved text-and-image generation","scene graph","automatic evaluation","benchmark","multimodal large language models","visual question answering","generative agents"],"falsifier":"Rephrase 100 ISG-Bench queries into new phrasings with identical intended output structure and run the structure predictor on them; any mispredicted structure (accuracy below 1.0) would show that responses with correct content but unusual wording are silently dropped from block and image scoring, undermining the reported alignment.","tokens_in":43628,"feed_emoji":"🖼️","tokens_out":7514,"duration_ms":59689,"temperature":0.7,"pith_summary":"The paper's central claim is that interleaved text-and-image generation can be judged automatically at four levels of detail, and that this automated judgment tracks human judgment closely. It introduces the Interleaved Scene Graph (ISG), which parses a multimodal query into a graph of text blocks, image blocks, and the relations between them, then grades a generated response from four angles: holistic quality, structural compliance, block-level fidelity, and image-specific content. On its own 1,150-sample benchmark, ISG-Bench, the block-level VQA score reaches a Pearson correlation of 0.718 with human ratings and the image-level yes/no accuracy reaches 0.907. Using this metric, the paper reports that state-of-the-art unified models score about 2.961 at the holistic level while the best compositional pipeline scores 6.262, and that an agent baseline it builds leads at every level. A sympathetic reader should care because current LLM-as-a-judge evaluations are coarse and ambiguous, and if ISG holds up it gives the field a fine-grained, interpretable measuring stick for a task now dominated by unified generative models.","feed_headline":"Scene graphs score interleaved text-image answers like a human judge","feed_subtitle":"Agrees with human raters and reveals unified models lag far behind compositional pipelines.","key_machinery":"The load-bearing object is the Interleaved Scene Graph, a representation in which text blocks and image blocks are nodes and open-vocabulary relations (such as 'describes' or 'the next step after') are edges. ISG first has an LLM predict the response's required structure from the query and rejects with holistic-only scoring if the generated response does not match that structure exactly. For responses that pass, it generates questions from the graph at block level (subject-object-relation triples) and image level (entities, attributes, and relations), and answers them with an MLLM-based VQA module using a 1-10 score or yes/no. This multi-level question-answer machinery is what turns a graph parse into a human-aligned number.","core_discovery":"On its own terms, the paper establishes that a scene-graph representation of a user query can drive a four-level automatic evaluation that outperforms prior evaluation methods in agreement with human judgment. ISG parses the query into structure, block-relation tuples, and image entity-attribute-relation tuples; an MLLM converts each element into questions, and a VQA module answers them against the generated response. The paper reports structural match accuracy of 1.000 on golden answers and human-alignment numbers of 0.718 (block-level scoring) and 0.907 (image-level yes/no). The benchmark built on this machinery, ISG-Bench, contains 1,150 vision-dependent queries with golden answers across 21 subcategories, and the evaluation of ten models leads the authors to conclude that unified models are far behind compositional pipelines, that vision-dominated tasks are the hardest, and that a plan-execute-refine agent outperforms all baselines at all four levels.","pith_inferences":["Editorial inference: ISG's numbers are tied to GPT-4o as both question generator and VQA module; if the framework were re-run with an open-weight MLLM, the reported Pearson correlations and model rankings could change, so the metric is currently contingent on one proprietary judge.","Editorial inference: the structural-match gate turns structure following into a binary on/off switch, zeroing block/image scores for any deviation; a graded or soft structural score would let the metric distinguish 'right content, wrong order' from 'no structure at all'.","Editorial inference: the 111% and 122% improvement figures compare two families of models on ISG's own scoring scale; a direct human preference test between a unified model output and an ISG-Agent output would provide a check of whether the ranking reproduces human taste rather than just human-annotated question answers."],"forward_implications":["If ISG is accepted as a metric, fine-grained automatic evaluation of interleaved generation becomes practical: it produces interpretable question-answer feedback for each block and image, not just one holistic score.","The benchmark results imply that current unified text-and-image models cannot yet follow structural or vision-centric instructions, so the bottleneck is instruction following and multi-image consistency, not raw generation ability.","The large gap between unified and compositional pipelines suggests a concrete research direction: better interleaved training data and pipelines, possibly built with agent-based synthesis, could close most of the 111% holistic gap.","Because golden answers substantially raise holistic judge agreement, any future interleaved benchmark should include reference answers rather than rely on free-form judging."],"supporting_citations":[{"why":"Supplies the VQA-based question-answer evaluation paradigm (TIFA) that ISG extends from text-to-image to interleaved generation.","marker":"Hu et al., 2023"},{"why":"Provides the scene graph representation (Visual Genome) that ISG borrows to define nodes, edges, and open-vocabulary relations.","marker":"Krishna et al., 2017"},{"why":"Shows how scene graphs are used to condition image generation, the conceptual anchor for using graphs to evaluate multimodal output.","marker":"Johnson et al., 2018"},{"why":"OpenLeaf is the interleaved generation benchmark and MLLM-as-a-judge baseline that ISG-Bench improves on by adding golden answers and multi-granular levels.","marker":"An et al., 2023"},{"why":"An interleaved generation benchmark that relies on holistic evaluation; its coarse scoring is what ISG's four-level protocol is designed to replace.","marker":"Liu et al., 2024d"},{"why":"GPT-4o is the proprietary MLLM used as the question-generation and VQA module for every ISG number reported; removing it would change all measurements.","marker":"OpenAI, 2024"}],"fun_headline_variants":["Scene-graph eval exposes weak unified interleaved generators","Interleaved text-image answers get a scene-graph grade from ISG","New ISG-Bench: unified models fail, agent pipeline wins by 122%","Scene-graph judge scores interleaved generation at four levels","Interleaved image-text AI: scene-graph questions reveal large gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the LLM that predicts the required output structure is essentially perfect, so any response with a mismatched structure is excluded from fine-grained scoring rather than being scored on its content.","fun_headline_variants_meta":{"raw":{"variants":["Scene-graph eval exposes weak unified interleaved generators","Interleaved text-image answers get a scene-graph grade from ISG","New ISG-Bench: unified models fail, agent pipeline wins by 122%","Scene-graph judge scores interleaved generation at four levels","Interleaved image-text AI: scene-graph questions reveal large gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000371,"raw_usage":{"total_tokens":2020,"prompt_tokens":1014,"completion_tokens":1006,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":911}},"tokens_in":630,"tokens_out":1006,"duration_ms":9326,"temperature":1.0,"reasoning_tokens":911,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:24:30.893786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rephrase 100 ISG-Bench queries into new phrasings with identical intended output structure and run the structure predictor on them; any mispredicted structure (accuracy below 1.0) would show that responses with correct content but unusual wording are silently dropped from block and image scoring, undermining the reported alignment.","supporting_citations":[],"review_version":1}