{"id":"aff3f940-5c00-4983-9bc3-15660ea5eeff","arxiv_id":"2608.13112","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Physics diagram generation becomes markedly more physically correct when training captions are replaced with a structured schema that separates drawn facts from inferred reasoning, and when evaluation checks individual physical facts.","lead":"This paper presents Princigram, a text-to-image system for physics diagrams trained with a structured, step-by-step description of the physics in each image instead of a flat caption. It reports large gains in physical faithfulness over existing generators, arguing that better supervision, not bigger models, is what makes scientific diagrams correct.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-supervision vs inference-prompt confound: baseline BAGEL/DiMOO are compared raw-prompt, while Princigram uses SP-CoT prompt expansion, so 'supervision is the key lever' is not yet isolated.","rationale":"The reader's stated weakest assumption was the reliability of the VLM judge and expert annotations. My concern is different but related: the headline comparisons conflate SP-CoT training supervision with SP-CoT inference-time prompt expansion, which directly undermines the causal claim that structured supervision, not prompting or architecture, is the key lever. The reader's rationale briefly notes the GenExam baseline ambiguity, so there is partial overlap. The concern is concrete and testable with a single control experiment that the paper does not report. Because this does not overturn the paper's conditional status, the reader's CONDITIONAL verdict is unchanged.","tokens_in":40318,"tokens_out":4967,"duration_ms":49685,"concrete_test":"Run unmodified BAGEL (and optionally DiMOO) with the same inference-time SP-CoT prompt-expansion pipeline (same prompt-expansion model and schema, no SP-CoT training) on VeriphyT2IBench and the GenExam physics subset, under the same judge and scoring. Report Local/Global/Strict and relaxed/strict scores. If BAGEL+SP-CoT-prompt reaches roughly 75 Local or 55 GenExam relaxed, the claimed supervision effect is not isolated; if it stays near 46/14, the confound is resolved and the supervision claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SP-CoT *training supervision* drives the gains rests on comparing unmodified baselines (BAGEL 46.38 Local, 13.8 GenExam relaxed) with Princigram-BAGEL (75.69 Local, 54.8 GenExam relaxed) in Tables 3 and 4. But the unmodified backbone is prompted with the raw user prompt, while Princigram at inference runs the SP-CoT prompt-expansion pipeline (Eq. 13, Section S1.3): an LLM first populates the full schema and the generator conditions on it. The comparison therefore varies two factors simultaneously: training supervision and inference-time structured prompting. Section S1.7 says \"for any baseline run in structured mode\" the prompt is expanded, yet no structured-mode baseline scores appear in Tables 3-6, and the only same-training ablation (Section S3.3) is qualitative. Consequently, the reported 29-35 point Local gains and 4x GenExam relaxed gains may be attributable partly or wholly to giving Princigram a structured 'thinking' prompt rather than to SP-CoT training. The paper's 'supervision is the key lever' conclusion requires that an untrained backbone with the same SP-CoT prompt expansion does not already reach Princigram-level scores.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Princigram, a physics-faithful scientific-diagram generator built around Structured Physical Chain-of-Thought (SP-CoT), a per-subdiscipline JSON schema that decomposes a physics diagram into five reasoning steps (scenario, parameters, structure, laws, synthesis) with a strict visible/inferred grounding split. The authors curate a corpus of 4.3 million physics images, of which 115,037 receive expert-level structured annotations, and train two unified multimodal backbones (BAGEL and DiMOO) on these annotations. At inference, a language model expands a raw user prompt into the SP-CoT schema before generation. They evaluate on the physics subset of GenExam and on a new benchmark, VeriphyT2IBench, whose binary yes/no questions are derived from each held-out diagram's own structured annotation. The central claim is that structured physics supervision, rather than architecture or scale, is the key lever for physically faithful diagram generation.","tokens_in":40592,"tokens_out":5197,"duration_ms":46936,"significance":"If the central claim holds, the paper makes a substantial contribution: a reusable, machine-parseable annotation framework for physics diagrams, a large corpus to support it, and evidence that dense structured supervision can improve physical faithfulness across two mechanistically different backbones. The formal specification in S1.3 (typed fields, grounding map, fidelity rule, serialization, and the shared training/evaluation objective) is a genuine strength and provides a clear substrate for future work. The external GenExam evaluation and the two-backbone comparison are also valuable design choices that partially mitigate concerns about benchmark circularity. However, because the main quantitative comparisons are confounded by inference-time prompting, the magnitude of the claimed supervision effect is not yet established; the qualitative same-training ablation supports the direction but does not quantify it.","major_comments":[{"comment":"The comparison between Princigram and the baselines in Tables 3 and 4 varies two factors simultaneously: training supervision and inference-time structured prompting. Section S1.7 states that \"for any baseline run in structured mode\" the original prompt is expanded into the subdiscipline schema, but no structured-mode baseline scores are reported anywhere in the paper. The raw BAGEL and DiMOO baselines are prompted with the unexpanded prompt, while Princigram uses the SP-CoT expansion of Eq. (13) at inference. Consequently, the reported gains (BAGEL from 46.38 to 75.69 Local on VeriphyT2IBench and from 13.8 to 54.8 relaxed on GenExam) may be inflated by the inference-time structured prompt alone. The central claim in Sections 1 and 3 that \"closing the supervision gap is the key lever\" requires that an untrained backbone prompted with the same SP-CoT expansion does not already reach Princigram-level scores. Please report structured-mode baseline results (e.g., BAGEL and DiMOO with the SP-CoT prompt expansion but without SP-CoT training) or provide a quantitative version of the Section S3.3 ablation on the full benchmarks.","section":"S1.7 / Tables 3-4"},{"comment":"VeriphyT2IBench is constructed from the same structured annotations used to train Princigram: the questions and gold answers are compiled from the SP-CoT fields (Section S1.8). The risk is that the benchmark measures adherence to the training schema rather than independent physical correctness, since the model has been trained to produce images consistent with that schema. The external GenExam results mitigate this concern, but the paper's largest and most detailed claims (Tables 4-6, Figure 3) rest on VeriphyT2IBench. To rule out that the scores are inflated by schema familiarity, the authors should report performance of structured-mode baselines on VeriphyT2IBench or add a human validation subset in which expert raters judge physical correctness and the judge's answers are compared against human judgments. In addition, the inter-annotator consistency of the expert gold annotations is not reported (Section S1.1); since every benchmark gold answer depends on these annotations, a consistency measure (e.g., Cohen's kappa on a sample) would strengthen the reliability of the benchmark.","section":"2.5 / S1.8"},{"comment":"The only same-training ablation (Figure 14) is qualitative and limited to three mechanics prompts. It supports the direction of the claim but does not quantify the effect of SP-CoT supervision on the benchmarks reported in the main tables. This is not a flaw by itself, but it underscores that the quantitative evidence for the \"supervision, not architecture\" claim currently comes from the confounded comparison in Tables 3 and 4. A quantitative version of this ablation (with and without SP-CoT, same pipeline) on VeriphyT2IBench and GenExam would directly test the central claim.","section":"S3.3 / Figure 14"}],"minor_comments":[{"comment":"The abstract says \"4.3 million physics images,\" Figure 2a says \"4.2M,\" and Table 2 totals 4,313,866; please harmonize the numbers.","section":"Abstract / Figure 2a / Table 2"},{"comment":"Section 2.3 states that \"the same model that reasons over the physics also renders the diagram,\" but inference (S1.3, Eq. 13) uses a separate prompt-expansion model Φ to populate the schema; please clarify whether Φ is the same backbone or a distinct model.","section":"S1.3 / Section 2.3"},{"comment":"The strict scoring tolerance is defined as a fraction of wrong answers per item, but the column labels \"Tol. 0%, 5%, 10%\" could be misread as a required accuracy; consider renaming the columns to indicate the maximum allowed error fraction.","section":"Table 6 / Eq. (16)"},{"comment":"The per-step faithfulness scores in Figure 3 are presented without error bars or sample sizes; reporting confidence intervals or the number of items per step would help assess the significance of the 26-42 point improvements claimed in the text.","section":"Figure 3"},{"comment":"The limitation that corpus-level annotations are machine-generated and unverified is correctly acknowledged in Section 3.3; it would be helpful to mention this caveat in the abstract or introduction where the 4.3M corpus is first described.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well-written and the formal framework is a genuine strength. The main gap is the missing structured-mode baseline, which is essential to support the central claim that SP-CoT training supervision, rather than inference-time structured prompting, drives the gains. If the authors can provide that evidence, the paper would be suitable for publication in a major venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The real contribution is the representation, not the model: SP-CoT is a typed, five-step JSON schema with a visible/inferred grounding split, instantiated across six physics subdisciplines. The team uses it to annotate 4.3M physics diagrams (115k expert-verified) and to build VeriphyT2IBench, where each diagram's own annotation becomes a bank of yes/no questions. That is a genuinely useful way to score physical faithfulness fact by fact. The second thing: the central claim—that structured training supervision is the key lever—is not yet isolated, because Princigram gets both structured training and a structured inference-time prompt, while the BAGEL and DiMOO baselines get neither.\n\nThe formal framework in S1.3 is careful and the math is sound: typed fields, fidelity rule, deterministic serializer, and an evaluator that reuses the same annotation. The Figure 14 ablation, same model and pipeline with and without the structured annotation, is the right kind of evidence, and training on two mechanistically different backbones makes an architecture-specific story less likely. Citations are appropriate; GenExam is used as an external yardstick and the paper's own limitations section is candid about the judge being a proxy.\n\nThe soft spots are real. The stress-test note is correct: Tables 3 and 4 compare raw-prompt baselines against Princigram with SP-CoT prompt expansion, so the 29–35 point Local gains and the fourfold GenExam gain could come substantially from a better inference prompt, not from SP-CoT training. The paper says baselines can be run in structured mode but never reports those numbers. Figure 14 doesn't resolve this because it is qualitative and doesn't compare an untrained backbone with prompt expansion alone. The in-house benchmark is also partly circular—its questions are derived from the same annotations used for training—though the external GenExam result partly offsets that. The evaluation depends on GPT-4o answering binary physics questions, with no human validation or inter-annotator consistency reported. And no code or data is released, which matters when the corpus is the contribution.\n\nWho should read it: anyone working on text-to-image faithfulness, scientific figure generation, or benchmark design. It deserves a serious referee. I'd accept it conditionally: require the missing structured-mode baseline and a prompt-expansion-only control on the untrained backbone. If those numbers show the inference prompt alone accounts for most of the gain, the 'supervision is the key lever' framing needs to be rewritten. If not, this is a strong paper.","headline":"Real contribution is a structured representation and data pipeline, and the in-house benchmark is clever, but the paper hasn't isolated training supervision from inference-time prompt expansion.","tokens_in":41135,"tokens_out":4058,"would_cite":true,"duration_ms":37227,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing flat captions with structured physics reasoning chains — not changing the architecture — yields diagrams whose forces, states, and equations are physically consistent.","keywords":["physics-faithful diagram generation","structured chain-of-thought","text-to-image generation","scientific diagrams","multimodal supervision","physical faithfulness evaluation","SP-CoT","VeriphyT2IBench"],"falsifier":"Have two trained experts independently score a random sample of generated images against the same binary checklists used in VeriphyT2IBench, then compare their answers with the vision-language judge's; if judge–expert agreement is much smaller than the roughly 29-point margin Princigram shows over its untrained backbone, the measured faithfulness gain is largely a property of the judge rather than of the diagrams.","tokens_in":40152,"feed_emoji":"📐","tokens_out":7033,"duration_ms":60604,"temperature":0.7,"pith_summary":"Text-to-image models can draw a plausible pendulum but routinely mis-point its forces, because web captions describe appearance, not physics. This paper argues that the bottleneck is supervision, not architecture: it replaces flat captions with Structured Physical Chain-of-Thought (SP-CoT), a fixed five-step template that separates what is visibly drawn from what is physically inferred and types all mathematics symbolically. On 4.3 million physics images, 115,037 of them expert-verified, the authors train Princigram, a generator conditioned on these structured annotations during both training and inference. The result is a generator that beats generic open and closed models on physics-diagram faithfulness, including a roughly fourfold gain over its own backbone on the physics subset of a public text-to-image exam. The paper also introduces a benchmark, VeriphyT2IBench, whose yes/no questions are compiled from each diagram's own annotation, so scores name which physical facts are wrong rather than giving one opaque number.","feed_headline":"Reasoning-chain captions make physics diagrams fourfold more faithful","feed_subtitle":"Structured physics supervision, not a larger model, drives a 1,283-item physics-diagram gain.","key_machinery":"The load-bearing object is Structured Physical Chain-of-Thought (SP-CoT), a fixed JSON schema instantiated per subdiscipline. Each field carries a type (entity, relation, or value) and a grounding tag: visible fields may record only elements actually drawn in the image, while inferred fields may be filled by physical reasoning from those drawn elements, and missing information must be left empty rather than guessed; all mathematics is stored as valid LaTeX. The same serialized annotation serves three purposes: pre-training and fine-tuning supervision for the generator, a structured thinking prompt that a language model populates from a user request at inference, and the source of the binary yes/no checklist used by the evaluator, because the serializer preserves keys and values verbatim. That single representation makes data construction, training, and evaluation share one machine-parseable object.","core_discovery":"The central claim is that replacing flat image–caption pairs with SP-CoT — a per-subdiscipline, five-step reasoning chain covering scenario, parameters, structure, laws, and synthesis, with a strict visible/inferred split and symbolic LaTeX — yields supervision that is dense, auditable, and aligned across six physics subdisciplines, and that a generator trained on it produces diagrams more consistent with the underlying physics than generic baselines. The evidence is the measured jump in faithfulness: on the in-house benchmark, the same unified multimodal backbone rises from 46.38 to 75.69 on local per-attribute accuracy and from 62.15 to 82.54 on global accuracy; on the physics subset of a public text-to-image exam, its relaxed score rises from 13.8 to 54.8, about fourfold. The gains concentrate on the middle reasoning steps (parameters, structure, laws), widen as prompts get harder, and appear on a second, architecturally different backbone, which the paper takes as evidence that the structured supervision, not the network, is the active ingredient. The paper also states plainly that fully faithful generation is unsolved: under zero-tolerance strict scoring, every model, including Princigram, scores at or near zero on almost every subject.","pith_inferences":["The visible/inferred tags are currently unused in scoring; a natural next step, left implicit by the paper, is to weight grounded picture checks more heavily than inferred reasoning checks, or to use the per-fact scores as a reward for preference optimization.","The same five-step structure should transplant to other quantitative sciences: chemistry, geometric optics, and biology diagrams have their own visually carried relations, so per-discipline schemas could let this style of supervision generalize without new architecture.","Because each benchmark question is tied to a named physical fact, the same machinery could be repurposed as automatic item-specific feedback for students or as a filter that rejects physically wrong generated figures before publication; the paper does not pursue these uses.","The evaluation rests on a single vision-language judge; a checkable extension is to measure judge–judge and judge–expert agreement on a sample, since cross-judge consistency is not reported."],"forward_implications":["On the physics subset of a public text-to-image exam, structured supervision lifts the relaxed score about fourfold (13.8 to 54.8) on the paper's primary backbone, bringing an open-weight model ahead of closed reference systems on several subjects.","On the in-house benchmark, the gain transfers across two mechanistically different unified backbones, with local-score improvements of roughly 29 and 35 points, which suggests the supervision rather than the architecture is doing the work.","The advantage is largest on the schema steps that require physical inference, such as Laws and Synthesis, and on hard prompts with many simultaneous constraints, exactly where appearance-based baselines collapse.","No current system, including Princigram, reliably produces a diagram in which every checked physical attribute is correct: zero-tolerance strict scores are near zero everywhere, so fully faithful generation remains open.","Because the checklist is compiled from each diagram's annotation, a low score names the specific physical facts a model gets wrong, such as a missing $mg\\sin\\theta$ component, rather than giving one holistic number."],"supporting_citations":[{"why":"Supplies the idea that explicit intermediate reasoning steps unlock model reasoning, which SP-CoT adapts to diagrams.","marker":"[43]"},{"why":"Provides the unified multimodal backbone that Princigram trains, and the unmodified version serves as the key baseline.","marker":"[24]"},{"why":"Provides the second, architecturally different unified backbone used to show that the gains follow the supervision rather than a specific network.","marker":"[22]"},{"why":"The public text-to-image exam whose physics subset measures the gap and the fourfold relaxed-score gain.","marker":"[44]"},{"why":"The vision-language model that generates the corpus-scale structured annotations before expert verification.","marker":"[50]"},{"why":"Establishes that caption quality, not only image quality, drives what a text-to-image generator learns, motivating the replacement of flat captions.","marker":"[41]"},{"why":"Represents the web-scale image-text corpora whose shallow alt-text captions create the data gap the paper targets.","marker":"[26]"},{"why":"Supplies the large-scale open dataset mined for physics imagery in the corpus construction pipeline.","marker":"[45]"}],"fun_headline_variants":["Structured physics CoT quadruples diagram faithfulness","Physics-aware reasoning lifts diagram accuracy fourfold","SP-CoT beats bigger models for physics diagrams","Explicit physics supervision, not model size, wins diagrams","Reasoning-chain supervision makes diagrams 4x more faithful"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rest on a vision-language judge correctly answering the yes/no physics questions and on expert-annotated gold answers whose inter-annotator consistency is not measured, so systematic judge or annotation error could inflate the score gaps.","fun_headline_variants_meta":{"raw":{"variants":["Structured physics CoT quadruples diagram faithfulness","Physics-aware reasoning lifts diagram accuracy fourfold","SP-CoT beats bigger models for physics diagrams","Explicit physics supervision, not model size, wins diagrams","Reasoning-chain supervision makes diagrams 4x more faithful"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000303,"raw_usage":{"total_tokens":1825,"prompt_tokens":1109,"completion_tokens":716,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":725,"completion_tokens_details":{"reasoning_tokens":642}},"tokens_in":725,"tokens_out":716,"duration_ms":7170,"temperature":1.0,"reasoning_tokens":642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:32:13.664767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two trained experts independently score a random sample of generated images against the same binary checklists used in VeriphyT2IBench, then compare their answers with the vision-language judge's; if judge–expert agreement is much smaller than the roughly 29-point margin Princigram shows over its untrained backbone, the measured faithfulness gain is largely a property of the judge rather than of the diagrams.","supporting_citations":[{"cited_title":"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models","cited_arxiv_id":null,"evidence_quote":"Supplies the idea that explicit intermediate reasoning steps unlock model reasoning, which SP-CoT adapts to diagrams."},{"cited_title":"does the image show𝑋?","cited_arxiv_id":null,"evidence_quote":"The vision-language model that generates the corpus-scale structured annotations before expert verification."},{"cited_title":"Improving Image Generation with Better Captions","cited_arxiv_id":null,"evidence_quote":"Establishes that caption quality, not only image quality, drives what a text-to-image generator learns, motivating the replacement of flat captions."},{"cited_title":"LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models","cited_arxiv_id":null,"evidence_quote":"Represents the web-scale image-text corpora whose shallow alt-text captions create the data gap the paper targets."}],"review_version":1}