{"id":"412b06dc-4012-4bfe-b374-b3a7ed664af0","arxiv_id":"2607.18091","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A 9B open-weights model trained with axis-decomposed conjunctive preference optimization (M-DPO) and a structural inventory beats GPT-Image-1.5 on scientific-diagram structural-fidelity benchmarks.","lead":"SciForma is a training pipeline that makes AI-generated scientific flowcharts and architecture diagrams checkable and more reliable by scoring them on three separate axes: components, arrows, and text. It adds a new preference-optimization step plus a verification-and-edit loop, and reports results approaching or beating proprietary image models on diagram-structure benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Under the paper's own Qwen3-VL rescoring, SciForma-9B (59.21) trails GPT-Image-1.5 (59.32) on SciFormaBench-2K; only SciForma-9B+Edit overtakes it.","rationale":"The reader's CONDITIONAL verdict is appropriate. My stress-test sharpens the evaluator-bias concern into a concrete internal contradiction: the paper's own cross-evaluator table reverses the exact headline comparison. This is more specific than the reader's general worry about VLM-inventory circularity, and it directly targets the strongest claim. I did not find an error in the M-DPO derivation, and the AIBench result plus the ablation ladder provide independent support. However, the no-edit SciForma-9B's superiority over GPT-Image-1.5 on SciFormaBench-2K is not robust across the two evaluators presented in the paper. That does not require a REJECT—the conditional acceptance already captures the need for qualification—but it should be made explicit in the verdict rationale.","tokens_in":33230,"tokens_out":5207,"duration_ms":382969,"concrete_test":"Rescore SciFormaBench-2K with Qwen3-VL-8B-Instruct (or another non-GPT VLM) at temperature 0 on all 2,000 samples for SciForma-9B and GPT-Image-1.5, using the exact Appendix I.2 prompts, and compute per-sample paired deltas with a bootstrap 95% CI. If the mean delta remains negative or the CI includes 0, the 'SciForma-9B exceeds GPT-Image-1.5' claim must be restricted to the GPT-5.4 judge or to the +Edit pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the no-edit SciForma-9B exceeds GPT-Image-1.5 on SciFormaBench-2K (Table 2a: 69.51 vs 68.96, +0.55 with GPT-5.4). The paper's own cross-evaluator check in Appendix E.2 (Table 10) rescoring the same benchmark with Qwen3-VL-8B shows the opposite for this exact pair: GPT-Image-1.5 scores 59.32, SciForma-9B 59.21, a 0.11 reversal. Only the inference-time-editing variant SciForma-9B+Edit (61.05) overtakes GPT-Image-1.5 under the independent evaluator. E.2 nevertheless claims the 'significance of our gains remains completely unaffected' and reports r=0.9945; a rank flip at the headline comparison is not captured by an aggregate correlation. Because the no-edit model is what the abstract and Section 5.2 tout as exceeding GPT-Image-1.5, and no confidence intervals are provided, the headline superiority on SciFormaBench-2K is evaluator-dependent rather than robust. The AIBench margin (+8.67) is independent and survives, so the overall contribution is not void; but the specific SciFormaBench-2K claim needs qualification or a direct head-to-head with a non-overlapping judge.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SciForma introduces a structural-inventory formalism for scientific methodology diagrams, decomposing diagram quality into Component (C), Arrow (A), and Text (T) axes. The paper contributes (1) SciFormaData-700K, a large curated generation/editing dataset; (2) SciFormaBench-2K, a 2,000-sample benchmark scored by VLM checklist verification; (3) M-DPO, a multi-dimensional conjunctive preference optimization objective that couples axis-specific losers through a multi-way Bradley–Terry loss; and (4) an inference-time iterative refinement loop with verification-gated editing. The main empirical claim is that SciForma-9B, a 9B open model fine-tuned from FLUX.2-klein-base-9B, exceeds all open-source baselines and GPT-Image-1.5 on both SciFormaBench-2K and AIBench, with additional support from a 36-subject user study. The paper also reports ablations showing M-DPO improves over SFT and over scalar or mean-DPO alternatives.","tokens_in":33571,"tokens_out":4249,"duration_ms":49645,"significance":"If the claims hold, this is a substantive contribution: a large, professionally relevant dataset; a benchmark with element-level, axis-decomposed structural verification; and a training objective that explicitly models conjunctive correctness. The paper is unusually thorough in its supporting evidence: an independent AIBench evaluation, a Qwen3-VL rescoring of the full leaderboard, a reflexivity check on ground-truth diagrams, multi-round judge-stability measurements, and an analysis of C/A/T axis orthogonality. The M-DPO formulation is clean and reduces to standard DPO when D=1. These strengths make the paper a candidate for publication, but the headline claim is currently overstated relative to the paper’s own cross-evaluator results.","major_comments":[{"comment":"The central claim that no-edit SciForma-9B exceeds GPT-Image-1.5 on SciFormaBench-2K is evaluator-dependent. With GPT-5.4, SciForma-9B scores 69.51 vs. GPT-Image-1.5's 68.96 (+0.55). Under the paper's own Qwen3-VL rescoring (Table 10), the same pair reverses: SciForma-9B 59.21 vs. GPT-Image-1.5 59.32 (−0.11). Only SciForma-9B+Edit (61.05) overtakes GPT-Image-1.5 under the independent evaluator. The statement in E.2 that 'the significance of our gains remains completely unaffected' is not supported: a rank flip at the headline comparison is not captured by the aggregate correlation r=0.9945. The AIBench margin is independent and survives, but the SciFormaBench-2K claim in the abstract and Section 5.2 should be qualified, or a direct head-to-head with a non-overlapping judge should be provided.","section":"§5.2 vs. Appendix E.2, Tables 2a and 10"},{"comment":"There is a circularity risk: the M-DPO reward signal (Qwen3-VL scoring on the C/A/T inventory) and the SciFormaBench-2K evaluation (GPT-5.4 scoring on the same inventory) are both built from the same structural checklist. The reflexivity check (Table 9) shows the evaluator can find elements it is told to look for, but it does not establish that the checklist is a complete or unbiased proxy for structural fidelity. The paper should explicitly test whether an independent, non-VLM or human structural check (e.g., a human judge re-extracting C/A/T from generated images, or a layout/color-semantics audit) corroborates the benchmark ordering, especially for the SciForma-9B vs. GPT-Image-1.5 comparison.","section":"§3.2, §4.5.1, §5.2"},{"comment":"No confidence intervals or significance tests are reported for the main benchmark comparisons. The headline margins are small: +0.55 on SciFormaBench-2K (GPT-5.4) and +0.20 over human originals on AIBench. Given per-sample variance and stochastic VLM scoring, bootstrap confidence intervals or paired tests over the 2,000/300 samples should be reported. Without them, the reader cannot distinguish a real improvement from evaluator noise.","section":"§5.2, Tables 2a/2b"},{"comment":"The user study description is inconsistent. The main text reports 36 subjects and 30 samples, while Appendix H.4 describes 50 prompts and majority voting by three annotators per pair. The appendix also says Fleiss' kappa is reported, but no kappa value appears. Since the user study is one of the three supporting lines for the benchmark, this discrepancy needs to be reconciled and the inter-annotator agreement actually reported.","section":"§5.7 vs. Appendix H.4"}],"minor_comments":[{"comment":"The phrase 'SciForma-9B exceeds ... GPT-Image-1.5' should be clarified to distinguish the no-edit model from SciForma-9B+Edit, since the no-edit model's lead reverses under the independent evaluator.","section":"Abstract/Section 5.2"},{"comment":"The claim of 'near-perfect correlation' (r=0.9945) is not an adequate substitute for reporting the specific SciForma-9B vs. GPT-Image-1.5 comparison under both evaluators. Consider adding a paired-difference table.","section":"Appendix E.2"},{"comment":"The GDRO baseline is described as ranking candidates with a single scalar reward; the abbreviation is potentially confusing with standard group distributionally robust optimization. A one-sentence clarification of the baseline's objective would help.","section":"§5.5, Table 4"},{"comment":"The reward-scoring prompt and the benchmark evaluation prompts are similar in spirit but not identical; the paper should state explicitly that the same prompt template was not used for reward construction and evaluation, or discuss any effect of prompt overlap.","section":"Appendix I.2"},{"comment":"The monotonic guard is defined as S_new >= S_old - epsilon with epsilon=0.03, but S is not precisely defined. The paper should state which score (average, per-axis, or a composite) is used for the guard.","section":"§4.6 and Appendix D.2"},{"comment":"GPT-Image-2 is listed as a proprietary baseline, but its margin over SciForma-9B+Edit is large (85.62 vs. 72.40). The paper's contribution would be clearer if the discussion acknowledged this residual gap rather than only emphasizing the GPT-Image-1.5 comparison.","section":"§5.2, Table 2a"}],"recommendation":"major_revision","confidential_remarks":"The paper is strong and well within the scope of the journal, but the headline claim needs to be made as robust as the underlying evidence. The cross-evaluator reversal for the no-edit model is the key issue; the authors should either qualify the claim to SciForma-9B+Edit, add a proper confidence-interval analysis, or provide a non-overlapping human evaluation of the SciForma-9B vs. GPT-Image-1.5 comparison. The circularity concern is real but not disqualifying given the independent AIBench result; still, the authors should address it explicitly. I would not reject the paper on the current evidence, but I would not accept it in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a serious, well-ablated systems paper, and M-DPO is worth understanding. The one thing to watch: the abstract-level claim that SciForma-9B beats GPT-Image-1.5 on SciFormaBench-2K does not survive their own cross-evaluator check for the no-edit model. Under Qwen3-VL rescoring (Appendix E.2, Table 10), SciForma-9B gets 59.21 vs GPT-Image-1.5's 59.32; only the +Edit variant overtakes it. That does not sink the paper, but the headline needs qualification.\n\nWhat's actually new: the decomposition of diagram fidelity into Component/Arrow/Text primitives, a 700K dataset with topology-aware captions and 70K edit triplets, and M-DPO, which pairs a shared winner against axis-specific losers under a multi-way Bradley–Terry objective. Equation (4) is a known InfoNCE/logistic form, but the dimension-anchored negative mining for structurally verifiable axes is the real contribution. The ablations support it: scalar DPO, Pareto DPO, mean-DPO, CaPO, and MCDPO all leave something behind; M-DPO lifts every axis, with the largest gain on Text. The comparison against continued SFT is also persuasive. I checked the gradient derivation in App. C.1; it is correct, and D=1 reduces to standard DPO as claimed.\n\nThe main soft spot is the shared construct: the M-DPO reward and SciFormaBench-2K are both built from VLM-extracted C/A/T checklists. That is partially mitigated by the reflexivity check (99.94%), the Qwen rescoring, and the independent AIBench result (+8.67 over GPT-Image-1.5). It is not eliminated: part of the measured gain is alignment to the authors' rubric. The more concrete problem is the rank flip mentioned above; reporting r=0.9945 does not capture that the exact advertised pair reversed. The margins on the home benchmark are also thin (0.55 and 0.20 points) with no confidence intervals.\n\nOther issues are more practical than fatal: code/data/checkpoints are promised but not shipped, and the pipeline leans on proprietary APIs (GPT-5.4, Qwen3-VL, SAM3) for labeling, scoring, and editing. The paper acknowledges this in Section 6. The failure-mode appendix (rotated text, dense-arrow errors) is honest and useful.\n\nCitation pattern looks fair; I did not see red flags or missing nearby work.\n\nWho reads this: anyone working on structured or diagram-aware image generation, and people designing conjunctive multi-axis alignment objectives. It deserves a serious referee; I would send it out for review. The revision should reframe the head-to-head claim, add confidence intervals, and ideally release the benchmark and evaluator prompts before the community re-checks.","headline":"Serious, well-ablated systems paper; the abstract's no-edit win over GPT-Image-1.5 on the home benchmark flips under their own Qwen rescoring, so the headline needs qualification rather than rejection.","tokens_in":34141,"tokens_out":3842,"would_cite":true,"duration_ms":43001,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 9-billion-parameter open model can now generate scientific diagrams whose components, arrows, and text are all structurally faithful, beating a strong proprietary image model on the paper's benchmarks.","keywords":["scientific diagram generation","structural fidelity","component-arrow-text decomposition","conjunctive preference optimization","multi-dimensional DPO","diffusion models","structural inventory","verification-gated editing"],"falsifier":"Rescore the benchmark outputs using human annotators who verify every component, arrow, and label against the reference, adding dimensions the inventory omits (rotated text, color semantics, layout aesthetics). If human per-axis error rates fail to reproduce the ordering SciForma-9B over the proprietary model, or if an adversarially constructed diagram that passes all component/arrow/text checklists is nevertheless judged structurally invalid by a reader, then the inventory-based verification is incomplete and the claimed structural-fidelity gain is an artifact of the evaluator.","tokens_in":33035,"feed_emoji":"🧩","tokens_out":6988,"duration_ms":69143,"temperature":0.7,"pith_summary":"SciForma aims to make scientific methodology diagrams structurally faithful: every module, arrow, and text label must be correct, because one reversed arrow or unreadable equation invalidates the whole figure. The paper argues that this kind of correctness is conjunctive, so quality should be decomposed into three independently verifiable axes—components, arrows, and text—rather than collapsed into a single scalar score. Its central contribution is M-DPO, a preference-optimization objective that vanishes only when a generated candidate beats an axis-specific loser on all three axes at once, and whose gradient automatically concentrates on whichever axis is failing most. The paper shows that a 9-billion-parameter open model trained this way surpasses all open-source baselines and a strong proprietary image model on both the paper's own structural-fidelity benchmark and an independent reading-comprehension benchmark. If that result holds, open research tools can produce diagrams whose logical structure can be trusted, closing most of the gap with closed commercial systems.","feed_headline":"Open 9B model beats proprietary rival on diagram structure","feed_subtitle":"Components, arrows, and text are checked jointly, so training attacks the failing axis and stops when all three pass.","key_machinery":"The load-bearing mechanism is M-DPO, a conjunctive preference objective over three axis-anchored preference pairs. For each prompt the system rolls out candidates and labels them under a structural inventory; the overall best candidate is the shared winner and one loser is chosen per axis for being weak on that axis while competitive elsewhere. The loss, derived from a multi-way Bradley–Terry model, is log(1 + sum_d exp(-beta * Delta_d)), where Delta_d is the relative preference margin for axis d. It vanishes only if the winner beats every axis-specific loser, and its per-axis gradient weight is a softmax over deficits, so the strongest update goes to the axis with the largest failure. This","core_discovery":"Structural fidelity in methodology diagrams is conjunctive: components, arrows, and text must all be right or the figure fails. The paper encodes this in a structural inventory, a checklist extracted from a reference image, used as both training reward and evaluation metric. Its M-DPO objective pairs a winning candidate with one loser per axis and optimizes a multi-way comparison that vanishes only when the winner beats every axis-specific loser, routing gradient weight to the weakest axis. The same inventory gates inference-time edits. The resulting 9B model exceeds all open-source baselines and the proprietary GPT-Image-1.5 on the paper's benchmark, and edges out human-drawn originals on a","pith_inferences":["As an editorial extension, the same 'conjunctive axes plus weakest-axis gradient routing' recipe could transfer to any domain where a single error invalidates a visual artifact—circuit schematics, UML models, game levels—provided a checklist can be extracted.","Because M-DPO's reward signal and the home benchmark's ranking both come from vision-language checklists, part of the reported gain may reflect alignment with that particular evaluator's notion of structure; an independent human-annotated error study would clarify how much is genuine topological understanding.","A testable extension is to apply the same conjunctive objective to long-form structured outputs such as procedures or instructions, where scalar rewards are known to dilute rare but critical errors."],"forward_implications":["A 9-billion-parameter open model can match or beat much larger and closed systems on structurally faithful diagram generation, so publication-quality diagram generation no longer requires proprietary APIs.","Because the reward and the evaluation share the same explicit checklists, failures become diagnosable: a low Arrow score tells the user exactly which arrows are missing or misrouted, enabling targeted repair.","Scalar post-training methods—averaging rewards or running online RL on one number—plateau or regress on these diagrams; the conjunctive formulation is what breaks the plateau, with the largest gains on the weakest axes.","Inference-time editing with a verification gate and rollback monotonically improves the output, so generation and correction can be unified in one model.","Pixel-space generation with structural supervision outperforms code-based diagram synthesis by roughly a factor of two on the home benchmark."],"fun_headline_variants":["Conjunctive check makes 9B model beat GPT-Image on diagram structure","SciForma: 9B fixes arrows, text, components jointly to beat GPT-Image","Diagram structure: one miss fails all, SciForma targets weakest axis","M-DPO: training attacks failing axis until all three pass","Open 9B exceeds proprietary GPT-Image-1.5 on diagram fidelity"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole pipeline assumes that a vision-language model's checklist of components, arrows, and text is a complete and unbiased measure of whether a diagram is structurally correct.","fun_headline_variants_meta":{"raw":{"variants":["Conjunctive check makes 9B model beat GPT-Image on diagram structure","SciForma: 9B fixes arrows, text, components jointly to beat GPT-Image","Diagram structure: one miss fails all, SciForma targets weakest axis","M-DPO: training attacks failing axis until all three pass","Open 9B exceeds proprietary GPT-Image-1.5 on diagram fidelity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000388,"raw_usage":{"total_tokens":1918,"prompt_tokens":817,"completion_tokens":1101,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":999}},"tokens_in":561,"tokens_out":1101,"duration_ms":18315,"temperature":1.0,"reasoning_tokens":999,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:01:03.100800+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rescore the benchmark outputs using human annotators who verify every component, arrow, and label against the reference, adding dimensions the inventory omits (rotated text, color semantics, layout aesthetics). If human per-axis error rates fail to reproduce the ordering SciForma-9B over the proprietary model, or if an adversarially constructed diagram that passes all component/arrow/text checklists is nevertheless judged structurally invalid by a reader, then the inventory-based verification is incomplete and the claimed structural-fidelity gain is an artifact of the evaluator.","supporting_citations":[],"review_version":1}