REVIEW 3 major objections 4 minor 39 references
Naturally occurring paper revisions — before/after figure pairs from manuscript version histories — are an effective training signal for instruction-driven scientific figure editing, and a skill-evolving agent built on them beats strong sin
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 23:39 UTC pith:ZVZPL4CY
load-bearing objection Genuinely new benchmark and a competent editing agent, but the headline claim about 'natural revisions' is undercut by the curator-authored instructions. the 3 major comments →
SciDiagramEdit: Learning to Edit Scientific Diagrams from Paper Revisions
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Central claim: author-drawn figure revisions encode the implicit visual grammar of scientific communication, and an agent can learn that grammar by contrasting its own edits against the author-drawn target. SciDiagramEdit is a benchmark of 364 before/after figure pairs mined from preprint version histories, each annotated with an instruction and a checklist of atomic changes, plus an agentic loop: an Editor edits SVG under a skill specification; a Judge multiplies checklist semantic faithfulness by pairwise aesthetic preference against the author's revision; a Coach patches the skill from traces, scores, and the target as a demonstration. The evolved skill improves accuracy across iterations
What carries the argument
Key machinery: the skill-evolution loop with a demonstration-aware Coach. An Editor edits SVG source under a skill specification S; a Judge scores each edit with r = r_aes · r_sem, gating checklist-based semantic faithfulness on pairwise aesthetic preference against the author's revision; a Coach consumes traces, verdicts, and the author's revised figure as a worked demonstration, and patches S. S is a set of files — top-level SKILL.md plus workflows/ of trigger-plus-rule entries. Evolution keeps a top-K frontier of candidates, admitting new ones only on validation improvement. The demonstration-aware comparison lets the Coach read concrete fixes such as 'tighten the viewBox after structural
Load-bearing premise
The whole training signal rests on the assumption that the curator-authored editing instructions faithfully reconstruct what the original authors intended; if those instructions are biased or incomplete reconstructions of the visual diff, the agent is learning to follow post-hoc descriptions rather than genuine revision intent.
What would settle it
Re-annotate a subset of the benchmark with the raw visual diff withheld from the annotator (or generated mechanically from pixel differences), then re-run the skill-evolution loop; if performance does not drop, the 'grounded in revision intent' claim is not load-bearing. Alternatively, have two independent teams author instructions for the same pairs and compare the evolved skills' test scores: instability across instruction sets would show the method is learning annotation artifacts.
If this is right
- Public preprint revision histories become a free, steadily replenishing source of training pairs for figure editing: every paper that re-renders its figures yields a before/after supervision signal.
- Because the agent edits vector primitives rather than re-rendering pixels, its outputs remain editable SVG that the researcher can inspect, correct, and co-edit — unlike raster regenerators that bake changes into pixels.
- The skill is portable: a skill evolved on the strongest backbone raises both semantic and aesthetic edit quality on weaker backbones without any retraining, suggesting the distilled rules are genuinely procedural.
- On semantic faithfulness, the vector-based method matches the strongest closed-source raster editor (checklist success 0.932 vs 0.882; win rate 0.756 vs 0.778), so users do not have to sacrifice fidelity to gain editability.
- The evolution loop lifts held-out validation accuracy over iterations, evidence that the revision-derived signal is learnable rather than memorized per training sample.
Where Pith is reading between the lines
- The strongest untested extension is to scale the loop up by orders of magnitude; the authors themselves note that a few hundred pairs and a few dozen evolution steps may under-use the signal. At scale, the skill library could consolidate into a general grammar of scientific visual communication applicable to generation as well as editing.
- A direct test of the premise would vary the annotation protocol — e.g., compare curator-authored instructions with instructions generated mechanically from the raw visual diff — to determine how much of the gain comes from genuine authorial intent versus from the checklist-based judge alone.
- A hybrid architecture seems a natural next step: use the vector-aware editor for structural precision and a raster regenerator for aesthetic polish, since the two failure modes in the qualitative results are complementary.
- The authors flag that their benchmark only covers explicitly stated instructions; the obvious harder regime is inference of intent from the paper's argument, and the same mined revision pairs already contain the surrounding textual context needed to train for that.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SciDiagramEdit, a benchmark of 364 before/after figure-edit pairs mined from arXiv version histories, with 2,628 atomic checklist questions and natural-language editing instructions. It proposes a vector-SVG editing agent whose skill specification is iteratively evolved by a Coach from execution traces, judged by a VLM-based semantic-faithfulness checklist and an aesthetic pairwise preference signal. The experiments compare the proposed agent against raster regenerators (GPT-Image-1.5/2) and single-pass vector baselines (AutoFigure-Edit with GPT-5.x), reporting higher checklist success, a transferred skill across backbones, and a small human study. The paper's central claim is that natural paper revisions are an effective training signal for instruction-driven figure editing.
Significance. If the central claim holds, the benchmark is a valuable resource: it is the only instruction-driven editing benchmark built on real scientific-diagram revisions, and the vector-source agent with an evolvable skill library is a plausible and useful design. The paper is transparent about the annotation interface, limitations, compute/API costs, and the small scale of the user study, and it releases the benchmark and evolved skill specification. However, the central claim is currently weakened by a construct-validity concern about the provenance of the instructions and by the absence of statistical rigor in the headline comparisons. These issues are fixable, but they need to be addressed before the main claims can be accepted.
major comments (3)
- [§3 / Appendix A.1] The abstract, §1, and §6 repeatedly state that each benchmark pair is 'grounded in the authors' own revision intent.' Appendix A.1, however, says the raw visual diff 'seldom makes the author's intent explicit' and that 'a useful editing instruction therefore has to be authored, not just extracted.' The curator interface iteratively refines a GPT-5.5-drafted instruction until it matches the *observed visual diff*, not until it matches evidence of what the original authors intended. Since both training and evaluation use these curator-authored instructions, the central claim that 'natural paper revisions are an effective training signal' is not established. The agent is learning to follow post-hoc descriptions of diffs. I recommend either reframing the claim to 'curator-annotated revision instructions,' or providing validity evidence: e.g., mining author-written revision notes or paper tex
- [§5.1, Tables 2 and 3] All quantitative comparisons are point estimates on a single test split without confidence intervals or significance tests. Several differences are small relative to the apparent noise: Ours achieves 0.932 checklist success vs GPT-Image-2's 0.882, but the instruction-following win rate is 0.756 vs 0.778 (i.e., numerically lower); Table 3 reports skill gains of +0.011 to +0.054 on semantic win rate depending on backbone. With a 2:1:3 split and 364 total samples, these magnitudes cannot be interpreted without bootstrap intervals, paired per-sample tests, or multiple evolution runs. The user study (Table 4) also reports only proportions from five participants, with no variance or significance testing and no tie option. Please add per-sample bootstrap CIs and paired tests for the main tables, and report inter-rater variability in the human study.
- [§4.3, Eq. (4)–(6) and §5.1] The same VLM judge J that provides the reward r in Eq. (6) and drives validation gating in Algorithm 1 is also the primary semantic metric reported in Table 2. Although the test split is held out and the loop gates on D_val, the Coach is explicitly optimizing for J's verdicts, so the headline semantic scores may partly reflect judge-specific biases rather than instruction-following quality. The human study is a useful partial check, but it covers only 30 instances and no human-judge agreement with J is reported. I recommend reporting agreement between the VLM judge and human raters, and adding an independent evaluation protocol (e.g., a different VLM or human checklist scoring) to guard against reward-hacking of the judge.
minor comments (4)
- [Abstract / §3 vs Appendix A.1] Please reconcile the terminology: 'grounded in the authors' own revision intent' and 'naturally occurring author-written instructions' imply author-authored language, while A.1 describes curator-authored instructions. Consistent phrasing would reduce reader confusion.
- [Table 3] The Δ superscripts for 'w/ evolved skill' and 'w/ transferred skill' are compact but hard to read; consider separate delta columns or making the baseline explicit in the column header.
- [§5.4 / Appendix A.3] The user study uses a forced-choice design with no tie option. This is defensible, but please report raw vote counts and per-participant variation, since the headline proportions are based on only five raters.
- [Appendix B.2] The demonstration-aware Coach ablation is qualitative only. A quantitative table comparing the two variants on the same metric would strengthen the central design claim, even if only on a validation subset.
Circularity Check
No circularity found: the results are empirical, computed on a held-out test split, and externally checked by a human study; the annotation-intent caveat is a construct-validity concern, not a circular step.
full rationale
SciDiagramEdit is an empirical benchmark and training paper, not a derivation, and its reported gains do not reduce by construction to its inputs. The skill-evolution loop trains on D_train, selects candidates by validation score (Algorithm 1; Eq. 8: S* = argmax_S rbar(S; D_val)), and all Table 2/3 scores are reported on the held-out D_test split. The semantic metric is a curatorial checklist and the same judge is used as reward and metric, but this is the standard train/val/test setup; the test numbers are not fitted. The human pairwise study (Table 4) and the qualitative failure cases (Fig. 12: 'our model captures the explicit edit but misses an implicit one') provide external, non-circular checks. Appendix A.1 does weaken the strong 'authors' own revision intent' framing: it admits 'the raw visual diff seldom makes the author's intent explicit' and 'A useful editing instruction therefore has to be authored, not just extracted,' meaning instructions are post-hoc curator reconstructions. That is a construct-validity limitation (the 'natural paper revision' signal may be contaminated by curator/LLM phrasing), and the paper's own Limitations section concedes it has not explored intent inference from context. But no equation or fitted parameter is defined as the claimed result; the concern is about external validity, not circularity. No load-bearing self-citation chain or imported uniqueness theorem appears; citations to skill-evolution work are contextual. Score 0.
Axiom & Free-Parameter Ledger
free parameters (2)
- Evolution loop hyperparameters =
frontier size K=3; batch size=8; epochs=2; edit budget=8; max analyst rounds=3
- Reward composite form =
r = r_aes * r_sem
axioms (3)
- domain assumption The author's revised figure F_ref is a correct editing target, and the curator-authored instruction I faithfully captures the edit intent.
- domain assumption The VLM Judge J's checklist answers and pairwise aesthetic preferences are reliable proxies for human judgment.
- domain assumption The AutoFigure-Edit decomposition into vector primitives plus embedded raster panels preserves enough structure for meaningful SVG editing.
read the original abstract
Editing the figures in a research paper is a routine and time-consuming part of everyday research practice: authors relabel components, rearrange panels, and restyle visuals as they revise their manuscripts. Automating this editing workflow under a natural-language instruction, however, is challenging, because a scientific figure is a dense infographic in which heterogeneous visual elements such as schematics, plots, photos, captions, and arrows are composed under a tight visual grammar to advance a specific argument. To address this, we present SciDiagramEdit, a benchmark and skill-evolution framework that learns from natural paper revisions and operates on the figure's editable vector source, where users can inspect and co-edit individual primitives alongside the agent. Our benchmark mines before/after figure pairs from arXiv version histories, each grounded in the authors' own revision intent. To accommodate the diversity of editing instructions, we adopt agentic learning via skill evolution: an agentic proposer continually refines the agent's skill specification from execution traces over multiple epochs. The resulting skill progressively lifts edit accuracy on a held-out validation set, providing evidence that natural paper revisions are an effective training signal for instruction-driven figure editing.
Figures
Reference graph
Works this paper leans on
-
[3]
Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18392--18402
2023
-
[5]
Siqi Chen, Xinyu Dong, Haolei Xu, Xingyu Wu, Fei Tang, Hang Zhang, Yuchen Yan, Linjuan Wu, Wenqi Zhang, Guiyang Hou, and 1 others. 2025. Svgenius: Benchmarking llms in svg understanding, editing and generation. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 13289--13296
2025
-
[7]
Google DeepMind . 2025. Nano banana pro ( Gemini image). Image generation and editing model
2025
-
[14]
OpenAI . 2025. GPT-Image : Image generation and editing. Image generation and editing model
2025
-
[15]
Juan A Rodriguez, Abhay Puri, Shubham Agarwal, Issam H Laradji, Pau Rodriguez, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. 2025. Starvector: Generating scalable vector graphics code from images and text. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 16175--16186
2025
-
[18]
Ronghuan Wu, Wanchao Su, and Jing Liao. 2025. Chat2svg: Vector graphics generation with large language models and image diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23690--23700
2025
-
[19]
Ximing Xing, Juncheng Hu, Guotao Liang, Jing Zhang, Dong Xu, and Qian Yu. 2025. Empowering llms to understand and generate complex vector graphics. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19487--19497
2025
-
[23]
Minjun Zhu, Zhen Lin, Yixuan Weng, Panzhong Lu, Qiujie Xie, Yifan Wei, Sifan Liu, QiYao Sun, and Yue Zhang. 2026 b . Autofigure: Generating and refining publication-ready scientific illustrations. In The Fourteenth International Conference on Learning Representations
2026
-
[24]
Aho and Jeffrey D
Alfred V. Aho and Jeffrey D. Ullman , title =. 1972
1972
-
[25]
Publications Manual , year = "1983", publisher =
1983
-
[26]
Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243
arXiv 1981
-
[27]
Scalable training of
Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of
-
[28]
Dan Gusfield , title =. 1997
1997
-
[29]
Tetreault , title =
Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =
2015
-
[30]
A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =
Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =
-
[31]
The Fourteenth International Conference on Learning Representations , year=
AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations , author=. The Fourteenth International Conference on Learning Representations , year=
-
[32]
arXiv preprint arXiv:2603.06674 , year=
AutoFigure-Edit: Generating Editable Scientific Illustration , author=. arXiv preprint arXiv:2603.06674 , year=
-
[33]
arXiv preprint arXiv:2601.23265 , year=
PaperBanana: Automating Academic Illustration for AI Scientists , author=. arXiv preprint arXiv:2601.23265 , year=
-
[34]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Instructpix2pix: Learning to follow image editing instructions , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[35]
arXiv preprint arXiv:2507.19457 , year=
Gepa: Reflective prompt evolution can outperform reinforcement learning , author=. arXiv preprint arXiv:2507.19457 , year=
-
[36]
arXiv preprint arXiv:2305.16291 , year=
Voyager: An open-ended embodied agent with large language models , author=. arXiv preprint arXiv:2305.16291 , year=
-
[37]
arXiv preprint arXiv:2603.25158 , year=
Trace2skill: Distill trajectory-local lessons into transferable agent skills , author=. arXiv preprint arXiv:2603.25158 , year=
-
[38]
TextGrad: Automatic "Differentiation" via Text , author=. arXiv preprint arXiv:2406.07496 , year=
-
[39]
arXiv preprint arXiv:2603.02766 , year=
Evoskill: Automated skill discovery for multi-agent systems , author=. arXiv preprint arXiv:2603.02766 , year=
-
[40]
arXiv preprint arXiv:2603.28052 , year=
Meta-harness: End-to-end optimization of model harnesses , author=. arXiv preprint arXiv:2603.28052 , year=
-
[41]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Chat2svg: Vector graphics generation with large language models and image diffusion models , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[42]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Empowering llms to understand and generate complex vector graphics , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[43]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Starvector: Generating scalable vector graphics code from images and text , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[44]
arXiv preprint arXiv:2504.06263 , year=
OmniSVG: A Unified Scalable Vector Graphics Generation Model , author=. arXiv preprint arXiv:2504.06263 , year=
-
[45]
Proceedings of the 33rd ACM International Conference on Multimedia , pages=
Svgenius: Benchmarking llms in svg understanding, editing and generation , author=. Proceedings of the 33rd ACM International Conference on Multimedia , pages=
-
[46]
arXiv preprint arXiv:2502.19453 , year=
Svgeditbench v2: A benchmark for instruction-based svg editing , author=. arXiv preprint arXiv:2502.19453 , year=
-
[47]
arXiv preprint arXiv:2601.04390 , year=
SciFig: Towards Automating Scientific Figure Generation , author=. arXiv preprint arXiv:2601.04390 , year=
-
[48]
arXiv preprint arXiv:2603.28068 , year=
AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generation , author=. arXiv preprint arXiv:2603.28068 , year=
-
[49]
arXiv preprint arXiv:2505.22126 , year=
Sridbench: Benchmark of scientific research illustration drawing of image generation model , author=. arXiv preprint arXiv:2505.22126 , year=
-
[50]
arXiv preprint arXiv:2604.27505 , year =
Hanzhong Guo and Jie Wu and Jie Liu and Yu Gao and Zilyu Ye and Linxiao Yuan and Xionghui Wang and Yizhou Yu and Weilin Huang , title =. arXiv preprint arXiv:2604.27505 , year =
-
[51]
International Conference on Learning Representations (ICLR) , year =
Xin Luo and Jiahao Wang and Chenyuan Wu and Shitao Xiao and Xiyan Jiang and Defu Lian and Jiajun Zhang and Dong Liu and Zheng Liu , title =. International Conference on Learning Representations (ICLR) , year =
-
[52]
2025 , note =
Nano Banana Pro (. 2025 , note =
2025
-
[53]
arXiv preprint arXiv:2508.20751 , year=
Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning , author=. arXiv preprint arXiv:2508.20751 , year=
-
[54]
arXiv preprint arXiv:2605.12500 , year=
SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture , author=. arXiv preprint arXiv:2605.12500 , year=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.