REVIEW 3 major objections 5 minor 44 references
Real Data Closes Synthetic-to-Real Gap in Optical Chemical Structure Recognition
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Labeled real training images, not more synthetic renders, close the synthetic-to-real gap in optical chemical structure recognition, with exact match on a held-out journal-figure benchmark rising from 0.154 to 0.459 as the real-data…
desk verdict A well-designed empirical study showing real training images beat synthetic scaling for OCSR, though the reason—depiction realism vs molecular provenance—is underdetermined; deserves peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is the controlled 3-by-6 factorial sweep: three pretrained vision-language model bases crossed with six real-data fractions (0%, 5%, 10%, 20%, 40%, 70%) under a fixed approximately 496k-sample mixture, a fixed 10k-step budget, and a fixed light vision-LoRA adaptation surface. This design isolates the real-data fraction from base choice and training recipe, and it is paired with a canonical exact-match scoring metric that parses predicted SMILES with RDKit and compares molecular identity through InChIKey; the same scoring protocol is applied to every configuration and every external comparison.
What would settle it
Re-run the controlled sweep with multiple independent seeds per cell at 0%, 20%, and 70% real data using the same fixed evaluation sets. If between-seed spread at 70% equals or exceeds the observed 0.060 between-base spread, or if base orderings flip across seeds, the paper's ranking and contraction conclusions would not survive.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that representative real supervision is the main thing that closes the synthetic-to-real gap in OCSR, and that this effect is separable from the choice of base model or adaptation strategy. The controlled 3-by-6 sweep crosses three pretrained vision-language model bases with six real-data fractions, holding mixture size, training budget, objective, and adaptation surface fixed, and shows mean real-document exact match rising monotonically from as low as 0.027 at 0% real data to between 0.674 and 0.734 at 70%, with the first 5% real data delivering about two-thirds of the total gain. The paper also shows that base-model rankings change with the real-data fraction, that the between-base spread contracts from 0.212 at 0% to 0.060 at 70%, and that extending LoRA to the vision tower helps one base by 22.8 to 34.6 points, helps another modestly, and leaves a third unchanged.
Load-bearing premise
The load-bearing premise is that each training cell in the controlled 18-run sweep is represented by one seed, while the paper's uncertainty intervals measure evaluation-set sampling only; if seed-to-seed training variation is large, the base ranking and the dose-response shape could change.
Editorial extensions
If this is right
- OCSR development should prioritize collecting labeled real depictions from patents, journals, and hand-drawn collections over further synthetic render scaling.
- The real-data fraction is an operating point, not something to maximize: real-document accuracy rises while rendered-clean accuracy falls by roughly 10 to 14 points across bases.
- Base-model choice must be revisited at the intended real-data fraction, since the ranking of the three bases changes as real data is added.
- Vision-tower adaptation should be justified by a matched within-base comparison on the target task, because its measured effect ranges from about 23 to 35 points for one base to zero for another.
- Synthetic-only accuracy is not a reliable predictor of real-document accuracy for OCSR.
Reading between the lines
- A testable extension: if the observed real-data dose response is a general phenomenon, then even a small labeled real set (around 5% of the mixture) may be the most cost-effective next step, since roughly two-thirds of the total gain arrives there.
- The paper's consensus-reconciliation labeling rule implies a bootstrapping loop in which the recognizer itself helps generate the real-data resource needed to improve it; that loop is described but not fully evaluated as a training-data acquisition strategy.
- The cross-task probes suggest the base-dependence pattern may generalize to other image-to-structure tasks, but single-run results make that an open question rather than a conclusion.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies optical chemical structure recognition (OCSR) with vision-language models, comparing fine-tuning on synthetic renders versus mixtures that include labeled real document depictions. Across three VLM bases and six real-data fractions in a controlled sweep, plus an independent Qwen dose series, the authors report that increasing the real-data fraction improves exact match on real document benchmarks (ACS, CLEF-IP, UOB, USPTO) while slightly lowering rendered-clean accuracy. They also report that vision-tower LoRA helps InternVL3-8B strongly, helps GLM-4.1V-9B modestly, and does not help Qwen2.5-VL-7B, and that base-model rankings change with the real-data fraction. The paper's main contribution is the claim that real labeled training images are the most effective way to close the synthetic-to-real gap, supported by a controlled experimental design and a common evaluation harness.
Significance. If the central claim is taken at face value, the practical implication is important: OCSR practitioners should prioritize collecting labeled real depictions over adding more synthetic renders or renderer diversity. The paper is commendable for its controlled 3x6 design, fixed mixture size and training budget, paired statistical tests, shared evaluation protocol across specialist recognizers and chemical VLMs, and unusually frank limitations section. The main weakness is that the real-data effect is measured under a confound between depiction realism and molecular provenance: every real training example is also a molecule from the real-world distribution, while every synthetic example is a molecule from the curated one-million set. Because the paper itself acknowledges this in Section 9, the core contribution needs additional experimental separation before the strong practical recommendation is fully supported.
major comments (3)
- [§4.3, §9, §A.2] The central claim that real labeled images close the synthetic-to-real gap is confounded with molecular provenance. In the controlled 3×6 sweep and the Qwen dose series, raising the real-data fraction simultaneously replaces the depiction style (real documents vs. RDKit/CoordGen/Indigo renders) and the molecular distribution (real-pool molecules such as USPTO patents and journal figures vs. the curated one-million-molecule set). Section 9 explicitly states that 'raising the real fraction adds images and molecules together,' and Section A.2 notes that the InChIKey blocklist removes exact identity overlap but not scaffold or source-style overlap. The observed ACS/CLEF-IP/USPTO gains could therefore be driven by training on molecules closer to the test distribution rather than by the realism of the training depictions. To support Contribution (1), the paper needs a control that separates depiction style from molecular provenance, for example synthetic renders of the real-pool molecules or at least a molecular-provenance-matched arm. Without such an arm, the claim that real depictions are the effective ingredient is underdetermined.
- [§4.1, §5.2] The paper does not establish that real data is 'the most effective way' to close the gap because the alternative interventions—synthetic scale, renderer diversity, degradation, and the verifiable RL objective—were not evaluated on the real-document benchmark suite. Table 2 reports these interventions only on rendered conditions, and the text itself says 'no conclusion about real-document transfer can be obtained from these runs alone.' The only other intervention measured on real documents is the vision-LoRA contrast, which shows no effect for Qwen. Thus the empirical content is that real data is the only tested intervention that improves real-document accuracy, not that it is more effective than alternatives that could have been evaluated on the same real benchmarks. At minimum, one strong synthetic-only recipe should be evaluated on ACS, CLEF-IP, UOB, and USPTO under the same protocol to give the comparative claim substance.
- [§9, §C.1] The ranking and between-base spread claims rest on single-seed cells. The controlled 18-cell sweep trains exactly one model per cell, and Section 9 acknowledges that the reported confidence intervals capture evaluation-set uncertainty, not training-seed variation. Section C.1 further states that the 0.060 between-base spread at 70% real data 'should not be interpreted as a stable ranking without multi-seed replication.' This directly undermines Contribution (4) and the associated discussion of base-model reordering, since those conclusions depend on the assumption that a single fine-tuning run represents each condition. The monotonic real-data trend is less vulnerable because it is replicated across three bases and an independent descriptive series, but the ranking-change and spread-shrinkage claims need multi-seed replication for at least the 0%, 5%, and 70% cells, or they should be explicitly downgraded to single-run observations.
minor comments (5)
- [§3.1] The sentence 'These paiors are not cells of a full base×real fraction×adaptation surface design' contains a typo: 'paiors' should be 'pairs'.
- [§1] The acronym is introduced as 'Optical Chemical Structure Recognition (OSCR)' but the standard and consistently used abbreviation in the paper is OCSR; please correct the introduction to read 'Optical Chemical Structure Recognition (OCSR)'.
- [§1] The phrase 'In computation chemistry' should be 'In computational chemistry'.
- [§5.2, Table 6] The table row 'b/c—11/11 16/27 25/43 22/22' is cryptic; the b/c values should be labeled explicitly as the discordant-pair counts for the McNemar test, and the formatting should be aligned with other tables.
- [§A.1] The description of the effective epoch count is clear, but the relationship between the 496k-sample mixture and the exact 10,000-step budget would benefit from stating the effective batch size explicitly in the main text rather than only in the appendix.
Circularity Check
No significant circularity: the real-data effect is measured on external held-out benchmarks, and the self-cited consensus-labeling method is an input to training, not the measured outcome.
full rationale
The paper's central claim is empirical rather than definitional: the real-data effect is quantified by canonical exact match on held-out ACS, CLEF-IP, UOB, and USPTO images, using reference labels and evaluation protocols external to the training mixtures. The controlled 3x6 sweep varies the real-data fraction while holding mixture size, step budget, objective, and adaptation surface fixed, and the metric is not defined in terms of any fitted parameter or training-mixture statistic. The only self-citation, the companion paper [Guan, 2026], supplies the consensus reconciliation rule used to construct weak training labels; Appendix B states that rule deterministically, and the paper explicitly treats the labels as inputs rather than as the measured outcome, noting that 'Consensus-derived examples remain weak labels until manually audited.' The evaluation benchmarks and released recognizers are independent of the fitted values, and the paper's own Section 9 flags real limitations—'raising the real fraction adds images and molecules together' and the single-seed design—but these are alternative explanations or uncertainty caveats, not reductions of a prediction to its own construction. No equation equates a claimed result with a fitting target by construction, no fitted parameter is renamed as a prediction, and no load-bearing premise is justified solely by an unverified self-citation. The derivation chain is therefore self-contained with respect to circularity, even though the experimental design has acknowledged confounds.
Assumptions & free parameters
assumptions (4)
- domain assumption Canonical exact match via RDKit parsing and InChIKey equality is the correct identity metric.
- domain assumption The evaluation labels and images in ACS, CLEF-IP, UOB, and USPTO are correct and independently sourced.
- ad hoc to paper The consensus reconciliation rule in Appendix B produces training labels of sufficient quality.
- domain assumption A single fine-tuning run per cell is representative of that training configuration.
Cite this review
Pith. "Pith review of Real Data Closes Synthetic-to-Real Gap in Optical Chemical Structure Recognition." pith.science (2026). https://pith.science/paper/XB5J3CZY
@misc{pith2026260809100,
author = {Pith},
title = {Pith review of: Real Data Closes Synthetic-to-Real Gap in Optical Chemical Structure Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/XB5J3CZY}},
note = {Machine review of arXiv:2608.09100}
}
read the original abstract
Millions of chemical structures appear in patents and papers only as drawings, and using that information at scale requires reading the drawings. OCSR appears nearly solved on synthetic images yet remains difficult on real documents: the starting recognizer, Qwen2.5-VL-7B, exceeds 91% accuracy on synthetic renders but falls below 16% on three real-world benchmarks (ACS, CLEF-IP, USPTO). To identify the main source of improvement, 21 recognizers were fine-tuned on mixtures of synthetically rendered structures and labeled real depictions from patents, journal figures, and hand-drawn collections, varying the vision language model (VLM) base, the fraction of real training data, and the vision-tower adaptation strategy. Labeled real training images make the largest difference. For Qwen2.5-VL, ACS exact match rises from 0.15 with no real data to 0.37 at 9.5% and 0.46 at 50.2%; a controlled experiment across three base models reproduces the trend. A vision-tower LoRA, in contrast, does nothing for Qwen (+0.00, paired p=1.00), substantially helps InternVL3-8B (+22.8 to +34.6 pt), and modestly helps GLM-4.1V-9B (+1.0 to +9.6 pt), so its value depends on the base model. The best configuration reaches 0.96 exact match on clean renders and 0.49, 0.65, 0.84, and 0.76 on ACS, CLEF-IP, UOB, and USPTO, respectively. Gaps between base models are largest without real data (0.21), shrink to 0.06 at 70% real data, and reorder the ranking; base model and real-data mixture must therefore be selected together. Small-scale experiments on handwritten image-to-LaTeX recognition and chart-to-table conversion show that base-model rankings also vary beyond chemistry. More generally, model and adaptation choices for visual structure recognition should be evaluated on the target task.
Figures
Reference graph
Works this paper leans on
-
[1]
Journal of Chemical Information and Modeling , volume=
Optical structure recognition software to recover chemical information: OSRA, an open-source solution , author=. Journal of Chemical Information and Modeling , volume=. 2009 , publisher=
work page 2009
-
[2]
Rajan, Kohulan and Zielesny, Achim and Steinbeck, Christoph , journal=
-
[3]
Brinkhaus, Henning Otto and Rajan, Kohulan and Schaub, Jonas and Zielesny, Achim and Steinbeck, Christoph , journal=
-
[4]
Qian, Yujie and Guo, Jiang and Tu, Zhengkai and Li, Zhening and Coley, Connor W and Barzilay, Regina , journal=. 2023 , publisher=
work page 2023
-
[5]
Chen, Yufan and Leung, Ching Ting and Huang, Yong and Sun, Jianwei and Chen, Hao and Gao, Hanyu , journal=
-
[6]
Bai, Shuai and others , journal=
- [7]
-
[8]
2025 , howpublished=
2025
Show all 44 references
-
[9]
2025 , note=
Li, Junxian and Zhang, Di and Wang, Xunzhi and Hao, Zeying and Lei, Jingdi and Tan, Qian and Zhou, Cai and Liu, Wei and Yang, Yaotian and Xiong, Xinrui and Wang, Weiyun and Chen, Zhe and Wang, Wenhai and Li, Wei and Zhang, Shufei and Su, Mao and Ouyang, Wanli and Li, Yuqiang a...
2025
-
[10]
2026 , note=
Zhao, Xuanle and Zeng, Shuxin and Cai, Xinyuan and Cheng, Xiang and Zhang, Duzhen and Chen, Xiuyi and Xu, Bo , booktitle=. 2026 , note=
2026
-
[11]
Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle=
-
[12]
Journal of Chemical Information and Modeling , volume=
Molecular Structure Extraction from Documents Using Deep Learning , author=. Journal of Chemical Information and Modeling , volume=. 2019 , publisher=
2019
-
[13]
Rajan, Kohulan and Brinkhaus, Henning Otto and Zielesny, Achim and Steinbeck, Christoph , year=
-
[14]
Heller, Stephen R and McNaught, Alan and Pletnev, Igor and Stein, Stephen and Tchekhovskoi, Dmitrii , journal=
-
[15]
Landrum, Greg and others , howpublished=
-
[16]
ICML , year=
Image-to-Markup Generation with Coarse-to-Fine Attention , author=. ICML , year=
-
[17]
Masry, Ahmed and Long, Do Xuan and Tan, Jia Qing and Joty, Shafiq and Hoque, Enamul , booktitle=
-
[18]
Guan, Yani , title =
-
[19]
Cognitive Science , volume=
Why a Diagram is (Sometimes) Worth Ten Thousand Words , author=. Cognitive Science , volume=. 1987 , doi=
1987
-
[20]
Physiological Reviews , volume=
The Calculating Brain , author=. Physiological Reviews , volume=. 2025 , doi=
2025
-
[21]
Nature Communications , volume =
Rajan, Kohulan and Brinkhaus, Henning Otto and Agea, Maria Isabel and Zielesny, Achim and Steinbeck, Christoph , title =. Nature Communications , volume =
-
[22]
and White, Andrew D
Andonian, Alex and Rodriques, Samuel G. and White, Andrew D. and Narayanan, Siddharth M. , title =
-
[23]
Rodriques, Andrew D
Alex Andonian, Samuel G. Rodriques, Andrew D. White, and Siddharth M. Narayanan. MarkushGlyph and OCSRGlyph : Improved chemical structure recognition, 2026. arXiv:2607.28532
2026 arXiv
-
[24]
Qwen2.5-VL technical report
Shuai Bai et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025
2025 arXiv
-
[25]
RanDepict : Random chemical structure depiction generator
Henning Otto Brinkhaus, Kohulan Rajan, Jonas Schaub, Achim Zielesny, and Christoph Steinbeck. RanDepict : Random chemical structure depiction generator. Journal of Cheminformatics, 14: 0 31, 2022
2022
-
[26]
MolNexTR : a generalized deep learning model for molecular image recognition
Yufan Chen, Ching Ting Leung, Yong Huang, Jianwei Sun, Hao Chen, and Hanyu Gao. MolNexTR : a generalized deep learning model for molecular image recognition. Journal of Cheminformatics, 16 0 (1): 0 141, 2024 a
2024
-
[27]
InternVL : Scaling up vision foundation models and aligning for generic visual-linguistic tasks
Zhe Chen et al. InternVL : Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, 2024 b . InternVL3, OpenGVLab, 2025
2024
-
[28]
Image-to-markup generation with coarse-to-fine attention
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M Rush. Image-to-markup generation with coarse-to-fine attention. In ICML, 2017
2017
-
[29]
Optical structure recognition software to recover chemical information: Osra, an open-source solution
Igor V Filippov and Marc C Nicklaus. Optical structure recognition software to recover chemical information: Osra, an open-source solution. Journal of Chemical Information and Modeling, 49 0 (3): 0 740--743, 2009
2009
-
[30]
GLM-4.1V-Thinking
GLM-V Team, Zhipu AI . GLM-4.1V-Thinking . arXiv preprint arXiv:2507.01006, 2025
2025 arXiv
-
[31]
VERDICT : On real journal depictions, agreement certifies a machine-read chemical structure and self-verification does not, 2026
Yani Guan. VERDICT : On real journal depictions, agreement certifies a machine-read chemical structure and self-verification does not, 2026. Companion paper
2026
-
[32]
InChI , the IUPAC international chemical identifier
Stephen R Heller, Alan McNaught, Igor Pletnev, Stephen Stein, and Dmitrii Tchekhovskoi. InChI , the IUPAC international chemical identifier. Journal of Cheminformatics, 7: 0 23, 2015
2015
-
[33]
LoRA : Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA : Low-rank adaptation of large language models. In ICLR, 2022
2022
-
[34]
RDKit : Open-source cheminformatics
Greg Landrum et al. RDKit : Open-source cheminformatics. https://www.rdkit.org, 2024
2024
-
[35]
Larkin and Herbert A
Jill H. Larkin and Herbert A. Simon. Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11 0 (1): 0 65--100, 1987. doi:10.1111/j.1551-6708.1987.tb00863.x
1987
-
[36]
ChemVLM : Exploring the power of multimodal large language models in chemistry area
Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang, Xinrui Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Wei Li, Shufei Zhang, Mao Su, Wanli Ouyang, Yuqiang Li, and Dongzhan Zhou. ChemVLM : Exploring the power of multimodal large ...
2025
-
[37]
ChartQA : A benchmark for question answering about charts with visual and logical reasoning
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA : A benchmark for question answering about charts with visual and logical reasoning. In Findings of ACL, 2022
2022
-
[38]
The calculating brain
Andreas Nieder. The calculating brain. Physiological Reviews, 105 0 (1): 0 267--314, 2025. doi:10.1152/physrev.00014.2024
2025
-
[39]
MolScribe : Robust molecular structure recognition with image-to-graph generation
Yujie Qian, Jiang Guo, Zhengkai Tu, Zhening Li, Connor W Coley, and Regina Barzilay. MolScribe : Robust molecular structure recognition with image-to-graph generation. Journal of Chemical Information and Modeling, 63 0 (7): 0 1925--1934, 2023
1925
-
[40]
DECIMER : towards deep learning for chemical structure recognition
Kohulan Rajan, Achim Zielesny, and Christoph Steinbeck. DECIMER : towards deep learning for chemical structure recognition. Journal of Cheminformatics, 12 0 (1): 0 65, 2020
2020
-
[41]
DECIMER.ai : an open platform for automated optical chemical structure identification, segmentation and recognition in scientific publications
Kohulan Rajan, Henning Otto Brinkhaus, Maria Isabel Agea, Achim Zielesny, and Christoph Steinbeck. DECIMER.ai : an open platform for automated optical chemical structure identification, segmentation and recognition in scientific publications. Nature Communications, 14 0 (5045), 2023 a
2023
-
[42]
DECIMER hand-drawn molecule images dataset
Kohulan Rajan, Henning Otto Brinkhaus, Achim Zielesny, and Christoph Steinbeck. DECIMER hand-drawn molecule images dataset. Zenodo, 2023 b
2023
-
[43]
Molecular structure extraction from documents using deep learning
Joshua Staker, Kyle Marshall, Robert Abel, and Carolyn M McQuaw. Molecular structure extraction from documents using deep learning. Journal of Chemical Information and Modeling, 59 0 (3): 0 1017--1029, 2019
2019
-
[44]
TinyChemVL : Advancing chemical vision-language models via efficient visual token reduction and complex reaction tasks
Xuanle Zhao, Shuxin Zeng, Xinyuan Cai, Xiang Cheng, Duzhen Zhang, Xiuyi Chen, and Bo Xu. TinyChemVL : Advancing chemical vision-language models via efficient visual token reduction and complex reaction tasks. In AAAI, 2026. arXiv:2511.06283
2026
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.