{"id":"4a4f86ad-a294-48d7-a4a9-b147b4bcf04c","arxiv_id":"2508.00311","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A vision-language model fine-tuned on the new CSFormula dataset claims state-of-the-art formula recognition across line, paragraph, and page levels.","lead":"This paper presents a system that reads math formulas from complex scientific documents by fine-tuning a general vision-language model, plus a new large dataset for training and evaluation. It claims the approach beats specialized formula-recognition models across many layouts and scientific fields.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim of surpassing specialized models is only as strong as the external evaluation; without a benchmark independent of CSFormula, the reported results may reflect in-distribution fitting rather than generalized capability.","rationale":"The reader's weakest assumption is that CSFormula accurately represents real-world complex scientific documents, which is about dataset representativeness. My concern is closely related but more specific and more load-bearing for the exact wording of the strongest claim: the claim 'surpasses specialized models in terms of accuracy and robustness' can only be tested against models trained and evaluated on established benchmarks. If CSFormula is used for both training and evaluation, the comparison is circular in the sense of distribution overlap, not in the logical sense. The abstract gives no indication that such external evaluation exists, so the headline claim is unverified. The proposed concrete test directly targets this gap: CROHME is the standard formula recognition benchmark with published specialized-model results, and running the released model on it with a held-out protocol would settle whether the gains transfer. I did not set a harsher verdict because the full text may well contain the missing external evaluation; the reader's CONDITIONAL verdict already captures the need for more evidence. My partial agreement reflects that the reader emphasized dataset representativeness, whereas I emphasize the need for an external benchmark to make the SOTA claim meaningful; both are valid but distinct concerns.","tokens_in":691,"tokens_out":2132,"duration_ms":23873,"concrete_test":"Apply the fine-tuned DocTron-Formula to the CROHME 2014 and CROHME 2016 test sets, using the official evaluation protocol and without any CROHME data in the training set. Compare the reported expression recognition accuracy against the published results of the specialized models cited in the paper. If DocTron-Formula does not match or exceed those published numbers on at least one of these standard benchmarks, then the abstract's claim of surpassing specialized models in accuracy and robustness is not supported and should be narrowed to in-distribution performance on CSFormula.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that DocTron-Formula surpasses specialized models in accuracy and robustness across styles, domains, and layouts. The only evidence available in the abstract is performance related to CSFormula, the same dataset introduced and used for fine-tuning. If the reported test results come from splits of CSFormula that share document sources, layout templates, or domain coverage with the training split, then the evaluation is in-distribution. In-distribution superiority over specialized models does not establish the abstract's broader claims of 'generalized formula recognition' or a 'new paradigm'; it establishes that the model fits CSFormula's distribution, which the authors control. The robustness claim is additionally unsupported without explicit out-of-distribution or stress tests. The load-bearing condition for the central claim is therefore that the evaluation includes at least one independent benchmark or held-out corpus not derived from CSFormula, with reported baselines from published specialized models. Since the full text is unavailable, this concern cannot be resolved from the abstract; however, it is a specific, testable gap rather than a generic appeal for more experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript (abstract only) proposes DocTron-Formula, a unified formula recognition framework built on general vision-language models, and introduces CSFormula, a large-scale dataset with line-, paragraph-, and page-level formula annotations across multiple scientific domains. The abstract claims that supervised fine-tuning on CSFormula achieves state-of-the-art accuracy and robustness, surpassing specialized models and establishing a new paradigm for automated understanding of complex scientific documents. No experimental details, benchmark names, dataset statistics, or quantitative results are provided in the available text.","tokens_in":878,"tokens_out":2603,"duration_ms":27343,"significance":"If substantiated, the work would be practically significant: a single general VLM-based pipeline that competes with or exceeds specialized formula recognizers across varied layouts and domains could simplify deployment and benefit scientific document understanding. The proposed CSFormula dataset could also be a valuable community resource. However, the significance is entirely contingent on the hidden experimental validation. The claims of state-of-the-art performance, robustness, and a new paradigm are plausible but currently unsupported, and the circularity risk arising from introducing and evaluating on the same dataset is a central concern.","major_comments":[{"comment":"The central claim that DocTron-Formula 'surpasses specialized models in terms of accuracy and robustness' is unverifiable without any quantitative results, benchmark names, baseline definitions, or error bars. The abstract must report concrete metrics on at least one public formula recognition benchmark (e.g., CROHME, ICDAR-2019, or similar) and on a held-out test split of CSFormula, along with the compared specialized models and their published scores.","section":"Abstract (last sentence)"},{"comment":"The paper both introduces CSFormula and uses it for fine-tuning, creating a risk of in-distribution evaluation. To support the claim of 'generalized formula recognition,' the authors must provide explicit train/test split statistics, state that no document sources or layout templates overlap between splits, and demonstrate performance on independent external corpora that are not derived from CSFormula.","section":"Abstract (CSFormula description)"},{"comment":"The term 'robustness' is not defined or evidenced. Robustness should be demonstrated through explicit out-of-distribution tests, such as cross-domain transfer, layout perturbations, or noisy rendering, with corresponding metrics. A single dataset's accuracy cannot support a standalone robustness claim.","section":"Abstract (robustness claim)"}],"minor_comments":[{"comment":"The abstract does not provide any statistics for CSFormula, such as the number of formulas per level (line, paragraph, page), the number of domains, the document sources, or the annotation procedure; these are needed for readers to assess the dataset's scale and challenge.","section":"Abstract (dataset description)"},{"comment":"The abstract asserts that CSFormula is 'multidisciplinary' but does not list which disciplines are included; adding a brief enumeration or a domain breakdown would clarify the scope.","section":"Abstract (multidisciplinary claim)"},{"comment":"No mention is made of code, model weights, or dataset release plans, which are important for reproducibility and would strengthen the contribution if included.","section":"Abstract (reproducibility)"}],"recommendation":"major_revision","confidential_remarks":"The submission is an abstract-only manuscript, so the review is necessarily limited. The main risk is that the claimed state-of-the-art results are based solely on in-distribution evaluation on the newly introduced CSFormula dataset. The authors should be required to submit a full paper with thorough external validation, detailed dataset documentation, and a clear separation between training and evaluation sources. If the full experiments are clean, the work could be a strong contribution; if not, the claims would need to be substantially narrowed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid new-application paper with a useful new dataset, but the abstract overreaches with \"new paradigm\" and the SOTA claim can't be assessed until we see the evaluation protocol. Worth sending to peer review, because if the eval is clean it becomes the default choice for formula-aware OCR.\n\nWhat's actually new: the CSFormula dataset, covering line/paragraph/page levels across multiple disciplines, and the straightforward recipe of supervised fine-tuning a general VLM on it. That's a real contribution—formula recognition has been stuck in specialized architectures, and showing that a general VLM can beat them with SFT is genuinely useful. The \"eliminating the need for specialized architectures\" framing is accurate and not hype.\n\nWhere the soft spots are: the central claim—surpassing specialized models in accuracy and robustness—rests entirely on results tied to CSFormula, the same dataset the authors introduce and use for fine-tuning. If the test splits share document sources or layout templates with training, then the results are in-distribution and don't justify the robustness claim. The full paper needs to name independent benchmarks and show held-out or OOD evaluation. The \"new paradigm\" sentence is also overdrawn; this is an application of SFT, not a new learning principle. Neither issue is fatal, but they are the difference between a 6 and a 7.\n\nOne more minor thing: the abstract gives no dataset statistics, no baseline names, no error bars. That's typical for an abstract, so I'm not docking it heavily, but it does mean the current version is unverifiable.\n\nWho this is for: anyone working on document understanding, OCR, or scientific literature mining. If the dataset is released and the eval is credible, it will get cited.\n\nRecommendation: send to peer review. Ask the authors to release the dataset, report on external benchmarks, and clarify train/test separation. If they deliver that, it deserves to be published.","headline":"A plausible and useful systems paper whose headline claim depends on evaluation details the abstract doesn't provide.","tokens_in":1393,"tokens_out":1247,"would_cite":false,"duration_ms":13961,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-tuning a general vision-language model on the large, multidisciplinary CSFormula dataset achieves state-of-the-art formula recognition at line, paragraph, and page levels.","keywords":["formula recognition","OCR","vision-language model","scientific document understanding","math formula dataset","supervised fine-tuning","layout complexity"],"falsifier":"A direct test would be to hold out a set of journal pages whose layout patterns do not appear in CSFormula and check whether DocTron-Formula still keeps its accuracy edge over specialized models; a sharp drop on such held-out pages would indicate the dataset does not capture real-world variability.","tokens_in":519,"feed_emoji":"🧮","tokens_out":2744,"duration_ms":28264,"temperature":0.7,"pith_summary":"This paper sets out to show that formula recognition in scientific documents does not need task-specific OCR architectures. The authors fine-tune a general vision-language model on CSFormula, a new large-scale dataset of structurally complex formulas collected at line, paragraph, and page levels, and report state-of-the-art accuracy across scientific domains and layout styles. The intended consequence is simpler, more robust automated reading of mathematical content in real-world papers. If true, this would let one general model handle formula extraction and broader document understanding without specialized components.","feed_headline":"One fine-tuned vision model beats specialized formula OCR","feed_subtitle":"New CSFormula dataset spans line, paragraph, and page layouts across many disciplines.","key_machinery":"The load-bearing component is CSFormula, a large-scale dataset of mathematically and structurally complex formulas with annotations at line, paragraph, and page levels, spanning multiple scientific disciplines. The mechanism that carries the argument is supervised fine-tuning: taking a general vision-language model and adapting it to formula recognition with this dataset, rather than designing a specialized architecture. This combination is what the paper claims yields the improved accuracy and robustness.","core_discovery":"On the paper's own terms, the central discovery is that straightforward supervised fine-tuning of a general vision-language model is sufficient to reach state-of-the-art formula recognition, eliminating the need for specialized architectures. The framework, called DocTron-Formula, unifies line, paragraph, and page-level formula recognition, and the authors introduce CSFormula, a large and challenging dataset covering multidisciplinary and structurally complex formulas. Experimental results are reported to show that the approach surpasses specialized models in both accuracy and robustness, establishing a new path for automated understanding of complex scientific documents.","pith_inferences":["If the central claim holds, formula recognition could be folded into a single vision-language model that also handles tables, figures, and text, rather than relying on a cascade of specialized OCR components.","The main unknown is dataset representativeness: a natural extension would be to test zero-shot transfer to page images from journals and layout styles entirely absent from CSFormula.","The phrase 'new paradigm' implies a broader shift toward unified document understanding, but the paper itself only demonstrates formula recognition, not other document tasks.","A direct practical test would be to compare the fine-tuned model against a general vision-language model that has never seen CSFormula, to isolate the contribution of the dataset."],"forward_implications":["A single general vision-language model can replace specialized formula-recognition systems without sacrificing accuracy.","Formula recognition can be handled jointly at line, paragraph, and page levels within one unified framework.","The CSFormula dataset provides a new benchmark for training and evaluating formula recognition on more complex, real-world-like inputs.","The approach suggests that broader scientific document understanding tasks may also be addressable with general models rather than task-specific pipelines.","If the reported robustness holds across domains, the method could simplify OCR pipelines for scientific literature digitization."],"supporting_citations":[],"fun_headline_variants":["General VLM fine-tuned to beat specialized formula OCR","CSFormula dataset helps general VLM top formula recognition","DocTron-Formula: fine-tuned VLM outdoes bespoke OCR","One vision model for line, paragraph, and page formula OCR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The key assumption is that CSFormula's mix of styles, disciplines, and page layouts represents the real-world distribution of scientific documents, so that fine-tuning on it transfers to unseen documents.","fun_headline_variants_meta":{"raw":{"variants":["General VLM fine-tuned to beat specialized formula OCR","CSFormula dataset helps general VLM top formula recognition","DocTron-Formula: fine-tuned VLM outdoes bespoke OCR","One vision model for line, paragraph, and page formula OCR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3185,"prompt_tokens":787,"completion_tokens":2398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":403,"completion_tokens_details":{"reasoning_tokens":2326}},"tokens_in":403,"tokens_out":2398,"duration_ms":18851,"temperature":1.0,"reasoning_tokens":2326,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:12:17.076182+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to hold out a set of journal pages whose layout patterns do not appear in CSFormula and check whether DocTron-Formula still keeps its accuracy edge over specialized models; a sharp drop on such held-out pages would indicate the dataset does not capture real-world variability.","supporting_citations":[],"review_version":1}