{"id":"4abe4428-c6e3-4a38-b2f4-89c85bbe51f1","arxiv_id":"2505.15948","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Open-weight language models, including a 0.6B-parameter model, reach higher field-level citation-parsing accuracy than GROBID and Crossref search, with the small model succeeding within 64 sampled attempts.","lead":"This paper tests whether open-weight language models can turn plain-text bibliographic references into structured XML tags. It reports that they outperform existing tools like GROBID and Crossref search, and that even a 0.6-billion-parameter model solves most fields within 64 trials, hinting at cheap deployment for scholarly indexing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Untested extraction/matching pipeline (Sec 2.1, 0.75 edit-distance filter) biases the evaluation set; all accuracies and SOTA comparisons may hold only on an easy subset.","rationale":"The reader's identified weakest assumption is exactly the load-bearing concern. The paper assembles its evaluation data through an LLM-based extraction and a string-similarity filter, and the authors candidly admit they did not test extraction accuracy. Every result in Sec 3, including the GROBID and Crossref comparisons, depends on this filtered dataset. This is not an internal inconsistency, but it is a real threat to external validity. The claims about outperforming state-of-the-art methods would be much stronger if the evaluation included citations that the extraction step finds hard. I considered other candidate concerns: the best-of-two-prompts reporting (max over CoT and non-CoT per field) inflates per-field accuracy; the GROBID comparison uses a single API endpoint rather than a tuned full parser; and the abstract's '2^5 passes' is inconsistent with the actual 2^6 = 64 samples. These are secondary. The dataset bias is more fundamental because it undermines the denominator of every accuracy figure. The paper is transparent about the limitation and the direction is plausible, so the conditional verdict is appropriate. No code or data are provided, which is another reason not to accept fully. The concrete test would settle the concern by re-evaluating on unfiltered and low-similarity samples, allowing a direct check of whether the observed superiority persists in harder citation formats.","tokens_in":8311,"tokens_out":3893,"duration_ms":35958,"concrete_test":"Take a random sample of 500 references from the original JATS files in each corpus, without the 0.75 similarity filter. Run the same Llama-3.1-8B-Instruct extraction on the source markdown and manually label extraction correctness. For correctly extracted pairs, run the full evaluation (all 11 LLMs, GROBID, Crossref). Separately, take 200 previously discarded low-similarity pairs, manually repair the plaintext to match the JATS, and evaluate the same systems on them. Compare field accuracies for article-title, surname, volume, issue, year, fpage, and coverage. If accuracies drop materially on the low-similarity or manually-repaired set, or if non-Latin and link-heavy references are over-represented among discarded pairs, the central claim overgeneralizes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that open-weight LLMs outperform GROBID and Crossref is computed entirely on a dataset filtered by an untested extraction step. In Sec 2.1, plaintext citations are produced by Llama-3.1-8B-Instruct, verified only for substring presence, then matched to JATS using edit-distance similarity with a 0.75 threshold. The authors explicitly state in the Discussion that they 'did not test the accuracy of this step.' Because matches below 0.75 are discarded, and no analysis of the discarded pairs is provided (Fig. 1 shows distributions but not content), the surviving 2,000 citations may be systematically easier: clean English text, no embedded links, no non-Latin scripts, no unusual punctuation. Every model accuracy, including the GROBID and Crossref comparisons, is measured only on this filtered subset. If the extraction step fails on a particular citation format, that format is silently removed from evaluation, so the headline 'outperforming state-of-the-art methods' does not generalize to the full population of citations. The 0.75 threshold is arbitrary, with no sensitivity analysis, and the matched subset is further reduced by sampling 1,000 per corpus. A secondary inconsistency: the abstract says Qwen3-0.6B parses 'in 2^5 passes,' but Sec 2.2 and Fig. 3 use 2^6 = 64 samples; the smaller claim is unsupported. The dataset bias remains the more load-bearing issue because it affects every reported accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This short report evaluates open-weight decoder-only language models on the task of parsing plaintext bibliographic citations into JATS XML. The authors construct a 2,000-citation evaluation set from the PKP/Garnett and Open Research Europe corpora by extracting plaintext citations with Llama-3.1-8B-Instruct and matching them to JATS annotations using an edit-distance similarity threshold of 0.75. They then report pass@1 accuracies for eleven models across eight citation sub-fields, taking the best result over a chain-of-thought and a non-chain-of-thought prompt for each field, and compare these to GROBID and Crossref. For the smallest model, Qwen3-0.6B, they additionally report pass@64 accuracy and argue that this indicates reinforcement learning with verifiable rewards (RLVR) is likely to be effective. The paper's central claim is that open-weight language models, even out of the box, outperform state-of-the-art citation parsing tools.","tokens_in":8619,"tokens_out":3632,"duration_ms":32655,"significance":"If the reported accuracies are robust, the result is practically valuable: a small, open-weight model that can replace heavier citation-parsing infrastructure would lower barriers to reference indexing, particularly for journals and repositories in the Global South. The paper also has commendable transparency in several respects: the full prompts are included in Appendix A, the qualitative example of a low-similarity match is informative, the carbon-cost calculation is detailed, and the Discussion explicitly acknowledges that the initial extraction step was not tested. The main limitation is empirical rather than conceptual: every reported accuracy, including the GROBID and Crossref comparisons, is computed on a subset of citations that survived an untested extraction and matching pipeline. Because the authors themselves flag this as an untested step, the headline claims should be regarded as conditional until the pipeline is validated.","major_comments":[{"comment":"The evaluation set is constructed by a pipeline whose accuracy is not measured. Plaintext citations are produced by Llama-3.1-8B-Instruct, verified only for substring presence, and then matched to JATS entries using an edit-distance similarity threshold of 0.75; all pairs below this threshold are discarded. The authors state in the Discussion that 'We did not test the accuracy of this step.' This is load-bearing because every model accuracy and every comparison to GROBID and Crossref is computed on the surviving 2,000 citations. If the extraction or matching step systematically fails for certain formats—for example, citations containing embedded links, non-Latin scripts, or unusual punctuation—those cases are silently removed from evaluation, and the claim of outperforming state-of-the-art methods may hold only on an easy subset. The example in Sec. 2.1 of an ORE reference with PubMed links that lower the similarity score illustrates exactly such a failure mode. The authors should (a) report how many candidate pairs were discarded by the 0.75 threshold and characterize those pairs, (b) manually annotate a random sample of the discarded pairs to estimate extraction and matching error rates, and (c) provide a sensitivity analysis of the threshold (e.g., 0.65 and 0.85) to show that the main conclusions are not an artifact of this particular cutoff.","section":"Sec. 2.1"},{"comment":"The evaluation protocol reports the highest pass@1 accuracy for each field across two prompt variants (with and without chain of thought). This per-field selection over prompts can inflate accuracy relative to a fixed prompting strategy, and it makes the comparison to GROBID and Crossref less straightforward because those baselines are evaluated with a single default configuration. The paper should either report results for each prompt variant separately or apply a consistent model-selection rule (e.g., choosing the better prompt on a held-out development set) rather than taking the per-field maximum. In addition, no confidence intervals or significance tests are reported; with 2,000 items, a 90% accuracy has a standard error of roughly 0.7 percentage points, and the claimed differences from baselines may be small relative to this uncertainty. The authors should include error bars or confidence intervals, at least for the headline comparisons.","section":"Sec. 2.2 and Fig. 2"},{"comment":"The abstract states that Qwen3-0.6B 'can parse all fields with high accuracy in 2^5 passes,' but Sec. 2.2 and Fig. 3 report sampling 2^6 = 64 completions. The pass@64 result therefore does not support the '2^5' claim as written; if the authors intend to claim 32-pass performance, they need to report pass@32. More substantively, pass@64 measures the probability that the correct field value appears in at least one of 64 independent samples; it is not pass@1 accuracy and does not by itself establish that RLVR will succeed. The connection to Yue et al. (2025) is suggestive, but the paper does not actually train an RLVR policy. The claim in Sec. 3.2 that the task 'is within distribution for the model' should be tempered to state that the model can sometimes produce the correct field within a large sample, and the inference to RLVR effectiveness should be presented as a hypothesis rather than a demonstrated result.","section":"Sec. 3.2 and Abstract"}],"minor_comments":[{"comment":"The abstract and the main text use '2^5' and '25' inconsistently with the actual 2^6 = 64 samples; please correct the exponent and ensure the abstract matches the experimental design.","section":"Abstract and Sec. 3.2"},{"comment":"The sentence 'We also sampled 2^6 = 64 completions for each citation, with a reasoning trace, from Qwen3-0.6B' appears in Sec. 2.2 but the results are presented in Sec. 3.2; consider restructuring so that the pass@64 experiment is described together with its motivation and results.","section":"Sec. 2.2"},{"comment":"The GROBID comparison reports only article-title and surname accuracy, while the language models are evaluated on eight fields. If other GROBID fields are shown in Fig. 2, the mapping from GROBID's TEI output to the JATS fields used here should be described; if not, the comparison should be labeled as partial.","section":"Sec. 3.1"},{"comment":"The paper does not state how the 1,000-citation sample was drawn from each corpus (random, stratified, or otherwise). Please specify the sampling procedure so that the representativeness of the final 2,000-citation set can be assessed.","section":"Sec. 2.1"},{"comment":"The paper does not include a data or code availability statement. Releasing the matched dataset, the extraction prompts, and the evaluation scripts would substantially strengthen reproducibility and allow other researchers to test the impact of the matching threshold.","section":"Data availability"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style short report with a practically motivated question and a commendably honest limitation section. The central problem is that the evaluation pipeline is untested and the threshold used to filter the dataset is arbitrary; this affects every reported accuracy, so the manuscript needs additional validation before the headline claims can be accepted. The issue is fixable within the scope of the paper: the authors should measure extraction accuracy on a manually labeled sample, characterize the discarded pairs, and add a threshold sensitivity analysis. I would not reject, but I would not accept without these additions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is the evaluation of eleven open-weight models on citation markup, including a 0.6B model that gets pass@64 above 90% on every field. That is a concrete, useful data point for anyone thinking about low-compute reference parsers. The paper is also refreshingly honest about its own limitations, and the carbon accounting is a nice touch. The authors did real work with an under-served problem, and the writing is clear.\n\nThe soft spot is exactly where the stress-test note lands. The evaluation set is built by extracting plaintext citations with Llama-3.1-8B-Instruct and matching them to JATS using edit-distance similarity with a 0.75 threshold. The authors openly say they did not test the accuracy of the extraction step. That means every reported accuracy, including the GROBID and Crossref comparisons, is computed on the subset that survived this untested filter. If the extraction or matching systematically fails on citations with links, non-Latin scripts, or unusual punctuation, those are silently dropped. The paper shows similarity distributions but not the content of the discarded pairs, and there is no sensitivity analysis for the 0.75 threshold. So the headline claim that LMs outperform state-of-the-art methods is, strictly, a claim about an easy subset of citations. That doesn't make the result useless, but it makes it conditional.\n\nThere is also a specific inconsistency: the abstract says Qwen3-0.6B parses “in 2^5 passes,” while Section 2.2 and Figure 3 use 2^6 = 64 completions. The smaller claim is unsupported. And no code or data is provided, which weakens the benchmark's value for reproducibility.\n\nBottom line: this is a workshop-grade report that answers a real question and gives a plausible signal that small LMs are worth pursuing. The missing validation of the data pipeline and the pass-count error mean the accuracy claims should be treated as provisional, not as a settled comparison with GROBID and Crossref. I'd send it to peer review because the empirical question is legitimate and the authors are transparent, but I'd ask for a revision that tests the extraction step, reports sensitivity to the matching threshold, and fixes the 2^5/2^6 discrepancy. If they can do that, the paper becomes a genuinely useful baseline for citation parsing in low-resource settings.","headline":"Useful short benchmark for small open-weight models on citation parsing, but the untested extraction/matching pipeline and a 2^5 vs 2^6 slip make the headline accuracy claims provisional.","tokens_in":9132,"tokens_out":2207,"would_cite":false,"duration_ms":21147,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-weight language models, out of the box, mark up citations as JATS XML more accurately than GROBID or Crossref, and a 0.6B model reaches high accuracy across repeated samples.","keywords":["citation parsing","language models","JATS XML","reference markup","open scholarly metadata","small language models","GROBID","Crossref"],"falsifier":"Reproduce the benchmark on a hand-verified sample of references drawn from the same two corpora, including non-English, link-rich, and locally formatted citations, and compare the language models' per-field accuracy against GROBID and Crossref on that independently labeled set; if the language models no longer outperform the baselines, the paper's central claim fails.","tokens_in":8106,"feed_emoji":"📚","tokens_out":14473,"duration_ms":111058,"temperature":0.7,"pith_summary":"The paper asks whether general-purpose, open-weight language models can replace specialized citation parsers. It reports that eleven such models, prompted with only two example annotations and no fine-tuning, identify the components of a plaintext citation—authors, article title, source, year, volume, issue, pages, and related fields—and emit them as JATS XML more accurately than GROBID and Crossref search on a 2,000-citation test set. The authors highlight that even the worst-performing language model beats GROBID's article-title accuracy by 8.76 percentage points, and most models beat GROBID on first-author surname. They also report that Qwen3-0.6B, the smallest model, gets every field above 90% accuracy in 64 sampled generations, which they take as evidence that a tiny model can be post-trained into a robust parser. If correct, citation and reference indexing could run on modest hardware, which matters for journals and archives in under-indexed regions.","feed_headline":"Small open language models beat GROBID and Crossref","feed_subtitle":"Scores above 90% on every field in 64-try sampling, pointing to low-cost, in-browser indexing.","key_machinery":"The mechanism is the few-shot JATS annotation prompt: every test citation is given to the model as plaintext along with two worked examples of plaintext-to-JATS conversions, and the model must output the JATS mixed-citation, optionally after a chain of thought. Accuracy is measured per field with exact match except a 10-character edit-distance tolerance for article-title and 5 for source, and malformed XML fails all fields. The second mechanism is the dataset-construction pipeline: plaintext citations are pulled from markdown-converted articles by an 8-billion-parameter language model and matched to JATS XML by edit-distance similarity, keeping only pairs with similarity at least 0.75 before sampling 2,000 for evaluation. For the small-model claim, the 64-sample pass@64 rate is the load-bearing quantity, because prior work has tied such pass@k success to whether reinforcement learning with verifiable rewards can learn the task.","core_discovery":"The paper's central claim is that citation parsing is no longer the preserve of bespoke extraction software: a decoder-only language model, given two plaintext/JATS example pairs and instructed to output only the XML annotation, can split a plaintext reference into its constituent fields with accuracy that matches or exceeds GROBID and Crossref. Field-level scoring is strict—exact match for most fields, with a small edit-distance allowance for article title and source—and an invalid XML response counts as wrong on every field. Against these baselines, Crossref's best article-title accuracy is 0.633 even on its high-confidence subset, GROBID's is 0.6674, and the worst language model in the study exceeds GROBID's article-title accuracy by 8.76 percentage points, with most models also beating GROBID's 0.8516 first-author-surname accuracy. For small-scale deployment, Qwen3-0.6B's 64 sampled completions contain the correct value for every sub-element in over 90% of citations, which the paper presents as evidence that the task lies within a small model's learned distribution and that reinforcement learning with verifiable rewards can turn those samples into a reliable single-pass parser.","pith_inferences":["Because the extraction-and-matching step was never validated, the accuracy numbers should be read as bounds on a filtered subset; a natural extension is to hand-label a stratified sample from the two original corpora and rerun the benchmark without any automated extraction.","The corpora used are predominantly English-language and North-American or European in origin; testing the same prompts on citations in Spanish, Portuguese, French, Arabic, or with publisher-added link text would show whether the claimed advantage survives the multilingual, link-rich contexts that motivated the study.","Constrained decoding that forces valid JATS XML, which the paper mentions but does not test, would eliminate the coverage penalty and is a cheaper complement to post-training for deployment.","Pass@64 success is not the same as single-pass pass@1 accuracy; the paper's small-model optimism would be directly tested by fine-tuning Qwen3-0.6B on a few thousand annotated citations and measuring whether pass@1 approaches the reported field accuracies."],"forward_implications":["A journal or archive could generate JATS-marked reference lists from plaintext with a generic open-weight model, without training a task-specific extractor.","The Qwen3-0.6B pass@64 result indicates that a post-trained tiny model is a realistic route to browser- or phone-scale citation parsing, which would lower the compute barrier for smaller publishers.","The same models can be applied offline to already-published articles, enriching reference metadata retrospectively in archives that current indexing services cover poorly.","Better reference markup directly improves the fidelity of citation and reference networks, which the paper argues is needed for fairer global research evaluation and for understanding how knowledge circulates outside the Global North."],"supporting_citations":[{"why":"Supplies the PKP corpus of 829 JATS-marked submitted manuscripts, one of the two source corpora for the matched citation dataset.","marker":"Garnett (2016)"},{"why":"Supplies the Open Research Europe corpus of 848 JATS-marked published articles, the other source corpus.","marker":"European Commission (2025)"},{"why":"Provides the markitdown tool that converted the article files to markdown, from which plaintext citations were then extracted.","marker":"Microsoft Corporation (2025)"},{"why":"Establishes the connection between pass@k performance and learnability by reinforcement learning with verifiable reward, which supports the claim that Qwen3-0.6B's pass@64 accuracy implies post-training will succeed.","marker":"Yue et al. (2025)"}],"fun_headline_variants":["Open language models beat citation-parsing tools","Small LMs outperform GROBID and Crossref on citations","Qwen-0.6B parses citations better than GROBID","Language models beat GROBID and Crossref on citation fields","Citation parsing falls to open-weight language models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies are measured only on the 2,000 citation pairs that survived a machine-driven extraction and matching step, and the paper explicitly says that step's own accuracy was never tested; if that step silently drops or distorts some citation styles, the results will not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Open language models beat citation-parsing tools","Small LMs outperform GROBID and Crossref on citations","Qwen-0.6B parses citations better than GROBID","Language models beat GROBID and Crossref on citation fields","Citation parsing falls to open-weight language models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1702,"prompt_tokens":1025,"completion_tokens":677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":598}},"tokens_in":641,"tokens_out":677,"duration_ms":5986,"temperature":1.0,"reasoning_tokens":598,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:10:03.030572+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the benchmark on a hand-verified sample of references drawn from the same two corpora, including non-English, link-rich, and locally formatted citations, and compare the language models' per-field accuracy against GROBID and Crossref on that independently labeled set; if the language models no longer outperform the baselines, the paper's central claim fails.","supporting_citations":[{"cited_title":"The XML Markup Evaluation Corpus , April 2016","cited_arxiv_id":null,"evidence_quote":"Supplies the PKP corpus of 829 JATS-marked submitted manuscripts, one of the two source corpora for the matched citation dataset."},{"cited_title":"Open Research Europe : Full article corpus","cited_arxiv_id":null,"evidence_quote":"Supplies the Open Research Europe corpus of 848 JATS-marked published articles, the other source corpus."},{"cited_title":"MarkItDown: A Python Tool for Converting Files to Markdown , 2025","cited_arxiv_id":null,"evidence_quote":"Provides the markitdown tool that converted the article files to markdown, from which plaintext citations were then extracted."}],"review_version":1}