REVIEW 3 major objections 5 minor 25 references
Citation Parsing and Analysis with Language Models
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Open-weight language models, out of the box, mark up citations as JATS XML more accurately than GROBID or Crossref, and a 0.6B model reaches high accuracy across repeated samples.
desk verdict Useful short benchmark for small open-weight models on citation parsing, but the untested extraction/matching pipeline and a 2^5 vs 2^6 slip make the headline accuracy claims provisional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the few-shot JATS annotation prompt: every test citation is given to the model as plaintext along with two worked examples of plaintext-to-JATS conversions, and the model must output the JATS mixed-citation, optionally after a chain of thought. Accuracy is measured per field with exact match except a 10-character edit-distance tolerance for article-title and 5 for source, and malformed XML fails all fields. The second mechanism is the dataset-construction pipeline: plaintext citations are pulled from markdown-converted articles by an 8-billion-parameter language model and matched to JATS XML by edit-distance similarity, keeping only pairs with similarity at least 0.75 before sampling 2,000 for evaluation. For the small-model claim, the 64-sample pass@64 rate is the load-bearing quantity, because prior work has tied such pass@k success to whether reinforcement learning with verifiable rewards can learn the task.
What would settle it
Reproduce the benchmark on a hand-verified sample of references drawn from the same two corpora, including non-English, link-rich, and locally formatted citations, and compare the language models' per-field accuracy against GROBID and Crossref on that independently labeled set; if the language models no longer outperform the baselines, the paper's central claim fails.
Extended reading notes
Core claim
The paper's central claim is that citation parsing is no longer the preserve of bespoke extraction software: a decoder-only language model, given two plaintext/JATS example pairs and instructed to output only the XML annotation, can split a plaintext reference into its constituent fields with accuracy that matches or exceeds GROBID and Crossref. Field-level scoring is strict—exact match for most fields, with a small edit-distance allowance for article title and source—and an invalid XML response counts as wrong on every field. Against these baselines, Crossref's best article-title accuracy is 0.633 even on its high-confidence subset, GROBID's is 0.6674, and the worst language model in the study exceeds GROBID's article-title accuracy by 8.76 percentage points, with most models also beating GROBID's 0.8516 first-author-surname accuracy. For small-scale deployment, Qwen3-0.6B's 64 sampled completions contain the correct value for every sub-element in over 90% of citations, which the paper presents as evidence that the task lies within a small model's learned distribution and that reinforcement learning with verifiable rewards can turn those samples into a reliable single-pass parser.
Load-bearing premise
The reported accuracies are measured only on the 2,000 citation pairs that survived a machine-driven extraction and matching step, and the paper explicitly says that step's own accuracy was never tested; if that step silently drops or distorts some citation styles, the results will not generalize.
Editorial extensions
If this is right
- A journal or archive could generate JATS-marked reference lists from plaintext with a generic open-weight model, without training a task-specific extractor.
- The Qwen3-0.6B pass@64 result indicates that a post-trained tiny model is a realistic route to browser- or phone-scale citation parsing, which would lower the compute barrier for smaller publishers.
- The same models can be applied offline to already-published articles, enriching reference metadata retrospectively in archives that current indexing services cover poorly.
- Better reference markup directly improves the fidelity of citation and reference networks, which the paper argues is needed for fairer global research evaluation and for understanding how knowledge circulates outside the Global North.
Reading between the lines
- Because the extraction-and-matching step was never validated, the accuracy numbers should be read as bounds on a filtered subset; a natural extension is to hand-label a stratified sample from the two original corpora and rerun the benchmark without any automated extraction.
- The corpora used are predominantly English-language and North-American or European in origin; testing the same prompts on citations in Spanish, Portuguese, French, Arabic, or with publisher-added link text would show whether the claimed advantage survives the multilingual, link-rich contexts that motivated the study.
- Constrained decoding that forces valid JATS XML, which the paper mentions but does not test, would eliminate the coverage penalty and is a cheaper complement to post-training for deployment.
- Pass@64 success is not the same as single-pass pass@1 accuracy; the paper's small-model optimism would be directly tested by fine-tuning Qwen3-0.6B on a few thousand annotated citations and measuring whether pass@1 approaches the reported field accuracies.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This short report evaluates open-weight decoder-only language models on the task of parsing plaintext bibliographic citations into JATS XML. The authors construct a 2,000-citation evaluation set from the PKP/Garnett and Open Research Europe corpora by extracting plaintext citations with Llama-3.1-8B-Instruct and matching them to JATS annotations using an edit-distance similarity threshold of 0.75. They then report pass@1 accuracies for eleven models across eight citation sub-fields, taking the best result over a chain-of-thought and a non-chain-of-thought prompt for each field, and compare these to GROBID and Crossref. For the smallest model, Qwen3-0.6B, they additionally report pass@64 accuracy and argue that this indicates reinforcement learning with verifiable rewards (RLVR) is likely to be effective. The paper's central claim is that open-weight language models, even out of the box, outperform state-of-the-art citation parsing tools.
Significance. If the reported accuracies are robust, the result is practically valuable: a small, open-weight model that can replace heavier citation-parsing infrastructure would lower barriers to reference indexing, particularly for journals and repositories in the Global South. The paper also has commendable transparency in several respects: the full prompts are included in Appendix A, the qualitative example of a low-similarity match is informative, the carbon-cost calculation is detailed, and the Discussion explicitly acknowledges that the initial extraction step was not tested. The main limitation is empirical rather than conceptual: every reported accuracy, including the GROBID and Crossref comparisons, is computed on a subset of citations that survived an untested extraction and matching pipeline. Because the authors themselves flag this as an untested step, the headline claims should be regarded as conditional until the pipeline is validated.
major comments (3)
- [Sec. 2.1] The evaluation set is constructed by a pipeline whose accuracy is not measured. Plaintext citations are produced by Llama-3.1-8B-Instruct, verified only for substring presence, and then matched to JATS entries using an edit-distance similarity threshold of 0.75; all pairs below this threshold are discarded. The authors state in the Discussion that 'We did not test the accuracy of this step.' This is load-bearing because every model accuracy and every comparison to GROBID and Crossref is computed on the surviving 2,000 citations. If the extraction or matching step systematically fails for certain formats—for example, citations containing embedded links, non-Latin scripts, or unusual punctuation—those cases are silently removed from evaluation, and the claim of outperforming state-of-the-art methods may hold only on an easy subset. The example in Sec. 2.1 of an ORE reference with PubMed links that lower the similarity score illustrates exactly such a failure mode. The authors should (a) report how many candidate pairs were discarded by the 0.75 threshold and characterize those pairs, (b) manually annotate a random sample of the discarded pairs to estimate extraction and matching error rates, and (c) provide a sensitivity analysis of the threshold (e.g., 0.65 and 0.85) to show that the main conclusions are not an artifact of this particular cutoff.
- [Sec. 2.2 and Fig. 2] The evaluation protocol reports the highest pass@1 accuracy for each field across two prompt variants (with and without chain of thought). This per-field selection over prompts can inflate accuracy relative to a fixed prompting strategy, and it makes the comparison to GROBID and Crossref less straightforward because those baselines are evaluated with a single default configuration. The paper should either report results for each prompt variant separately or apply a consistent model-selection rule (e.g., choosing the better prompt on a held-out development set) rather than taking the per-field maximum. In addition, no confidence intervals or significance tests are reported; with 2,000 items, a 90% accuracy has a standard error of roughly 0.7 percentage points, and the claimed differences from baselines may be small relative to this uncertainty. The authors should include error bars or confidence intervals, at least for the headline comparisons.
- [Sec. 3.2 and Abstract] The abstract states that Qwen3-0.6B 'can parse all fields with high accuracy in 2^5 passes,' but Sec. 2.2 and Fig. 3 report sampling 2^6 = 64 completions. The pass@64 result therefore does not support the '2^5' claim as written; if the authors intend to claim 32-pass performance, they need to report pass@32. More substantively, pass@64 measures the probability that the correct field value appears in at least one of 64 independent samples; it is not pass@1 accuracy and does not by itself establish that RLVR will succeed. The connection to Yue et al. (2025) is suggestive, but the paper does not actually train an RLVR policy. The claim in Sec. 3.2 that the task 'is within distribution for the model' should be tempered to state that the model can sometimes produce the correct field within a large sample, and the inference to RLVR effectiveness should be presented as a hypothesis rather than a demonstrated result.
minor comments (5)
- [Abstract and Sec. 3.2] The abstract and the main text use '2^5' and '25' inconsistently with the actual 2^6 = 64 samples; please correct the exponent and ensure the abstract matches the experimental design.
- [Sec. 2.2] The sentence 'We also sampled 2^6 = 64 completions for each citation, with a reasoning trace, from Qwen3-0.6B' appears in Sec. 2.2 but the results are presented in Sec. 3.2; consider restructuring so that the pass@64 experiment is described together with its motivation and results.
- [Sec. 3.1] The GROBID comparison reports only article-title and surname accuracy, while the language models are evaluated on eight fields. If other GROBID fields are shown in Fig. 2, the mapping from GROBID's TEI output to the JATS fields used here should be described; if not, the comparison should be labeled as partial.
- [Sec. 2.1] The paper does not state how the 1,000-citation sample was drawn from each corpus (random, stratified, or otherwise). Please specify the sampling procedure so that the representativeness of the final 2,000-citation set can be assessed.
- [Data availability] The paper does not include a data or code availability statement. Releasing the matched dataset, the extraction prompts, and the evaluation scripts would substantially strengthen reproducibility and allow other researchers to test the impact of the matching threshold.
Circularity Check
No significant circularity: the paper is an empirical benchmark with external ground-truth labels and external baselines; noted weaknesses are validity and consistency issues, not circular reasoning.
full rationale
The central claims are measured accuracies on a held-out evaluation set. Ground-truth field labels come from JATS-marked-up references in two pre-existing corpora (Garnett/PKP and ORE), which are independent of the language models being evaluated. The evaluation compares each model's output against those external labels using exact match, with small edit-distance tolerances only for article-title and source. No parameter is fitted to the test set, so none of the reported accuracies are forced by construction. The GROBID and Crossref comparisons use external services, so the 'outperforming state-of-the-art methods' claim is not derived from the paper's own assumptions. The dataset construction does use an LLM-based extraction step (Llama-3.1-8B-Instruct) and an edit-distance similarity filter with a 0.75 threshold, and the authors explicitly state in the Discussion that they 'did not test the accuracy of this step.' This is a genuine threat to external validity and generalizability, but it is not circularity: the matching step does not define the accuracy metric in terms of the models' predictions, and the ground-truth labels remain external. The small-model inference that pass@64 accuracy predicts RLVR success relies on Yue et al. (2025), an external empirical finding rather than a self-citation, and is an extrapolation rather than a definitional equivalence. There is also an internal inconsistency between the abstract's '2^5 passes' and Section 2.2's '2^6 = 64' samples, but that is a reporting error, not circularity. No step in the paper reduces by construction to its inputs, so the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- similarity threshold for citation matching =
0.75
- edit-distance tolerance for article-title matches =
10
- edit-distance tolerance for source matches =
5
- decoding hyperparameters =
temperature 0.6/0.7, top_p 0.95/0.8, top_k 20
assumptions (5)
- domain assumption The JATS XML markup in the PKP and ORE corpora is correct ground truth for citation fields.
- domain assumption The Llama-3.1-8B-Instruct plaintext extraction and edit-distance matching produce a representative set of correct citation pairs.
- domain assumption Pass@64 success of Qwen3-0.6B indicates that reinforcement learning with verifiable reward will be effective, following Yue et al. (2025).
- domain assumption The selected 2,000 citations sampled from the matched pools are representative of the target deployment data.
- domain assumption GROBID and Crossref search are appropriate state-of-the-art baselines for comparison.
Cite this review
Pith. "Pith review of Citation Parsing and Analysis with Language Models." pith.science (2026). https://pith.science/paper/V4J62QZ6
@misc{pith2026250515948,
author = {Pith},
title = {Pith review of: Citation Parsing and Analysis with Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/V4J62QZ6}},
note = {Machine review of arXiv:2505.15948}
}
abstract
A key type of resource needed to address global inequalities in knowledge production and dissemination is a tool that can support journals in understanding how knowledge circulates. The absence of such a tool has resulted in comparatively less information about networks of knowledge sharing in the Global South. In turn, this gap authorizes the exclusion of researchers and scholars from the South in indexing services, reinforcing colonial arrangements that de-center and minoritize those scholars. In order to support citation network tracking on a global scale, we investigate the capacity of open-weight language models to mark up manuscript citations in an indexable format. We assembled a dataset of matched plaintext and annotated citations from preprints and published research papers. Then, we evaluated a number of open-weight language models on the annotation task. We find that, even out of the box, today's language models achieve high levels of accuracy on identifying the constituent components of each citation, outperforming state-of-the-art methods. Moreover, the smallest model we evaluated, Qwen3-0.6B, can parse all fields with high accuracy in $2^5$ passes, suggesting that post-training is likely to be effective in producing small, robust citation parsing models. Such a tool could greatly improve the fidelity of citation networks and thus meaningfully improve research indexing and discovery, as well as further metascientific research.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
An analysis of the suitability of OpenAlex for bibliometric analyses
Juan Pablo Alperin, Jason Portenoy, Kyle Demes, Vincent Larivière, and Stefanie Haustein. An analysis of the suitability of OpenAlex for bibliometric analyses. arXiv preprint arXiv:2404.17663, 2024
arXiv 2024
-
[3]
Scientific and scholarly journals in Latin America and the Caribbean
Ana Marıa Cetto, Octavio Alonso-Gamboa, PG Altbach, and D Teferra. Scientific and scholarly journals in Latin America and the Caribbean . Knowledge dissemination in Africa: The role of the scholarly journal, ed. Philip G. Altbach and Damtew Teferra, pp.\ 99--126, 1998
work page 1998
-
[4]
Wonjun Choi, Hwamook Yoon, Mihwan Hyun, Hyejin Lee, Jaewook Seol, Kangsan Dajeong Lee, Young Joon Yoon, and Hyesoo Kong. Building an annotated corpus for automatic metadata extraction from multilingual journal article references. PLOS ONE, 18, 2023. URL https://api.semanticscholar.org/CorpusID:256031950
work page 2023
-
[5]
Reference Coverage Analysis of OpenAlex compared to Web of Science and Scopus
Jack Culbert, Anne Hobert, Najko Jahn, Nick Haupka, Marion Schmidt, Paul Donner, and Philipp Mayr. Reference coverage analysis of openalex compared to web of science and Scopus . arXiv preprint arXiv:2401.16359, 2024
work page Pith review arXiv 2024
-
[6]
Uptime Institute Global Data Center Survey 2024 , jul 2024
Douglas Donnellan, Andy Lawrence, Daniel Bizo, et al. Uptime Institute Global Data Center Survey 2024 , jul 2024. URL https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2024.GlobalDataCenterSurvey.Report.pdf
work page 2024
-
[7]
Open Research Europe : Full article corpus
European Commission . Open Research Europe : Full article corpus. https://open-research-europe.ec.europa.eu, 2025. Complete collection of all articles published on Open Research Europe; accessed 19 March 2025
work page 2025
-
[8]
The XML Markup Evaluation Corpus , April 2016
Alex Garnett. The XML Markup Evaluation Corpus , April 2016. URL https://pkp.sfu.ca/2016/04/18/the-xml-markup-evaluation-corpus/
work page 2016
Show all 25 references
-
[9]
Ghost work: How to stop Silicon Valley from building a new global underclass
Mary L Gray and Siddharth Suri. Ghost work: How to stop Silicon Valley from building a new global underclass . Harper Business, 2019
2019
-
[10]
Ai is taking water from the desert
Karen Hao. Ai is taking water from the desert. The Atlantic, March 2024. URL https://www.theatlantic.com/technology/archive/2024/03/ai-water-climate-microsoft/677602/
2024
-
[11]
How are exclusively data journals indexed in major scholarly databases? An examination of four databases
Chenyue Jiao, Kai Li, and Zhichao Fang. How are exclusively data journals indexed in major scholarly databases? An examination of four databases. Scientific Data, 10 0 (1): 0 737, 2023. Publisher: Nature Publishing Group UK London
2023
-
[12]
An end-to-end pipeline for bibliography extraction from scientific articles
Bikash Joshi, Anthi Symeonidou, Syed Mazin Danish, and Floris Hermsen. An end-to-end pipeline for bibliography extraction from scientific articles. Proceedings of the Second Workshop on Information Extraction from Scientific Publications, 2023. URL https://api.semanticscholar....
2023
-
[13]
Recalibrating the scope of scholarly publishing: A modest step in a vast decolonization process
Saurabh Khanna, Jon Ball, Juan Pablo Alperin, and John Willinsky. Recalibrating the scope of scholarly publishing: A modest step in a vast decolonization process. Quantitative Science Studies, 3 0 (4): 0 912--930, 2022. Publisher: MIT Press One Broadway, 12th Floor, Cambridge,...
2022
-
[14]
Self-Refine: Iterative Refinement with Self-Feedback , 2023
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterativ...
2023 arXiv
-
[15]
MarkItDown: A Python Tool for Converting Files to Markdown , 2025
Microsoft Corporation . MarkItDown: A Python Tool for Converting Files to Markdown , 2025. URL https://github.com/microsoft/markitdown. Version 0.1.1
2025
-
[16]
The journal coverage of Web of Science and Scopus : a comparative analysis
Philippe Mongeon and Adèle Paul-Hus. The journal coverage of Web of Science and Scopus : a comparative analysis. Scientometrics, 106: 0 213--228, 2016. Publisher: Springer
2016
-
[17]
NVIDIA H200 Tensor Core GPU Datasheet
NVIDIA Corporation . NVIDIA H200 Tensor Core GPU Datasheet . Technical report, NVIDIA Corporation, jul 2024. URL https://www.megware.com/fileadmin/user_upload/LandingPage Preliminary specifications
2024
-
[18]
Synchromesh: Reliable code generation from pre-trained language models , 2022
Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models , 2022. URL https://arxiv.org/abs/2201.11227
2022 arXiv
-
[19]
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models . ArXiv, abs/2109.05093, 2021. URL https://www.aclanthology.org/2021.emnlp-main.779.pdf
2021 arXiv
-
[20]
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters , 2024
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters , 2024. URL https://arxiv.org/abs/2408.03314
2024 arXiv
-
[21]
Environmental Protection Agency
U.S. Environmental Protection Agency . Greenhouse Gas Equivalencies Calculator – Calculations and References . https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator-calculations-and-references, 2024. Accessed: 2025-05-20
2024
-
[22]
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? , 2025
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? , 2025. URL https://arxiv.org/abs/2504.13837
2025 arXiv
-
[23]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[24]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[25]
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.