Pith. sign in

REVIEW 2 major objections 14 references

Digging Up Citations: FOSSIL, a Dataset and Workflow for Reference Extraction in Law and the Humanities

T0 review · 2 major / 0 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new dataset of footnote citations and a specialized Grobid pipeline nearly double extraction quality over the default tool.

desk verdict FOSSIL supplies the first open multilingual dataset of footnote citations from law and humanities papers plus a Grobid tweak that lifts micro-F1 from 0.36 to 0.72 on their labels, but the labels are still incomplete. read the letter →

arxiv 2606.01109 v1 pith:JJ5F2A7C submitted 2026-05-31 cs.DL cs.CL

Luca Foppiano , Christian Boulanger This is my paper · ORCID
classification cs.DLcs.CL
keywords citationextractionfootnoteslawandhumanitiesdatasetGrobidreferenceparsingdigitallibrariesannotationworkflow
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces FOSSIL, an openly licensed multilingual dataset of 96 law and humanities articles containing more than 7,600 annotated footnote references. It supplies a collaborative annotation tool, a seven-annotator workflow, and a customized Grobid model trained to handle the interleaved commentary, cross-references, and stylistic variety typical of footnotes. End-to-end tests show the specialized pipeline raises micro-F1 from 0.36 to 0.72, mainly by lifting recall, while leaving room for improvement on mixed-content cases. Existing citation tools were built for end-of-document bibliographies in the sciences and perform poorly on the footnote-heavy style of law and humanities scholarship.

What carries the argument

FOSSIL dataset of annotated footnote-embedded references paired with the specialized Grobid pipeline for extracting citations from footnotes rather than end-of-document lists.

What would settle it

Re-running the end-to-end evaluation on a fresh collection of law and humanities articles whose footnotes were annotated independently of the original seven-annotator process and finding that the specialized pipeline no longer outperforms default Grobid.

Watch

Extended reading notes

Core claim

The authors create FOSSIL together with PDF-TEI Editor and a documented annotation workflow, then train a Grobid specialization for footnote-based citations; in end-to-end evaluation the specialized pipeline raises micro-F1 from 0.36 to 0.72 over default Grobid, driven by higher recall, although cross-references and mixed-content footnotes remain challenging.

Load-bearing premise

The seven-annotator workflow and its labels form a reliable gold standard that captures the range of footnote styles, languages, and mixed-content cases found in actual law and humanities articles.

Editorial extensions

If this is right

  • Automated processing of legal and humanities literature can now reach substantially higher recall for footnote references.
  • Openly licensed training data becomes available for further refinement of citation parsers aimed at SSH fields.
  • Remaining performance gaps on cross-references and mixed-content footnotes are now quantifiable and can guide targeted improvements.
  • The annotation workflow offers a reusable template for creating gold standards in additional languages or citation styles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same specialization approach could be tested on historical or non-English footnote corpora outside the current 96 articles.
  • Higher-quality footnote extraction might enable large-scale mapping of citation practices across entire law and humanities literatures.
  • The dataset could support studies that compare citation density or style differences between law, history, and philosophy subfields.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper introduces FOSSIL, an openly licensed multilingual dataset of 96 annotated scholarly articles from law and humanities containing over 7,600 footnote-embedded references. It describes the PDF-TEI Editor annotation tool, a documented seven-annotator workflow, and a Grobid specialization for footnote-based citations. In end-to-end evaluation the specialized pipeline achieves micro-F1 of 0.72 versus 0.36 for default Grobid, driven mainly by recall gains, while noting substantial remaining headroom for cross-references and mixed-content footnotes. The manuscript is presented as an extended abstract of work in progress, with segmentation, parsing, and cross-reference annotations still ongoing.

Significance. Creation of a new, openly licensed gold-standard resource for footnote citation extraction in SSH fields addresses a clear gap, as most existing tools target end-of-document bibliographies in the sciences. If the annotations prove reliable, the reported near-doubling of extraction quality provides concrete evidence that domain-specific specialization can yield substantial gains and supplies a reusable benchmark for future work on complex, multilingual, mixed-content footnotes.

major comments (2)
  1. [Abstract] Abstract: The headline result (micro-F1 rising from 0.36 to 0.72) is evaluated end-to-end against the FOSSIL labels, yet the abstract states that 'annotations of citations segmentation and parsing, and cross-reference resolution are ongoing.' Because the evaluation depends on these still-incomplete labels, the reported performance figure and the claim of nearly doubled quality cannot yet be treated as stable.
  2. [Abstract] Abstract: The seven-annotator workflow is presented as the foundation for the dataset and evaluation, but no inter-annotator agreement statistics or consistency metrics are supplied. Without these quantities it is impossible to determine whether the 0.72 micro-F1 reflects reliable reference boundaries or provisional labeling that may shift once the remaining annotation tasks are finished.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful review and for recognizing the value of FOSSIL as a resource for footnote citation extraction. We address the two major comments point by point below, with proposed revisions where appropriate.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The headline result (micro-F1 rising from 0.36 to 0.72) is evaluated end-to-end against the FOSSIL labels, yet the abstract states that 'annotations of citations segmentation and parsing, and cross-reference resolution are ongoing.' Because the evaluation depends on these still-incomplete labels, the reported performance figure and the claim of nearly doubled quality cannot yet be treated as stable.

    Authors: The reported evaluation measures reference boundary detection (the core extraction task) against the completed annotations of the 7,600+ footnote references. The ongoing work concerns only the subsequent tasks of intra-footnote segmentation, bibliographic field parsing, and cross-reference resolution; these are not required for the boundary-level evaluation presented. The 0.72 micro-F1 therefore rests on the stable reference-identification labels. We will revise the abstract to explicitly separate the completed annotation components from the ongoing ones and to qualify the performance claim accordingly. revision: yes

  2. Referee: [Abstract] Abstract: The seven-annotator workflow is presented as the foundation for the dataset and evaluation, but no inter-annotator agreement statistics or consistency metrics are supplied. Without these quantities it is impossible to determine whether the 0.72 micro-F1 reflects reliable reference boundaries or provisional labeling that may shift once the remaining annotation tasks are finished.

    Authors: We agree that quantitative inter-annotator agreement metrics would strengthen confidence in the labels. The documented workflow incorporated multiple rounds of joint review and adjudication, but formal IAA statistics were not computed for this extended abstract. We will expand the description of the workflow to detail the consistency measures employed and will include any available agreement figures or a clear statement of this limitation in the revised manuscript. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical dataset creation and held-out evaluation

full rationale

The paper introduces a new multilingual dataset of footnote citations (FOSSIL) together with an annotation workflow and a specialized Grobid model. Its central claim is an end-to-end micro-F1 improvement measured on annotations produced by that workflow. No equations, fitted parameters, or first-principles derivations appear; the reported scores are direct empirical measurements against the created labels rather than quantities defined by construction from the inputs. Self-citations are absent from the load-bearing steps. The contribution is therefore self-contained against external benchmarks (the held-out annotations) and receives the default non-circularity finding.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

The paper introduces no free parameters, new axioms, or invented entities. It relies on the standard assumption that manual annotation by domain-aware annotators produces usable gold labels for training and evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Digging Up Citations: FOSSIL, a Dataset and Workflow for Reference Extraction in Law and the Humanities." pith.science (2026). https://pith.science/paper/JJ5F2A7C

@misc{pith2026260601109,
  author       = {Pith},
  title        = {Pith review of: Digging Up Citations: FOSSIL, a Dataset and Workflow for Reference Extraction in Law and the Humanities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JJ5F2A7C}},
  note         = {Machine review of arXiv:2606.01109}
}
read the original abstract

Citation extraction tools are designed for the structured end-of-document bibliographies of the natural sciences, but law and humanities scholarship cites references primarily in footnotes, where bibliographic data is interleaved with commentary and cross-references and varies widely across languages and styles. To address the scarcity of suitable gold-standard resources, we present FOSSIL (Footnote-based Open-access SSH Scientific Instance Labels), an openly licensed multilingual dataset of 96 annotated scholarly articles containing over 7,600 footnote-embedded references, together with PDF-TEI Editor (a collaborative web annotation tool), a documented seven-annotator workflow, and a Grobid specialization for footnote-based citations. In end-to-end evaluation, the specialized pipeline nearly doubles extraction quality over default Grobid (micro-F1 from 0.36 to 0.72), driven largely by improved recall, while showing that substantial headroom remains for cross-references and mixed-content footnotes. This extended abstract presents work in progress; annotations of citations segmentation and parsing, and cross-reference resolution are ongoing.

Figures

Figures reproduced from arXiv: 2606.01109 by the authors.

Figure 1
Figure 1. End-to-end evaluation on author surname, journal title, and publica [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 11 canonical work pages

  1. [1]

    A., & Mayr, P

    Backes, T., Iurshina, A., Shahid, M. A., & Mayr, P. (2024). Comparing free ref- erence extraction pipelines.International Journal on Digital Libraries, 25(4), 841–853. https://doi.org/10.1007/s00799-024-00404-6

    work page doi:10.1007/s00799-024-00404-6 2024

  2. [2]

    (2024–present)

    Boulanger, C. (2024–present). Pdf-tei editor. https://github.com/mpilhlt/pdf- tei-editor/

    2024

  3. [3]

    Boulanger, C. (2026). The potential of LLMs for constructing a socio-legal knowledge graph [part-4]. In A. Simons, A. W¨ uthrich, M. Zichert, & G. Graßhoff (Eds.),Understanding science with large language models? potentials for the history, philosophy, and sociology of science. tran- script

    2026

  4. [4]

    (2022, June)

    Boulanger, C., & Iurshina, A. (2022, June). Extracting bibliographic references from footnotes with excite-docker. In T. Backes, A. Iurshina, & P. Mayr (Eds.),Proceedings of the workshop on understanding literature refer- ences in academic full text(pp. 26–33, Vol. 3220). CEUR. http://ceur- ws.org/Vol-3220/#paper3

    2022

  5. [5]

    (2022, February)

    Cioffi, A. (2022, February). Data for testing and evaluating references extraction and parsing tools. https://doi.org/10.5281/zenodo.6182066

    work page doi:10.5281/zenodo.6182066 2022

  6. [6]

    Cioffi, A., & Peroni, S. (2022). Structured references from pdf articles: Assess- ing the tools for bibliographic reference extraction and parsing.ArXiv. https://doi.org/10.48550/arXiv.2205.14677

    work page doi:10.48550/arxiv.2205.14677 2022

  7. [7]

    Colavizza, G., & Romanello, M. (2017). Annotated references in the historiogra- phy on venice: 19th–21st centuries.Journal of Open Humanities Data. https://doi.org/10.5334/johd.9

    work page doi:10.5334/johd.9 2017

  8. [8]

    Colavizza, G., & Romanello, M. (2019). Citation mining of humanities journals: The progress to date and the challenges ahead.Journal of European Periodical Studies,4(11). https://doi.org/10.21825/jeps.v4i1.10120 Grobid. (2008–present). https://github.com/grobidOrg/grobid

    work page doi:10.21825/jeps.v4i1.10120 2019

Show all 14 references
  1. [9]

    Hosseini, A., Ghavimi, B., Boukhers, Z., & Mayr, P. (2019). Excite – a toolchain to extract, match and publish open literature references.2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL), 432–433. https://doi. org/10.1109/JCDL.2019.00105

    2019 doi

  2. [10]

    (2025, February)

    Karkee, M., Raza, S., & Karkee, M. (2025, February). Comprehensive analysis of transparency and accessibility of chatgpt, deepseek, and other sota large language models. https://doi.org/10.20944/preprints202502.1608.v1 Llamore. (2025–present). https://github.com/mpilhlt/llamor...

    2025 doi

  3. [11]

    Pagnotta, O. (2024). Cex project - dataset and gold standard.Zenodo. https: //doi.org/10.5281/zenodo.10535653

    2024 doi

  4. [12]

    Sarin, P., & Alperin, J. P. (2025). Citation parsing and analysis with language models.ArXiv. https://doi.org/10.48550/arXiv.2505.15948

    2025 doi

  5. [13]

    Simons, A., Zichert, M., & W¨ uthrich, A. (2026). Large language models for history, philosophy, and sociology of science: Interpretive uses, method- ological challenges, and critical perspectives.Studies in History and 6 Philosophy of Science,117, 102151. https://doi.org/10.1...

    2026 doi

  6. [14]

    (2026, March)

    Zhu, Y., Colavizza, G., & Romanello, M. (2026, March). Benchmarking large language models on reference extraction and parsing in the social sci- ences and humanities [arXiv:2603.13651 [cs]]. https://doi.org/10.48550/ arXiv.2603.13651 7

    2026

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.