Pith. sign in

REVIEW 3 major objections 5 minor 9 references

Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Nearly a million public-domain books from Harvard Library's Google Books digitization — 983,004 volumes, about 242B tokens — are released with documented provenance, per-volume metadata, and cleaned OCR text.

desk verdict A genuinely useful 242B-token book corpus with strong provenance documentation; the rights claims are overstated — roughly a fifth of volumes are US-public-domain only, and the distributed dataset is noncommercial. read the letter →

arxiv 2506.08300 v1 pith:AUTDR27K submitted 2025-06-10 cs.CL cs.DL

Matteo Cargnelutti , Catherine Brobston , John Hess , Jack Cushman , Kristi Mukk , Aristana Scourtas , Kyle Courtney , Greg Leppert
show 3 more authors
Amanda Watson Martha Whitehead Jonathan Zittrain
This is my paper
classification cs.CLcs.DL
keywords publicdomaincorpuslargelanguagemodeltrainingdataOCRpost-processingHathiTrustrightsdatabaseGoogleBooksdigitizationmultilingualtextnear-duplicatedetectiondatasetprovenance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that Harvard Library's Google Books digitization can be converted into a documented, reusable public-domain training corpus, and reports doing so at record scale: 983,004 volumes totaling 242B tokens (o200k_base), drawn from a 1,075,899-volume collection spanning over 250 languages. The authors argue this matters because publicly available, high-quality training data is scarce and because most existing corpora are English-dominated web text with weak provenance chains. The release pairs each volume's raw OCR text with a post-processed version for the five most common languages and attaches bibliographic, source, and generated metadata designed to let users filter the collection for their own uses. The central promise, if correct, is that a documented multi-hundred-billion-token public-domain book corpus now exists as a building block for language-model research.

What carries the argument

The load-bearing mechanism is the barcode-keyed volume record: each Harvard barcode, prefixed with 'hvd', is matched to the HathiTrust API, and the returned rights status decides whether the volume enters the release. Around that gate the paper assembles a processing stack: a custom retrieval pipeline for the Google Return Interface (GRIN); a multilingual BERT classifier fine-tuned to assign each volume one of 20 Library of Congress top-level topics; a Simhash-based near-duplicate detector using 7-character shingles; and an OCR line-type classifier — a static embedding model distilled from LaBSE — that labels each OCR line as paragraph chunk, heading, page number, running head, separator, or noise, guiding a post-processing pass that reassembles sentences and removes page furniture.

What would settle it

Take a random sample of roughly 1,000 released barcodes, re-query the HathiTrust API, and compare the returned rights codes with the dataset's records; then independently verify the public-domain basis of a subsample of volumes dated after 1928, for example by checking for US government authorship or non-US publication. If a material share of post-1928 volumes have no valid public-domain basis, or if the barcode-to-hvd matching misaligns for a large fraction of the sample, the claim that this is a public-domain corpus would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the public-domain portion of Harvard Library's Google Books digitization — 983,004 volumes, 91.37% of the 1,075,899-volume collection, about 242B o200k_base tokens — has been retrieved, analyzed, documented, and released. Inclusion in the release is gated by a single external judgment: each volume's barcode is prefixed with 'hvd' and matched against the HathiTrust rights database, and only volumes returning pd, pdus, or cc-zero statuses are included. The released text comprises the original OCR output plus a post-processed version for English, German, French, Italian, and Spanish, together with per-volume topic labels from a fine-tuned classifier, text-level language distributions, near-duplicate flags, OCR-quality scores, and text statistics. The authors present the dataset not as a finished training set but as a substrate for community filtering and future iteration.

Load-bearing premise

The entire release depends on one external judgment: the HathiTrust rights database, matched to each volume through a simple barcode prefix, correctly labels everything it marks as public domain, so if those determinations or that match are wrong for a significant number of volumes, the dataset's central promise fails regardless of how well the processing pipeline works.

Editorial extensions

If this is right

  • Researchers gain a documented public-domain corpus of roughly 242B tokens, with 230 languages each contributing at least a million detected tokens.
  • The corpus is concentrated in the nineteenth and early twentieth centuries (about 60% of datable volumes fall between 1820 and 1920), so it fits historical research and long-context work better than contemporary-web-style pretraining.
  • The 41,366 volumes flagged as likely near-duplicates are listed in the dataset rather than removed, letting each user decide whether to exclude them from training runs.
  • Because post-processing removed only 0.97% of characters while raising tokenizability scores and lowering detected sentence length, the cleaned text offers a more machine-usable alternative to the raw OCR without replacing it.
  • At preliminary launch the text is released under a noncommercial click-through license with raw scan images withheld, so the practical reach of the corpus depends on how those access terms evolve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if other Google Books partner libraries adopt the same barcode-prefix matching and pipeline, the approach generalizes into a family of institutional public-domain corpora that could multiply the supply of documented book text for training.
  • Beyond the paper: a direct test of the post-processing claim would be to train matched small models on original versus post-processed text and compare perplexity or downstream-task scores; the paper reports proxy metrics but does not run that experiment.
  • Beyond the paper: because roughly 32% of volumes lack usable dates and the rights determinations are US-centric, users outside the United States should expect to perform their own rights review despite the authors' good-faith assessment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This technical report (arXiv:2506.08300) describes the construction of Institutional Books 1.0, a corpus of OCR-extracted text from Harvard Library's participation in the Google Books project. The authors document retrieval from GRIN, rights determination via HathiTrust's API, and a processing pipeline that includes language detection, topic classification, near-duplicate detection, OCR artifact analysis, text metrics, and a line-type classifier used to post-process OCR text for the five most frequent languages. The released dataset is claimed to contain 983,004 public-domain volumes, approximately 242B o200k_base tokens, with both original and post-processed OCR text plus extensive metadata. The paper also releases the processing pipeline and the fine-tuned classifier.

Significance. If the claims are properly qualified, this is a significant resource: a multilingual, multi-century book corpus at the 100B-token scale with unusually detailed provenance documentation, per-volume metadata, duplicate flags, OCR quality scores, and both original and post-processed text. The authors ship the pipeline and classifier, report collection-level statistics in extensive tables and appendices, and are explicit about several limitations. The main contributions, however, are currently presented with an unqualified 'public domain' label that is contradicted by the paper's own rights breakdown, and the usability improvements are supported by metrics that are partly constructed by the post-processing heuristics themselves.

major comments (3)
  1. [Abstract, §2, §5.3, Appendix J] The unqualified 'public domain' claim is not supported for the released dataset. Appendix J reports 196,171 volumes with HathiTrust status 'pdus' (18.23% of the collection, roughly 19.95% of the 983,510 rights-positive set), and §5.3 concedes that these volumes may be in the public domain in the United States but still subject to copyright or other rights protections in other jurisdictions. In addition, §1 states that the OCR text is initially released under a noncommercial license with a click-through, so the compiled dataset artifact is not public domain or CC0 even though many underlying volumes may be. Because 'a public dataset containing ... volumes ... for which there is no known copyright' is Contribution 2, the abstract and contribution list must be revised to say 'public domain in the United States' and to state the dataset's actual license and access terms.
  2. [§4.9.2, §4.8, Figures 14–17] The evidence that post-processing improves 'usability' is partly circular. The two reported improvements are the o200k_base tokenizability score and average detected sentence length, and the post-processing pipeline was explicitly designed to merge line fragments, remove hyphenations, and strip page numbers and running heads. These operations directly change token-per-word ratios and sentence segmentation results, so observing movement in those metrics is partly by construction. To support the claim that the post-processed text is easier to filter, read, and use in ML/NLP contexts, the paper needs an independent evaluation—for example, human readability annotation, downstream task performance, perplexity, or an external OCR-quality measure. As it stands, Figures 14–17 show that the heuristics move their own targets, not that the output is more usable.
  3. [§4.9.1, Appendix I] The line-type detector's reported 71% accuracy is measured against 10% of the same Phi-4-generated annotations used for training, with no human-validated held-out set described. Because the training labels were produced by Phi-4 and the benchmark is drawn from the same machine-generated distribution, the benchmark largely measures agreement with the labeling model rather than accuracy on real OCR lines. This matters because the line-type detections are a load-bearing component of the post-processing claim; independent, manually verified evaluation of the detector's predictions is needed before the post-processed output can be presented as an accuracy improvement.
minor comments (5)
  1. [§5.2] The percentage '91,41%' appears to be a typographical error and should read '91.41%'.
  2. [Appendix A] The field description for language_gen says 'ISO 693-3' in two places; this should be 'ISO 639-3'.
  3. [Appendix A] Table App. A4 is numbered twice: once for text_analysis_gen and once for identifiers_src; the second table should be renumbered, with subsequent tables adjusted accordingly.
  4. [§4.7] The spelling of the OCRoscope tool is inconsistent: both 'OCROscope' and 'OCRoscope' appear in the text; the paper should use a single spelling.
  5. [§4.2.2, Table 1] The token counts for openai/gpt-4 and microsoft/phi-4 are identical (275,637,216,999); this is plausible if phi-4 uses the same cl100k_base tokenizer, but the paper should state that explicitly to avoid the appearance of a copying error.

Circularity Check

2 steps flagged · score 4.0 of 10

Quality-evaluation metrics partly reduce to the post-processing heuristics; the dataset release itself is independent.

  1. self definitional [Section 4.9.2 (post-processing results), with metric defined in Section 4.8.1 and pipeline in Section 4.9.1]
    "This score, ranging from 0.0 to 100.0, indicates how efficiently o200k_base can encode this text. Specifically, it measures how close to 1.25 tokens per word the text is. / The first is that the average o200k_base "tokenizability" score (see Section 4.8.1) is consistently higher for the post-processed texts."

    The tokenizability metric is defined as closeness to 1.25 o200k_base tokens per word. The post-processing pipeline was explicitly designed to remove page numbers, running heads, hyphenations, and line breaks, and to reassemble paragraphs; the paper itself attributes the score rise to "the hyphenations removed... as well as the partial removal of page numbers and running heads." Removing non-word tokens and rejoining hyphen-split words directly lowers the tokens-per-word ratio toward the target, so the reported improvement is partly guaranteed by the construction of the metric. The average-sentence-length comparison is likewise computed on the same text segmentation that the heuristics change.

  2. fitted input called prediction [Section 4.9.1, "Line-level type detection" (training-data generation and benchmarking)]
    "235,168 OCR lines from 5,000 randomly sampled pages were annotated and 10% of these samples were set aside for benchmarking purposes. / The resulting model yielded a 71% accuracy rate against our benchmarking data, which matched our previously described target for coarse signal."

    The line-type classifier is fine-tuned on labels generated by microsoft/phi-4, and the "benchmarking data" is a 10% holdout of that same generation run. The reported 71% "accuracy rate" therefore measures agreement between the student classifier and Phi-4's annotations, not agreement with any human-verified ground truth. Because the model was fit precisely to reproduce that label source, high held-out accuracy is the expected consequence of fitting to the teacher's distribution. The number is presented as an accuracy figure, but the target of the prediction is the generated label set — the same input that defined the training objective — so it cannot independently validate the line-type detection.

full rationale

The central contribution — the release of OCR text and metadata for 983,004 HathiTrust-identified public-domain volumes — is a self-contained data-publication effort and does not reduce to a fitted prediction. The rights determination is an authoritative external lookup (HathiTrust), and the paper itself discloses the 'pdus' caveat and the noncommercial click-through license in Sections 1 and 5.3 and Appendix J; whether the 'public domain' claim is overbroad for roughly a fifth of the corpus is a correctness or legal-scope concern, not a circularity concern. The circularity is confined to the evaluation of the processing claims. Section 4.9 measures success with the same metrics ('tokenizability', average sentence length) that the line-reassembly and removal heuristics directly target, so the reported gains are partly by construction. Separately, Section 4.9 reports a 71% 'accuracy' for the line-type classifier against benchmarking data drawn from the same Phi-4-generated labels it was trained to reproduce, so that number validates self-consistency rather than external ground truth. The topic classifier's 97.8% figure is explicitly disclaimed by the paper in Section 4.5.2 as reproducing existing classification patterns, so it is not treated here as independent external validation. Because these issues affect secondary quality claims rather than the identity or scale of the released dataset, the overall circularity score is moderate.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central output is a dataset rather than a theory, so the ledger is dominated by hand-chosen processing thresholds and domain assumptions about external sources. No new physical or formal entities are introduced.

free parameters (5)
  • textless_token_threshold = 100 o200k_base tokens
    Hand-chosen threshold in Section 4.2.1 below which volumes are treated as textless; this changes the release count and token totals.
  • simhash_shingle_length = 7 characters
    Chosen through trial and error in Section 4.6.1 to minimize false positives in near-duplicate detection.
  • char_count_difference_threshold = 15%
    Heuristic in Section 4.6.1 used to reject false positive duplicate pairs with different continuous character counts.
  • language_token_exclusion_threshold = 1,000 o200k_base tokens
    Section 4.4.1: volume-level language statistics below 1,000 tokens were excluded as likely noise after manual review.
  • tokenizability_target = 1.25 tokens per word
    Section 4.8.1 defines the tokenizability score as closeness to 1.25 o200k_base tokens per word, a rough approximation of average compression.
assumptions (5)
  • domain assumption HathiTrust rights statuses (pd, pdus, cc-zero) are accurate and remain valid for the released volumes.
    Section 5.1 and 5.2: the public domain release is based exclusively on HathiTrust API records matched by the hvd barcode prefix; no independent legal verification is provided.
  • domain assumption Google Books OCR text is a sufficiently faithful rendering of the scanned volumes for the release to be usable as a text corpus.
    Sections 4.2 and 4.7: OCR quality scores and manual samples are used, but no systematic comparison against the original scans is reported, and raw images are withheld.
  • ad hoc to paper The Phi-4-generated line labels used to train the OCR line-type detector are reliable enough to serve as ground truth for benchmarking.
    Section 4.9.1 and Appendix I: 235,168 lines from 5,000 pages were labeled by Phi-4 at temperature 0; the 71% accuracy figure is measured against a 10% holdout of those same generated labels.
  • ad hoc to paper Tokenizability and average sentence length are meaningful proxies for the usability of OCR text in ML/NLP pipelines.
    Section 4.8 and 4.9.2: these metrics are the only quantitative evidence that post-processing improves text, and no downstream task or human evaluation is provided.
  • domain assumption franc's trigram-based language detection with the Python port's 414-language support yields accurate volume-level and text-level language distributions for this corpus.
    Section 4.4.1: the authors selected franc based on coverage and cost and validated manually in early rounds, but no quantitative accuracy benchmark on this corpus is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability." pith.science (2026). https://pith.science/paper/AUTDR27K

@misc{pith2026250608300,
  author       = {Pith},
  title        = {Pith review of: Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUTDR27K}},
  note         = {Machine review of arXiv:2506.08300}
}
read the original abstract

Large language models (LLMs) use data to learn about the world in order to produce meaningful correlations and predictions. As such, the nature, scale, quality, and diversity of the datasets used to train these models, or to support their work at inference time, have a direct impact on their quality. The rapid development and adoption of LLMs of varying quality has brought into focus the scarcity of publicly available, high-quality training data and revealed an urgent need to ground the stewardship of these datasets in sustainable practices with clear provenance chains. To that end, this technical report introduces Institutional Books 1.0, a large collection of public domain books originally digitized through Harvard Library's participation in the Google Books project, beginning in 2006. Working with Harvard Library, we extracted, analyzed, and processed these volumes into an extensively-documented dataset of historic texts. This analysis covers the entirety of Harvard Library's collection scanned as part of that project, originally spanning 1,075,899 volumes written in over 250 different languages for a total of approximately 250 billion tokens. As part of this initial release, the OCR-extracted text (original and post-processed) as well as the metadata (bibliographic, source, and generated) of the 983,004 volumes, or 242B tokens, identified as being in the public domain have been made available. This report describes this project's goals and methods as well as the results of the analyses we performed, all in service of making this historical collection more accessible and easier for humans and machines alike to filter, read and use.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 6 canonical work pages

  1. [1]

    As such—and because we anticipate this collection to have a variety of use cases—we have chosen to focus our efforts on non-destructive, collection-level deduplication

    nature of the underlying data and target model behaviors (Albalak et al., 2024). As such—and because we anticipate this collection to have a variety of use cases—we have chosen to focus our efforts on non-destructive, collection-level deduplication. Our goal with this experiment was therefore to identify near-duplicate OCR-extracted texts. For that purpos...

    work page 2024

  2. [8]

    From source

    [online] Association for Computational Linguistics, pp.16–27. doi:https://doi.org/10.18653/v1/2024.vardial-1.2. Manku, G.S., Jain, A. and Das Sarma, A. (2007). Detecting near-duplicates for web crawling. In: Proceedings of the 16th international conference on World Wide Web . [online] Association for Computing Machinery, pp.141–150. doi:https://doi.org/10...

    arXiv 2007

  3. [11]

    This text would likely need further processing (or re-OCR) before it could be used. Conversely, the very nature of some of these metrics, and in particular those concerned with lexical complexity, can be misleading when used across contexts and languages (Oh and Pellegrino, 2022). As such we have chosen not to use them to evaluate the underlying quality o...

    work page 2022

  4. [17]

    public domain,

    might be an additional clue pointing in that direction. For example, the text analysis data of the source texts shows that 3,001 volumes, primarily written in English, have an average detected sentence length of at least 500 characters. That figure decreased to 480 for the post-processed text. Further analysis is needed to confirm this trend and we encour...

    arXiv doi:10.48550/arxiv.2303.09540 2023

  5. [30]

    Swedigh” instead of “ Swedish

    Volume-level token counts < 1000 for a given language were discarded. Language Total detected tokens % of total detected tokens eng 105,918,942,360 43.83% deu 41,803,724,013 17.30% fra 33,852,308,477 14.01% ita 9,763,407,270 4.04% lat 7,718,749,717 3.19% spa 5,424,427,269 2.24% rus 4,956,088,535 2.05% ell 3,498,189,810 1.45% nld 3,006,044,500 1.24% heb 2,...

    work page 1914

  6. [158]

    tokenizability

    It ranged from 45 to 207 with an average of 76 for the post-processed texts. We hypothesize that these metrics, while varying in accuracy, indicate that post-processing these texts made them easier to work with in certain ML/NLP contexts. Because very little text was removed in the process (-0.97% characters based on compared continuous character counts),...

    work page 2016

  7. [2002]

    tokenizability

    and grouped together volumes with identical hashes. Overlap in these hashes suggested the presence of near duplicate OCR-extracted texts. While often used for web content (Manku, Jain and Das Sarma, 2007), research suggests that Simhash can effectively be used on the OCR-extracted text of books (Vladimir et al., 2015). Through trial and error, we identifi...

    work page 2007

  8. [2023]

    Available at: https://proceedings.neurips.cc/paper_files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html

    [online] pp.50358–50376. Available at: https://proceedings.neurips.cc/paper_files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html. Oh, Y.M. and Pellegrino, F. (2022). Towards robust complexity indices in linguistic typology A corpus-based assessment. Studies in Language. International Journal sponsored by the Foundation ‘Foundati...

    arXiv doi:10.48550/arxiv.2303.08774 2022

Show all 9 references
  1. [2024]

    Using the prompt described in Appendix I, we labeled individual lines from sample pages by presenting them to an 8-bit quantized version of Phi-4

    to generate the training dataset for this experiment. Using the prompt described in Appendix I, we labeled individual lines from sample pages by presenting them to an 8-bit quantized version of Phi-4. For each item to annotate, we provided the model with the current, previous,...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.