Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning XGLM-4.5B with QLoRA on a 600 MB UnifiedCrawl-Amharic corpus extracted from Common Crawl reduces perplexity from 35.6 to 19.6 and raises few-shot QA F1 from 8.0 to 9.9.

desk verdict Solid data-engineering contribution undermined by circular perplexity evaluation and overstated dataset sizes; worth reviewing, but the adaptation claim needs major rework. read the letter →

arxiv 2411.14343 v1 pith:YTMYTTIY submitted 2024-11-21 cs.CL cs.AI

Bethel Melesse Tessema (1) , Akhil Kedia (2) , Tae-Sun Chung (1) ((1) Ajou University , (2) Independent Researcher) This is my paper · ORCID
classification cs.CLcs.AI
keywords low-resourcelanguagesCommonCrawlmonolingualcorpusextractionQLoRAAmharicdatadeduplicationXGLMconsumer-hardwareNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a low-resource language can be given a usable language-model corpus by filtering the entire Common Crawl archive at low cost, and that fine-tuning a multilingual model on that corpus with lightweight adapters improves the language considerably. Applying the pipeline to Amharic produced a 600 MB deduplicated dataset, larger than the available Amharic sources (mC4, OSCAR, Wikipedia). QLoRA adaptation of XGLM-4.5B on this data reduced language-modeling perplexity from 35.6 to 19.6 and raised few-shot question-answering F1 from 8.0 to 9.9, within a day on a single consumer GPU. The same gains did not carry over to supervised training on the small AmQA set, which the authors attribute to that dataset's limited size and narrowness. If the pattern holds for other languages, the approach offers an affordable route to adapting open multilingual models to languages poorly served by pre-training data.

What carries the argument

The load-bearing mechanism is a streamed filtering pipeline. Common Crawl's columnar index carries a per-URL content-language annotation along with WARC record offset and length, so the system decides which pages to keep and downloads only those byte ranges rather than the whole archive or index. Around this core sit an in-memory analytical database for column filtering, an HTML-to-text extraction step, and exact-substring deduplication with a substring length threshold of 50 and a minimum document length of 100 characters. On the modeling side, QLoRA, quantized frozen model weights combined with trainable low-rank adapters, is what makes a 4.5B model trainable on a consumer GPU; the paper uses LoRA rank 2 and one epoch of training.

What would settle it

Take a random sample of UnifiedCrawl-Amharic documents and have native speakers, or an independent language identifier, label the language; if a substantial share turns out to be mislabeled or code-switched, the extraction's precision is in doubt. A quantitative version is to build a same-size Amharic corpus from the same WARC records re-annotated with an independent language identifier, fine-tune XGLM-4.5B with the same QLoRA settings, and compare perplexity and few-shot F1 against the paper's numbers.

Watch

Extended reading notes

Core claim

The central claim is that the largest public web archive, Common Crawl, can be mined for a low-resource language without downloading or storing the corpus. The pipeline keeps only URLs whose content-language annotation is exactly the target language, uses the index's byte offsets to fetch the relevant WARC records via HTTP range requests, extracts plain text from HTML, and applies exact-substring deduplication. For Amharic this yielded 600 MB after deduplication reduced the raw 4 GB extraction by 85%, a corpus larger than mC4 (1.2 GB), OSCAR (500 MB), and other prior sources. Fine-tuning XGLM-4.5B with QLoRA on this corpus for one epoch lowered perplexity from 35.6 to 19.6 and raised few-shot AmQA F1 from 8.0 to 9.9 and exact match from 1.3 to 2.3. The authors also report that QLoRA on the 4.5B model outperforms full fine-tuning of the 564M model at equal compute, while supervised downstream training on AmQA showed no 4.5B improvement.

Load-bearing premise

The pipeline's correctness rests on Common Crawl's per-URL language annotations being accurate enough to identify target-language pages, since the paper filters on the content-language column and reports no independent validation of the extracted Amharic text against a native corpus or human sample.

Editorial extensions

If this is right

  • For any language whose Common Crawl share is tiny, the released pipeline can produce a monolingual corpus in days on a consumer internet connection with under 10 GB of RAM, so the method should transfer beyond the seven languages tested.
  • Adapter-based fine-tuning of a 4.5B model on such a corpus is within consumer hardware (less than one GPU-day on an RTX 3070 or RTX 3090), which makes low-resource adaptation affordable in practice.
  • At equal compute, QLoRA on the 4.5B XGLM outperforms full fine-tuning of the 564M model on both perplexity and few-shot QA, favoring larger frozen models with adapters over smaller fully-trained models.
  • The reported gains appear in language modeling and few-shot prompting; supervised fine-tuning on the small AmQA set did not improve for the 4.5B model, so the adaptation value should be expected on tasks that draw on broad language knowledge rather than narrow supervised training.
  • Dataset quality controls (exact-substring deduplication and short-document filtering) cut the raw 4 GB Amharic extraction to 600 MB, and the retained text still produces the reported gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper reports language-modeling perplexity on the same UnifiedCrawl-Amharic corpus used for adaptation without stating a held-out split, part of the drop from 35.6 to 19.6 could reflect train/test overlap; evaluating on an independent native Amharic corpus would separate corpus familiarity from genuine improvement.
  • The single-language URL filter trades recall for precision; the paper's estimated maximum corpus sizes are three to six times larger when pages with minor other-language content are kept, so testing the larger, noisier version would show where the filter should sit.
  • All seven test languages have small Common Crawl shares, so a natural next test is a language with heavy code-switching or a script that the crawl's language annotation handles poorly, where annotation errors would be most visible.
  • Since the pipeline stores no intermediate corpora and fetches only byte ranges, it could be re-run on each new Common Crawl snapshot cheaply, yielding continuously updated low-resource corpora instead of one-time dumps.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces UnifiedCrawl, a pipeline for extracting monolingual text corpora for low-resource languages from the entire Common Crawl corpus using only consumer-grade hardware. The pipeline filters Common Crawl's columnar index with DuckDB, downloads only the relevant WARC records via HTTP range requests, extracts text with Trafilatura, and applies substring deduplication. The authors report datasets for seven low-resource languages, with Amharic as the main case study, and fine-tune XGLM-564M and XGLM-4.5B with QLoRA on UnifiedCrawl-Amharic. They report large perplexity reductions during fine-tuning and small gains in few-shot question answering on AmQA, concluding that the method significantly boosts performance on low-resource languages.

Significance. The dataset extraction contribution is potentially valuable: a reproducible, low-cost method for obtaining larger monolingual corpora for under-resourced languages, with released code, would be a useful community resource. However, the central model-adaptation claim is currently supported only by an in-distribution perplexity measurement and a small, statistically uncharacterized few-shot gain. If the evaluation is strengthened with held-out and externally validated metrics, the contribution could be solid; as written, the evidence for the headline claim is not yet convincing.

major comments (4)
  1. [Section 5.2.1, Table 4] The headline perplexity reduction in Table 4 (XGLM-4.5B from 35.6 to 19.6) is computed on UnifiedCrawl-Amharic itself: Section 4.3.1 states that perplexity is measured 'during fine-tuning' on the UnifiedCrawl-Amharic dataset, and Table 6's note confirms that LM PPL is reported on UnifiedCrawl-Amharic. Because the QLoRA model is optimized on that corpus while the baseline XGLM has never seen it, the comparison conflates memorization with generalization; a held-out split of UnifiedCrawl or an external Amharic language-modeling test set is required before claiming a '45% improvement' in language modeling.
  2. [Section 5.2.2, Table 5] The few-shot AmQA gains (F1 from 8.0 to 9.9, EM from 1.3 to 2.3) are reported without variance, confidence intervals, or the size of the test set, and the absolute gains are small. Since the few-shot examples are randomly selected from the AmQA train set and the prompt is subject to truncation, the results may be highly sensitive to prompt selection; the authors should report standard errors across multiple prompt draws and state the number of test instances.
  3. [Section 6.3, Table 8] The supervised AmQA results in Table 8 show no improvement for XGLM-4.5B after UnifiedCrawl fine-tuning (F1 35.0 vs. 34.7, EM 20.5 vs. 20.0), a fact the authors acknowledge in the text. This directly contradicts the abstract's claim that fine-tuning on UnifiedCrawl 'significantly boosts performance on the low-resource language' in a general sense; the manuscript should either restrict the claim to the few-shot and language-modeling settings or provide evidence that the adaptation transfers to supervised downstream tasks.
  4. [Section 3.1.1] The entire extraction pipeline relies on the 'content_language' column of the Common Crawl index to identify target-language documents, but the manuscript does not validate these labels against manually annotated samples or an external Amharic corpus. Since UnifiedCrawl is the only training signal for the adaptation experiments, reporting the precision and recall of the language filter (e.g., by manually inspecting a random sample of the final dataset or comparing against an existing Amharic corpus) is necessary to establish that the dataset actually consists of Amharic text.
minor comments (6)
  1. [Section 4.2.2] The authors state that they performed a grid search for the learning rate but do not report the chosen values; they should report the learning rate, batch size, sequence length, and number of training steps for both model sizes.
  2. [Section 5.1.1] Table 2 and the text give different dataset sizes: the text reports a 4GB raw extraction and a 600MB deduplicated UnifiedCrawl-Amharic, while Table 3 lists 4000MB for Amharic without specifying whether this is before or after deduplication; the table captions and the comparison with prior datasets should state the processing stage.
  3. [Section 4.3.4] The few-shot evaluation uses 10 random context-question-answer examples from the AmQA train set, but the prompt formatting, the random seed, and the number of test examples are not specified; this information is needed for reproducibility.
  4. [Table 1] The table header lists 'Sudanese (sun)' but the language is Sundanese (ISO 639-3 'sun'), which is spoken in Indonesia; the label should be corrected throughout.
  5. [Figure 3] The pipeline diagram labels the text extraction step 'Trailtura', which is a typo for Trafilatura; the diagram should also clarify the order of the deduplication and short-document filtering steps.
  6. [Section 6.3, Table 8] Reporting perplexity on the AmQA dataset is unusual because perplexity is a language-modeling metric and the text does not define the corpus or normalization used; please clarify what PPL is measured on and how it is computed.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline PPL improvement is measured on the QLoRA training corpus itself, so the main quantitative evidence is a fit-to-training-set effect; only the small few-shot AmQA gain is independent.

  1. fitted input called prediction [Section 4.3.1 (Language Modeling Evaluation) and Section 5.2.1, Table 4]
    "For evaluating the model's capabilities, we compare the perplexity of our model during fine-tuning using QLoRA (Dettmers et al., 2023) on our UnifiedCrawl-Amharic dataset and the original XGLM model, for both variants. ... We present the results in table 4 ... we analyze the model's perplexity (PPL) during the training process to measure its language modeling capability. ... LM PPL is reported on UnifiedCrawl-Amharic"

    The QLoRA adapters are trained to minimize next-token cross-entropy on UnifiedCrawl-Amharic, and Table 4 reports PPL on that same corpus, as the quoted text and Table 6 note state. The XGLM-4.5B drop from 35.6 to 19.6 therefore compares a model optimized on this exact corpus against a baseline that never saw it. This is a training-fit number, not a held-out prediction: it measures memorization/fit to the optimization target and is forced by the training objective, so it cannot by itself establish generalizable language modeling improvement.

full rationale

The paper's central claim of 'significantly boosts performance' relies on the perplexity drop in Table 4, and that perplexity is computed on UnifiedCrawl-Amharic, which is precisely the corpus used for QLoRA fine-tuning (Sections 4.3.1 and 5.2.1; Table 6 note). This is the fitted-input-called-prediction pattern: the model was optimized to reduce exactly this cross-entropy, so the reported improvement largely reflects fit to the training set rather than generalization. The few-shot AmQA results (F1 8.0 to 9.9, EM 1.3 to 2.3) are independent, held-out-style evidence and are not circular, although they are small and reported without variance or test-set size. The supervised AmQA comparison in Table 8 explicitly shows no gain for the 4.5B model (F1 35.0 vs 34.7), which the paper acknowledges. The dataset extraction and size comparison are self-contained and non-circular. Overall, one load-bearing evaluation reduces to a fit by construction, but the paper retains independent few-shot evidence and a dataset contribution, so the score is 6 rather than higher.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on domain assumptions about Common Crawl annotation reliability, QLoRA's fidelity to full fine-tuning, and deduplication/generalization. The empirical free parameters are hand-set thresholds and hyperparameters that affect final corpus size and model scores. No invented entities are introduced; UnifiedCrawl is a dataset and pipeline artifact rather than a new physical or ontological entity.

free parameters (5)
  • Deduplication substring length = 50 characters
    Chosen arbitrarily following (Penedo et al., 2023); affects final dataset size and quality.
  • Minimum document length = 100 characters
    Chosen arbitrarily following (Penedo et al., 2023); documents shorter than this are discarded.
  • LoRA rank = 2
    Chosen based on prior work; affects adapter capacity and model performance.
  • Learning rate (Amharic fine-tuning) = not reported (grid-searched)
    Grid-searched per Section 4.2.2, exact values omitted; affects final results.
  • Few-shot example count = 10
    Chosen because longer prompts are truncated; affects F1/EM scores.
assumptions (4)
  • domain assumption Common Crawl's URL language annotations are reliable proxies for document language.
    The entire pipeline selects documents by the CC index 'content_language' field; annotation errors directly propagate into dataset purity (Section 3.1.1).
  • domain assumption QLoRA fine-tuning preserves most of the quality of full fine-tuning on this data.
    The paper adopts QLoRA based on cited literature; if QLoRA underperforms on low-resource data, the model adaptation results would differ (Section 2.5).
  • domain assumption Exact substring deduplication improves model generalization in this setting.
    Adopted from Lee et al. without ablating on this data; it shapes the final dataset and could affect both size and training behavior (Section 3.1.4).
  • domain assumption The AmQA dataset is a meaningful benchmark for Amharic question answering.
    The paper evaluates on AmQA but does not validate its quality or coverage; results on this dataset are used as evidence for downstream performance (Section 4.3.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages." pith.science (2026). https://pith.science/paper/YTMYTTIY

@misc{pith2026241114343,
  author       = {Pith},
  title        = {Pith review of: UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YTMYTTIY}},
  note         = {Machine review of arXiv:2411.14343}
}
read the original abstract

Large language models (LLMs) under-perform on low-resource languages due to limited training data. We present a method to efficiently collect text data for low-resource languages from the entire Common Crawl corpus. Our approach, UnifiedCrawl, filters and extracts common crawl using minimal compute resources, yielding mono-lingual datasets much larger than previously available sources. We demonstrate that leveraging this data to fine-tuning multilingual LLMs via efficient adapter methods (QLoRA) significantly boosts performance on the low-resource language, while minimizing VRAM usage. Our experiments show large improvements in language modeling perplexity and an increase in few-shot prompting scores. Our work and released source code provide an affordable approach to improve LLMs for low-resource languages using consumer hardware. Our source code is available here at https://github.com/bethelmelesse/unifiedcrawl.

Figures

Figures reproduced from arXiv: 2411.14343 by the authors.

Figure 1
Figure 1. High-Level Overview of our Method 1.2 Proposed Method To address the aforementioned challenges, we present a novel approach aimed at overcoming data scarcity for low-resource languages, and leverage efficient methods for training LLMs on low-cost hardware. Our proposed method involves the devel￾opment of an efficient and cost-effective data collection strategy to extract comprehensive textual content specific to a g… view at source ↗
Figure 2
Figure 2. Our Dataset is much Larger than all Prior Works on the collected dataset. This innovative tech￾nique facilitates the utilization of exception￾ally large models on consumer-grade GPUs, thereby enhancing accessibility and affordabil￾ity in training [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. UnifiedCrawl: Data Extraction Framework 3.1 Data Collection Framework The raw text data for low-resource languages is gathered from the common crawl dataset. The common crawl dataset is extremely large, at approximately 100 TeraBytes per crawl archive, with multiple archives per year1 . Due to its sheer size, it is difficult to directly down￾load raw text dataset from the corpus. In this subsection, we propose an ef… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of Languages in Common Crawl except English A.2 Distribution of Languages in Common Crawl after Top 60 cymtat tgk sompus fry cos zul xho mri bih sna ina ile lug crs run zha 0.000% 0.002% 0.004% 0.006% 0.008% 0.010% 0.012% 0.014% Languages -> Percentage of …
Figure 5
Figure 5. Figure 5: Distribution of Languages in Common Crawl after Top 60 B Overview of Multilingual LLMs See table 9 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluating Binary Decision Biases in Large Language Models: Implications for Fair Agent-Based Financial Simulations

    cs.LG 2025-01 conditional novelty 5.0 of 10

    GPT models asked to make random binary choices show large, version-specific biases; only GPT-4o-Mini came close to a 50/50 split in one-shot tests.

Reference graph

Works this paper leans on

13 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [4]

    In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 901– 910, Marseille, France

    Deduplication of scholarly documents using locality sensitive hashing and word em- beddings. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 901– 910, Marseille, France. European Language Re- sources Association. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding- enhanced bert with disenta...

    work page 2021

  2. [7]

    ArXiv preprint, abs/2307.10485

    Fingpt: Democratizing internet-scale data for financial large language models. ArXiv preprint, abs/2307.10485. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilin- gual denoising pre-training for neural machine translation. Transactions of the Association for Computational Ling...

    arXiv 2020

  3. [8]

    ArXiv preprint, abs/2309.09958

    An empirical study of scaling instruct- tuned large multimodal models. ArXiv preprint, abs/2309.09958. Haoran Luo, Haihong E, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, and Wei Lin

    arXiv

  4. [9]

    ArXiv preprint, abs/2310.08975

    Chatkbqa: A generate-then-retrieve frame- work for knowledge base question answering with fine-tuned large language models. ArXiv preprint, abs/2310.08975. Rohin Manvi, Samar Khanna, Gengchen Mai, Mar- shall Burke, David B. Lobell, and Stefano Ermon

    arXiv

  5. [10]

    ArXiv preprint , abs/2310.06213

    Geollm: Extracting geospatial knowledge from large language models. ArXiv preprint , abs/2310.06213. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, A...

    arXiv

  6. [11]

    In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991– 16111, Toronto, Canada

    Crosslingual generalization through mul- titask finetuning. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991– 16111, Toronto, Canada. Association for Compu- tational Linguistics. OpenAI. 2022. Introducing ChatGPT - openai.com. https://openai.com/blog/chatgpt. [Accessed 15-12-...

    arXiv 2022

  7. [12]

    In Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 4003–4012, Mar- seille, France

    CCNet: Extracting high quality mono- lingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference , pages 4003–4012, Mar- seille, France. European Language Resources Association. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf...

    work page 2020

  8. [13]

    In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11682–11703, Toronto, Canada

    BLOOM+1: Adding language support to BLOOM for zero-shot prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11682–11703, Toronto, Canada. Associa- tion for Computational Linguistics. A Distribution of Languages in Common Crawl A.1 Distribution of Languages in Common Crawl exc...

    work page 2019

Show all 13 references
  1. [2008]

    Internet-Draft draft-kunze-warc-00, Internet En- gineering Task Force

    The WARC File Format (Version 0.16). Internet-Draft draft-kunze-warc-00, Internet En- gineering Task Force. Work in Progress. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. 2022. Deduplicat- ing training...

    2022

  2. [2014]

    RFC 7233

    Hypertext Transfer Protocol (HTTP/1.1): Range Requests. RFC 7233. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: accurate quantiza- tion for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations...

    2023

  3. [2020]

    In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 8440–8451, Online

    Unsupervised cross-lingual representa- tion learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 8440–8451, Online. Asso- ciation for Computational Linguistics. Warcio Contributors. 2017. GitHub - we- brecorder/...

    2017

  4. [2022]

    An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Sys- tems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. Edward J. Hu,...

    2022

  5. [2023]

    Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1–240:113. CommonCrawl. 2007. Common Crawl - Open Repository of Web Crawl Data - common- crawl.org. https://commoncrawl.org/. [Ac- cessed 15-12-2023]. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, ...

    2007

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.