{"id":"36d3c4d6-f51e-473e-a08c-881350edc805","arxiv_id":"2411.14343","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Filtering Common Crawl through a database-driven index scan and range-request WARC downloads yields large low-resource-language corpora, and QLoRA fine-tuning on the Amharic subset reduces perplexity and slightly improves few-shot QA for XGLM.","lead":"UnifiedCrawl is a tool that pulls text for seven low-resource languages from the entire Common Crawl archive using a regular computer, producing corpora that are often larger than older datasets. The paper also shows that fine-tuning a multilingual model on the Amharic version with QLoRA cuts perplexity and nudges up few-shot question answering scores, making low-resource LLM adaptation more affordable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Perplexity win may be an artifact of evaluating on the fine-tuning corpus; held-out PPL and supervised QA do not support 'significantly boosts performance'.","rationale":"Reading the paper in good faith, the contributions are an efficient Common Crawl extraction pipeline, a released codebase, and a QLoRA-based adaptation recipe on consumer hardware. The code release and the resource-conscious engineering are genuine and likely useful for low-resource language data collection. The central claim, however, is that fine-tuning on UnifiedCrawl 'significantly boosts performance on the low-resource language.' The strongest evidence for that claim is the perplexity reduction in Table 4, but the paper's own evaluation description indicates this perplexity is computed on the training corpus, not on held-out Amharic text. That makes the improvement expected from fitting the training data and does not demonstrate better language modeling on unseen Amharic. The few-shot AmQA gain is small and unreplicated, and the supervised AmQA result is null for the 4.5B model, which Section 6.3 explicitly acknowledges. These observations do not invalidate the data-engineering contribution, but they do mean the headline performance claim is only as strong as a held-out evaluation would make it. The reader's weakest-assumption choice, Common Crawl language-label accuracy, is a real domain risk, but it is secondary: if the model showed clear held-out gains, noisy labels would not by themselves refute the pipeline's value, and if held-out gains vanish, the pipeline's contribution to the performance claim is not established regardless of label quality. The reader's rationale does mention the training-data perplexity issue, so we partially agree. A concrete held-out PPL test, together with variance-aware few-shot evaluation, would settle whether the central claim survives. Given the real engineering contribution and the absence of evidence of any deliberate distortion, CONDITIONAL remains the appropriate verdict, pending that held-out evaluation and release of the final corpora.","tokens_in":18971,"tokens_out":3888,"duration_ms":40116,"concrete_test":"Re-run the XGLM-4.5B comparison with a clean held-out split: randomly hold out at least 10% of UnifiedCrawl-Amharic documents before QLoRA training, compute PPL on that held-out set for both the original XGLM-4.5B and the adapted model, and additionally report AmQA few-shot F1/EM with standard errors over at least 3 random prompt seeds. If held-out PPL does not improve substantially (e.g., does not drop well below the original model's held-out PPL) and few-shot gains remain within noise, the central 'significant boost' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative evidence for the headline claim is the perplexity drop in Table 4 (XGLM-4.5B: 35.6 to 19.6). However, Section 4.3.1 says the authors compare perplexity 'during fine-tuning' on UnifiedCrawl-Amharic, and Section 5.2.1 evaluates PPL 'during the training process'; Table 6's note also states that LM PPL is reported on UnifiedCrawl-Amharic. No held-out split is described. Measuring PPL on the same corpus used for QLoRA training measures fit and memorization, not language ability, while the baseline XGLM is evaluated on text it never trained on. This asymmetry can by itself produce a large PPL drop, so Table 4 does not establish that the model learned generalizable Amharic. The only held-out-style evidence, the few-shot AmQA scores in Table 5, is small (F1 8.0 to 9.9, EM 1.3 to 2.3), reported without variance or test-set size, and the supervised AmQA evaluation in Table 8 shows no gain for the 4.5B model (F1 35.0 vs. 34.7). The 'significantly boosts performance' claim therefore rests almost entirely on an evaluation protocol that cannot distinguish memorization from generalization. This is more load-bearing than the Common Crawl annotation-noise concern: even with perfect language labels, the current evidence would not support the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UnifiedCrawl, a pipeline for extracting monolingual text corpora for low-resource languages from the entire Common Crawl corpus using only consumer-grade hardware. The pipeline filters Common Crawl's columnar index with DuckDB, downloads only the relevant WARC records via HTTP range requests, extracts text with Trafilatura, and applies substring deduplication. The authors report datasets for seven low-resource languages, with Amharic as the main case study, and fine-tune XGLM-564M and XGLM-4.5B with QLoRA on UnifiedCrawl-Amharic. They report large perplexity reductions during fine-tuning and small gains in few-shot question answering on AmQA, concluding that the method significantly boosts performance on low-resource languages.","tokens_in":19231,"tokens_out":3725,"duration_ms":32350,"significance":"The dataset extraction contribution is potentially valuable: a reproducible, low-cost method for obtaining larger monolingual corpora for under-resourced languages, with released code, would be a useful community resource. However, the central model-adaptation claim is currently supported only by an in-distribution perplexity measurement and a small, statistically uncharacterized few-shot gain. If the evaluation is strengthened with held-out and externally validated metrics, the contribution could be solid; as written, the evidence for the headline claim is not yet convincing.","major_comments":[{"comment":"The headline perplexity reduction in Table 4 (XGLM-4.5B from 35.6 to 19.6) is computed on UnifiedCrawl-Amharic itself: Section 4.3.1 states that perplexity is measured 'during fine-tuning' on the UnifiedCrawl-Amharic dataset, and Table 6's note confirms that LM PPL is reported on UnifiedCrawl-Amharic. Because the QLoRA model is optimized on that corpus while the baseline XGLM has never seen it, the comparison conflates memorization with generalization; a held-out split of UnifiedCrawl or an external Amharic language-modeling test set is required before claiming a '45% improvement' in language modeling.","section":"Section 5.2.1, Table 4"},{"comment":"The few-shot AmQA gains (F1 from 8.0 to 9.9, EM from 1.3 to 2.3) are reported without variance, confidence intervals, or the size of the test set, and the absolute gains are small. Since the few-shot examples are randomly selected from the AmQA train set and the prompt is subject to truncation, the results may be highly sensitive to prompt selection; the authors should report standard errors across multiple prompt draws and state the number of test instances.","section":"Section 5.2.2, Table 5"},{"comment":"The supervised AmQA results in Table 8 show no improvement for XGLM-4.5B after UnifiedCrawl fine-tuning (F1 35.0 vs. 34.7, EM 20.5 vs. 20.0), a fact the authors acknowledge in the text. This directly contradicts the abstract's claim that fine-tuning on UnifiedCrawl 'significantly boosts performance on the low-resource language' in a general sense; the manuscript should either restrict the claim to the few-shot and language-modeling settings or provide evidence that the adaptation transfers to supervised downstream tasks.","section":"Section 6.3, Table 8"},{"comment":"The entire extraction pipeline relies on the 'content_language' column of the Common Crawl index to identify target-language documents, but the manuscript does not validate these labels against manually annotated samples or an external Amharic corpus. Since UnifiedCrawl is the only training signal for the adaptation experiments, reporting the precision and recall of the language filter (e.g., by manually inspecting a random sample of the final dataset or comparing against an existing Amharic corpus) is necessary to establish that the dataset actually consists of Amharic text.","section":"Section 3.1.1"}],"minor_comments":[{"comment":"The authors state that they performed a grid search for the learning rate but do not report the chosen values; they should report the learning rate, batch size, sequence length, and number of training steps for both model sizes.","section":"Section 4.2.2"},{"comment":"Table 2 and the text give different dataset sizes: the text reports a 4GB raw extraction and a 600MB deduplicated UnifiedCrawl-Amharic, while Table 3 lists 4000MB for Amharic without specifying whether this is before or after deduplication; the table captions and the comparison with prior datasets should state the processing stage.","section":"Section 5.1.1"},{"comment":"The few-shot evaluation uses 10 random context-question-answer examples from the AmQA train set, but the prompt formatting, the random seed, and the number of test examples are not specified; this information is needed for reproducibility.","section":"Section 4.3.4"},{"comment":"The table header lists 'Sudanese (sun)' but the language is Sundanese (ISO 639-3 'sun'), which is spoken in Indonesia; the label should be corrected throughout.","section":"Table 1"},{"comment":"The pipeline diagram labels the text extraction step 'Trailtura', which is a typo for Trafilatura; the diagram should also clarify the order of the deduplication and short-document filtering steps.","section":"Figure 3"},{"comment":"Reporting perplexity on the AmQA dataset is unusual because perplexity is a language-modeling metric and the text does not define the corpus or normalization used; please clarify what PPL is measured on and how it is computed.","section":"Section 6.3, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is well founded: the principal evidence for the adaptation claim is an in-distribution perplexity measurement on the fine-tuning corpus, and the only held-out-style evaluation (few-shot QA) shows a small, unreplicated gain. The dataset construction contribution is plausible and reproducible, so I recommend a major revision rather than rejection. The authors should be asked to add held-out language-modeling evaluation, characterize the variance of the few-shot results, and temper the abstract's claims to match the supervised-transfer evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful part of this paper is the data-engineering pipeline. The authors show that by filtering Common Crawl's columnar index with DuckDB, pulling only targeted WARC records via HTTP range requests, extracting text in memory with Trafilatura, and running exact-substring dedup across 43 archives, you can build a low-resource-language corpus on a consumer machine in days. That is real, clearly described, and the code release makes it checkable. The dataset-size claims versus OSCAR/mC4/CC-100 are striking, and the pipeline is language-agnostic in principle. I buy that part.\n\nThe soft spot is the model-adaptation evaluation, and the stress-test note is right: the headline perplexity drop (XGLM-4.5B 35.6 -> 19.6) is measured \"during the training process\" on UnifiedCrawl-Amharic itself, i.e., training-set perplexity. That measures fit, not generalization. The baseline XGLM is evaluated on text it never saw, so the comparison is apples-to-oranges and the 45% improvement is not evidence of generalizable language ability. The few-shot AmQA gains (F1 8.0 -> 9.9, EM 1.3 -> 2.3) are tiny, reported without variance or test-set size, and their own supervised QA table shows no gain for the 4.5B model. So the abstract's \"significantly boosts performance\" is not supported by the evidence as presented. That is load-bearing: the paper's second contribution is the model adaptation, and that claim needs held-out perplexity on a separate corpus, plus a few-shot evaluation with proper setup and error bars.\n\nMinor but real: dataset sizes are reported inconsistently. Section 5.1.1 says the final deduplicated Amharic corpus is 600MB, but Tables 2 and 3 headline 4GB. That overstates the resource by roughly 7x. Also, the pipeline relies on Common Crawl's content-language annotations with no validation against an external corpus; for very low-resource languages those labels are noisy, and the authors don't quantify the impact.\n\nVerdict: the extraction method is a solid engineering contribution worth peer review, but the evaluation needs major revision before the adaptation claims can be taken at face value. I'd send it to a referee with instructions to focus on the evaluation protocol.","headline":"Solid data-engineering contribution undermined by circular perplexity evaluation and overstated dataset sizes; worth reviewing, but the adaptation claim needs major rework.","tokens_in":19839,"tokens_out":1926,"would_cite":true,"duration_ms":17326,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning XGLM-4.5B with QLoRA on a 600 MB UnifiedCrawl-Amharic corpus extracted from Common Crawl reduces perplexity from 35.6 to 19.6 and raises few-shot QA F1 from 8.0 to 9.9.","keywords":["low-resource languages","Common Crawl","monolingual corpus extraction","QLoRA","Amharic","data deduplication","XGLM","consumer-hardware NLP"],"falsifier":"Take a random sample of UnifiedCrawl-Amharic documents and have native speakers, or an independent language identifier, label the language; if a substantial share turns out to be mislabeled or code-switched, the extraction's precision is in doubt. A quantitative version is to build a same-size Amharic corpus from the same WARC records re-annotated with an independent language identifier, fine-tune XGLM-4.5B with the same QLoRA settings, and compare perplexity and few-shot F1 against the paper's numbers.","tokens_in":18738,"feed_emoji":"🌍","tokens_out":9420,"duration_ms":77000,"temperature":0.7,"pith_summary":"The paper claims that a low-resource language can be given a usable language-model corpus by filtering the entire Common Crawl archive at low cost, and that fine-tuning a multilingual model on that corpus with lightweight adapters improves the language considerably. Applying the pipeline to Amharic produced a 600 MB deduplicated dataset, larger than the available Amharic sources (mC4, OSCAR, Wikipedia). QLoRA adaptation of XGLM-4.5B on this data reduced language-modeling perplexity from 35.6 to 19.6 and raised few-shot question-answering F1 from 8.0 to 9.9, within a day on a single consumer GPU. The same gains did not carry over to supervised training on the small AmQA set, which the authors attribute to that dataset's limited size and narrowness. If the pattern holds for other languages, the approach offers an affordable route to adapting open multilingual models to languages poorly served by pre-training data.","feed_headline":"Crawled Amharic text cuts XGLM perplexity from 35.6 to 19.6","feed_subtitle":"A 600 MB corpus filtered from Common Crawl, plus QLoRA adapters, lifts few-shot QA F1 from 8.0 to 9.9 on one consumer GPU.","key_machinery":"The load-bearing mechanism is a streamed filtering pipeline. Common Crawl's columnar index carries a per-URL content-language annotation along with WARC record offset and length, so the system decides which pages to keep and downloads only those byte ranges rather than the whole archive or index. Around this core sit an in-memory analytical database for column filtering, an HTML-to-text extraction step, and exact-substring deduplication with a substring length threshold of 50 and a minimum document length of 100 characters. On the modeling side, QLoRA, quantized frozen model weights combined with trainable low-rank adapters, is what makes a 4.5B model trainable on a consumer GPU; the paper uses LoRA rank 2 and one epoch of training.","core_discovery":"The central claim is that the largest public web archive, Common Crawl, can be mined for a low-resource language without downloading or storing the corpus. The pipeline keeps only URLs whose content-language annotation is exactly the target language, uses the index's byte offsets to fetch the relevant WARC records via HTTP range requests, extracts plain text from HTML, and applies exact-substring deduplication. For Amharic this yielded 600 MB after deduplication reduced the raw 4 GB extraction by 85%, a corpus larger than mC4 (1.2 GB), OSCAR (500 MB), and other prior sources. Fine-tuning XGLM-4.5B with QLoRA on this corpus for one epoch lowered perplexity from 35.6 to 19.6 and raised few-shot AmQA F1 from 8.0 to 9.9 and exact match from 1.3 to 2.3. The authors also report that QLoRA on the 4.5B model outperforms full fine-tuning of the 564M model at equal compute, while supervised downstream training on AmQA showed no 4.5B improvement.","pith_inferences":["Because the paper reports language-modeling perplexity on the same UnifiedCrawl-Amharic corpus used for adaptation without stating a held-out split, part of the drop from 35.6 to 19.6 could reflect train/test overlap; evaluating on an independent native Amharic corpus would separate corpus familiarity from genuine improvement.","The single-language URL filter trades recall for precision; the paper's estimated maximum corpus sizes are three to six times larger when pages with minor other-language content are kept, so testing the larger, noisier version would show where the filter should sit.","All seven test languages have small Common Crawl shares, so a natural next test is a language with heavy code-switching or a script that the crawl's language annotation handles poorly, where annotation errors would be most visible.","Since the pipeline stores no intermediate corpora and fetches only byte ranges, it could be re-run on each new Common Crawl snapshot cheaply, yielding continuously updated low-resource corpora instead of one-time dumps."],"forward_implications":["For any language whose Common Crawl share is tiny, the released pipeline can produce a monolingual corpus in days on a consumer internet connection with under 10 GB of RAM, so the method should transfer beyond the seven languages tested.","Adapter-based fine-tuning of a 4.5B model on such a corpus is within consumer hardware (less than one GPU-day on an RTX 3070 or RTX 3090), which makes low-resource adaptation affordable in practice.","At equal compute, QLoRA on the 4.5B XGLM outperforms full fine-tuning of the 564M model on both perplexity and few-shot QA, favoring larger frozen models with adapters over smaller fully-trained models.","The reported gains appear in language modeling and few-shot prompting; supervised fine-tuning on the small AmQA set did not improve for the 4.5B model, so the adaptation value should be expected on tasks that draw on broad language knowledge rather than narrow supervised training.","Dataset quality controls (exact-substring deduplication and short-document filtering) cut the raw 4 GB Amharic extraction to 600 MB, and the retained text still produces the reported gains."],"supporting_citations":[{"why":"Supplies the entire web archive and the per-URL content-language column that the entire filtering pipeline depends on.","marker":"CommonCrawl (2007)"},{"why":"Provides the in-memory analytical database used to filter the large columnar index without bulk downloads.","marker":"Raasveldt and Mühleisen (2019)"},{"why":"Performs HTML-to-text extraction from the downloaded WARC records.","marker":"Barbaresi (2021)"},{"why":"Supplies the exact-substring deduplication method that cuts dataset size and removes repeated boilerplate.","marker":"Lee et al. (2022)"},{"why":"Defines QLoRA, the quantized low-rank adapter method used to fine-tune the 4.5B model on consumer hardware.","marker":"Dettmers et al. (2023)"},{"why":"Provides the XGLM multilingual model used as the adaptation target and baseline.","marker":"Lin et al. (2022)"},{"why":"Supplies the AmQA dataset used for downstream few-shot and supervised question-answering evaluation.","marker":"Abedissa et al. (2023)"}],"fun_headline_variants":["600MB Amharic from Common Crawl, QLoRA fine-tune cuts perplexity 35.6→19.6","Amharic QLoRA on 4.5B XGLM: Common Crawl mining cuts perplexity 35.6→19.6","Cheap Amharic: range-request Common Crawl URLs, QLoRA lifts QA F1 to 9.9","Mining Common Crawl without download: 600MB Amharic, QLoRA lifts XGLM few-shot QA","600MB Amharic mined from Common Crawl cuts XGLM perplexity 35.6 to 19.6"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline's correctness rests on Common Crawl's per-URL language annotations being accurate enough to identify target-language pages, since the paper filters on the content-language column and reports no independent validation of the extracted Amharic text against a native corpus or human sample.","fun_headline_variants_meta":{"raw":{"variants":["600MB Amharic from Common Crawl, QLoRA fine-tune cuts perplexity 35.6→19.6","Amharic QLoRA on 4.5B XGLM: Common Crawl mining cuts perplexity 35.6→19.6","Cheap Amharic: range-request Common Crawl URLs, QLoRA lifts QA F1 to 9.9","Mining Common Crawl without download: 600MB Amharic, QLoRA lifts XGLM few-shot QA","600MB Amharic mined from Common Crawl cuts XGLM perplexity 35.6 to 19.6"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002139,"raw_usage":{"total_tokens":8306,"prompt_tokens":960,"completion_tokens":7346,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":7186}},"tokens_in":576,"tokens_out":7346,"duration_ms":41560,"temperature":1.0,"reasoning_tokens":7186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:16:41.507323+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of UnifiedCrawl-Amharic documents and have native speakers, or an independent language identifier, label the language; if a substantial share turns out to be mislabeled or code-switched, the extraction's precision is in doubt. A quantitative version is to build a same-size Amharic corpus from the same WARC records re-annotated with an independent language identifier, fine-tune XGLM-4.5B with the same QLoRA settings, and compare perplexity and few-shot F1 against the paper's numbers.","supporting_citations":[],"review_version":1}