REVIEW 3 major objections 2 minor 1 cited by
Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus
T0 review · 3 major / 2 minor · reviewed 2026-05-07 · grok-4.3
Pith's one-line read Mistral 7B with QLoRA rank 16 reaches mean perplexity 5.03 on Tajik text generation after release of the largest open Tajik web corpus.
desk verdict The paper releases a sizable new Tajik corpus and runs straightforward PEFT benchmarks on it, but the results rest on a narrow subsample and perplexity alone. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Tajik Web Corpus of 319,298 documents used as training data together with perplexity and cross-entropy evaluation across full fine-tuning, LoRA, and QLoRA applied to models from the GPT-2 family through Mistral 7B.
What would settle it
Human raters assigning low quality scores to text samples generated by the Mistral 7B QLoRA model, or a new independent collection of Tajik documents producing substantially higher perplexity under the same fine-tuned model.
Extended reading notes
Core claim
By constructing the Tajik Web Corpus the author enables the first systematic comparison of parameter-efficient fine-tuning strategies for Tajik text generation, finding that Mistral 7B equipped with QLoRA at rank 16 attains the lowest mean perplexity of 5.03 on a 10,000-document subsample while remaining computationally practical.
Load-bearing premise
A random 10,000-document subsample from the web corpus represents the overall distribution of Tajik text and that perplexity alone measures generation quality adequately.
Editorial extensions
If this is right
- QLoRA at rank 16 provides the best observed trade-off between perplexity and memory use for Mistral 7B on Tajik.
- Increasing LoRA rank from 8 to 16 produces statistically insignificant perplexity gains at higher memory cost.
- Full fine-tuning of GPT-2 family models yields lower perplexity than LoRA but induces catastrophic forgetting.
- Encoder-only models such as XLM-RoBERTa deliver the highest perplexity and are least suitable for the generative task.
Reading between the lines
- The same corpus-construction and benchmarking approach could be replicated for other low-resource languages that have substantial web presence.
- Perplexity results would be strengthened by adding human evaluation or downstream task metrics such as Tajik question answering.
- The demonstrated viability of QLoRA suggests that similar efficiency gains are possible when adapting models to other Cyrillic-script low-resource languages.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Tajik Web Corpus (319,298 documents, ~1.11B characters) as the largest open Tajik resource and benchmarks 17 PEFT configurations (full fine-tuning, LoRA, QLoRA at ranks 8/16) across autoregressive, encoder-decoder, and encoder-only models on a 10k-document random subsample. Quality is measured via perplexity and cross-entropy loss, with auxiliary GPU memory and training time metrics. The central empirical claim is that Mistral-7B with QLoRA (r=16) yields the best result (mean perplexity 5.03, std 0.03), while full fine-tuning on small GPT-2 models produces lower perplexity but causes catastrophic forgetting; the work offers practical recommendations for low-resource Tajik generation.
Significance. If the benchmarking results hold under more rigorous validation, the paper contributes by publicly releasing the largest verified Tajik corpus and delivering the first systematic PEFT comparison for Tajik generative modeling. This supports low-resource language research, with explicit strengths in corpus scale, the breadth of 17 configurations, and reported standard deviations across runs. The practical recommendations on architecture and strategy selection (balancing quality against memory) are potentially useful for similar Cyrillic low-resource settings.
major comments (3)
- [Methods / Data Preparation] Methods / Data section: The experiments rely on a random 10,000-document subsample drawn from the full 319,298-document corpus, yet no sampling procedure, deduplication details, train-test split criteria, domain stratification, or statistical representativeness test (e.g., Kolmogorov-Smirnov on token distributions) is described. This assumption is load-bearing for the generalizability of all reported perplexity rankings and the claim that QLoRA r=16 is optimal.
- [Evaluation / Results] Evaluation section: Perplexity and cross-entropy are the sole quality metrics; no human ratings, BLEU/ROUGE, or downstream task performance (e.g., Tajik QA or summarization) is reported. Because the goal is text generation, next-token perplexity alone does not establish that lower values correspond to better generation quality or coherence, directly affecting the headline ranking of Mistral QLoRA.
- [Results / Discussion] Results section: The paper states that full fine-tuning on GPT-2 models induces catastrophic forgetting, but provides no quantitative measure (e.g., degradation on a held-out pre-training task, generation diversity metrics, or comparison of output entropy). This omission weakens the contrast drawn between full fine-tuning and PEFT strategies.
minor comments (2)
- [Abstract] Abstract: The claim of 'statistically insignificant improvement' when increasing LoRA rank from 8 to 16 is stated without naming the test, degrees of freedom, or p-value threshold used.
- [Tables] Tables: Ensure every table reporting mean perplexity and standard deviation explicitly states the number of independent runs or random seeds underlying the statistics.
Simulated Author's Rebuttal
We appreciate the referee's thorough review and constructive feedback on our manuscript. Below, we provide detailed responses to each major comment, indicating the revisions we plan to make to address the concerns.
read point-by-point responses
-
Referee: [Methods / Data Preparation] Methods / Data section: The experiments rely on a random 10,000-document subsample drawn from the full 319,298-document corpus, yet no sampling procedure, deduplication details, train-test split criteria, domain stratification, or statistical representativeness test (e.g., Kolmogorov-Smirnov on token distributions) is described. This assumption is load-bearing for the generalizability of all reported perplexity rankings and the claim that QLoRA r=16 is optimal.
Authors: We thank the referee for highlighting this important point regarding reproducibility and generalizability. In the original manuscript, the subsample was selected as a simple random sample of 10,000 documents from the full corpus to enable efficient benchmarking while maintaining scale. To address the concern, we will revise the Methods section to explicitly describe the sampling procedure, including the use of a fixed random seed for reproducibility, the absence of additional deduplication beyond the corpus-level cleaning already performed (as described in Section 3), and the train-validation-test split ratios (80/10/10). We will also include a brief analysis of representativeness by comparing key statistics such as average document length and character distribution between the subsample and the full corpus. While a full Kolmogorov-Smirnov test on token distributions was not performed due to computational constraints, we will add a note acknowledging this limitation and justifying the random sampling approach for this initial benchmarking study. These changes will be incorporated in the revised version. revision: yes
-
Referee: [Evaluation / Results] Evaluation section: Perplexity and cross-entropy are the sole quality metrics; no human ratings, BLEU/ROUGE, or downstream task performance (e.g., Tajik QA or summarization) is reported. Because the goal is text generation, next-token perplexity alone does not establish that lower values correspond to better generation quality or coherence, directly affecting the headline ranking of Mistral QLoRA.
Authors: We agree that perplexity alone does not fully capture generation quality, particularly coherence and semantic correctness in text generation tasks. Perplexity was selected as the primary metric because it directly measures the model's predictive performance on the language modeling objective, which is central to our benchmarking of fine-tuning strategies for low-resource settings. To strengthen the evaluation, we will add a new subsection in the Results discussing the limitations of perplexity and include qualitative examples of generated text from the top-performing models in an appendix to illustrate coherence. We note that human evaluations or downstream task benchmarks (such as QA) would provide valuable additional insights but were beyond the scope of this work due to the lack of existing Tajik evaluation datasets and resources for annotation. We will update the Discussion to recommend these as future work. revision: partial
-
Referee: [Results / Discussion] Results section: The paper states that full fine-tuning on GPT-2 models induces catastrophic forgetting, but provides no quantitative measure (e.g., degradation on a held-out pre-training task, generation diversity metrics, or comparison of output entropy). This omission weakens the contrast drawn between full fine-tuning and PEFT strategies.
Authors: The claim of catastrophic forgetting in fully fine-tuned small models was based on observed degradation in generation quality, where outputs became repetitive and lost Tajik-specific characteristics, contrasting with the coherent outputs from PEFT methods. However, we acknowledge that this was not supported by quantitative metrics in the original submission. We will revise the Results section to include quantitative evidence, such as measuring the entropy of generated sequences and the proportion of unique n-grams to demonstrate reduced diversity in full fine-tuning cases. Additionally, we will compare performance on a small held-out set of English text to show forgetting of pre-trained capabilities. These additions will provide a more rigorous contrast between full fine-tuning and PEFT approaches. revision: yes
Circularity Check
No circularity: purely empirical benchmarking with direct measurements
full rationale
The paper reports corpus creation (319k documents), random subsampling to 10k, and direct experimental benchmarking of 17 model/fine-tuning configurations via perplexity, loss, memory, and time. No derivations, equations, fitted parameters renamed as predictions, or self-citation chains appear; results are raw empirical outputs on held-out text with no reduction to inputs by construction. The work is self-contained against external benchmarks.
Assumptions & free parameters
free parameters (1)
- LoRA rank
assumptions (1)
- domain assumption Perplexity on held-out text is a valid proxy for text generation quality in low-resource language settings.
Cite this review
Pith. "Pith review of Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus." pith.science (2026). https://pith.science/paper/63R7DN2V
@misc{pith2026260503742,
author = {Pith},
title = {Pith review of: Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus},
year = {2026},
howpublished = {\url{https://pith.science/paper/63R7DN2V}},
note = {Machine review of arXiv:2605.03742}
}
read the original abstract
We release the Tajik Web Corpus (319k docs, 1.11B chars) and benchmark generative LLMs on prompt continuation in Tajik, a low-resource Cyrillic-script language. Seventeen configurations across nine architectures are evaluated under three fine-tuning strategies: full fine-tuning, LoRA, and QLoRA (ranks 8 and 16). Because perplexity is not directly comparable across model families with different tokenizers, generation quality is assessed through perplexity interpreted within each family, complemented by qualitative analysis performed by a native Tajik speaker. Computational cost is measured via GPU memory and training time. The best quality-cost trade-off is achieved by Mistral 7B with QLoRA rank 8: perplexity 5.11 (within its tokenizer family), coherent Tajik output confirmed by the native speaker, 14.21 GB GPU memory, and approximately 33 minutes of training. Increasing the rank to 16 yields a negligible improvement for Mistral (perplexity 5.03, pairwise p > 0.05) while consuming about 1 GB more memory. Full fine-tuning of small GPT-2 models obtains lower numeric perplexity but leads to catastrophic forgetting (English or gibberish output); in contrast, QLoRA preserves multilingual pretrained knowledge and generates meaningful Tajik text. Encoder-only models perform worst (perplexity approximately 59), confirming their unsuitability for autoregressive generation. To our knowledge, this is the first systematic PEFT benchmark for Tajik text generation. Practical recommendations include using Mistral 7B with QLoRA r=8, avoiding full fine-tuning of small GPT-2 models, and adopting the released corpus and benchmark.
Forward citations
Cited by 1 Pith paper
-
Large Language Models for Low-Resource Languages: A Conceptual Framework for an Electronic Explanatory Dictionary of the Tajik Language
The authors present the first holistic conceptual architecture for an LLM-driven electronic explanatory dictionary of Tajik, but the system is neither built nor evaluated.
Reviewed May 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.