Pith. sign in

REVIEW 3 major objections 6 minor 16 references

Soro: A Lightweight Foundation Model and Chatbot for Tajik

T0 review · 3 major / 6 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tajik-only continual pretraining on 1.9B tokens turns open Gemma 3 into deployable Soro chatbots that beat same-size baselines on new Tajik school and language tests, and still run after FP8/INT4 compression for schools with weak connectivi

desk verdict Solid applied Tajik specialization with open benchmarks and a real 100-school pilot; the main soft spot is possible textbook train–test overlap, not empty claims. read the letter →

arxiv 2605.27379 v2 pith:6LP47DHL submitted 2026-04-09 cs.AI cs.CL

Stanislav Liashkov , Haitz Sáez de Ocáriz Borde , Azizjon Azimi , Khushbakht Shoymardonov , Shuhratjon Khalilbekov , Bonu Boboeva This is my paper · ORCID
classification cs.AIcs.CL
keywords Tajiklanguagemodelscontinualpretraininglow-resourceNLPinstructiontuningmodelmergingquantizationeducationalAIdeploymentlanguage-specificbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a focused, Tajik-only adaptation of open Gemma 3 models is enough to produce practical conversational systems for a language that mainstream multilingual training largely ignores. The authors start from 12B and 27B Gemma 3 checkpoints, continually pretrain on a curated 1.9-billion-token Tajik corpus that mixes filtered web text, PDFs, and Ministry-aligned school materials, then instruction-tune on 40K teacher-style Tajik examples and linearly merge back about 20 percent of the original Gemma instruction weights. They also release a new suite of Tajik multiple-choice benchmarks covering general knowledge, linguistic competence, national history and literature, and university-entrance material. On those benchmarks the final Soro models substantially outperform same-size Gemma 3 instruction-tuned baselines while largely holding English MMLU, and FP8 and INT4 quantization keep most of the Tajik gains while cutting memory enough for edge or single-GPU use. The work is already running in a government- and UNICEF-linked pilot across 100 schools in five Tajik cities, with a stated path toward national scale-out under the country’s AI strategy.

What carries the argument

The three-stage pipeline of Tajik-only LoRA continual pretraining on a 1.9B-token curated corpus, supervised instruction tuning on 40K teacher-style examples, and linear weight merging (~80% Soro / ~20% Gemma 3-IT), together with the open-sourced Tajik evaluation suite that makes the gains measurable.

What would settle it

A decontamination or overlap audit between the released Tajik History and Literature benchmarks and the educational subset of the 1.9B-token pretraining corpus: if many questions or near-paraphrases already appear in training, the claimed generalization gains shrink; independent human evaluation on held-out local topics would also test whether synthetic instruction data introduced systematic bias.

Watch

Extended reading notes

Core claim

After Tajik-only continual pretraining, instruction tuning, and an approximately 80/20 linear merge with the original Gemma 3 instruction-tuned weights, the resulting Soro (27B) and Soro Lite (12B) models achieve clear average gains over same-size Gemma 3-IT baselines on a new suite of Tajik benchmarks while retaining most English capability; FP8 and GPTQ INT4 quantization preserve the bulk of those Tajik gains at far lower memory cost, making school and edge deployment feasible.

Load-bearing premise

That the new school-aligned Tajik benchmarks are sufficiently independent of the same class of Ministry textbooks that were transcribed or OCR’d into the pretraining corpus, so reported gains reflect general competence rather than memorization of training material.

Editorial extensions

If this is right

  • Low-resource languages with similar sparse digital footprints can obtain large, measurable gains by continual pretraining a strong open multilingual base on a carefully curated monolingual corpus rather than training from scratch.
  • Linear merging with a modest fraction of the original instruction-tuned weights can recover general English and world knowledge lost during language-specific adaptation without erasing the new language gains.
  • FP8 and INT4 quantization that preserve most specialized-language accuracy enable single-GPU and consumer-GPU deployment of 12B–27B models in schools and offices that lack data-center hardware or reliable connectivity.
  • Open-sourcing language-specific school and entrance-exam benchmarks fills an evaluation vacuum and lets others measure progress on the same culturally grounded tasks.
  • A curriculum-aligned training corpus plus teacher-style instruction data supports real classroom use cases (lesson help, diagnostics, AI literacy) already being piloted at national scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If textbook-to-benchmark contamination is low, the same recipe is a practical template for other Central Asian and Persian-related languages that share Cyrillic or Arabic-script under-representation.
  • The large gap on the pure linguistic competence set (TajLib) suggests that continual pretraining mainly improves internal grammar and morphology representations, not only surface fact recall.
  • Teacher survey scores that rate language quality highest and factual accuracy lower imply that future preference or RAG layers on verified local knowledge bases would address the remaining deployment friction more than further pretraining alone.
  • The edge-quantization story makes the model transferable to other low-connectivity public-sector settings beyond education, provided local knowledge bases and safety filters are added.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Soro (27B) and Soro Lite (12B), Tajik-specialized conversational LLMs obtained from Gemma 3 via Tajik-only continual pretraining on a curated 1.9B-token corpus (web, PDFs, curriculum materials), supervised instruction tuning on 40K teacher-style examples, and an ~80/20 linear merge with Gemma 3-IT. The authors release a suite of Tajik multiple-choice benchmarks (Tajik MMLU, Tajik-FactQA, History, Literature, TajLib, Curated), report average accuracy gains of roughly 6–8+ points over same-size Gemma 3-IT while largely retaining English MMLU, show that FP8 and GPTQ INT4 preserve most Tajik gains, and describe an education-sector pilot across 100 schools with a teacher feedback survey.

Significance. If the reported Tajik gains are largely free of train–eval contamination and statistically reliable, this is a concrete, reproducible contribution for a severely under-resourced language: open Tajik benchmarks on Hugging Face, a documented continual-pretraining + SFT + merge recipe under modest compute, quantization results that enable edge deployment, and a government-backed school pilot with quantitative teacher ratings. The tokenizer fertility analysis motivating Gemma 3, the merge-ratio sweep (Fig. 3), and the pre-merge / post-merge / quantized ablations are useful methodological details for other low-resource adaptation efforts. The work is more systems-and-deployment than algorithmic novelty, but that is appropriate for the stated goal of deployable Tajik language technology.

major comments (3)
  1. Independence of school-aligned benchmarks from the pretraining corpus is not established and is load-bearing for the headline claim of generalizable Tajik gains. §4.1 states that continual pretraining includes secondary-school materials (grades 5–11 textbooks, manuals, lecture notes) from Ministry sources via OCR/transcription. Appendices A.4 and A.6 state that Tajik History (1,400 items) and Tajik Literature (698 items) are derived from the same class of officially approved grades 5–11 textbooks. No n-gram, embedding, or passage-level decontamination between the educational subset and these benchmarks is reported. Part of the multi-benchmark average lift in Fig. 4 (and larger lifts on History/Literature) may therefore reflect memorization. Please report decontamination statistics and either (i) re-evaluate after removing overlapping items or (ii) report headline averages excluding Histo
  2. Statistical support for accuracy claims is thin. Fig. 4, Table 6, and the per-benchmark figures report point accuracies only—no bootstrap CIs, standard errors, or significance tests on Soro vs Gemma 3-IT (or vs other open models). With fixed multiple-choice sets and deterministic logit extraction (Appendix B), resampling over items is straightforward. Without uncertainty estimates, statements such as “substantially outperforms” and “preserves most Tajik-language gains” under quantization cannot be assessed for robustness, especially on smaller sets (e.g., Tajik-FactQA, n=436).
  3. Synthetic data provenance and quality controls are under-specified relative to their role in both pretraining and SFT. FineWeb-Edu is translated into Tajik with Gemini 2.5 Flash (§4.1), and the 40K instruction set is also Gemini-generated with a human audit of only “several hundred” examples and a qualitative “low defect rate” (§4.2)—no defect-rate numbers, inter-annotator agreement, or subject-stratified failure modes. Because the chatbot’s pedagogical style and much of the educational content rest on this pipeline, the manuscript should quantify audit outcomes and discuss residual risks (hallucinated facts, English interference, style homogenization) as limitations on the educational claims.
minor comments (6)
  1. Abstract and §6 claim average gains on the order of 6–8+ points and “8–13%” in the conclusion; align the wording (percentage points vs percent) and state explicitly which model variants and which benchmark average are meant.
  2. Fig. 4 and related bar charts would be clearer with error bars once item-level uncertainty is computed; also label whether scores are raw-logit or chat-template format (Appendix B describes both).
  3. Table 2 fertility comparison is useful; briefly note whether fertility was measured with the same normalization (whitespace, punctuation) for Cyrillic Tajik vs English so readers can interpret the 2.380 vs 2.798 gap.
  4. §5.2 merge experiment: state whether the 80/20 optimum was chosen on a held-out split of the Tajik suite or on the full evaluation set used for final reporting, to avoid selection bias on the merge weight.
  5. Appendix E teacher survey (n=53, convenience sample of pilot schools) is a welcome addition; the limitations paragraph already notes selection bias—consider moving a one-sentence caveat into the main §8.4 so readers of the deployment case study see it without opening the appendix.
  6. Minor consistency: date line “April 2026” / arXiv May 2026 and survey dates “March 16–30, 2026” are fine if intentional; ensure all URLs and HF dataset names remain stable at camera-ready.

Circularity Check

1 steps flagged · score 1.0 of 10

No derivation-by-construction circularity; only a minor merge-ratio selection on the same Tajik suite used for headline scores.

  1. fitted input called prediction [§5.2 Post-training Linear Merging; Figure 3]
    "We evaluate merged models across a range of interpolation ratios, varying the Gemma 3-IT contribution from 5% to 90%, and assess each variant on our Tajik benchmark suite. As shown in Figure 3, the optimal mixture for both model sizes is approximately 20% Gemma 3-IT and 80% Soro"

    The merge weight is selected by maximizing mean accuracy on the same Tajik suite used for the headline Soro vs Gemma comparison. The reported merged scores are therefore the best point on that grid rather than an independent out-of-sample prediction of the merge ratio. Effect is small (~1.5–2 pp) and pre-merge gains already stand alone, so this is minor reporting circularity, not a forced identity of the main claim.

full rationale

Soro is an empirical adaptation paper, not a first-principles derivation. The load-bearing chain is: start from Gemma 3 → continual pretrain on a stated 1.9B-token Tajik corpus → SFT on 40K examples → optional linear merge → score multiple-choice accuracy via logit extraction on held-out-style benchmarks. None of the six circularity patterns apply in a strong form. There is no self-definitional identity (no quantity defined as the thing it is said to predict), no uniqueness theorem imported from the authors, no ansatz smuggled in via self-citation, and no renaming of a known closed-form result. The Borde (2025) self-citation is only positioning (“Type II Partial AI Sovereignty”) and is not used to force any numerical claim. Pre-merge checkpoints already beat same-size Gemma 3-IT on the Tajik suite (Fig. 4: 56.4 vs 50.4 for 12B; 62.3 vs 55.8 for 27B), so the central continual-pretraining claim does not reduce to the merge step. The only mild issue is that the final 80/20 merge weight is chosen by maximizing mean accuracy on the same Tajik benchmark suite later reported as the main result (Fig. 3); that is ordinary test-set hyperparameter selection, not a forced identity. Train/eval overlap between Ministry textbooks in the pretraining corpus and school-aligned History/Literature benchmarks is a contamination/correctness risk, not circular derivation under these criteria, and is therefore not scored here. Overall circularity is negligible.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central performance claim rests on standard LLM adaptation assumptions plus several paper-specific modeling choices (LoRA ranks, merge weight, synthetic teacher data, curriculum-aligned eval). No new physical entities; the ‘invented’ objects are the model family and benchmarks as engineering artifacts. Free parameters are training/merge knobs selected for Tajik mean accuracy rather than derived constants.

free parameters (4)
  • Linear merge weight (Soro vs Gemma-3-IT)
    Chosen by sweeping 5–90% Gemma contribution and picking ~80% Soro / 20% Gemma for max mean Tajik accuracy (Fig. 3); directly shapes the reported final scores.
  • LoRA rank/alpha (pretrain 128/64; SFT 32)
    Hand-set ranks and alphas (Tables 3–4) control capacity of adaptation; not derived from theory, fitted to stability and corpus size.
  • Learning rates and step budgets
    7e-5/5e-5 pretrain and 3e-5/1e-5 SFT with fixed steps/epochs are manual schedules that determine how much Tajik is absorbed vs forgotten.
  • GPTQ calibration set size/source
    128 Tajik Wikipedia samples and group size 128 (Appendix C.2) are calibration choices that affect INT4 accuracy retention.
assumptions (5)
  • domain assumption Continual monolingual pretraining on a strong multilingual base improves target-language competence without full from-scratch training.
    Core pipeline premise in §2 and §5; standard in low-resource adaptation literature the paper cites.
  • domain assumption LoRA updates on attention and MLP projections plus full embed_tokens training suffice to absorb Tajik linguistic knowledge from 1.9B tokens.
    §5.1 configuration; assumes low-rank updates capture the needed language shift.
  • ad hoc to paper Gemini 2.5 Flash translations of FineWeb-Edu and synthetic teacher-style SFT data are faithful enough for educational Tajik use after light human audit.
    §4.1–4.2; quality of a large fraction of training signal depends on this generator-plus-audit assumption.
  • domain assumption Logit-based A/B/C/D scoring on author-built MCQs is a valid proxy for real classroom assistant quality.
    §6 and Appendix B; open-ended pilot feedback is only qualitative.
  • domain assumption Linear weight averaging of related checkpoints combines complementary capabilities without destructive interference at the chosen ratio.
    §5.2 post-training merge via mergekit; justified by model similarity, not proved.
invented entities (2)
  • Soro / Soro Lite model family independent evidence
    purpose: Deployable Tajik conversational LLMs under local compute/connectivity constraints.
    Primary artifact; independent evidence is benchmark tables and pilot deployment, not external theory.
  • Tajik evaluation suite (Tajik-FactQA, TajLib, Tajik History, Literature, Curated, Tajik MMLU) independent evidence
    purpose: Measure linguistic, cultural, and curriculum-aligned competence where standard benchmarks lack Tajik coverage.
    Author-constructed metrics that define success; open-sourced but not independently validated before this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Soro: A Lightweight Foundation Model and Chatbot for Tajik." pith.science (2026). https://pith.science/paper/6LP47DHL

@misc{pith2026260527379,
  author       = {Pith},
  title        = {Pith review of: Soro: A Lightweight Foundation Model and Chatbot for Tajik},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6LP47DHL}},
  note         = {Machine review of arXiv:2605.27379}
}
read the original abstract

We present Soro, a family of Tajik-specialized conversational large language models (LLMs) designed for real-world deployment under tight compute and connectivity constraints in Tajikistan. Starting from open-weight Gemma 3 checkpoints, we perform Tajik-only continual pretraining on a curated 1.9-billion-token corpus spanning filtered web text, PDF documents, and curriculum-aligned educational materials, followed by supervised instruction tuning on 40K Tajik teacher-style examples. To enable rigorous evaluation despite the limited coverage of Tajik in standard benchmarks, we introduce a suite of Tajik benchmarks covering general knowledge, linguistic competence, and school- and university entrance-exam domains, and we open-source them on Hugging Face. Across these Tajik benchmarks, Soro substantially outperforms same-size Gemma 3 baselines while retaining strong English performance on standard datasets. We further show that FP8 and INT4 quantization of Soro preserves most Tajik-language gains while reducing memory requirements for edge deployment, supporting an ongoing education-sector pilot and planned scale-out across schools in Tajikistan.

Figures

Figures reproduced from arXiv: 2605.27379 by the authors.

Figure 1
Figure 1. Training and evaluation loss during continual pretraining of Soro Lite Base and Soro Base [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Training and evaluation loss during supervised instruction tuning of Soro Lite and Soro. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Impact of the linear merging weight on mean accuracy across Tajik benchmarks. The optimal [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Average accuracy on Tajik benchmarks across open-source models and full-precision Soro [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Average accuracy on Tajik benchmarks for quantized Soro variants compared to full-precision [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Geographic distribution of Soro deployments. (a) Initial 30-school pilot in Dushanbe; (b) [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: MMLU Benchmark A.2 Tajik Curated General Benchmark The Tajik Curated General Benchmark comprises 2,266 questions distributed across six domains: History, Biology, Literature, Tajik Language (grammar, morphology, and syntax), Law, and Geography. The questions were manua…
Figure 8
Figure 8. Figure 8: Tajik Curated Benchmark A.3 Tajik Factual Benchmark (Tajik-FactQA) Tajik-FactQA consists of 436 multiple-choice questions drawn from the Centralized University Entrance Examinations (Markazi millii test¯ı) administered to high-school graduates seeking university admiss…
Figure 9
Figure 9. Figure 9: Tajik Factual Benchmark A.4 Tajik History (high-school level) The Tajik History benchmark contains 1,400 multiple-choice questions covering the national history curriculum for grades 5–11. Questions derive from officially approved school textbooks and span topics inclu…
Figure 10
Figure 10. Figure 10: Tajik History high-school level benchmark [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: Tajik Linguistic Benchmark (TajLib) A.6 Tajik Literature (high-school level) The Tajik Literature benchmark contains 698 multiple-choice questions covering the national literature curriculum for grades 5–11. Questions derive from officially approved school textbooks a…
Figure 12
Figure 12. Figure 12: Tajik Literature Benchmark A.7 Tajik MMLU Tajik MMLU is a Tajik-language translation of the original English MMLU benchmark, covering the same 57 subject categories and preserving the original question structure and answer options. Its role is complementary to English…
Figure 13
Figure 13. Figure 13: Tajik MMLU Benchmark 8https://huggingface.co/datasets/zehnlab/tajik-mmlu 30 [PITH_FULL_IMAGE:figures/full_fig_p030_13.png]
Figure 14
Figure 14. Figure 14: Raw format prompt structure for English evaluation. [PITH_FULL_IMAGE:figures/full_fig_p031_14.png]
Figure 15
Figure 15. Figure 15: Raw format prompt structure for Tajik evaluation. [PITH_FULL_IMAGE:figures/full_fig_p031_15.png]
Figure 16
Figure 16. Figure 16: Chat format prompt structure for English evaluation. [PITH_FULL_IMAGE:figures/full_fig_p032_16.png]
Figure 17
Figure 17. Figure 17: Chat format prompt structure for Tajik evaluation. [PITH_FULL_IMAGE:figures/full_fig_p032_17.png]
Figure 18
Figure 18. Figure 18: Mean teacher ratings across six survey dimensions ( [PITH_FULL_IMAGE:figures/full_fig_p042_18.png]
Figure 19
Figure 19. Figure 19: Distribution of teacher ratings across survey dimensions ( [PITH_FULL_IMAGE:figures/full_fig_p042_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 1 canonical work pages

  1. [1]

    Haitz Sáez de Ocáriz Borde

    URLhttps://arxiv.org/abs/2401.10440. Haitz Sáez de Ocáriz Borde. Sovereign AI vs AI Polyglots. SSRN Working Paper 5639470, SSRN, October 2025. URL https://ssrn.com/abstract=5639470. Date written: October 18,

    arXiv 2025

  2. [2]

    Posted: November 19, 2025. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen,...

    arXiv doi:10.18653/v1/2024.emnlp-industry.36 2025

  3. [3]

    URLhttps://arxiv.org/abs/2312.11011. OpenAI. gpt-oss-120b & gpt-oss-20b Model Card, 2025. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavar...

    work page Pith review arXiv doi:10.1145/325334.325242 2025

  4. [4]

    Adamush-shu’aro

    Python library throughout. C.1 Dynamic FP8 We apply dynamic FP8 quantization [Kuzmin et al., 2024] by converting both weights and activations from BF16 to 8-bit floating-point format at inference time. Quantization targets all Linear layers across the transformer blocks, while the language modeling head (lm_head) is deliberately excluded to preserve outpu...

    2024

  5. [5]

    Present-Future Tense: expresses an action occurring at the moment of speech or in the near future (Example:Man mekhonam- I read / I will read)

  6. [6]

    Past Definite Tense: an action completed in the past (Example:Man khondam- I read)

  7. [7]

    Past Indefinite Tense: a past action with a known result or relevance to the present (Example: Man khondaam- I have read)

  8. [8]

    Past Continuous/Narrative Tense: a continuous or repetitive action in the past (Example:Man mekhondam- I used to read / I was reading)

Show all 16 references
  1. [9]

    Past Perfect (Remote Past): an action completed before another past action (Example:Man khonda budam- I had read)

  2. [10]

    Fann Niagara

    Compound Future Tense: an action that will strictly take place in the future (Example:Man khoham khond- I shall read). 36 Example 6: Multi-Turn Conversation — Lake Iskanderkul 37 English Translation User (Turn 1):Hello! I would like to get information about Lake Iskanderkul. W...

  3. [11]

    Keep the water clean: Do not throw trash into the lake or the Iskandardarya river and avoid using chemical detergents near the water

  4. [12]

    Take your trash with you: Collect all plastic and paper waste to return it to special trash collection points or to the city

  5. [13]

    Protect the flora: Avoid picking rare flowers or breaking tree branches, as they are vital compo- nents of the ecosystem

  6. [14]

    Fire safety: Light fires only in designated areas to prevent forest fires and avoid damaging the topsoil

  7. [15]

    Masnavi-i Ma’navi

    Respect wildlife: Do not interfere with the lives of birds and wild animals or disturb their peace. 38 Example 7: Multi-Turn Conversation — Mawlana Jalaluddin Balkhi (Rumi) 39 English Translation User (Turn 1):I would like to receive information about Mawlana Jalaluddin Balkhi...

    2026

  8. [16]

    Favorable

    Lesson preparation usefulness: How useful is Soro for lesson preparation (plans, materials, tests)? 4.Ease of use: How easy is Soro to use? 5.Willingness to adopt: How willing are you to use Soro in your daily work as a teacher? 6.Recommendation likelihood: Would you recommend...

    2015

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.