Pith. sign in

REVIEW 4 major objections 5 minor 12 references

LKValues: Aligning Large Language Models with Sri Lankan Societal Values

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A survey of 205 Sri Lankans yields 40 values that, when fine-tuned into Qwen models, lift Sinhala value-judgment accuracy from 49% to 80% and cut invalid outputs to zero.

desk verdict A genuinely useful Sinhala-English value alignment resource, but the 'majority-endorsed' inventory is statistically fragile and the training/benchmark circularity narrows what the gains mean. read the letter →

arxiv 2607.20410 v1 pith:YTLYNOH5 submitted 2026-07-22 cs.CL

Nethmi Muthugala , Supryadi , Surangika Ranathunga , Nisansa de Silva , Ruijie Tao , Ovindu Gunatunga , Pengyun Zhu , Shaowei Zhang
show 2 more authors
Jingting Zheng Deyi Xiong
This is my paper
classification cs.CL
keywords valuealignmentSriLankansocietalvaluesSinhalalow-resourcelanguagessurvey-basedidentificationinstructionfine-tuningculturalbenchmarkLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that culturally grounded, survey-derived resources can close the value-alignment gap for low-resource languages. It builds LKValues, a bundle of 40 majority-endorsed Sri Lankan societal values, a 150k-instance bilingual instruction corpus, and a 1,000-item evaluation benchmark. The core claim is that fine-tuning on these resources substantially improves Qwen-family models' value-sensitive judgment in both English and Sinhala, reducing invalid outputs and cross-lingual disparities — though gains are model-family dependent. A sympathetic reader would care because it offers a replicable, country-specific alternative to Western-centric alignment, and shows that scale and recency alone don't guarantee culturally aware behavior.

What carries the argument

The central mechanism is the survey-to-benchmark pipeline: 40 majority-endorsed Sri Lankan societal values derived from a trilingual survey (with a >50% endorsement threshold) organize both the instruction dataset LKvaluesIT and the evaluation benchmark LKvaluesBench. The survey values act as the controlling taxonomy — every training instance is tagged with one value, and every benchmark item maps to one primary value. Fine-tuning uses LKvaluesIT mixed with 50K general-purpose Sinhala instruction examples, and evaluation uses a strict A/B/BOTH/0 forced-choice protocol with dual prompt framings (Sri Lankan-specific vs. Universal).

What would settle it

A replication study with a larger, more demographically representative Sri Lankan sample (e.g., stratified by ethnicity, religion, and region) that recomputes endorsement proportions and re-derives the value inventory: if the majority-endorsed set changes substantially, the benchmark and all fine-tuning results built on it would need to be revisited.

Watch

Extended reading notes

Core claim

The paper claims that LKValues — the first survey-grounded resource suite for Sri Lankan value alignment — improves value-sensitive judgment of LLMs in Sinhala and English. From a trilingual survey of 205 respondents, the authors derive 40 majority-endorsed values, then build LKvaluesIT (150k scenario-based instruction instances) and LKvaluesBench (1,000 evaluation instances). Fine-tuning Qwen3.5-4B-Base with full SFT raises its accuracy from 0.610 to 0.863 overall and from 0.491 to 0.801 in Sinhala, while reducing invalid outputs from 18.20% to 0.00%. Gains are not universal: the same one-epoch LoRA recipe does not transfer to Aya-Expanse-8B, whose adapted variant drops on the benchmark. Th

Load-bearing premise

The load-bearing premise is that the 40 values derived from a 205-respondent online survey (with a >50% endorsement threshold) are a valid operationalization of Sri Lankan societal values, and that LKvaluesBench items correctly measure alignment to those values.

Editorial extensions

If this is right

  • If the central claim holds, fine-tuning on LKValues can substantially improve Sinhala value-sensitive judgment in Qwen-family models, reducing invalid outputs to zero in the best case.
  • LKvaluesBench can serve as a diagnostic for low-resource cross-lingual alignment gaps: even strong multilingual models like Command-R-08-2024 show large English–Sinhala accuracy gaps.
  • The replicable pipeline — survey, value tagging, scenario extraction, bilingual evaluation — could be applied to other countries and low-resource languages.
  • Smaller fine-tuned models can outperform much larger baselines on country-specific value judgment, suggesting that culturally grounded supervision can compensate for scale.
  • Prompt wording (Sri Lankan-specific vs. Universal) has smaller effects than language, indicating that Sinhala competence and task-format adherence matter more than framing alone.
  • The model-family dependence (Qwen gains vs. Aya drop) implies that optimal alignment strategies require model-specific tuning of training duration, learning rate, and adaptation method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 40-value inventory may be an artifact of its 205-respondent online sample and its >50% endorsement threshold; a more representative or larger sample could yield a different value set, and the benchmark's gold labels would shift accordingly.
  • A testable extension: the same pipeline could be applied to Tamil, the third official language of Sri Lanka, to see whether a trilingual resource suite narrows the Sinhala–English gap further or reveals value differences across linguistic communities.
  • The distinction between LKvaluesIT (generation) and LKvaluesBench (judgment) suggests a broader principle: alignment resources may need to separate explanatory fluency from forced-choice judgment, since models can excel at one while failing the other.
  • The finding that Aya-Expanse drops after LoRA fine-tuning hints that multilingual-strong models may require different training schedules or judgment-formatted supervision, not just more data — a hypothesis the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents LKValues, a Sinhala-English resource suite for aligning LLMs with Sri Lankan societal values. The authors collect a trilingual survey from 205 respondents, derive 40 values endorsed by more than 50% of the sample, and use this inventory to construct LKvaluesIT (a 150k-instance instruction corpus from Sri Lankan news) and LKvaluesBench (a 1,000-instance bilingual value-sensitive judgment benchmark). They evaluate proprietary and open-weight models, then fine-tune Qwen3.5-4B, Qwen3.5-9B, and Aya-Expanse-8B using LKvaluesIT mixed with general Sinhala instruction data. The headline claim is that LKValues fine-tuning substantially improves Qwen-family models, e.g., Qwen3.5-4B-Base improves from 0.610 AC / 0.491 SI AC / 18.20% invalid outputs to 0.863 AC / 0.801 SI AC / 0.00% invalid outputs with full SFT (Table 1), while gains remain model-family dependent, with Aya-Expanse-8B not benefiting under the same one-epoch LoRA recipe.

Significance. If the resource is taken at face value, LKValues is a useful contribution to low-resource, country-specific value alignment: the dataset is publicly released, grounded in Sri Lankan news, bilingual, and evaluated through a broad protocol with human validation and reported inter-annotator agreement for several stages. The negative result for Aya-Expanse is also informative, as it documents that a single fine-tuning recipe does not transfer across model families. However, the central claim is currently supported only under the authors' own operationalization of Sri Lankan values. The survey sample is small, convenience-based, and statistically fragile near the 50% threshold, and both the training data and the evaluation benchmark are derived from the same value inventory. The resource is valuable as a replicable pipeline and a first benchmark, but the stronger generalization claims in the abstract and conclusion need additional external validation before they can be accepted at face value.

major comments (4)
  1. [§3, Appendix A.7, Fig. 3] The 'majority-endorsed' label is not statistically supported at the reported precision. With n≈205 and an 85% confidence MOE of about ±5 pp, values such as Equality (52.48±5.06), Justice (52.66±5.14), and Belonging (55.94±5.03) have intervals that cross 50%; at the conventional 95% level the intervals widen further. Because the same 40-value inventory is used to construct LKvaluesIT and LKvaluesBench (§4), both the training supervision and the benchmark gold labels inherit this uncertainty. The paper should either use an uncertainty-aware criterion (e.g., requiring the lower confidence bound to exceed 50%), report sensitivity analyses removing values whose CI crosses the threshold, or explicitly demote 'majority-endorsed' to 'survey-endorsed under the authors' operationalization.' In addition, Figure 3 is captioned 'all 40 Sri Lankan societal values' but tabulates 51 rows; the retained 4
  2. [§1, §4.1, §4.2] There is a circularity between training and evaluation: LKvaluesIT value labels and LKvaluesBench gold labels are both derived from the same 40-value survey inventory, and the benchmark items are created or adapted to fit those values. The reported gains (e.g., Qwen3.5-4B base 0.610 AC to 0.863 AC FullSFT in Table 1) therefore measure the model's ability to reproduce the authors' operationalization of Sri Lankan values, not alignment with an independently established set of societal values. This does not invalidate the resource, but it weakens the generalization claim in the abstract and conclusion. I ask for at least one additional validation: human-model agreement on LKvaluesBench, a held-out set of items not constructed from the inventory, or a comparison against an external value-alignment benchmark.
  3. [§4.2, §4.3] Benchmark gold-label reliability is not quantified. Quality control reports Fleiss' κ=0.81 for value tagging and κ=0.82/0.75 for scenario generation (Section 4.3), but no inter-annotator agreement is reported for the LKvaluesBench A/B/BOTH/0 labels, which are the direct evaluation target. This is especially important because 509 of the 1,000 benchmark items are LLM-generated and only 'human-verified.' Please provide per-item agreement statistics (e.g., κ or adjudication rate), the number of annotators per item, and a clear description of how disagreements were resolved.
  4. [Table 1, Table 8, Table 11] The reported headline numbers differ across tables without definitional reconciliation. Table 1 reports Qwen3.5-4B-FullSFT-LKV AC=0.863 and Qwen3.5-9B-LoRA-LKV AC=0.736; Table 8 reports the similar-sounding 'Value-align. AC' as 0.868 and 0.717, and Table 11's language averages for Aya-Expanse-8B-LoRA-LKV ((0.730+0.408)/2=0.569) match Table 1 but not Table 8 (0.555). If the appendix metric is computed on a different subset (e.g., retained value categories) or with different averaging, this must be stated explicitly. Without clarification, a reader cannot verify the central quantitative claim.
minor comments (5)
  1. [§3] Typo: 'The survey designs to capture' should be 'The survey is designed to capture.'
  2. [§1] The citation 'hae (2022)' appears garbled; this likely refers to the World Values Survey (Haerpfer et al., 2020) or similar and should be corrected.
  3. [§1] The citation 'Political compass1 (Röttger et al., 2024)' is confusing: Röttger et al. is a critical methodological paper, not the Political Compass instrument itself. Please clarify how this source was used for question selection.
  4. [Appendix A.2] Footnote markers such as '3,000 years16' and 'Charter for a Pluralistic Sri Lankan Society17' do not have corresponding footnotes in the visible text; please fix the reference formatting.
  5. [Various tables] Minor presentation issues: Table 3 caption says '40 values' but the table lists 51 rows; Table 15 is very long and could be summarized in the main text with a pointer to the appendix; several model names in the references have inconsistent spacing (e.g., 'Pengyun Zhu 1').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the survey-derived value inventory is an explicit operationalization and the held-out benchmark provides an empirical, non-tautological result.

full rationale

The paper's derivation chain is: trilingual survey → 40 majority-endorsed values → LKvaluesIT (training) and LKvaluesBench (evaluation). The authors explicitly state that "the survey directly determines the value labels and evaluation categories used throughout LKValues." This is a transparent operationalization, not a hidden reduction: the benchmark is constructed from SinhalaMMLU items plus human-verified generated scenarios, and is not a subset of the training data. The fine-tuning gains (e.g., Qwen3.5-4B-Base 0.610→0.863 AC) are empirical results on a held-out test set, not mathematical consequences of the value inventory. The fact that Aya-Expanse-8B-LoRA-LKV degrades after fine-tuning shows the outcome is not forced by construction. Self-citations (Xu et al. 2024; Zhu et al. 2026) appear only in related work and are not load-bearing. The statistical fragility of the >50% endorsement threshold near the MOE boundary is a validity and robustness concern, not a circularity concern. The Limitations section appropriately concedes sampling limitations, but these do not make the derivation circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the survey-derived value inventory and the datasets built from it. These are design choices rather than physical parameters, but they are load-bearing. The most fragile assumptions are the representativeness of the survey and the validity of the operationalized values and benchmark.

free parameters (4)
  • Endorsement threshold (>50%) = 50%
    Hand-chosen threshold in Section 3 that determines which of the 51 candidate values are retained as the final 40-value inventory. A different threshold would change the value set and all downstream resources.
  • Confidence level (85%, z=1.44) = 85%
    Arbitrary statistical choice for computing margins of error in the survey; does not affect the retained values but characterizes uncertainty.
  • LoRA hyperparameters = r=16, alpha=32, dropout=0.05, LR=1e-4, 1 epoch
    Chosen via pilot experiments; affects the fine-tuning results. The paper acknowledges that this recipe does not transfer to Aya-Expanse.
  • General-purpose Sinhala training mixture size = 50,000 instances
    Hand-chosen balance between LKvaluesIT and general Sinhala instruction data; could affect trade-offs between value alignment and general instruction following.
assumptions (4)
  • domain assumption Survey sample representativeness
    The 205 online volunteers are treated as sufficient to estimate Sri Lankan societal values, despite acknowledged subgroup imbalance (Section 3, Limitations).
  • domain assumption Validity of the 40-value operationalization
    The retained values, as mapped from survey items, are assumed to capture Sri Lankan societal values (Section 3, Appendix A.5).
  • domain assumption Quality of LLM-generated scenarios and translations
    GPT-5-Mini-generated scenarios and Google Translate Sinhala translations are assumed to preserve value semantics; limited back-translation checking is provided (Section 4.3).
  • domain assumption Benchmark validity
    LKvaluesBench items (including 509 LLM-generated, human-verified items) are assumed to correctly measure value-sensitive judgment (Section 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of LKValues: Aligning Large Language Models with Sri Lankan Societal Values." pith.science (2026). https://pith.science/paper/YTLYNOH5

@misc{pith2026260720410,
  author       = {Pith},
  title        = {Pith review of: LKValues: Aligning Large Language Models with Sri Lankan Societal Values},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YTLYNOH5}},
  note         = {Machine review of arXiv:2607.20410}
}
read the original abstract

Value alignment of Large Language Models (LLMs) has been shown to be culturally biased toward Western norms. This results in the mishandling of local values in multilingual societies such as Sri Lanka that have their unique cultural dynamics. Existing benchmarks overlook Sri Lankan-contextualized values in its official language Sinhala, hindering culturally sensitive evaluation and fine-tuning. To bridge this gap, we propose LKValues, the first survey-grounded resource suite for Sri Lankan value alignment. From a trilingual survey of 205 respondents, blending adapted global frameworks and LLM-elicited local constructs, we derive 40 majority-endorsed societal values. Using these values, we construct LKvaluesIT, a Sinhala-English news-derived instruction corpus containing 150k scenario-based instances, and LKvaluesBench, a value-sensitive evaluation benchmark of 1,000 instances. We evaluate a set of proprietary and open-weight LLMs with LKvaluesBench. We fine-tune three open-weight base models (Qwen3.5-4B-Base, Qwen3.5-9B-Base, and Aya-Expanse-8B-Base). Our experiments show that newer and larger LLMs still exhibit low-resource and cultural value-alignment gaps. LKValues fine-tuning improves Qwen-family models in English and Sinhala, reducing invalid outputs and cross-lingual disparities, though gains remain model-family dependent. These highlight LKValues efficacy in embedding Sri Lankan values, offering a replicable pipeline for low-resource, country-specific pluralist value alignment. The dataset is publicly available at https://github.com/NextME14/LKValues.

Figures

Figures reproduced from arXiv: 2607.20410 by the authors.

Figure 1
Figure 1. The flowchart shows the process for deriving Sri Lankan societal values, starting with selecting questions [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. End-to-end pipeline for curating the Sri Lankan value-aligned instruction and benchmark datasets, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overall Value Endorsement per Value at 85% Confidence Level. The table shows endorsement percentages [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Table showing endorsement percentages and margins of error (MOE) by gender for Sri Lankan societal [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Heatmap of endorsement rates (%) for Sri Lankan societal values across ethnic subgroups, with each [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Examples from LKvaluesIT (English split). Each instance contains an instruction (scenario), a target value label, and a short value-grounded explanation. still value-relevant. We therefore re-tagged the un￾matched items and merged the results using exact￾match deduplic…
Figure 7
Figure 7. Figure 7: The keyword list used in value tagging and scenario extraction prompt for [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: System prompt used for scenario extraction in [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Examples from LKvaluesBench (English split). Each instance contains a question, two candidate statements, a gold label (A/B/BOTH/0), and a mapped pri￾mary value. are categorized according to subjects taught in Sri Lankan government examinations. Regardless of difficult…
Figure 10
Figure 10. Figure 10: System prompts and output rule used for LKvaluesBench evaluation under Sri Lankan-specific and Universal prompting conditions. Instruct performs weakly in both languages, sug￾gesting that instruction tuning alone is insufficient for this culturally grounded judgment t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 5 linked inside Pith

  1. [1]

    common” items and addi- tionally included “unique

    Value-latent filtering: items were screened for whether the response could be interpreted as en- dorsing or rejecting an underlying value such as family orientation, respect norms, moral permis- sibility, civic engagement etc. 2) Sri Lankan rel- evance filtering: items were further retained only when the underlying value was judged to be cul- turally inte...

    2022

  2. [2]

    Mid- sized or smaller models show more variability

    Language-Asymmetric Sensitivity: Larger models like Qwen3.5-9B-Base, Aya-Expanse-8B- Base exhibit moderate ∆SL differences between English and Sinhala, indicating that SL prompts benefit one language more than the other. Mid- sized or smaller models show more variability. For instance, Qwen3.5-4B-LoRA-LKV benefits in Sin- hala (+0.028) but drops slightly ...

  3. [3]

    Indika Liyanage, Liyanage, and V oogd

    On the alignment of large language mod- els with global human opinion.arXiv preprint arXiv:2509.01418. Indika Liyanage, Liyanage, and V oogd. 2018.Multilin- gual education yearbook 2018. Springer. 10 London School of Economics and Political Science

    arXiv 2018

  4. [4]

    Guidance document (PDF)

    Using data from the internet and social media in research: Ethics & consent. Guidance document (PDF). Oshan Mudannayake. 2022. Sri lanka news dataset. OpenAI. GPT-5 [large language model]. OpenAI API documentation. Retrieved January 5, 2026. Giada Pistilli, Alina Leidinger, Yacine Jernite, Atoosa Kasirzadeh, Alexandra Sasha Luccioni, and Margaret Mitchell...

    arXiv 2022

  5. [6]

    Sri Lankan soci- etal values

    Self-pluralising culture alignment for large language models.Preprint, arXiv:2410.12971. Pardis Sadat Zahraei and Ehsaneddin Asgari. 2025. I am aligned, but with whom? mena values benchmark for evaluating cultural alignment and multilingual bias in llms.arXiv preprint arXiv:2510.13154. Wenlong Zhao, Debanjan Mondal, Niket Tandon, Dan- ica Dillion, Kurt Gr...

    arXiv 2025

  6. [8]

    Model outputs are normalized with a case- insensitive regular expression that extracts the first valid label from A, B, BOTH, or 0; outputs that can- not be normalized are counted as invalid. Results are written row-by-row to JSONL logs for recovery, and per-model Excel files are saved immediately after each model finishes, with separate English, Sinhala,...

    2024

  7. [10]

    Proprietary and Open-Weight Models: Frontier proprietary systems such as DeepSeek- V3, Kimi-K2, and Gemma-4-31B-IT display small ∆SL magnitudes, indicatingrobustness to prompt framing. Open-weight models, par- ticularly smaller ones like Tiny-Aya-Fire and Tiny-Aya-Global, showhigh invalid rates(up to 37.7%), underscoring format brittleness and difficulty ...

  8. [11]

    Effectiveness of Local Fine-Tuning: Fine- tuning on the LKValues datasets reduces the cross- lingual accuracy gap and improves label adherence in Sinhala without significantly sacrificing English performance.This targeted adaptation highlights gaps in current large proprietary and open-weight models: despite high overall accuracy, they may misalign with l...

    arXiv 2024

Show all 12 references
  1. [12]

    Justice,

    Presumption of Innocence: Until proven guilty, every individual should be considered innocent. The arrest should be based on strong, credible evidence, not mere suspicion. 2. Respect for the Dead and Their Family: The death of a person, especially under mysterious circumstance...

    2024

  2. [2022]

    Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, and Mona Diab

    World values survey: Round seven – country- pooled datafile version 6.0. Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models.arXiv preprint arXiv:2402.13231. 9 Noam Benkler, Drisana Mosaphir, Scot...

    2024 arXiv

  3. [2024]

    Gemma Team

    Value kaleidoscope: Engaging ai with plural- istic human values, rights, and duties.Proceedings of the AAAI Conference on Artificial Intelligence, 38(18):19937–19947. Gemma Team. 2025a. Gemma 3. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue ...

    2025 arXiv

  4. [2025]

    Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov

    Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning.Preprint, arXiv:2501.12948. Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. 2024. Modular pluralism: Pluralistic align- ment via mul...

    2024 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.