Pith. sign in

REVIEW 3 major objections 5 minor 16 references

9B model matches humans on agreement, fails on validity

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 02:05 UTC pith:HCMSMHTL

load-bearing objection AMALIA-9B matches human coders on agreement but fails a construct-validity test; the gap may reflect decomposition difficulty rather than shortcut reliance. the 3 major comments →

arxiv 2607.08731 v1 pith:HCMSMHTL submitted 2026-07-09 cs.CL cs.AIcs.CY

Validity of LLMs as data annotators: AMALIA on authority

Manuel Pita This is my paper
classification cs.CL cs.AIcs.CY
keywords agreementamaliaauthorityconstructcorpusinstrumentmodelmodels
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

A language model can produce annotation codes that agree with trained human coders almost as well as models eight to thirteen times its size, yet still fail to measure the theoretical construct it was asked to code. This paper demonstrates that distinction on Portugal's national 9-billion-parameter model, AMALIA, using the moral foundation of authority as a test case. The central mechanism is the recovery gap: when the model's holistic prompt is decomposed into the atomic clauses the construct's theory defines and recombined by an explicit rule, AMALIA recovers only about half of its original performance. A larger multilingual model closes the same gap on the same Portuguese corpus under identical instructions, which localises the failure to the construct-model pair rather than to the corpus or the language. Error analysis shows AMALIA tends to code texts as authority when it detects moral outrage near a powerful figure, rather than tracking the evaluative stance towards hierarchy that the construct requires. The paper argues that agreement with human coders is a measure of reliability, not validity, and that sovereign-LLM programmes should test the evidential route by which agreement is achieved, not just the agreement itself.

Core claim

The recovery gap, defined as the difference between a model's performance with a holistic prompt and its performance when that prompt is decomposed into theory-specified atomic clauses and recombined by an explicit integration rule, stays open for AMALIA-9B (Δ = +0.358 under Portuguese instructions, +0.436 under English instructions) while closing for a 120B-parameter model on the same corpus (Δ = +0.028). This means AMALIA reaches correct codes via surface correlates rather than the inferences the construct's theory specifies, and the failure is attributable to the model rather than to the corpus or the instructions.

What carries the argument

The recovery gap Δ = F1_undecomposed − F1_decomposed, where the decomposed prompt breaks the construct into atomic yes/no clauses (detection, distinction, appraisal) recombined by a Boolean integration rule derived from moral foundations theory. The gap measures how much of a model's agreement with human coders survives decomposition into the theory's own evidential route.

Load-bearing premise

The recovery gap is interpreted as evidence that the model relies on surface shortcuts, but this interpretation assumes that the clause decomposition calibrated on a 120-billion-parameter model is at an executable grain for a 9-billion-parameter model. If the decomposition is simply too fine for AMALIA's capacity, the gap would reflect decomposition difficulty rather than shortcut reliance, and the validity conclusion would not follow.

What would settle it

If AMALIA were recalibrated with a coarser clause decomposition matched to its own capacity and still showed an open recovery gap, the validity failure would be confirmed. Conversely, if a coarser calibration closed the gap, the failure would be attributed to grain mismatch rather than to shortcut reliance.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper evaluates Portugal's national language model, AMALIA-9B, as a measurement instrument for annotating the moral foundation of authority/subversion. The central methodological contribution is the recovery gap (Δ = F1_undecomposed − F1_decomposed), which measures how much of a model's holistic annotation performance survives when the prompt is decomposed into theory-specified atomic clauses and recombined via an explicit integration rule. A calibrated English instrument (from prior work by the same author, Pita 2026) is transferred to AMALIA-9B and to a European Portuguese transcreation of the Moral Foundations Reddit Corpus (748 texts). The study is pre-registered with out-of-sample confirmatory tests on 448 unseen texts. The main finding: AMALIA agrees with human coders within ~6 F1 points of much larger models (Llama-3.3-70B, GPT-OSS-120B), but its recovery gap remains open (Δ_pt = +0.358, Δ_en = +0.436), meaning decomposition recovers only about half of holistic performance. GPT-OSS-120B closes the gap on the same Portuguese corpus (Δ = +0.028), isolating the model rather than the corpus or construct as the failure locus. Error analyses (S1–S3) suggest AMALIA relies on surface correlates such as moral outrage near authority figures.

Significance. The paper addresses a genuinely important question for the growing class of national/sovereign LLMs: whether agreement with human annotators suffices to certify a model as a valid measurement instrument for theoretical constructs, or whether the evidential route by which codes are produced must also be tested. The recovery-gap framework is a principled diagnostic that goes beyond standard agreement metrics, and the pre-registered, out-of-sample design with bootstrap CIs and reference-model controls is commendable. The transcreation pipeline (blind generation, five-criterion verification gate, human adjudication, back-translation audit) is a reusable methodological contribution for cross-linguistic construct validation. The cost-effectiveness of the entire study (~€30, 2.35 GPU-hours) demonstrates that validity auditing is feasible at any programme's budget. The policy implication—that sovereign-LLM benchmark batteries should test construct validity, not just agreement—is timely and well-argued. The paper ships open data, code, and pre-registration, enabling independent replication.

major comments (3)
  1. §5 and the S3 result (Table 3) expose a confound that is not fully resolved. The detection clause D_entity fires on only 17.9% (pt) / 12.3% (en) of texts in a corpus where ~49% are authority-positive. Since the integration rule (Expression 1) requires D for the A1/A2 path, an overly conservative D will mechanically suppress true positives in the decomposed condition, inflating Δ regardless of whether the holistic prompt uses shortcuts. The paper acknowledges this ('A construct calibrated to detection grade on a 120-billion-parameter model exceeds what a nine-billion-parameter model can execute clause by clause', §5), but then continues to interpret the gap as evidence of shortcut reliance. The two interpretations—shortcut reliance vs. decomposition grain mismatch—are not disentangled. The S1/S2 error analyses provide some independent evidence for the shortcut interpretation, but they are
  2. The S1 and S2 error analyses rest on an LLM reading panel (Claude Fable 5) with 63% pre-adjudication agreement, and the two readers are instances of the same model prompted under opposed lenses. The paper acknowledges this measures prompt-induced divergence, not independent human judgement (§5). While the controls (blind to ground truth, correct-rejection controls, agreement-only subsample raising S1 to 88%) are reasonable, the 63% agreement rate is low enough that the S1/S2 findings should be treated as suggestive rather than confirmatory. The paper labels them exploratory, which is appropriate, but the abstract and conclusion lean on 'error analysis suggests reliance on surface correlates' as if it were more settled than the evidence supports. The authors should either soften the framing in the abstract/conclusion or provide human-rater validation on a subset.
  3. The study evaluates a single construct (authority) on a single corpus (MFRC, Reddit register). The paper acknowledges this ('a single counterexample, not a verdict on national models'), but the policy recommendation in §5—that sovereign-LLM programmes should adopt recovery-gap batteries—is strong relative to the evidence base of one construct. The authority construct may be unusually dependent on relational inference (who obeys whom, stance appraisal) in ways that make it a hard case for decomposition transfer. A second construct (e.g., harm/care, which has a different inferential structure) would substantially strengthen the claim that the finding generalizes. This is not necessarily blocking for a single-construct study, but the policy framing should be calibrated to the evidence scope.
minor comments (5)
  1. Table 4 reports full-corpus (748-text) figures while Table 3 reports confirmatory (448-text) figures. Figure 1 plots AMALIA at 448-text estimates and reference models at 748-text estimates. This mixing of sample sizes across the same comparison figure is defensible (AMALIA's confirmatory set is the pre-registered endpoint) but should be more prominently flagged in the figure caption or a footnote.
  2. The logistic regression result in §4.2 ('β≈1.65' for the equality distinction clause, '30% of the undecomposed prompt's positives fired no clause at all') is described as exploratory but is one of the most informative diagnostics in the paper. Consider promoting it to a table or giving it more space, as it directly speaks to the shortcut mechanism.
  3. The paper uses 'transcreation' without defining it until §3 ('translation that preserves referents, stance and illocutionary force while adapting idiom and register'). Moving this definition earlier (e.g., to the introduction) would aid readers unfamiliar with the term.
  4. The reference to 'Claude Fable 5' (model id claude-fable-5, July 2026) in §3 may not be verifiable by readers at publication time. If this model is not publicly released, the S1/S2 protocol's reproducibility is compromised. A note on availability or a fallback model specification would help.
  5. §3, footnote 4: the deduplication artefact in the MFRC (37.6% reduction in positive codes) is significant and should be mentioned in the main text, not only in a footnote, as it affects the corpus's positive-class prevalence and thus the baseline for all F1 comparisons.

Circularity Check

0 steps flagged

No significant circularity; self-citation provides methodology but does not force the empirical result

full rationale

The paper's central claim — that AMALIA-9B exhibits a large recovery gap (Δ ≈ 0.35–0.47) indicating reliance on surface correlates rather than theory-specified inferences — is an empirical finding, not a consequence of the method's construction. The recovery gap (Expression 2: Δ = F1_undecomposed − F1_decomposed) is a mechanical computation over two independently obtained F1 scores against external ground truth (MFRC human annotations). The method (grain calibration, recovery gap, clause decomposition) and the calibrated English instrument both come from Pita 2026, a self-citation by the sole author. This self-citation is load-bearing for the methodology. However, it does not force the result: the same method produces Δ ≈ 0 for GPT-OSS-120B on the same Portuguese corpus under the same instructions, demonstrating that the framework can yield both 'pass' and 'fail' outcomes. The integration rule (Expression 1) is derived from moral foundations theory (Graham et al. 2013; Atari et al. 2023), not from the author's prior work. The English calibration was performed against external ground truth on other models (GPT-OSS, Llama), not on AMALIA's data. The interpretation of a large Δ as evidence of shortcuts is explicitly stated to require independent error analysis (S1/S2/S3), which the paper provides — these are exploratory but not circular. The paper's own acknowledgment that the clause grain calibrated on a 120B model may be too fine for a 9B model is a validity concern (correctness risk), not a circularity: the gap is still mechanically computed, and the alternative explanation (decomposition difficulty) is a competing hypothesis the paper discusses, not a tautology. No step in the derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The framework's core entities and assumptions come from Pita 2026. The present paper adds no new invented entities but relies on the prior work's operationalization of construct validity through decomposition.

free parameters (2)
  • Interpretation bands for recovery gap = Δ≥0.10 open, Δ<0.05 closed
    Pre-registered thresholds that define what counts as a validity failure. These are set by the author's prior work, not derived from theory.
  • S3 detection clause firing rate bound = <40%
    Pre-registered threshold for the D_entity clause firing rate, set from pilot data.
axioms (4)
  • domain assumption The recovery gap (F1_undecomposed - F1_decomposed) is a valid diagnostic for construct validity: if decomposition fails to reproduce holistic performance, the model is using shortcuts rather than theory-specified inferences.
    §2: The paper assumes that the recovery gap measures whether the model follows the construct's theory. This is the foundational assumption of the entire framework, from Pita 2026.
  • domain assumption The clause decomposition (7 binary clauses, 3 components, integration rule) faithfully represents moral foundations theory's authority/subversion construct.
    §2, Table 1, Logical Expression 1: The decomposition into D, A1, A2, A3 clauses and the rule (D∧(A1∨A2))∨A3⇒authority is asserted to capture the theory. This is the author's operationalization, validated in prior work.
  • domain assumption Ground-truth codes from the English MFRC transfer validly to the European-Portuguese transcreated corpus.
    §3: 'The ground-truth codes are inherited from the source texts, and the validity of that transfer is defended by verification, not assumed.' The paper acknowledges residual drift risk.
  • ad hoc to paper The LLM reading panel (Claude Fable 5) can reliably classify the basis of annotation errors.
    §3, §5: S1 and S2 endpoints rely on an LLM panel with 63% pre-adjudication agreement to classify error bases and evidence grounding. The paper acknowledges this is exploratory.
invented entities (1)
  • Recovery gap (Δ) independent evidence
    purpose: Quantifies the divergence between undecomposed and decomposed prompt performance as a validity diagnostic
    The metric is computed against external ground truth (MFRC human annotations) and is falsifiable: a model that closes the gap (GPT-OSS-120B, Δ=0.028) demonstrates the gap is not tautological.

pith-pipeline@v1.1.0-glm · 20186 in / 4158 out tokens · 255280 ms · 2026-07-10T02:05:32.260060+00:00 · methodology

0 comments
read the original abstract

A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size. Yet agreement is reliability, not validity. For theoretical constructs that must be inferred rather than read from surface features, the question is whether the model follows the construct's theory or reaches the right code by correlated shortcuts. We test this with the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined by the theory's explicit rule. If calibration closes that gap, some portability should survive across models and languages; where it does not, the construct-model instrument is the likely locus of failure. We ask whether a calibrated English instrument transfers to AMALIA-9B and to European Portuguese. For one construct and one corpus, it does not. Decomposition recovers only about half of AMALIA's holistic performance, and error analysis suggests reliance on surface correlates, especially moral outrage near authority figures. An open multilingual LLM closes the gap on the same Portuguese corpus under the same instructions, pointing away from the corpus as the main explanation. AMALIA can still screen and pre-code at scale, but it cannot yet measure this construct well enough to stand alone. The study is a single counterexample, not a verdict on national models; it argues that sovereign-LLM benchmark batteries should test not only agreement with human coders, but the evidential route by which that agreement is warranted.

Figures

Figures reproduced from arXiv: 2607.08731 by Manuel Pita.

Figure 1
Figure 1. Figure 1: Recovery gap ∆ = F u 1 − F d 1 for each model; larger ∆ means more of the model’s agreement with human coders is not reproduced when the construct is decomposed into its theory clauses, that is, a larger validity shortfall. AMALIA-9B is plotted at its 448-text confirmatory gaps with 95% bootstrap confidence intervals (the pre-registered endpoint in [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · 3 internal anchors

  1. [1]

    Mohammad Atari, Jonathan Haidt, Jesse Graham, Sena Koleva, Sean T

    doi: 10.1093/pnasnexus/pgae245. Mohammad Atari, Jonathan Haidt, Jesse Graham, Sena Koleva, Sean T. Stevens, and Morteza Dehghani. Morality beyond the WEIRD: How the nomological network of morality varies across cultures.Journal of Personality and Social Psychology, 125(5):1157–1188,

    work page doi:10.1093/pnasnexus/pgae245

  2. [2]

    doi: 10.1037/pspp0000470

    ISSN 0022-3514. doi: 10.1037/pspp0000470. URLhttps://doi.org/10.1037/pspp0000470. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    work page doi:10.1037/pspp0000470 1901

  3. [3]

    FotiosFitsilis, MariaKamilaki, BasilisGatos, VassilisKatsouros, andGeorgeMikros

    doi: 10.1037/h0040957. FotiosFitsilis, MariaKamilaki, BasilisGatos, VassilisKatsouros, andGeorgeMikros. TheHellenicParliament’s Approach to Digital Sovereignty for the Era of Artificial Intelligence.International Journal of Parliamentary 13 Validity of LLMs as data annotatorsA Preprint Studies, 6(1):155–167,

    work page doi:10.1037/h0040957

  4. [4]

    doi: 10.1163/26668912-bja10127

    ISSN 2666-8912. doi: 10.1163/26668912-bja10127. URLhttps://doi.org/ 10.1163/26668912-bja10127. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. ChatGPT outperforms crowd workers for text- annotation tasks.Proceedings of the National Academy of Sciences, 120(30):e2305016120,

    work page doi:10.1163/26668912-bja10127

  5. [5]

    Gilardi, M

    doi: 10.1073/pnas.2305016120. Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P. Wojcik, and Peter H. Ditto. Moral foundations theory: The pragmatic validity of moral pluralism. In Patricia Devine and Ashby Plant, editors,Advances in Experimental Social Psychology, volume 47, pages 55–130. Academic Press,

    work page doi:10.1073/pnas.2305016120

  6. [6]

    Suramya Jadhav, Abhay Shanbhag, Amogh Thakurdesai, Ridhima Sinare, and Raviraj Joshi

    doi: 10.1016/B978-0-12-407236-7.00002-4. Suramya Jadhav, Abhay Shanbhag, Amogh Thakurdesai, Ridhima Sinare, and Raviraj Joshi. On limitations of llm as annotator for low resource languages. InProceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025), pages 277–282,

    work page doi:10.1016/b978-0-12-407236-7.00002-4 2025

  7. [7]

    EuroLLM-9B: Technical Report

    Pedro Henrique Martins, João Alves, Patrick Fernandes, Nuno M Guerreiro, Ricardo Rei, Amin Farajian, Mateusz Klimaszewski, Duarte M Alves, José Pombal, Nicolas Boizard, et al. Eurollm-9b: Technical report. arXiv preprint arXiv:2506.04079,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    David Eduardo Pereira, Daniela Thuaslar Simão Gomes, and Claudio E

    doi: 10.1037/0003-066X.50.9.741. David Eduardo Pereira, Daniela Thuaslar Simão Gomes, and Claudio E. C. Campelo. Evaluating LLMs on Argument Mining Tasks in Brazilian Portuguese Debate Data.Journal of the Brazilian Computer Society, 31(1):1279–1299,

    work page doi:10.1037/0003-066x.50.9.741

  9. [9]

    doi: 10.5753/jbcs.2025.5824

    ISSN 1678-4804. doi: 10.5753/jbcs.2025.5824. URLhttps://doi.org/10.5753/ jbcs.2025.5824. Kunat Pipatanakul and Pittawat Taveekitworachai. Typhoon-S: Minimal Open Post-Training for Sovereign Large Language Models. Arxiv preprint 10.48550/arXiv.2601.18129,

    work page doi:10.5753/jbcs.2025.5824 2025

  10. [10]

    Manuel Pita

    URLhttps://arxiv.org/abs/ 2601.18129. Manuel Pita. Correct codes for the wrong reasons? Validating LLMs as measurement instruments for theoretical constructs.arXiv preprint,

    work page arXiv

  11. [11]

    Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs

    doi: 10.48550/arXiv.2606.28574. URL https://arxiv.org/ abs/2606.28574. Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguis...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2606.28574 2025

  12. [12]

    NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models

    doi: 10.18653/v1/2025.naacl-long.120. URL https://arxiv.org/abs/2404.12464. Steve Rathje, Dan-Mircea Mirea, Ilia Sucholutsky, Raja Marjieh, Claire E. Robertson, and Jay J. Van Bavel. GPT is an effective tool for multilingual psychological text analysis.Proceedings of the National Academy of Sciences, 121(34):e2308950121,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2025.naacl-long.120 2025

  13. [13]

    Rathje,et al., GPT is an effective tool for multilingual psychological text analysis

    doi: 10.1073/pnas.2308950121. André da Fonseca Schuck, Gabriel Lino Garcia, João Renato Ribeiro Manesco, Pedro Henrique Paiola, and João Paulo Papa. Evaluating Large Language Models for Brazilian Portuguese Sentiment Analysis: A Comparative Study of Multilingual State-of-the-Art vs. Brazilian Portuguese Fine-Tuned LLMs.Journal of the Brazilian Computer So...

    work page doi:10.1073/pnas.2308950121

  14. [14]

    doi: 10.5753/jbcs.2025.5793

    ISSN 1678-4804. doi: 10.5753/jbcs.2025.5793. URLhttps://doi.org/10.5753/jbcs.2025.5793. Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Glória-Silva, Miguel Faria, Marcos Treviso,...

    work page doi:10.5753/jbcs.2025.5793 2025

  15. [15]

    ISBN 979-8-89176-387-6

    Association for Computational Linguistics. ISBN 979-8-89176-387-6. URLhttps://aclanthology.org/2026.propor-1.38/. Petter Törnberg. Large language models outperform expert coders and supervised classifiers at annotating political social media messages.Social Science Computer Review, 43(6):1181–1195,

    work page 2026

  16. [16]

    doi: 10.48550/arXiv.2208.05545. 15

    work page doi:10.48550/arxiv.2208.05545