REVIEW 4 major objections 5 minor 17 references
Inducing Human-like Biases in Moral Reasoning Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Fine-tuning BERT, RoBERTa, and DeBERTa on moral-reasoning data, fMRI data, or both does not raise their BrainScore against human fMRI activity; larger models are more brain-aligned to begin with.
desk verdict First BrainScore measurement for moral reasoning reports a null fine-tuning effect, but the null claim is under-supported without sensitivity or significance tests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing instrument is the BrainScore metric: a Pearson correlation between predicted and actual fMRI activity, where predictions come from a fitted regression that maps [CLS] token activations of a language model to 1,024 regions of interest in the DiFuMo functional atlas. The fMRI stimulus set is a published moral-judgment dataset, preprocessed through a self-supervised learning pipeline, with four temporal sampling strategies (average of all time points, LAST, MIDDLE, SENTENCES) and four region groups (Theory of Mind, moral, language, vision) used as target and control partitions. The metric's job is to detect whether fine-tuning shifts the model's internal geometry toward the geometry of human moral reasoning; the paper's conclusion is that, at this resolution, it does not.
What would settle it
Re-running the LAST-sampling fine-tuning protocol on a moral-reasoning fMRI dataset with, say, ten times more held-out trials, and observing a BrainScore improvement over the pretrained baseline that exceeds the standard deviations in Table 2, would overturn the paper's central claim; a positive-control experiment that recovers a known injected alignment signal would rule out measurement insensitivity.
Extended reading notes
Core claim
On the authors' own terms, the discovery is a measured absence: fine-tuning language models on moral-reasoning data does not make their [CLS]-token representations more predictive of human fMRI activity. Using a BrainScore protocol that regresses model activations onto 1,024 DiFuMo atlas parcels and correlates predictions with held-out fMRI, the authors find no significant improvement for any model or sampling scheme. BERT-large-cased goes from 0.217 to 0.213, RoBERTa-large from 0.173 to between 0.144 and 0.156, and DeBERTa-v2-xlarge from 0.271 to between 0.237 and 0.273, all within one standard deviation of the baseline. The same fine-tuning does raise ETHICS commonsense accuracy, sometimes above the scores reported for the original benchmark, so the null is specific to neural alignment rather than to learning in general. The paper is also the first to attempt this measurement in the moral-reasoning domain.
Load-bearing premise
The BrainScore measurement assumes that regressing [CLS] token activations onto 1,024 DiFuMo atlas parcels and correlating predictions with held-out fMRI activity is sensitive enough to detect a genuine improvement in representational alignment; if the regression fit is noisy, the held-out set is small, or the hemodynamic lag is mis-specified, the observed null result could be a false negative rather than evidence that fine-tuning cannot work.
Editorial extensions
If this is right
- Model scale, not task-specific fine-tuning, is the main driver of brain alignment in moral reasoning: the largest pretrained model's BrainScore exceeds the fine-tuned scores of smaller models.
- Adding fMRI data to ETHICS fine-tuning does not improve ETHICS accuracy beyond ETHICS-only fine-tuning, so neural data is not an obvious free lunch for task performance.
- BrainScore and benchmark accuracy decouple: a model can learn the ETHICS task while its representational alignment with fMRI stays flat.
- The null result is stable across fMRI sampling methods (average, LAST, MIDDLE, SENTENCES), making it unlikely to be an artifact of a single time-point choice.
- Researchers pursuing brain-aligned moral AI should expect that current fine-tuning recipes and dataset sizes are insufficient, and should treat moral-reasoning neural alignment as an open problem.
Reading between the lines
- Inference: the flat BrainScore may reflect that the [CLS]-head readout used for prediction discards the layer-wise features where alignment could change; probing individual attention heads or intermediate layers might reveal fine-tuning effects the aggregate score masks.
- Inference: the result is consistent with the idea that representational convergence with brains is primarily a pretraining-scale phenomenon; a decisive test would be to compare BrainScore across model families at matched parameter counts before and after much larger-scale moral fine-tuning.
- Inference: a promising next experiment is to fine-tune on naturalistic moral narratives or on a contrastive objective that aligns intermediate layers directly to fMRI parcels, rather than on classification labels, and to check whether BrainScore rises on held-out subjects.
- Inference: because the held-out fMRI set is small and the regression is fit per atlas parcel, the null could be a false negative; re-running the same protocol on a moral fMRI dataset with more subjects or trials would be the most direct way to test the robustness of the conclusion.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript asks whether fine-tuning encoder language models (BERT-base/large, RoBERTa-large, DeBERTa-v2-xlarge) on moral reasoning behavioral data (ETHICS), fMRI data (Koster-Hale et al. 2013), or both, improves their BrainScore, defined as the correlation between model activations and held-out fMRI activity fitted to 1,024 DiFuMo parcels. The authors report two main findings: (1) fine-tuning can improve ETHICS Commonsense accuracy, sometimes beyond the original benchmark report, but adding fMRI data does not consistently beat ETHICS-only fine-tuning; and (2) BrainScores do not significantly improve over pre-trained models after any fine-tuning protocol, while larger models tend to have higher BrainScores. The paper concludes that the negative result supports the need for larger moral reasoning datasets.
Significance. The paper is, to my knowledge, the first to measure and attempt to improve BrainScore in the moral reasoning domain, and the scale of the fine-tuning effort (450 runs, four models) is a strength. If the null result were rigorously established, it would be a useful cautionary data point for the NeuroAI literature and for claims that human brain data can straightforwardly induce human-like representations in LLMs. The authors also release their code and are unusually explicit about the failure of their intervention. The main limitation is that the null result is only as informative as the BrainScore measurement, and the manuscript does not currently demonstrate that this measurement is sensitive to the targeted moral/ToM dimension or that the observed differences are statistically meaningful.
major comments (4)
- [Section 4 / Table 2] The central negative claim that fine-tuning does not significantly improve BrainScore is not supported by inferential statistics. The table reports only means and standard deviations, and the standard deviations (0.09–0.12) are as large as or larger than the differences being interpreted (e.g., DeBERTa 0.271 vs 0.273 and 0.237; RoBERTa 0.173 vs 0.144–0.156). I recommend reporting per-run paired comparisons between fine-tuned and pre-trained models, confidence intervals on the differences, and, ideally, a noise ceiling for the BrainScore metric so that a null result can be distinguished from an insensitive measurement.
- [Section 3.4 / Appendix B] The interpretation of the null result depends on the sensitivity of BrainScore to the moral-reasoning/ToM dimension, and the paper's own Appendix B states that 'our models were not able to achieve better brain scores than the control group, meaniing that our experiment did not achieve the desired effect.' Since vision was chosen as a control expected not to improve, this admission suggests that the measured alignment may be dominated by generic stimulus- or language-related variance rather than by moral reasoning. Please provide evidence that the metric is sensitive to the targeted dimension, for example a positive control where a known alignment-boosting manipulation increases BrainScore, or a demonstration that BrainScore tracks ETHICS accuracy across models.
- [Section 3.4] The BrainScore pipeline is described only qualitatively, which makes the headline null result difficult to evaluate or reproduce from the text. The manuscript should specify the regression inputs (which layers and tokens enter the regressor), the train/test split (subject-level or item-level, number of examples), the number of fMRI time points and the hemodynamic lag values, the cross-validation scheme, and the exact formula for aggregating layer-wise scores. Without these details, it is hard to rule out overfitting of the 1,024-parcel regression or misspecification of the temporal alignment as explanations for the null result.
- [Section 4 / Table 1] The claim that 'we could not improve accuracy by fine-tuning on the fMRI data only or on a combination of fMRI and ETHICS, compared to fine-tuning purely on ETHICS' is based on small numbers of runs with wide, overlapping confidence intervals (e.g., RoBERTa-large has 4 runs for ETHICS-only and 18 for the combined condition, with CS test set intervals of 53.0–91.4 and 49.7–91.6). This does not establish equivalence; a paired or Bayesian comparison across matched runs is needed before concluding that fMRI fine-tuning has no effect on ETHICS accuracy.
minor comments (5)
- [Section 3.2] The text refers to 'Appendix A' for the vision control group, but Appendix A is the Author Contributions section; the relevant material appears in Appendix B. Please correct the cross-reference.
- [Appendix B] There is a typo in the sentence 'our models were not able to achieve better brain scores than the control group, meaniing that our experiment did not achieve the desired effect'; 'meaniing' should be 'meaning'.
- [Table 3] The abbreviation 'A VG' in the Sampling column is not defined in the table caption; Section 4 defines 'VG' as an average of all time points, but the table should be self-contained.
- [Appendix B (Figures 13–20)] The figure captions mention one-tailed Bonferroni-corrected significance, but the text never states which test was used, how many comparisons were corrected for, or whether the comparisons are across models, layers, or subjects. Please add this information.
- [Table 2] The column header 'Brain Score Mean Brain Score St. Dev.' is awkwardly worded and ambiguous; splitting it into 'Mean Brain Score' and 'Standard Deviation' would improve readability.
Circularity Check
No significant circularity: BrainScore is measured on held-out fMRI data after regression fitting, and fine-tuning uses separate splits, so no prediction reduces to its inputs by construction.
full rationale
The paper's central measurement, BrainScore (Section 3.4), is not defined in terms of the quantities it is used to test. For each model, activations are regressed onto fMRI parcel time courses on a fitting split, and the Pearson correlation is computed against held-out brain activity; the fine-tuned models are evaluated with the same held-out protocol, so a fine-tuned model's score is not forced by construction (Table 2). The ETHICS accuracies in Table 1 also use standard train/test splits. No fitted parameter is renamed as a prediction, and no load-bearing claim is justified only by a self-citation. The absence of significance tests and the sensitivity of the metric are validity concerns, not circularity. The Appendix B admission that vision-control scores were not beaten weakens the interpretation of the null result but does not make the derivation circular. The paper is self-contained as an empirical study, so the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption fMRI BOLD activity in DiFuMo parcels reflects the cognitive processes involved in moral reasoning.
- domain assumption The [CLS] token's hidden activations provide a comparable representation to whole-brain fMRI patterns.
- domain assumption Hemodynamic lag sampling correctly aligns text stimulus times to fMRI responses.
- standard math Pearson correlation and linear regression are sufficient statistics for representational alignment.
Cite this review
Pith. "Pith review of Inducing Human-like Biases in Moral Reasoning Language Models." pith.science (2026). https://pith.science/paper/ATCXOI7I
@misc{pith2026241115386,
author = {Pith},
title = {Pith review of: Inducing Human-like Biases in Moral Reasoning Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/ATCXOI7I}},
note = {Machine review of arXiv:2411.15386}
}
read the original abstract
In this work, we study the alignment (BrainScore) of large language models (LLMs) fine-tuned for moral reasoning on behavioral data and/or brain data of humans performing the same task. We also explore if fine-tuning several LLMs on the fMRI data of humans performing moral reasoning can improve the BrainScore. We fine-tune several LLMs (BERT, RoBERTa, DeBERTa) on moral reasoning behavioral data from the ETHICS benchmark [Hendrycks et al., 2020], on the moral reasoning fMRI data from Koster-Hale et al. [2013], or on both. We study both the accuracy on the ETHICS benchmark and the BrainScores between model activations and fMRI data. While larger models generally performed better on both metrics, BrainScores did not significantly improve after fine-tuning.
Figures
Figures from the paper (17 more)
Reference graph
Works this paper leans on
-
[5]
doi: 10.1038/s41583-023-00705-w
ISSN 1471-0048. doi: 10.1038/s41583-023-00705-w. URL https: //www.nature.com/articles/s41583-023-00705-w . Publisher: Nature Publishing Group. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with Disentangled Attention, October
-
[7]
URL http://arxiv.org/ abs/2008.02275. arXiv:2008.02275 [cs]. Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis,
arXiv 2008
-
[10]
URL http://arxiv.org/abs/2105.08209. arXiv:2105.08209 [cs]. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach, July
-
[11]
URL http://arxiv.org/abs/1907.11692. arXiv:1907.11692 [cs]. Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-Score: Which Artificial Neural Network for Object Recognition is most ...
arXiv 1907
-
[12]
Pages: 407007 Section: New Results
URL https://www.biorxiv.org/content/ 10.1101/407007v2. Pages: 407007 Section: New Results. Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias in natural language pro- cessing models, October
-
[13]
Inducing brain-relevant bias in natural language processing models
URL http://arxiv.org/abs/1911.03268. arXiv:1911.03268 [cs, q-bio]. Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C. Love, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Martin N. Hebart, Nori Jacoby, Qiuyi Zhang, Raja Ma...
work page Pith review arXiv 1911
-
[14]
URL http://arxiv.org/abs/2310.13018. arXiv:2310.13018 [cs, q-bio]. Armin W. Thomas, Christopher Ré, and Russell A. Poldrack. Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data, January
-
[15]
URL http://arxiv.org/abs/2206. 11417. arXiv:2206.11417 [q-bio]. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quent...
Show all 17 references
-
[16]
arXiv:1910.03771 [cs]
URL http://arxiv.org/abs/1910.03771. arXiv:1910.03771 [cs]. Tal Yarkoni, Russell A. Poldrack, Thomas E. Nichols, David C. Van Essen, and Tor D. Wager. Large-scale automated synthesis of human functional neuroimaging data. Nature Methods, 8 (8):665–670, August
1910 arXiv
-
[2011]
doi: 10.1038/nmeth.1635
ISSN 1548-7105. doi: 10.1038/nmeth.1635. URL https://www. nature.com/articles/nmeth.1635. Publisher: Nature Publishing Group. A Author Contributions Artem Karpov – implementation of the fine tuning, experiments for fine tuning, reports for the fine tuning experiments, most of ...
-
[2013]
URL https://www.pnas.org/ doi/10.1073/pnas.1207992110
doi: 10.1073/pnas.1207992110. URL https://www.pnas.org/ doi/10.1073/pnas.1207992110. Publisher: Proceedings of the National Academy of Sciences. Wojciech Kry´sci´nski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. BookSum: A Collection of Datasets for Lo...
-
[2019]
arXiv:1810.04805 [cs]
URL http://arxiv.org/ abs/1810.04805. arXiv:1810.04805 [cs]. Adrien Doerig, Rowan P. Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W. Lindsay, Konrad P. Kording, Talia Konkle, Marcel A. J. van Gerven, Nikolaus Kriegeskorte, and Tim C. Kietzmann. The neuroconnec...
-
[2020]
doi: 10.1016/j.neuroimage.2020.117126
ISSN 1053-8119. doi: 10.1016/j.neuroimage.2020.117126. URL https://www.sciencedirect.com/science/ article/pii/S1053811920306121. Joel Dapello, Kohitij Kar, Martin Schrimpf, Robert Geary, Michael Ferguson, David D. Cox, and James J. DiCarlo. Aligning Model and Macaque Inferior ...
2020
-
[2021]
arXiv:2006.03654 [cs]
URL http://arxiv.org/abs/2006.03654. arXiv:2006.03654 [cs]. 3Code available at: https://github.com/ajmeek/Inducing-human-like-biases-in-moral-reasoning-LLMs 5 Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning AI With...
2006 arXiv
-
[2022]
URL http://biorxiv.org/lookup/doi/10.1101/2022.07.01. 498495. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May
2022 doi
-
[2023]
arXiv:2212.10898 [cs, q-bio]
URL http://arxiv.org/abs/2212.10898. arXiv:2212.10898 [cs, q-bio]. Brian Christian. The alignment problem: Machine learning and human values . WW Norton & Company,
-
[2024]
Jorie Koster-Hale, Rebecca Saxe, James Dungan, and Liane L
URL https://arxiv.org/abs/2405.07987. Jorie Koster-Hale, Rebecca Saxe, James Dungan, and Liane L. Young. Decoding moral judgments from neural representations of intentions. Proceedings of the National Academy of Sciences , 110 (14):5648–5653, April
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.