REVIEW 4 major objections 5 minor 40 references
The way a loan application is written out as text — JSON, sentence fragments, or fluent prose — materially changes both how accurately LLMs approve loans and how evenly they treat male and female applicants, and in zero-shot mode none of th
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Serialization format and in-context examples change both accuracy and gender fairness of LLM loan approvals, with finance-tuned models often showing larger disparities.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection The benchmark is useful, but the fairness conclusions don't hold until degenerate predictions are separated from real disparities. the 4 major comments →
Accept or Deny? Evaluating LLM Fairness and Performance in Loan Approval across Table-to-Text Serialization Approaches
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On the paper's own terms, the central claim is that serialization — converting each tabular loan application into text — is a first-order factor in both performance and fairness of LLM-based loan approval. Across seven formats and ten open models, the natural-language formats GReaT and LIFT delivered the highest F1 scores in several settings but also the largest equality-of-opportunity gaps between gender groups. In zero-shot prompting every LLM scored below a logistic regression benchmark, several effectively approving or denying everyone; adding 2–8 gender-balanced in-context examples improved F1 by 4.9–59.6% yet did not uniformly reduce bias, pushing EO differences above 0.10 in some conf
What carries the argument
The machine doing the work is the serialization pipeline plus the fairness metrics it is measured against. Each applicant row is rendered in seven formats — JSON and List as bare key-value pairs, Text, HTML, and LaTeX as structured or marked-up strings, GReaT as 'feature is value' sentences, and LIFT as fluent prose ('A 32-year-old female is applying for a loan for 48 months...'). The same ten models read these strings zero-shot or with 2–8 gender-balanced in-context examples. Fairness is scored by equality of opportunity (true-positive rate equal across gender) and statistical parity (approval rate equal across gender), and the paper's headline is that these two differences move as much wit
Load-bearing premise
The fairness comparisons assume that gender-specific approval and true-positive rates estimated from roughly 120 test applicants in the smallest dataset are stable enough to compare; the paper reports them without confidence intervals, so some of the disparities it highlights could be sampling noise rather than systematic bias.
What would settle it
Recompute the equality-of-opportunity and statistical-parity values from the paper's own test splits with bootstrap confidence intervals (e.g., 1,000 resamples per model–format–dataset cell). If the 95% interval around FinMA-7B-full's 0.16 EO gap on Germany or the >0.10 gaps at n=8 on Ghana includes zero, the claim that serialization and financial fine-tuning systematically drive gender bias loses support; if the intervals exclude zero, the claim holds.
If this is right
- Practitioners should treat serialization choice as a decision variable with fairness consequences, not a formatting detail: the same model can flip from fair to biased purely from how the table is written out.
- Zero-shot LLM deployment for loan approval is not yet justified: every tested LLM underperformed logistic regression, and several effectively defaulted to uniform approval or denial.
- Financial fine-tuning does not automatically confer accuracy or fairness; the finance-tuned FinMA-7B-full model produced the largest gender disparities on Ghana and Germany.
- In-context learning is a performance lever but not a fairness guarantee: adding examples lifted F1 by up to 59.6% yet sometimes pushed equality-of-opportunity differences above 0.10.
- Robust evaluation of LLM lending tools requires multiple serializations and regionally diverse data, since format–dataset interactions can matter more than which model is used.
Where Pith is reading between the lines
- The authors leave implicit that the smallest test splits make several headline disparities statistically fragile: Ghana's 614 rows yield roughly 123 test rows, so a per-gender equality-of-opportunity gap like FinMA-7B-full's 0.16 could easily sit within sampling error; a bootstrap analysis would test whether the format–fairness link is systematic.
- A testable extension the paper does not run: prompt the same models with deliberately imbalanced in-context examples to separate the effect of example demographics from the effect of serialization format on fairness.
- If format–region interactions dominate model identity, a practical next step is serialization-robust inference — requiring a model's prediction to stay fixed across all seven formats and flagging formats where it flips — as a fairness guardrail in deployment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks ten open-source LLMs on loan-approval classification using three geographically distinct datasets (Ghana, Germany, U.S.), seven table-to-text serialization formats, and both zero-shot and few-shot in-context learning. Performance is measured by weighted F1 and fairness by equality of opportunity (EO) and statistical parity (SP) differences across gender. The central claims are that serialization format significantly affects both performance and fairness, that formats such as GReaT and LIFT improve F1 while worsening fairness disparities, that zero-shot LLMs generally underperform a logistic regression baseline and often default to uniform predictions, and that in-context learning improves F1 but has context-dependent fairness effects. The paper also discusses model size, instruction tuning, and prompt sensitivity, with supplementary token-attribution experiments.
Significance. If supported, the paper would provide a useful broad empirical resource: ten models, seven serializations, three datasets, and standard fairness metrics, with resources released on Hugging Face and a reproducible lm-eval harness. The descriptive observation that serialization and ICL are first-order variables for LLM loan approval is practically important, as is the recommendation to keep classical baselines as decision support. However, the headline fairness conclusions are currently stronger than the evidence: degeneracy of predictions, small per-group test samples, and the absence of uncertainty quantification all affect the load-bearing claims about 'amplifying' bias and about specific serialization formats 'exacerbating fairness disparities.' The paper is best treated as a valuable descriptive benchmark that needs targeted statistical and conceptual fixes before its fairness conclusions can be accepted.
major comments (4)
- [§3.5, Table 4, Fig. 5, Discussion] Fairness comparisons are confounded by degenerate predictions. A constant classifier has EO=SP=0 by definition, and the paper itself reports zero/one baselines with 0.00 on both metrics and later states that zero-shot LLMs 'frequently default to uniform approval or denial.' Many zero entries for LLMs in Table 4 and Fig. 5 may therefore reflect constant predictions, not fairness. Comparing FinMA-7B-full's nonzero EO/SP against such entries conflates predictive variability with bias. The claims that FinMA-7B-full 'amplifies historical gender bias' and that certain serializations 'exacerbate fairness disparities' are not established unless the analysis conditions on non-degeneracy (e.g., reports prediction entropy or positive rate, or restricts to models with non-constant outputs).
- [§3.5, Table 4, Fig. 5, §4.5] No confidence intervals, bootstrap estimates, or significance tests are reported for the fairness metrics. For the Ghana dataset the test set is about 20% of 614 rows (~123), so after splitting by gender, EO differences such as FinMA-7B-full's 0.16 on Germany or the various Ghana values are estimated from tens of applicants per group. The statement that serialization 'significantly affects' fairness, and the stronger assertion that financial models 'amplify existing historical gender bias,' require uncertainty quantification or at minimum explicit acknowledgment that the observed differences may be sampling noise. The same issue affects the few-shot fairness comparisons in Fig. 7.
- [Abstract, §4.4, Figs. 3-4] The abstract's numerical claim that ICL improves performance by 4.9-59.6% is not tied to a table or equation in the main text, and the supporting figures (Figs. 3-4) do not show per-condition error bars or confidence intervals. Since the size of the improvement varies widely across datasets and formats, this headline range needs an explicit definition of what is being compared, which model/format conditions are included, and how the variability across prompts or splits is accounted for. Without that, the number is not verifiable from the paper as written.
- [Discussion vs. Figure 2, §4.1-4.2] The summary statement that 'in zero-shot scenarios, all LLMs perform worse than a Logistic Regression model baseline' is contradicted by the paper's own Figure 2 caption and Section 4.2, which name GReaT on Ghana, List/Text on Germany, and Gemma-2-27b-it on the U.S. as showing 'modest improvements' over the logistic regression baseline. If the claim is intended only for the JSON serialization, it needs to say so explicitly; as written, the discrepancy between 'all' and 'most' undermines a supporting conclusion.
minor comments (5)
- [§4.10, Fig. 7] The text says 'In the German dataset... Few-shot examples (e.g., n=8) can lead to significant fairness disparities... reaching differences of over 0.10 for some serialization methods in the Ghana datasets.' The mixed German/Ghana reference is confusing and should be corrected.
- [Table 3, §4.5] The sentence that FinMA-7B-full is 'the only one fine-tuned for finance' contradicts Table 3, which lists both FinMA-7B-full and FinMA-7B-NLP as fine-tuned on financial data.
- [Throughout, Table 8] Inconsistent capitalizations appear: 'GReat'/'great' vs. 'GReaT', 'LlaMA-3-70B' vs. 'LLaMA-3-70B', and 'Latex' vs. 'LaTeX'. There is also a typo 'purpuse' in Table 8.
- [Appendix H] The token-attribution results are described as inconclusive and are relegated to the appendix; the main text should either briefly state this or remove references to explaining model behavior in the introduction. The appendix currently contains many large figures with limited interpretation.
- [§B] The evaluation setup mentions three prompts per dataset but does not describe how generations were parsed or what 'log-likelihood prediction' means for multi-token outputs. A short description of the parsing/filtering of predictions would improve reproducibility.
Circularity Check
No circularity: empirical benchmark against external ground-truth labels; no quantity is defined in terms of the target result.
full rationale
This paper is an empirical evaluation. The reported quantities (weighted F1, equality of opportunity, statistical parity) are measured from model predictions on held-out test splits against ground-truth labels. No parameter is fitted to the reported metrics and then presented as a prediction: the Logistic Regression baseline is trained with default scikit-learn parameters on the training split and used only as an external reference point, not as part of the LLM claims. In-context examples are selected by gender balance ('top-n examples balanced by gender'), not by optimizing F1 or fairness, so the fairness and performance outcomes are not forced by construction. The serialization formats are pre-existing external methods (JSON, GReaT, LIFT, Text, HTML, Latex) with citations to prior work; the paper does not define its dependent variables in terms of its independent variables. The zero/one baselines having zero EO/SP is a mathematical property of constant classifiers, but the paper reports these as baselines rather than deriving its central claim from that property. The skeptic's concern about degenerate predictions is a validity threat to fairness comparisons, not a circularity: it does not show that any claimed result reduces to its input by definition or by self-citation. Self-citations (Kanubala et al. 2024; Majumdar et al. 2025) are contextual and not load-bearing for the empirical conclusions. The derivation chain is therefore self-contained against external benchmarks and no circular step is present.
Axiom & Free-Parameter Ledger
axioms (5)
- domain assumption Gender is a binary attribute (male/female) and is the sole sensitive attribute evaluated
- domain assumption Weighted F1, EO, and SP are appropriate evaluation measures for loan approval decisions
- domain assumption The three datasets are comparable proxies for loan approval in their respective regions
- domain assumption LLM log-likelihood scoring over the possible label tokens adequately represents the model's decision
- domain assumption The three hand-written prompts per dataset adequately span prompt sensitivity
Cite this review
Pith. "Pith review of Accept or Deny? Evaluating LLM Fairness and Performance in Loan Approval across Table-to-Text Serialization Approaches." pith.science (2026). https://pith.science/paper/PHTSF7BB
@misc{pith2026250821512,
author = {Pith},
title = {Pith review of: Accept or Deny? Evaluating LLM Fairness and Performance in Loan Approval across Table-to-Text Serialization Approaches},
year = {2026},
howpublished = {\url{https://pith.science/paper/PHTSF7BB}},
note = {Machine review of arXiv:2508.21512}
}
read the original abstract
Large Language Models (LLMs) are increasingly employed in high-stakes decision-making tasks, such as loan approvals. While their applications expand across domains, LLMs struggle to process tabular data, ensuring fairness and delivering reliable predictions. In this work, we assess the performance and fairness of LLMs on serialized loan approval datasets from three geographically distinct regions: Ghana, Germany, and the United States. Our evaluation focuses on the model's zero-shot and in-context learning (ICL) capabilities. Our results reveal that the choice of serialization (Serialization refers to the process of converting tabular data into text formats suitable for processing by LLMs.) format significantly affects both performance and fairness in LLMs, with certain formats such as GReat and LIFT yielding higher F1 scores but exacerbating fairness disparities. Notably, while ICL improved model performance by 4.9-59.6% relative to zero-shot baselines, its effect on fairness varied considerably across datasets. Our work underscores the importance of effective tabular data representation methods and fairness-aware models to improve the reliability of LLMs in financial decision-making.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298--306
work page 2021
-
[4]
Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. 2025. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37:76930--76966
work page 2025
-
[5]
Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. 2024. Many-shot in-context learning. arXiv preprint arXiv:2404.11018
Pith/arXiv arXiv 2024
-
[6]
Carlos Alejandro Aguirre, Kuleen Sasse, Isabel Alyssa Cachola, and Mark Dredze. 2024. https://doi.org/10.18653/v1/2024.nlp4pi-1.4 Selecting shots for demographic fairness in few-shot learning with large language models . In Proceedings of the Third Workshop on NLP for Positive Impact, Miami, Florida, USA. Association for Computational Linguistics
-
[7]
Microsoft Research AI4Science and Microsoft Azure Quantum. 2023. The impact of large language models on scientific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361
Pith/arXiv arXiv 2023
-
[8]
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29
2016
-
[9]
Vadim Borisov, Kathrin Se ler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. 2022. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.06280
Pith/arXiv arXiv 2022
-
[10]
Han Ding, Yinheng Li, Junhao Wang, and Hang Chen. 2024. Large language model agent in financial trading: A survey. arXiv preprint arXiv:2408.06361
arXiv 2024
-
[11]
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. 2022. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35:11763--11784
work page 2022
-
[12]
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214--226
2012
-
[13]
Duanyu Feng, Yongfu Dai, Jimin Huang, Yifang Zhang, Qianqian Xie, Weiguang Han, Zhengyu Chen, Alejandro Lopez-Lira, and Hao Wang. 2023. Empowering many, biasing a few: Generalist credit scoring through large language models. arXiv preprint arXiv:2310.00566
Pith/arXiv arXiv 2023
-
[14]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. https...
-
[15]
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635--E3644
2018
-
[16]
FM Haque and Md Mahedi Hassan. 2024. Bank loan prediction using machine learning techniques. arXiv preprint arXiv:2410.08886
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[17]
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29
2016
-
[18]
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. 2023. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549--5581. PMLR
work page 2023
-
[19]
Noah Hollmann, Samuel M \"u ller, Katharina Eggensperger, and Frank Hutter. 2022. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848
Pith/arXiv arXiv 2022
-
[20]
Sukriti Jaitly, Tanay Shah, Ashish Shugani, and Razik Singh Grewal. 2023. Towards better serialization of tabular data for few-shot classification. arXiv preprint arXiv:2312.12464
Pith/arXiv arXiv 2023
-
[21]
Loan approval prediction dataset
Kaggle . Loan approval prediction dataset. https://www.kaggle.com/altruistdelhite04/loan-prediction-problem-dataset. Accessed: 2024-07-19
work page 2024
-
[22]
Deborah D Kanubala, Isabel Valera, and Kavya Gupta. 2024. Fairness beyond binary decisions: A case study on german credit. European Workshop on Algorithmic Fairness
work page 2024
-
[23]
Siavash Kazemian, Cosmin Munteanu, and Gerald Penn. 2022. https://doi.org/10.18653/v1/2022.finnlp-1.10 A taxonomical NLP blueprint to support financial decision making through information-centred interactions . In Proceedings of the Fourth Workshop on Financial Technology and Natural Language Processing (FinNLP), pages 89--98, Abu Dhabi, United Arab Emira...
-
[24]
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. 2020. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint arXiv:2009.07896
Pith/arXiv arXiv 2020
-
[25]
Yaxuan Kong, Yuqi Nie, Xiaowen Dong, John M Mulvey, H Vincent Poor, Qingsong Wen, and Stefan Zohren. 2024. Large language models for financial and investment management: Applications and benchmarks. Journal of Portfolio Management, 51(2)
work page 2024
-
[26]
Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of the ACM collective intelligence conference, pages 12--24
2023
-
[27]
Nikita Kozodoi, Johannes Jacob, and Stefan Lessmann. 2022. Fairness in credit scoring: Assessment, implementation and profit implications. European Journal of Operational Research, 297(3):1083--1094
work page 2022
- [28]
-
[29]
Meta. 2024. I ntroducing M eta L lama 3: T he most capable openly available L L M to date --- ai.meta.com. https://ai.meta.com/blog/meta-llama-3/. [Accessed 01-06-2024]
work page 2024
-
[30]
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Nick Barnes, and Ajmal Mian. 2023. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435
Pith/arXiv arXiv 2023
-
[31]
Roberto Navigli, Simone Conia, and Bj \"o rn Ross. 2023. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality, 15(2):1--21
work page 2023
-
[32]
Frank Gyimah Sackey and Peter Nkrumah Amponsah. 2018. Gender discrimination in commercial banks’ credit markets in ghana: a decomposition and counterfactual analysis. African Journal of Business and Economic Research, 13(2):121--140
work page 2018
-
[33]
Ananya Singha, Jos \'e Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. 2023. Tabular representation, noisy operators, and impacts on table structure understanding tasks in llms. arXiv preprint arXiv:2310.10358
Pith/arXiv arXiv 2023
-
[34]
Statlog. Statlog (german credit data). https://archive.ics.uci.edu/ml/datasets/Statlog+ Accessed: 2024-07-19
work page 2024
-
[35]
Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 645--654
2024
-
[36]
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. arXiv preprint arXiv:1906.08976
Pith/arXiv arXiv 2019
-
[37]
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Anto...
Pith/arXiv arXiv 2024
-
[38]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288
Pith/arXiv arXiv 2023
-
[39]
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. https://arxiv.org/abs/2306.05443 Pixiu: A large language model, instruction data and evaluation benchmark for finance . Preprint, arXiv:2306.05443
Pith/arXiv arXiv 2023
-
[40]
Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Jesujoba O Alabi, Xiaoyu Shen, Dietrich Klakow, and Marius Mosbach. 2024. The impact of demonstrations on multilingual in-context learning: A multidimensional analysis. arXiv preprint arXiv:2402.12976
work page internal anchor Pith review Pith/arXiv arXiv 2024
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.