REVIEW 4 major objections 5 minor 40 references
Accept or Deny? Evaluating LLM Fairness and Performance in Loan Approval across Table-to-Text Serialization Approaches
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The way a loan application is written out as text — JSON, sentence fragments, or fluent prose — materially changes both how accurately LLMs approve loans and how evenly they treat male and female applicants, and in zero-shot mode none of th
desk verdict The benchmark is useful, but the fairness conclusions don't hold until degenerate predictions are separated from real disparities. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machine doing the work is the serialization pipeline plus the fairness metrics it is measured against. Each applicant row is rendered in seven formats — JSON and List as bare key-value pairs, Text, HTML, and LaTeX as structured or marked-up strings, GReaT as 'feature is value' sentences, and LIFT as fluent prose ('A 32-year-old female is applying for a loan for 48 months...'). The same ten models read these strings zero-shot or with 2–8 gender-balanced in-context examples. Fairness is scored by equality of opportunity (true-positive rate equal across gender) and statistical parity (approval rate equal across gender), and the paper's headline is that these two differences move as much wit
What would settle it
Recompute the equality-of-opportunity and statistical-parity values from the paper's own test splits with bootstrap confidence intervals (e.g., 1,000 resamples per model–format–dataset cell). If the 95% interval around FinMA-7B-full's 0.16 EO gap on Germany or the >0.10 gaps at n=8 on Ghana includes zero, the claim that serialization and financial fine-tuning systematically drive gender bias loses support; if the intervals exclude zero, the claim holds.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that serialization — converting each tabular loan application into text — is a first-order factor in both performance and fairness of LLM-based loan approval. Across seven formats and ten open models, the natural-language formats GReaT and LIFT delivered the highest F1 scores in several settings but also the largest equality-of-opportunity gaps between gender groups. In zero-shot prompting every LLM scored below a logistic regression benchmark, several effectively approving or denying everyone; adding 2–8 gender-balanced in-context examples improved F1 by 4.9–59.6% yet did not uniformly reduce bias, pushing EO differences above 0.10 in some conf
Load-bearing premise
The fairness comparisons assume that gender-specific approval and true-positive rates estimated from roughly 120 test applicants in the smallest dataset are stable enough to compare; the paper reports them without confidence intervals, so some of the disparities it highlights could be sampling noise rather than systematic bias.
Editorial extensions
If this is right
- Practitioners should treat serialization choice as a decision variable with fairness consequences, not a formatting detail: the same model can flip from fair to biased purely from how the table is written out.
- Zero-shot LLM deployment for loan approval is not yet justified: every tested LLM underperformed logistic regression, and several effectively defaulted to uniform approval or denial.
- Financial fine-tuning does not automatically confer accuracy or fairness; the finance-tuned FinMA-7B-full model produced the largest gender disparities on Ghana and Germany.
- In-context learning is a performance lever but not a fairness guarantee: adding examples lifted F1 by up to 59.6% yet sometimes pushed equality-of-opportunity differences above 0.10.
- Robust evaluation of LLM lending tools requires multiple serializations and regionally diverse data, since format–dataset interactions can matter more than which model is used.
Reading between the lines
- The authors leave implicit that the smallest test splits make several headline disparities statistically fragile: Ghana's 614 rows yield roughly 123 test rows, so a per-gender equality-of-opportunity gap like FinMA-7B-full's 0.16 could easily sit within sampling error; a bootstrap analysis would test whether the format–fairness link is systematic.
- A testable extension the paper does not run: prompt the same models with deliberately imbalanced in-context examples to separate the effect of example demographics from the effect of serialization format on fairness.
- If format–region interactions dominate model identity, a practical next step is serialization-robust inference — requiring a model's prediction to stay fixed across all seven formats and flagging formats where it flips — as a fairness guardrail in deployment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks ten open-source LLMs on loan-approval classification using three geographically distinct datasets (Ghana, Germany, U.S.), seven table-to-text serialization formats, and both zero-shot and few-shot in-context learning. Performance is measured by weighted F1 and fairness by equality of opportunity (EO) and statistical parity (SP) differences across gender. The central claims are that serialization format significantly affects both performance and fairness, that formats such as GReaT and LIFT improve F1 while worsening fairness disparities, that zero-shot LLMs generally underperform a logistic regression baseline and often default to uniform predictions, and that in-context learning improves F1 but has context-dependent fairness effects. The paper also discusses model size, instruction tuning, and prompt sensitivity, with supplementary token-attribution experiments.
Significance. If supported, the paper would provide a useful broad empirical resource: ten models, seven serializations, three datasets, and standard fairness metrics, with resources released on Hugging Face and a reproducible lm-eval harness. The descriptive observation that serialization and ICL are first-order variables for LLM loan approval is practically important, as is the recommendation to keep classical baselines as decision support. However, the headline fairness conclusions are currently stronger than the evidence: degeneracy of predictions, small per-group test samples, and the absence of uncertainty quantification all affect the load-bearing claims about 'amplifying' bias and about specific serialization formats 'exacerbating fairness disparities.' The paper is best treated as a valuable descriptive benchmark that needs targeted statistical and conceptual fixes before its fairness conclusions can be accepted.
major comments (4)
- [§3.5, Table 4, Fig. 5, Discussion] Fairness comparisons are confounded by degenerate predictions. A constant classifier has EO=SP=0 by definition, and the paper itself reports zero/one baselines with 0.00 on both metrics and later states that zero-shot LLMs 'frequently default to uniform approval or denial.' Many zero entries for LLMs in Table 4 and Fig. 5 may therefore reflect constant predictions, not fairness. Comparing FinMA-7B-full's nonzero EO/SP against such entries conflates predictive variability with bias. The claims that FinMA-7B-full 'amplifies historical gender bias' and that certain serializations 'exacerbate fairness disparities' are not established unless the analysis conditions on non-degeneracy (e.g., reports prediction entropy or positive rate, or restricts to models with non-constant outputs).
- [§3.5, Table 4, Fig. 5, §4.5] No confidence intervals, bootstrap estimates, or significance tests are reported for the fairness metrics. For the Ghana dataset the test set is about 20% of 614 rows (~123), so after splitting by gender, EO differences such as FinMA-7B-full's 0.16 on Germany or the various Ghana values are estimated from tens of applicants per group. The statement that serialization 'significantly affects' fairness, and the stronger assertion that financial models 'amplify existing historical gender bias,' require uncertainty quantification or at minimum explicit acknowledgment that the observed differences may be sampling noise. The same issue affects the few-shot fairness comparisons in Fig. 7.
- [Abstract, §4.4, Figs. 3-4] The abstract's numerical claim that ICL improves performance by 4.9-59.6% is not tied to a table or equation in the main text, and the supporting figures (Figs. 3-4) do not show per-condition error bars or confidence intervals. Since the size of the improvement varies widely across datasets and formats, this headline range needs an explicit definition of what is being compared, which model/format conditions are included, and how the variability across prompts or splits is accounted for. Without that, the number is not verifiable from the paper as written.
- [Discussion vs. Figure 2, §4.1-4.2] The summary statement that 'in zero-shot scenarios, all LLMs perform worse than a Logistic Regression model baseline' is contradicted by the paper's own Figure 2 caption and Section 4.2, which name GReaT on Ghana, List/Text on Germany, and Gemma-2-27b-it on the U.S. as showing 'modest improvements' over the logistic regression baseline. If the claim is intended only for the JSON serialization, it needs to say so explicitly; as written, the discrepancy between 'all' and 'most' undermines a supporting conclusion.
minor comments (5)
- [§4.10, Fig. 7] The text says 'In the German dataset... Few-shot examples (e.g., n=8) can lead to significant fairness disparities... reaching differences of over 0.10 for some serialization methods in the Ghana datasets.' The mixed German/Ghana reference is confusing and should be corrected.
- [Table 3, §4.5] The sentence that FinMA-7B-full is 'the only one fine-tuned for finance' contradicts Table 3, which lists both FinMA-7B-full and FinMA-7B-NLP as fine-tuned on financial data.
- [Throughout, Table 8] Inconsistent capitalizations appear: 'GReat'/'great' vs. 'GReaT', 'LlaMA-3-70B' vs. 'LLaMA-3-70B', and 'Latex' vs. 'LaTeX'. There is also a typo 'purpuse' in Table 8.
- [Appendix H] The token-attribution results are described as inconclusive and are relegated to the appendix; the main text should either briefly state this or remove references to explaining model behavior in the introduction. The appendix currently contains many large figures with limited interpretation.
- [§B] The evaluation setup mentions three prompts per dataset but does not describe how generations were parsed or what 'log-likelihood prediction' means for multi-token outputs. A short description of the parsing/filtering of predictions would improve reproducibility.
Circularity Check
No circularity: empirical benchmark against external ground-truth labels; no quantity is defined in terms of the target result.
full rationale
This paper is an empirical evaluation. The reported quantities (weighted F1, equality of opportunity, statistical parity) are measured from model predictions on held-out test splits against ground-truth labels. No parameter is fitted to the reported metrics and then presented as a prediction: the Logistic Regression baseline is trained with default scikit-learn parameters on the training split and used only as an external reference point, not as part of the LLM claims. In-context examples are selected by gender balance ('top-n examples balanced by gender'), not by optimizing F1 or fairness, so the fairness and performance outcomes are not forced by construction. The serialization formats are pre-existing external methods (JSON, GReaT, LIFT, Text, HTML, Latex) with citations to prior work; the paper does not define its dependent variables in terms of its independent variables. The zero/one baselines having zero EO/SP is a mathematical property of constant classifiers, but the paper reports these as baselines rather than deriving its central claim from that property. The skeptic's concern about degenerate predictions is a validity threat to fairness comparisons, not a circularity: it does not show that any claimed result reduces to its input by definition or by self-citation. Self-citations (Kanubala et al. 2024; Majumdar et al. 2025) are contextual and not load-bearing for the empirical conclusions. The derivation chain is therefore self-contained against external benchmarks and no circular step is present.
Assumptions & free parameters
assumptions (5)
- domain assumption Gender is a binary attribute (male/female) and is the sole sensitive attribute evaluated
- domain assumption Weighted F1, EO, and SP are appropriate evaluation measures for loan approval decisions
- domain assumption The three datasets are comparable proxies for loan approval in their respective regions
- domain assumption LLM log-likelihood scoring over the possible label tokens adequately represents the model's decision
- domain assumption The three hand-written prompts per dataset adequately span prompt sensitivity
Cite this review
Pith. "Pith review of Accept or Deny? Evaluating LLM Fairness and Performance in Loan Approval across Table-to-Text Serialization Approaches." pith.science (2026). https://pith.science/paper/PHTSF7BB
@misc{pith2026250821512,
author = {Pith},
title = {Pith review of: Accept or Deny? Evaluating LLM Fairness and Performance in Loan Approval across Table-to-Text Serialization Approaches},
year = {2026},
howpublished = {\url{https://pith.science/paper/PHTSF7BB}},
note = {Machine review of arXiv:2508.21512}
}
read the original abstract
Large Language Models (LLMs) are increasingly employed in high-stakes decision-making tasks, such as loan approvals. While their applications expand across domains, LLMs struggle to process tabular data, ensuring fairness and delivering reliable predictions. In this work, we assess the performance and fairness of LLMs on serialized loan approval datasets from three geographically distinct regions: Ghana, Germany, and the United States. Our evaluation focuses on the model's zero-shot and in-context learning (ICL) capabilities. Our results reveal that the choice of serialization (Serialization refers to the process of converting tabular data into text formats suitable for processing by LLMs.) format significantly affects both performance and fairness in LLMs, with certain formats such as GReat and LIFT yielding higher F1 scores but exacerbating fairness disparities. Notably, while ICL improved model performance by 4.9-59.6% relative to zero-shot baselines, its effect on fairness varied considerably across datasets. Our work underscores the importance of effective tabular data representation methods and fairness-aware models to improve the reliability of LLMs in financial decision-making.
Figures
Figures from the paper (12 more)
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298--306
work page 2021
-
[4]
Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. 2025. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37:76930--76966
work page 2025
-
[5]
Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. 2024. Many-shot in-context learning. arXiv preprint arXiv:2404.11018
arXiv 2024
-
[6]
Carlos Alejandro Aguirre, Kuleen Sasse, Isabel Alyssa Cachola, and Mark Dredze. 2024. https://doi.org/10.18653/v1/2024.nlp4pi-1.4 Selecting shots for demographic fairness in few-shot learning with large language models . In Proceedings of the Third Workshop on NLP for Positive Impact, Miami, Florida, USA. Association for Computational Linguistics
-
[7]
Microsoft Research AI4Science and Microsoft Azure Quantum. 2023. The impact of large language models on scientific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361
arXiv 2023
-
[8]
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29
2016
Show all 40 references
-
[9]
Vadim Borisov, Kathrin Se ler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. 2022. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.06280
2022 arXiv
-
[10]
Han Ding, Yinheng Li, Junhao Wang, and Hang Chen. 2024. Large language model agent in financial trading: A survey. arXiv preprint arXiv:2408.06361
2024
-
[11]
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. 2022. Lift: Language-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35:1...
2022
-
[12]
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214--226
2012
-
[13]
Duanyu Feng, Yongfu Dai, Jimin Huang, Yifang Zhang, Qianqian Xie, Weiguang Han, Zhengyu Chen, Alejandro Lopez-Lira, and Hao Wang. 2023. Empowering many, biasing a few: Generalist credit scoring through large language models. arXiv preprint arXiv:2310.00566
2023 arXiv
-
[14]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...
2024 doi
-
[15]
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635--E3644
2018
-
[16]
FM Haque and Md Mahedi Hassan. 2024. Bank loan prediction using machine learning techniques. arXiv preprint arXiv:2410.08886
2024 arXiv
-
[17]
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29
2016
-
[18]
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. 2023. Tabllm: Few-shot classification of tabular data with large language models. In International Conference on Artificial Intelligence and Statistics, pages 5549--5581. PMLR
2023
-
[19]
Noah Hollmann, Samuel M \"u ller, Katharina Eggensperger, and Frank Hutter. 2022. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848
2022 arXiv
-
[20]
Sukriti Jaitly, Tanay Shah, Ashish Shugani, and Razik Singh Grewal. 2023. Towards better serialization of tabular data for few-shot classification. arXiv preprint arXiv:2312.12464
2023 arXiv
-
[21]
Loan approval prediction dataset
Kaggle . Loan approval prediction dataset. https://www.kaggle.com/altruistdelhite04/loan-prediction-problem-dataset. Accessed: 2024-07-19
2024
-
[22]
Deborah D Kanubala, Isabel Valera, and Kavya Gupta. 2024. Fairness beyond binary decisions: A case study on german credit. European Workshop on Algorithmic Fairness
2024
-
[23]
Siavash Kazemian, Cosmin Munteanu, and Gerald Penn. 2022. https://doi.org/10.18653/v1/2022.finnlp-1.10 A taxonomical NLP blueprint to support financial decision making through information-centred interactions . In Proceedings of the Fourth Workshop on Financial Technology and ...
2022 doi
-
[24]
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. 2020. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint arXiv:2009.07896
2020 arXiv
-
[25]
Yaxuan Kong, Yuqi Nie, Xiaowen Dong, John M Mulvey, H Vincent Poor, Qingsong Wen, and Stefan Zohren. 2024. Large language models for financial and investment management: Applications and benchmarks. Journal of Portfolio Management, 51(2)
2024
-
[26]
Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of the ACM collective intelligence conference, pages 12--24
2023
-
[27]
Nikita Kozodoi, Johannes Jacob, and Stefan Lessmann. 2022. Fairness in credit scoring: Assessment, implementation and profit implications. European Journal of Operational Research, 297(3):1083--1094
2022
-
[28]
Ayan Majumdar, Deborah D Kanubala, Kavya Gupta, and Isabel Valera. 2025. A causal framework to measure and mitigate non-binary treatment discrimination. arXiv preprint arXiv:2503.22454
2025
-
[29]
Meta. 2024. I ntroducing M eta L lama 3: T he most capable openly available L L M to date --- ai.meta.com. https://ai.meta.com/blog/meta-llama-3/. [Accessed 01-06-2024]
2024
-
[30]
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Nick Barnes, and Ajmal Mian. 2023. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435
2023 arXiv
-
[31]
Roberto Navigli, Simone Conia, and Bj \"o rn Ross. 2023. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality, 15(2):1--21
2023
-
[32]
Frank Gyimah Sackey and Peter Nkrumah Amponsah. 2018. Gender discrimination in commercial banks’ credit markets in ghana: a decomposition and counterfactual analysis. African Journal of Business and Economic Research, 13(2):121--140
2018
-
[33]
Ananya Singha, Jos \'e Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. 2023. Tabular representation, noisy operators, and impacts on table structure understanding tasks in llms. arXiv preprint arXiv:2310.10358
2023 arXiv
-
[34]
Statlog (german credit data)
Statlog. Statlog (german credit data). https://archive.ics.uci.edu/ml/datasets/Statlog+ Accessed: 2024-07-19
2024
-
[35]
Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 645--654
2024
-
[36]
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. arXiv preprint arXiv:1906.08976
2019 arXiv
-
[37]
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...
2024 arXiv
-
[38]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288
2023 arXiv
-
[39]
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. https://arxiv.org/abs/2306.05443 Pixiu: A large language model, instruction data and evaluation benchmark for finance . Preprint, arXiv:2306.05443
2023 arXiv
-
[40]
Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Jesujoba O Alabi, Xiaoyu Shen, Dietrich Klakow, and Marius Mosbach. 2024. The impact of demonstrations on multilingual in-context learning: A multidimensional analysis. arXiv preprint arXiv:2402.12976
2024 arXiv
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.