Pith. sign in

REVIEW 4 major objections 5 minor 33 references

Instruction-Based Fine-tuning of Open-Source LLMs for Predicting Customer Purchase Behaviors

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning Mistral-7B on bank transactions written as natural-language instructions predicts a customer's next purchase category, and in this study it outperforms CNN and LSTM baselines on a held-out bank's data.

desk verdict A legitimate but under-evidenced empirical claim that LoRA-tuned Mistral beats CNN/LSTM on next merchant category prediction; the cross-bank setup is nice, but the paper needs significance testing and class-wise transparency before I'd trust the headline. read the letter →

arxiv 2502.15724 v1 pith:SB6OAUTQ submitted 2025-01-28 cs.IR cs.AI

Halil Ibrahim Ergul , Selim Balcisoy , Burcin Bozkaya This is my paper · ORCID
classification cs.IRcs.AI
keywords largelanguagemodelsinstructiontuningLoRAmerchantcategorypredictionfinancialtransactionscustomerpurchasebehaviorclassimbalancesequential
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a general-purpose open-source large language model, fine-tuned on bank transactions rewritten as natural-language instruction prompts, can predict a customer's next purchase category more accurately than standard CNN and LSTM sequential models. This matters because next-category prediction is what credit-card rewards and targeted marketing campaigns are built on, and the comparison is deliberately strict: models are trained on one bank's customers and tested on another bank's customers. In the paper's headline result, fine-tuned Mistral Instruct 7B v.2 reaches a weighted F1 of 0.66 on the second bank's data at sequence length 9, versus 0.62 for the CNN and 0.60 for the LSTM, with the largest gains in the clothing, grocery, and gas-station categories. If the result holds, banks could get better minority-class coverage and a way to fold demographics and transaction history into a single natural-language model without hand-built feature engineering.

What carries the argument

The load-bearing machinery is instruction fine-tuning of a frozen open-source LLM via LoRA. Each customer's demographic fields and recent transactions are rendered into a first-person natural-language prompt, such as 'I am 48 years old, married male, secondary school graduate, and I work as a private employee', followed by the chronological category and spending list, with the next merchant category as the output. LoRA adds two small trainable update matrices per adapted layer, so the pretrained Mistral weights stay frozen while a low-rank parameter set $\Theta$ is optimized by maximizing the log-likelihood of the output tokens given the input sequence and preceding tokens. The same prompt format is used for Bank A training and Bank B testing, and the evaluation varies input sequence length at test time (last-4, last-7, last-9, last-14) even though the model was trained only at last-9.

What would settle it

Re-run the Bank A training and Bank B evaluation multiple times, or bootstrap the 1,000 Bank B test customers, and compute confidence intervals for the weighted F1 at sequence length 9; if the CNN's 0.62 and the fine-tuned Mistral's 0.66 intervals overlap, the claimed superiority is not established.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that instruction-fine-tuning Mistral Instruct 7B v.2 with LoRA on Bank A transactions converted to natural-language prompts produces a next-merchant-category predictor that, tested on Bank B, reaches a weighted F1 of 0.66 at sequence length 9, above the CNN at 0.62 and the LSTM at 0.60. The gain is concentrated in the three named merchant categories: at the trained sequence length, clothing F1 is 0.62 versus near zero for the neural baselines, gas stations reaches 0.40 to 0.50 versus roughly 0.20 to 0.24, and grocery reaches 0.52 to 0.59 versus roughly 0.35 to 0.47. The paper attributes the advantage to the LLM's semantic understanding of categories and demographic context, and to its stronger handling of class imbalance in a dataset where grocery, clothing, and gas stations make up 11.2 to 31.3 percent of transactions and the remaining 45.5 percent are lumped as Other.

Load-bearing premise

The load-bearing premise is that the single-run F1 scores in Tables 1 and 2 are stable enough that the 0.04 to 0.06 weighted-F1 advantage for Mistral and the large class-level gaps reflect real model differences rather than chance.

Editorial extensions

If this is right

  • A weighted F1 of 0.66 on Bank B at sequence length 9, versus 0.62 for the CNN and 0.60 for the LSTM, means the fine-tuned LLM is a competitive alternative to bespoke sequential models for next-category prediction.
  • Class-level results at the trained sequence length show the LLM handling the minority clothing and gas-station categories far better than the neural baselines, which supports the paper's argument about imbalanced financial datasets.
  • Testing at last-4, last-7, and last-14 despite training only at last-9 suggests the fine-tuned model transfers across input sequence lengths without additional retraining.
  • Because demographic and transaction information is embedded in the instruction prompt rather than engineered as numeric features, adding new customer attributes would require editing the template rather than redesigning the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the class-level differences are taken at face value, the real payoff is in minority-category coverage; the weighted-F1 advantage could be diluted by the unshown 'Other' class, which holds 45.5 percent of transactions.
  • The sharp swing in the Clothing score across sequence lengths (F1 0.62 at last-9 but 0.12 at last-7) suggests the LLM's advantage may depend on sequence position or prompt formatting, which repeated runs could clarify.
  • The template-based design makes it cheap to test whether the model is using semantics or surface cues: systematically reword the demographic and amount fields and measure how the category predictions move.
  • For deployment, the accuracy gain over the CNN and LSTM must be weighed against inference cost and latency; the paper's claims are about predictive accuracy, not operational efficiency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates whether fine-tuned open-source LLMs can predict the next merchant category of a customer's bank transactions. Transaction and demographic data are converted into natural-language instruction samples, and a Mistral-7B-Instruct-v0.2 model is fine-tuned with LoRA on 8,154 customers from Bank A and tested on 1,000 customers from Bank B. The authors compare this model against an averaging baseline, a CNN, and an LSTM at input sequence lengths of 4, 7, 9, and 14 transactions. The central claim is that the fine-tuned Mistral model achieves a weighted F1 of 0.66 at last-9 versus 0.62 for CNN and 0.60 for LSTM, and that it outperforms the sequential baselines specifically on the Grocery, Clothing, and Gas Stations categories, which the paper attributes to the LLM's semantic understanding and better handling of minority classes.

Significance. If the empirical claims are correct, the paper demonstrates a practical use of open-source instruction-tuned LLMs for a realistic financial prediction task with cross-bank generalization, and it makes a concrete contribution by showing how tabular transaction data can be formatted as natural-language instructions. The study design has notable strengths: a held-out test bank, a clearly described preprocessing pipeline, class-imbalance discussion, and a public code repository. However, the headline comparison currently rests on single-run F1 values without uncertainty quantification, and the class-level results show large fluctuations that the text does not address. The significance of the result is therefore conditional on the authors providing statistical support for the performance gaps.

major comments (4)
  1. [Section 2.1, Table 1] The central claim that the fine-tuned Mistral model 'significantly outperforms' CNN and LSTM is not supported by any error bars, confidence intervals, or significance tests. The reported advantage at last-9 is 0.66 versus 0.62 and 0.60 on a single test set of 1,000 customers. Given that Clothing and Gas Stations each comprise roughly 11-12% of transactions, per-class support is around 110-120 examples, and the 0.04-0.06 F1 gaps could easily arise from sampling noise. Please report bootstrap confidence intervals, a paired significance test such as McNemar's test, or repeated runs with different seeds and random test samples.
  2. [Section 2.2, Table 2] The assertion of 'remarkable consistency' across categories and sequence lengths is directly contradicted by Table 2. Mistral's Clothing F1 is 0.620 at last-9, 0.480 at last-4, 0.120 at last-7, and 0.220 at last-14, and its Gas Stations F1 ranges from 0.400 to 0.500 across lengths. This variability is of the same order as the claimed advantage over the baselines, so the paper needs to explain this instability or temper the consistency claim.
  3. [Section 2.2, Table 2] The headline weighted F1 scores in Table 1 are dominated by the 'Other' class, which the paper reports as 45.5% of Bank A transactions and 'similar' in Bank B, yet Table 2 reports class-wise F1 only for Clothing, Gas Stations, and Grocery. Without the 'Other' class F1, the overall weighted F1 does not directly support the abstract's claim about the three named categories. Please report class-wise F1 for all four classes and, where possible, per-class statistical tests.
  4. [Section 4.6] The evaluation strategy states that models were trained only on last-9 sequences but tested on last-4, last-7, and last-14 sequences. The paper should clarify whether the CNN and LSTM were retrained or adapted for the non-9 lengths, since the results are used to make general claims about 'consistency across different sequence lengths' in Section 2.2.
minor comments (5)
  1. [Abstract] The phrase 'transaction datagrocery' is missing a space; it should read 'transaction data—grocery'.
  2. [Section 4.4] The text refers to the 'SFFtrainer library'; this appears to be a typo for 'SFTTrainer' from the TRL library.
  3. [Table 1] The table layout is difficult to parse: model names appear in the middle of the table, sequence-length labels repeat, and the raw-model rows are interleaved with baseline rows. Please use separate panels per model or otherwise restructure the table for clarity.
  4. [Section 2.1] The sentence 'The bold scores were depicted by comparing LSTM and CNN with Mistral model' is unclear, and the bold formatting is not visible in the manuscript text. Please state explicitly which comparisons are bolded.
  5. [Section 2.2] The paper uses the word 'significantly' repeatedly in the Discussion and Conclusion without any statistical test; please reserve this term for results with quantified uncertainty.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical benchmark comparison with independently labeled test data and external baselines.

full rationale

The paper's central claim is an empirical model comparison: a fine-tuned Mistral Instruct model is trained on Bank A transaction data and evaluated on held-out Bank B data, with predictions compared against ground-truth merchant categories from the data. The baselines (averaging, CNN, LSTM) are external methods, not derived from the LLM's own outputs or fitted parameters. No fitted constant is relabeled as a prediction, no self-citation is used to justify the core result, and no uniqueness theorem or ansatz is imported from the authors' prior work. The evaluation labels come from the transaction data, not from the model or its training objective. The main weaknesses identified in the reader's take, such as the absence of error bars and the instability of Clothing F1 across sequence lengths, concern statistical robustness and significance, which are correctness or reproducibility risks rather than circularity. Under the given criteria, these do not constitute any form of self-definitional, fitted-input, citation-borne, or renamed-result circularity. The derivation chain, insofar as one exists, is self-contained against external data and benchmarks, so the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical comparison is straightforward, but the paper's generalizability claims rely on several unstated assumptions about representation faithfulness, output parsing, and test-set representativeness.

free parameters (2)
  • Top-3 category selection = Grocery, Clothing, Gas Stations; rest as Other
    Chosen by hand based on frequency (Section 4.2); affects class balance and F1 results.
  • Minimum transaction and category thresholds = 10 transactions; 2 distinct categories
    Post-hoc data filter (Section 4.2) to exclude users like truck drivers; effect on the test set is not analyzed.
assumptions (4)
  • domain assumption Natural-language representation preserves all predictive information in the tabular transaction data
    Section 4.3 converts transactions to narrative prompts; no ablation compares the narrative format against the raw tabular format.
  • domain assumption Bank B, after identical preprocessing, is representative for cross-bank generalization
    Figure 1 and Section 2 assume generalization from Bank A to Bank B without distributional analysis of category shares or demographics.
  • domain assumption LLM output can be mapped deterministically to one of four categories
    Section 4.6 describes predictions but not how free-text answers like 'Gas stations' are parsed or what happens with off-vocabulary outputs.
  • ad hoc to paper Weighted F1 across all categories is an appropriate summary for claims about the three named categories
    The 'Other' class holds 45.5% of transactions; weighted F1 is dominated by it, yet the paper emphasizes performance on Grocery, Clothing, and Gas Stations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Instruction-Based Fine-tuning of Open-Source LLMs for Predicting Customer Purchase Behaviors." pith.science (2026). https://pith.science/paper/SB6OAUTQ

@misc{pith2026250215724,
  author       = {Pith},
  title        = {Pith review of: Instruction-Based Fine-tuning of Open-Source LLMs for Predicting Customer Purchase Behaviors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SB6OAUTQ}},
  note         = {Machine review of arXiv:2502.15724}
}
read the original abstract

In this study, the performance of various predictive models, including probabilistic baseline, CNN, LSTM, and finetuned LLMs, in forecasting merchant categories from financial transaction data have been evaluated. Utilizing datasets from Bank A for training and Bank B for testing, the superior predictive capabilities of the fine-tuned Mistral Instruct model, which was trained using customer data converted into natural language format have been demonstrated. The methodology of this study involves instruction fine-tuning Mistral via LoRA (LowRank Adaptation of Large Language Models) to adapt its vast pre-trained knowledge to the specific domain of financial transactions. The Mistral model significantly outperforms traditional sequential models, achieving higher F1 scores in the three key merchant categories of bank transaction data (grocery, clothing, and gas stations) that is crucial for targeted marketing campaigns. This performance is attributed to the model's enhanced semantic understanding and adaptability which enables it to better manage minority classes and predict transaction categories with greater accuracy. These findings highlight the potential of LLMs in predicting human behavior.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 17 canonical work pages

  1. [1]

    ”A Survey of Large Language Models.” arXiv preprint arXiv:2303.18223 (2023)

    Zhao, Wayne Xin, et al. ”A Survey of Large Language Models.” arXiv preprint arXiv:2303.18223 (2023)

    arXiv 2023

  2. [2]

    ”PaLM: Scaling Language Modeling with Path- ways.” arXiv preprint arXiv:2204.02311 (2022)

    Chowdhery, Aakanksha, et al. ”PaLM: Scaling Language Modeling with Path- ways.” arXiv preprint arXiv:2204.02311 (2022)

    arXiv 2022

  3. [3]

    ”Holistic Evaluation of Language Models.” arXiv preprint arXiv:2211.09110 (2023)

    Liang, Percy, et al. ”Holistic Evaluation of Language Models.” arXiv preprint arXiv:2211.09110 (2023). 15

    arXiv 2023

  4. [4]

    ”Emergent Abilities of Large Language Models.” arXiv preprint arXiv:2206.07682 (2022)

    Wei, Jason, et al. ”Emergent Abilities of Large Language Models.” arXiv preprint arXiv:2206.07682 (2022)

    arXiv 2022

  5. [5]

    ”Forecasting purchase categories by trans- actional data: A comparative study of classification methods.” Lecture Notes in Computer Science

    Shikov, Egor, and Klavdiya Bochenina. ”Forecasting purchase categories by trans- actional data: A comparative study of classification methods.” Lecture Notes in Computer Science. Springer International Publishing, 2019. 249-262

    work page 2019

  6. [6]

    ”Can Generative AI improve social science?” Proceedings of the National Academy of Sciences 121.21 (2024)

    Bail, Christopher A. ”Can Generative AI improve social science?” Proceedings of the National Academy of Sciences 121.21 (2024)

    work page 2024

  7. [7]

    ”A Survey on Large Language Models for Recommendation.” arXiv preprint arXiv:2305.19860 (2023)

    Wu, Likang, et al. ”A Survey on Large Language Models for Recommendation.” arXiv preprint arXiv:2305.19860 (2023)

    arXiv 2023

  8. [8]

    ”Recommendation as Language Processing (RLP): A Uni- fied Pretrain, Personalized Prompt I& Predict Paradigm (P5).” arXiv preprint arXiv:2203.13366 (2023)

    Geng, Shijie, et al. ”Recommendation as Language Processing (RLP): A Uni- fied Pretrain, Personalized Prompt I& Predict Paradigm (P5).” arXiv preprint arXiv:2203.13366 (2023)

    arXiv 2023

Show all 33 references
  1. [9]

    Bao, Keqin, et al. ”TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation.” RecSys ’23: Proceedings of the 17th ACM Conference on Recommender Systems (2023): 152-160

    2023

  2. [10]

    ”Zero-Shot Recommenda- tion as Language Modeling.” arXiv preprint arXiv:2112.04184 (2021)

    Sileo, Damien, Wout Vossen, and Robbe Raymaekers. ”Zero-Shot Recommenda- tion as Language Modeling.” arXiv preprint arXiv:2112.04184 (2021)

    2021 arXiv

  3. [11]

    ”Zero-Shot Recommender Systems.” arXiv preprint arXiv:2105.08318 (2021)

    Ding, Hao, et al. ”Zero-Shot Recommender Systems.” arXiv preprint arXiv:2105.08318 (2021)

    2021 arXiv

  4. [12]

    ”PALR: Personalization Aware LLMs for Recommendation.” arXiv preprint arXiv:2305.07622 (2023)

    Yang, Fan, et al. ”PALR: Personalization Aware LLMs for Recommendation.” arXiv preprint arXiv:2305.07622 (2023)

    2023 arXiv

  5. [13]

    ”Learning Vector-Quantized Item Representation for Trans- ferable Sequential Recommenders.” arXiv preprint arXiv:2210.12316 (2023)

    Hou, Yupeng, et al. ”Learning Vector-Quantized Item Representation for Trans- ferable Sequential Recommenders.” arXiv preprint arXiv:2210.12316 (2023)

    2023 arXiv

  6. [14]

    ”Instruction Tuning for Large Language Models: A Survey.” arXiv preprint arXiv:2308.10792 (2024)

    Zhang, Shengyu, et al. ”Instruction Tuning for Large Language Models: A Survey.” arXiv preprint arXiv:2308.10792 (2024)

    2024

  7. [15]

    ”A Survey on Data Selection for LLM Instruction Tuning.” arXiv preprint arXiv:2402.05123 (2024)

    Wang, Jiahao, et al. ”A Survey on Data Selection for LLM Instruction Tuning.” arXiv preprint arXiv:2402.05123 (2024)

    2024 arXiv

  8. [16]

    ”Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation.” arXiv preprint arXiv:1609.09152 (2016)

    He, Ruining, and Julian McAuley. ”Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation.” arXiv preprint arXiv:1609.09152 (2016)

    2016 arXiv

  9. [17]

    ”Learning and adaptivity in interac- tive recommender systems.” ICEC ’07: Proceedings of the Ninth International Conference on Electronic Commerce (2007): 75-84

    Mahmood, Tariq, and Francesco Ricci. ”Learning and adaptivity in interac- tive recommender systems.” ICEC ’07: Proceedings of the Ninth International Conference on Electronic Commerce (2007): 75-84

    2007

  10. [18]

    ”Factoriz- ing personalized Markov chains for next-basket recommendation.” WWW ’10: 16 Proceedings of the 19th International Conference on World Wide Web (2010): 811-820

    Rendle, Steffen, Christoph Freudenthaler, and Lars Schmidt-Thieme. ”Factoriz- ing personalized Markov chains for next-basket recommendation.” WWW ’10: 16 Proceedings of the 19th International Conference on World Wide Web (2010): 811-820

    2010

  11. [19]

    ”Deep Learning for Sequential Recommendation: Algorithms, Influential Factors, and Evaluations.” arXiv preprint arXiv:1905.01997 (2020)

    Fang, Hui, et al. ”Deep Learning for Sequential Recommendation: Algorithms, Influential Factors, and Evaluations.” arXiv preprint arXiv:1905.01997 (2020)

    2020 arXiv

  12. [20]

    ”Sequential Recommender Systems: Challenges, Progress and Prospects.” IJCAI-2019: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (2019)

    Wang, Shoujin, et al. ”Sequential Recommender Systems: Challenges, Progress and Prospects.” IJCAI-2019: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (2019)

    2019

  13. [21]

    ”Session-based Recommendations with Recurrent Neural Networks.” arXiv preprint arXiv:1511.06939 (2016)

    Hidasi, Balzs, et al. ”Session-based Recommendations with Recurrent Neural Networks.” arXiv preprint arXiv:1511.06939 (2016)

    2016 arXiv

  14. [22]

    ”Sequential User-based Recur- rent Neural Network Recommendations.” RecSys ’17: Proceedings of the Eleventh ACM Conference on Recommender Systems (2017): 152-160

    Donkers, Tim, Benedikt Loepp, and Jrgen Ziegler. ”Sequential User-based Recur- rent Neural Network Recommendations.” RecSys ’17: Proceedings of the Eleventh ACM Conference on Recommender Systems (2017): 152-160

    2017

  15. [23]

    ”A Simple Convolutional Generative Network for Next Item Recommendation.” arXiv preprint arXiv:1808.05163 (2018)

    Yuan, Fajie, et al. ”A Simple Convolutional Generative Network for Next Item Recommendation.” arXiv preprint arXiv:1808.05163 (2018)

    2018 arXiv

  16. [24]

    ”LoRA: Low-Rank Adaptation of Large Language Models.” arXiv preprint arXiv:2106.09685 (2021)

    Hu, Edward J., et al. ”LoRA: Low-Rank Adaptation of Large Language Models.” arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  17. [25]

    ”PEFT: State-of-the-art Parameter-Efficient Fine- Tuning methods.” GitHub repository

    Mangrulkar, Sourab, et al. ”PEFT: State-of-the-art Parameter-Efficient Fine- Tuning methods.” GitHub repository. https://github.com/huggingface/peft (2022)

    2022

  18. [26]

    ”TRL: Transformer Reinforcement Learning.” GitHub repository

    von Werra, Leandro, et al. ”TRL: Transformer Reinforcement Learning.” GitHub repository. https://github.com/huggingface/trl

  19. [27]

    ”Mistral 7B.” arXiv preprint arXiv:2310.06825 (2023)

    Jiang, Albert Q., et al. ”Mistral 7B.” arXiv preprint arXiv:2310.06825 (2023)

    2023 arXiv

  20. [28]

    ”Behavioral attributes and financial churn prediction.” EPJ Data Science 7.1 (2018): 1-18

    Kaya, Erdem, et al. ”Behavioral attributes and financial churn prediction.” EPJ Data Science 7.1 (2018): 1-18

    2018

  21. [29]

    ”Money Walks: Implicit Mobility Behavior and Financial Well-Being.” PLOS ONE 10.8 (2015): e0136628

    Singh, Vivek Kumar, Burcin Bozkaya, and Alex Pentland. ”Money Walks: Implicit Mobility Behavior and Financial Well-Being.” PLOS ONE 10.8 (2015): e0136628

    2015

  22. [30]

    ”Training language models to follow instructions with human feedback.” arXiv preprint arXiv:2203.02155 (2022)

    Ouyang, Long, et al. ”Training language models to follow instructions with human feedback.” arXiv preprint arXiv:2203.02155 (2022)

    2022 arXiv

  23. [31]

    ”Self-Instruct: Aligning Language Models with Self- Generated Instructions.” arXiv preprint arXiv:2212.10560 (2023)

    Wang, Yizhong, et al. ”Self-Instruct: Aligning Language Models with Self- Generated Instructions.” arXiv preprint arXiv:2212.10560 (2023)

    2023 arXiv

  24. [32]

    ”Stanford Alpaca: An Instruction-following LLaMA model.” GitHub repository

    Taori, Rohan, et al. ”Stanford Alpaca: An Instruction-following LLaMA model.” GitHub repository. https://github.com/tatsu-lab/stanford alpaca (2023). 17

    2023

  25. [33]

    ”LLaMA: Open and Efficient Foundation Language Models.” arXiv preprint arXiv:2302.13971 (2023)

    Touvron, Hugo, et al. ”LLaMA: Open and Efficient Foundation Language Models.” arXiv preprint arXiv:2302.13971 (2023). 18

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.