Pith. sign in

REVIEW 3 major objections 5 minor 15 references

Be.FM: Open Foundation Models for Human Behavior

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Be.FM, a family of open foundation models fine-tuned on behavioral data, predicts and simulates human decisions across economic games, surveys, and reasoning tasks.

desk verdict The MobLab evaluation is in-sample, so the headline behavior-prediction claim is unsupported; the benchmark suite and transparent appendices are still worth a look. read the letter →

arxiv 2505.23058 v1 pith:JOEA7Y6F submitted 2025-05-29 cs.AI cs.CEcs.CL

Yutong Xie , Zhuoheng Li , Xiyuan Wang , Yijun Pan , Qijia Liu , Xingzhi Cui , Kuang-Yu Lo , Ruoyi Gao
show 5 more authors
Xingjian Zhang Jin Huang Walter Yuan Matthew O. Jackson Qiaozhu Mei
This is my paper
classification cs.AIcs.CEcs.CL
keywords foundationmodelshumanbehaviormodelingbehavioralsciencepredictioneconomicgamesBigFivepersonalitylargelanguagedistributionmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Be.FM, a family of open large language models built by fine-tuning Llama 3.1 (8B and 70B) on a curated portfolio of behavioral data: abstracts from behavioral-science journals, human-play records from five classic economic games on MobLab, and Big Five personality survey responses. Its central claim is that a single model trained this way can predict and simulate human decisions across scenarios, infer characteristics of individuals and populations, generate insights about contexts, and apply behavioral knowledge, replacing task-specific models. The most concrete evidence is that Be.FM-generated behavior distributions in Dictator, Ultimatum, Trust, Public Goods, and Bomb games are closer to human MobLab distributions (lower Wasserstein distance) than those of base Llama models or GPT-4o, and that it also outperforms those baselines at predicting Big Five scores and inferring age from behavior. The paper frames these capabilities through a behavioral model y=F(K,x,c), in which knowledge K, subject characteristics x, and context c jointly determine choices y.

What carries the argument

The central object is Be.FM itself, a supervised fine-tune with low-rank adaptation (LoRA) applied across all layers of Llama 3.1-8B/70B, using the Alpaca instruction template. The organizing identity is y=F(K,x,c), which casts every task as a prediction over one of these variables given the others; the benchmark suite then tests each direction: behavior from characteristics and context, characteristics from behavior, context from behavior shifts, and new knowledge from existing knowledge. For the headline capability, the key metric is the Wasserstein distance between model-generated and human behavior distributions, which turns 'predicting behavior' into 'matching the full population distribution rather than a single point.'

What would settle it

Hold out a set of MobLab sessions or games from training and measure Be.FM's Wasserstein distance to human behavior on those held-out data; if the advantage over base Llama models shrinks or vanishes, the claim that Be.FM generalizes to unseen behavior is falsified. A stronger test: run a new economic game not in the training portfolio with human subjects and compare Be.FM's predicted distribution to the observed one.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that a behavioral foundation model can be trained from heterogeneous, loosely structured behavioral data and then serve a broad suite of behavioral-science tasks without task-specific retraining. The authors show that distribution-level behavior simulation in economic games improves substantially over general-purpose LLMs, and that individual-level predictions of survey outcomes and demographics, contextual-factor inference, and knowledge application also improve. They position this as evidence that the latent function F mapping knowledge, subject, and context to behavior is learnable by a language model fine-tuned on a diverse data portfolio.

Load-bearing premise

The economic-game results are evaluated at the distributional level on the same MobLab data used for training, with no train/test split (Appendix A.2.2), so the headline behavior-prediction claim rests on the assumption that reproducing the training distribution counts as predicting behavior.

Editorial extensions

If this is right

  • If the claim holds, behavioral science can use one open model for many tasks—behavior simulation, trait inference, context inference, and reasoning—instead of bespoke models per task.
  • Population-level behavior simulation with distribution matching becomes a standard evaluation target, making Wasserstein distance a meaningful yardstick for AI-human alignment.
  • Fine-tuning on behavioral data transfers to out-of-distribution tasks such as IEO contest questions and Dictator-game treatment design, suggesting the model internalizes general behavioral principles rather than only memorizing game outputs.
  • The same vocabulary can run in reverse: from survey responses to demographics, and from behavior shifts to experimental conditions, which makes the framework bidirectional and useful for intervention design.
  • Small open models can beat much larger commercial models on behavioral prediction, which lowers the barrier to reproducible behavioral modeling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If replicated on properly held-out economic-game data, this approach would make inexpensive in silico pilots of behavioral experiments possible: design variations could be scored by predicted human distributions before any subjects are recruited.
  • The framework suggests a unified substrate for both studying humans and auditing AI: the same model that predicts human choices can be treated as a simulated subject, bridging AI-for-behavioral-science and behavioral-science-of-AI.
  • One testable extension is whether training on survey and literature data alone improves game prediction; if it does, the representation of behavioral knowledge is transferable, not data-specific. If it does not, the game results may depend on direct exposure to game data.
  • Because the authors explicitly report no train/test split for the MobLab evaluation, an external validation on new games, new subject pools, or new experimental conditions is needed before the generalization claim can be taken at face value.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Be.FM, a family of open foundation models for human behavior, built by fine-tuning Llama 3.1 8B and 70B on a portfolio of behavioral data: AER literature abstracts, MobLab economic-game records, Big Five survey responses, and (in future versions) observational data. The authors claim four capabilities: predicting and simulating behavior across scenarios, inferring subject characteristics, generating insights about contextual factors, and applying behavioral-science knowledge. Evaluations include Wasserstein-distance comparisons of simulated versus human behavior distributions in seven economic-game settings, Big Five personality-score prediction, age prediction from personality scores, qualitative Dictator-game context inference, AER research-workflow reasoning, and International Economics Olympiad multiple-choice questions. The central quantitative claim is that Be.FM narrows the gap between AI-generated and human behavior distributions in economic games (Table 3).

Significance. If the central claim held, Be.FM would be a useful open foundation model for behavioral science, and the paper's benchmark suite would be a valuable contribution to a largely underexplored area. The training recipes, data-formatting templates, and the inclusion of multiple task families are useful scaffolding for future work. The paper is also commendably honest in some places: Section 3.5 explicitly notes that expert models can outperform Be.FM on individual tasks and that context-inference evaluation lacks a quantitative procedure. However, the headline economic-game result is compromised by the absence of a train/test split, and the individual-level Big Five results are weaker than a simple XGBoost baseline. The current evidence does not establish the abstract's claim that Be.FM can predict and simulate behavior across diverse scenarios in a generalizable sense.

major comments (3)
  1. [Appendix A.2.2 and Section 3.1, Table 3] The economic-game evaluation is circular. The same 82,057 MobLab observations are used to construct the training entries described in Appendix A.2.2 ("In total, we construct 82,057 such data entries for model training") and to form the ground-truth human distributions against which model outputs are compared in Table 3 and Figure 2 (Appendix B.1). Appendix A.2.2 explicitly states: "Since the evaluation regarding this data collection is performed at the distributional level, we do not separate the data into training and test sets." Consequently, low Wasserstein distance in Table 3 indicates in-sample distribution matching or memorization, not prediction of unseen behavior. This is load-bearing for the abstract and Section 3.1 claim that Be.FM can "predict and simulate behavior across diverse scenarios." The authors should re-run the evaluation on a held-out split of the MobLab data (e.g., by session or subject), or substantially reframe the claim as in-sample simulation of a known population and provide appropriate caveats throughout.
  2. [Section 3.1, Table 4 and Appendix B.1, Table 8] The main-text Table 4 omits the strongest baseline. In Appendix Table 8, XGBoost achieves MAE 6.03 and Spearman's correlation 0.252 on Big Five score prediction, while Be.FM 8B and 70B achieve MAE 7.27 with Spearman correlations of 0.101 and 0.128, respectively. Thus on both individual-level metrics, a standard non-LLM regressor outperforms Be.FM on held-out data. The main text should include XGBoost in the results table and qualify the claim that Be.FM demonstrates "advanced ability" in individual-level survey prediction; the current presentation overstates the model's capability relative to the evidence.
  3. [Section 3.3 and Appendix B.3] The context-inference capability is supported only by qualitative model outputs in Tables 10-13. There is no quantitative evaluation metric, no baseline comparison, and no scoring of whether the proposed designs are correct or novel. The authors themselves acknowledge in Section 3.5 that "a quantitative evaluation procedure is needed to measure the success of context inference." As presented, the claim that Be.FM can "generate insights about contextual factors that influence behavior" remains anecdotal and is not commensurable with the other benchmark results.
minor comments (5)
  1. [Section 3.1, paragraph after Table 3] The text says the exception is "the Responder role in a trust game," but the listed games include the Ultimatum Game as Proposer and Responder and the Trust Game as Investor and Banker; the Responder exception should refer to the Ultimatum Game, not the Trust Game.
  2. [Title and header] The paper title in the header reads "Be.FM: A Family of Open Foundation Models for Human Behaviors," while the abstract title reads "Be.FM: Open Foundation Models for Human Behavior." These should be made consistent.
  3. [Abstract and Section 1] The manuscript says "The models are available upon request:LINK," but no actual link or repository is provided. Since the paper emphasizes that Be.FM is an open foundation model, a working download link or clear availability statement is needed for reproducibility.
  4. [Section 2.1 and Table 1] Section 2.1 reports 68,779 MobLab subjects while Table 1 reports 68,780; the numbers should be reconciled.
  5. [Table 4 caption] The caption says "Boldtexts highlight the best performance in each game," but Table 4 reports Big Five personality prediction, not a game; the caption should say "each task."

Circularity Check

1 steps flagged · score 6.0 of 10

Behavior-distribution prediction in Sec. 3.1 (Table 3, Figure 2) evaluates on the same MobLab records used for training: Appendix A.2.2 explicitly omits a train/test split, so the headline economic-game prediction reduces to in-sample distribution matching.

  1. fitted input called prediction [Appendix A.2.2; Sec. 3.1, Table 3, Figure 2]
    "Since the evaluation regarding this data collection is performed at the distributional level, we do not separate the data into training and test sets. ... In total, we construct 82,057 such data entries for model training. ... Individual sessions in this data are used for model training, and a different task is used for evaluation, to predict the behavioral distributions at the population level."

    The 'ground-truth human distributions' in Table 3 and Figure 2 are the empirical distributions of the same 82,057 MobLab records that are formatted as supervised fine-tuning entries in Appendix A.2.2. Because no train/test split is made, every individual choice used to build the target histogram is also a training label. The evaluation task, predicting the population-level distribution, is a deterministic aggregate of those training labels; a model can achieve low Wasserstein distance by reproducing the frequencies it has memorized during fine-tuning. Calling this 'predict and simulate behavior across diverse scenarios' equates prediction with in-sample reproduction.

full rationale

The one substantive circular step is in Sec. 3.1 and Appendix A.2.2: Be.FM is fine-tuned on all 82,057 MobLab observations and then scored by Wasserstein distance against the empirical distributions of those same observations. The no-split design is explicitly stated, and because the population distribution is a deterministic aggregate of the training labels, low distance reflects memorization or frequency matching rather than prediction of unseen behavior. This weakens the first headline capability, 'predict and simulate behavior across diverse scenarios,' which is the most prominent behavior-prediction claim in the abstract, Figure 1, and Sec. 3.1. The other benchmark tasks are not circular: Big Five personality prediction uses a held-out set of 1,963 subjects (Appendix B.1), age prediction uses the same held-out subjects (B.2), AER workflow reasoning uses 300 held-out publications (B.4), and IEO questions were not used in training (Sec. 3.4). No load-bearing self-citation or imported uniqueness theorem was found; citations to Mei et al. 2024 and Xie et al. 2024 are data/method references rather than the derivation mechanism. The circularity is therefore partial: one of four claimed capabilities, but the one most prominently associated with predicting behavior, reduces by construction. Score 6.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numerical free parameters are fit in the paper; the model is trained via standard SFT. The key assumptions are domain-level: that behavior factors as in Eq. 1, that MobLab data is representative, that GPT-4o summaries are valid ground truth, and that Big Five self-reports capture personality. No new physical or theoretical entities are introduced.

assumptions (4)
  • domain assumption Human behavior is determined by subject characteristics x, context c, and behavioral knowledge K via a latent function F (Eq. 1).
    The entire training and benchmark design is organized around this framework, asserted without empirical validation.
  • domain assumption The MobLab subject pool's behavior distributions are valid ground truth for human behavior in these games.
    The evaluation treats MobLab records as ground truth, but it is a single online platform's subject pool and may not represent broader populations.
  • domain assumption GPT-4o-extracted research workflows are valid targets for evaluating research reasoning.
    Training and evaluation for literature tasks use GPT-4o to summarize abstracts into contexts, ideas, methods, outcomes, and impacts, so the evaluation measures alignment with GPT-4o summaries, not independently verified research insight.
  • domain assumption Big Five survey responses are an appropriate representation of personality characteristics.
    The model is trained and evaluated on self-reported survey data, which carries known response biases; the paper does not address this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Be.FM: Open Foundation Models for Human Behavior." pith.science (2026). https://pith.science/paper/JOEA7Y6F

@misc{pith2026250523058,
  author       = {Pith},
  title        = {Pith review of: Be.FM: Open Foundation Models for Human Behavior},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JOEA7Y6F}},
  note         = {Machine review of arXiv:2505.23058}
}
read the original abstract

Despite their success in numerous fields, the potential of foundation models for modeling and understanding human behavior remains largely unexplored. We introduce Be.FM, one of the first open foundation models designed for human behavior modeling. Built upon open-source large language models and fine-tuned on a diverse range of behavioral data, Be.FM can be used to understand and predict human decision-making. We construct a comprehensive set of benchmark tasks for testing the capabilities of behavioral foundation models. Our results demonstrate that Be.FM can predict behaviors, infer characteristics of individuals and populations, generate insights about contexts, and apply behavioral science knowledge.

Figures

Figures reproduced from arXiv: 2505.23058 by the authors.

Figure 1
Figure 1. Be.FM is a foundation model designed for modeling human behavior. Trained on a diverse portfolio of behavioral datasets, Be.FM has the capabilities to: (1) Predict and simulate behavior across diverse scenarios; (2) Infer motivations and other characteristics of individual humans as well as broader populations from their behaviors; (3) Generate insights about contextual factors that influence human behaviors; and (4… view at source ↗
Figure 2
Figure 2. Behavior distributions of human players (the first row) and models (remaining rows) in classic behavioral [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages

  1. [2]

    Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712,

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712,

    arXiv

  2. [3]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Let- man, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  3. [4]

    Generative agent simulations of 1,000 people.arXiv preprint arXiv:2411.10109,

    Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S Bernstein. Generative agent simulations of 1,000 people.arXiv preprint arXiv:2411.10109,

    arXiv

  4. [5]

    Large language models show human-like social desirability biases in survey responses.arXiv preprint arXiv:2405.06058,

    Aadesh Salecha, Molly E Ireland, Shashanka Subrahmanya, João Sedoc, Lyle H Ungar, and Johannes C Eich- staedt. Large language models show human-like social desirability biases in survey responses.arXiv preprint arXiv:2405.06058,

    arXiv

  5. [6]

    Fast adaptation with behavioral foundation models.arXiv preprint arXiv:2504.07896,

    Harshit Sikchi, Andrea Tirinzoni, Ahmed Touati, Yingchen Xu, Anssi Kanervisto, Scott Niekum, Amy Zhang, Alessandro Lazaric, and Matteo Pirotta. Fast adaptation with behavioral foundation models.arXiv preprint arXiv:2504.07896,

    arXiv

  6. [7]

    edu/2023/03/13/alpaca.html

    URLhttps://crfm.stanford. edu/2023/03/13/alpaca.html. 12 Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, and Matteo Pirotta. Zero-shot whole-body humanoid control via behavioral foundation models.arXiv preprint arXiv:2504.11054,

    arXiv 2023

  7. [8]

    How different ai chatbots behave? benchmarking large language models in behavioral economics games.arXiv preprint arXiv:2412.12362,

    Yutong Xie, Yiyao Liu, Zhuang Ma, Lin Shi, Xiyuan Wang, Walter Yuan, Matthew O Jackson, and Qiaozhu Mei. How different ai chatbots behave? benchmarking large language models in behavioral economics games.arXiv preprint arXiv:2412.12362,

    arXiv

  8. [9]

    For each model, the mean absolute error (MAE), Spearman’s correlation coefficient, and Wasserstein distance are computed by comparing the age prediction results and ground truths

    Model MAE(↓)Spearman’s corr(↑)W-distance(↓) Be.FM8B 9.05 0.108 ‡ 5.39 Be.FM70B 9.16 0.134 ‡ 5.70 Llama 3.1 8B 10.50 0.038 8.62 Llama 3.1 70B 10.36 0.029 7.75 GPT 4o-mini 10.17 0.030 8.07 GPT 4o 11.01 0.048 ‡ 9.12 Be.FM8B ∗ 9.21 0.096 ‡ 5.56 Be.FM70B ∗ 9.35 0.107 ‡ 5.66 XGBoost 8.97 0.168 ‡ 5.03 Table 9: Performances on inferring subjects’ ages given Big F...

    work page

Show all 15 references
  1. [12]

    Behavior distribution prediction.To train the model to predict behavioral distributions, we format each human-play record as a single data entry

    Since the evaluation regarding this data collection is performed at the distributional level, we do not separate the data into training and test sets. Behavior distribution prediction.To train the model to predict behavioral distributions, we format each human-play record as a...

    2024

  2. [15]

    loss" from giving rather than the

    18 Run List of Outputted Experiment Designs 1 (1) Social Identity Influence – Emphasizing a shared identity may reduce sharing due to perceived competition within the group. (2) Anonymity Variation – Reducing anonymity might lead to lower sharing as dictators manage their self...

    2019

  3. [50]

    The response should only be a number from 10 to 50 in square brackets

    means very prone to stress/anxiety. The response should only be a number from 10 to 50 in square brackets. Demographics prediction.Another modeling direction involves predicting demographic attributes from survey responses. Each subject’s record is converted into a data entry,...

    2024

  4. [2022]

    Massw: A new dataset and benchmark tasks for ai-assisted scientific workflows.arXiv preprint arXiv:2406.06357,

    Xingjian Zhang, Yutong Xie, Jin Huang, Jinge Ma, Zhaoying Pan, Qijia Liu, Ziyang Xiong, Tolga Ergen, Dongsub Shim, Honglak Lee, et al. Massw: A new dataset and benchmark tasks for ai-assisted scientific workflows.arXiv preprint arXiv:2406.06357,

    arXiv

  5. [2023]

    Centaur: a foundation model of human cognition.arXiv preprint arXiv:2410.20268,

    Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria K Eckstein, Noémi Éltet˝o, et al. Centaur: a foundation model of human cognition.arXiv preprint arXiv:2410.20268,

    arXiv

  6. [2024]

    doi: 10.18653/v1/2024.acl-demos.38

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-demos.38. URLhttps://aclanthology.org/2024.acl-demos.38/. 13 A Model Configuration and Training A.1 Backbone Foundation Model and Implementation Details Model.We used LLaMA-3.1-8B-Instruct2 and LLaMA-3.1-70B-...

    2024 doi

  7. [2025]

    This could normally be a problem, a research question, or a research gap that has not been successfully addressed by previous work

    14 - Context: The status quo of related literature or reality which motivated this study. This could normally be a problem, a research question, or a research gap that has not been successfully addressed by previous work. This is anything that happened before this study. - Key...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.