{"id":"80c20355-873f-4edb-af36-552ce764bf56","arxiv_id":"2508.02679","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper proposes an LLM agent simulation of student behavior, initialized with personality data and driven by smartphone sensing, and claims GPT-4o-mini outperforms Gemini-2.5-flash at predicting stress, sleep, and social levels.","lead":"This paper builds simulated college students as LLM agents that consume smartphone sensing data and answer mental health surveys. It compares two LLMs on how well these agents reproduce stress, sleep, and social outcomes from a real dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The counterfactual/intervention claim rests on an unvalidated causal proxy: LLM EMA predictions may reflect priors or prompt artifacts rather than sensing-driven student states.","rationale":"The Reader's weakest assumption is exactly the load-bearing point: the LLM's EMA-like responses must proxy real students' EMA responses for any simulated intervention to transfer. I agree, and I sharpen it: predictive accuracy on historical data is necessary but not sufficient for the counterfactual uses advertised in the abstract. The no-sensing ablation directly tests whether the sensing stream is doing causal work, and the cross-student counterfactual tests whether the simulation's response to changed sensing inputs matches real student response directions. Because the provided text stops at the Introduction, no metrics or ablations are available to assess this now; the honest status is therefore conditional rather than accepted or rejected. If the checks pass, the comparative prediction claim can be evaluated on its reported metrics; if they fail, the intervention narratives should be removed or reframed as illustrative only. The GitHub repository is a useful artifact but does not by itself settle validity.","tokens_in":1316,"tokens_out":6961,"duration_ms":84688,"concrete_test":"Run a sensing ablation and an intervention-consistency check on the StudentLife test split. (1) No-sensing baseline: give the LLM agent personality questionnaires and date/calendar context with all daily sensing features removed, and compare prediction error with the full-sensing condition; if the baseline is within one standard error, the 'guided by sensing data' claim is unsupported. (2) Cross-student counterfactual: for a held-out pair of students in the same calendar week, feed student A's sensing features and ask for the predicted EMA, then feed student B's features and compare the predicted EMA difference to the real EMA difference; if the simulated direction disagrees on a majority of pairs, the intervention channel is not credible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim (abstract: 'GPT-4o-mini outperforms Gemini-2.5-flash in predicting students' stress, sleep, and social levels') is testable against held-out StudentLife EMA data. The load-bearing assumption is the Reader's: an agent initialized with personality prompts and fed daily sensing features must have a response function that is a valid proxy for a real student's response function. This is needed specifically for the abstract's intervention claim, where the authors 'manipulat[e] activity patterns via sensing signals' and draw conclusions about student well-being. Held-out predictive accuracy does not establish that proxy. A concrete failure mode is that the LLM leans on generic priors about student life (e.g., stress at midterms) rather than on the individual sensing stream; average predictions could look good while the intervention channel is a no-op or points in a non-human-aligned direction. The provided GitHub repository is positive reproducibility evidence, but a repository link cannot validate the causal proxy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an LLM agent-based simulation framework that reproduces student activities and mental-health self-reports from the StudentLife dataset. Each agent is initialized with personality-questionnaire responses and driven by smartphone sensing features over a simulated semester; the agents emit predicted behaviors, EMA-style self-reports, and follow-up personality responses. The authors report that GPT-4o-mini outperforms Gemini-2.5-flash in predicting stress, sleep, and social levels, and they argue that the framework enables counterfactual interventions, peer-influence studies, and hypothetical interviews that go beyond the original dataset. A GitHub repository is provided as reproducibility evidence.","tokens_in":1488,"tokens_out":1987,"duration_ms":25345,"significance":"If the central claim is substantiated, the framework would be a useful testbed for generating hypotheses about student well-being interventions that cannot be run on the original observational data, and the open-source implementation is a positive reproducibility feature. However, the comparative performance claim is presented without any quantitative evidence in the abstract, and the transfer of intervention results to real students rests on a proxy-validity assumption that is not yet established. The work is therefore potentially valuable but currently under-evidenced.","major_comments":[{"comment":"The central quantitative claim, 'GPT-4o-mini outperforms Gemini-2.5-flash in predicting students' stress, sleep, and social levels,' is stated without any accuracy metric, effect size, confidence interval, or statistical test. As written, this claim is unverifiable and constitutes the paper's primary empirical contribution; the manuscript must report the actual prediction results, including the comparison protocol and uncertainty, before the claim can be assessed.","section":"Abstract"},{"comment":"The intervention claim—that the authors can 'manipulat[e] activity patterns via sensing signals' and draw conclusions about student well-being—requires that an agent's EMA response function be a valid proxy for a real student's response function conditional on the sensing stream. Held-out predictive accuracy does not establish this proxy validity. A concrete failure mode is that the LLM leverages generic priors about student life (e.g., stress around midterms) rather than the individual sensing data, in which case average predictions may look plausible while the intervention channel is a no-op or points in a direction misaligned with human behavior. The paper should include a validation or ablation specifically testing whether changes in sensing inputs produce changes in agent EMA outputs that match known or plausible human responses.","section":"Section 1 (Introduction) and Abstract"},{"comment":"The prediction task partially re-expresses the input: the agents are prompted with sensing data that correlates with the target EMA outcomes, so it is unclear how much of the reported performance reflects genuine behavioral modeling rather than information leakage from the prompt. The manuscript should clarify whether the EMA items are distinct from the sensing features, and should describe how the 'various prompting techniques, memory systems, and activity-based mental state management strategies' were selected; if configuration was tuned on the evaluation data, the reported comparison may be overfitted. A clear train/validation/test split for configuration selection is needed.","section":"Abstract and Section 1 (Introduction)"}],"minor_comments":[{"comment":"The abstract uses inconsistent grammatical number: 'LLM agent was initialized' is followed by 'These agents predict'; revise for consistency.","section":"Abstract"},{"comment":"The caption 'Agent-Based Simulation of Dartmouth College Students' is vague about whether the simulation reproduces the existing StudentLife cohort or generates new synthetic students; clarify the caption to match the actual scope.","section":"Figure 1 caption"},{"comment":"The GitHub repository name 'DarthMouthStudentSimulator' is reminiscent of 'Dartmouth' but may confuse readers; ensure the repository is permanently archived with a versioned DOI or release tag to support the reproducibility claim.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is currently a high-level proposal with the empirical core compressed into a single unsupported sentence in the abstract. The proxy-validity issue is the main scientific risk: without a demonstration that agent EMA responses respond to sensing manipulation in a human-aligned way, the intervention results are not interpretable. The paper may also be a better fit for a workshop or a dataset/benchmark venue unless the comparative evaluation is expanded into a full validation study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is real: an LLM agent-based simulation of student EMA responses grounded in smartphone sensing data from StudentLife, with counterfactual interventions as the payoff. That combination is not something I've seen, and the GitHub repo is a reasonable reproducibility gesture. The idea of testing intervention scenarios that aren't in the original dataset is worth taking seriously.\n\nBut what we have in front of us is only the abstract and the first page of the intro. The central claim, that GPT-4o-mini beats Gemini-2.5-flash on stress, sleep, and social prediction, is stated with no effect sizes, no confidence intervals, no baseline comparisons. That's the softest spot. The authors say they 'investigated various prompting techniques, memory systems, and mental state management strategies'—that's a lot of tuning, and without a fixed protocol it's easy to overfit the best configuration to this particular dataset. They also don't compare to a plain ML model on the sensing features, which is the natural bar for showing the LLM agent adds value.\n\nThe stress-test concern about the causal proxy is on target. The intervention claim only works if the LLM's response function is a valid proxy for a real student's response function. Held-out average accuracy doesn't establish that. A specific failure mode is the agent leaning on generic priors about student life (midterm stress, weekend socializing) rather than the individual sensing stream. A shuffle test or per-person sensitivity analysis would help, and the letter should ask for that.\n\nStill, I wouldn't call this a damaged paper. It's an under-supported one, and the missing validation is probably in the full methods section we don't have. The authors show clear thinking and honest framing—they don't oversell the simulation as a substitute for real data, just as a tool for exploring new scenarios.\n\nMy bottom line: this deserves a serious referee. The question is important, the framework is new, and the code is available, so a reviewer can actually check the claims. It needs revision, not rejection—at minimum baseline comparisons, per-individual metrics, and a direct test of the intervention proxy. I'd be glad to see it through review.","headline":"Promising LLM-agent + sensing framework, but the headline comparison is asserted without numbers and the intervention proxy needs validation before the results carry weight.","tokens_in":1981,"tokens_out":1796,"would_cite":false,"duration_ms":24115,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM agents, steered by smartphone sensing data and personality questionnaires, can simulate individual students' daily behaviors and self-reported mental health across a semester, with GPT-4o-mini beating…","keywords":["LLM agents","agent-based simulation","student mental health","smartphone sensing","ecological momentary assessment","StudentLife dataset","behavioral modeling","GPT-4o-mini"],"falsifier":"Take a held-out group of students from StudentLife who were not used to build or prompt the agents, and compare each agent's daily EMA stress, sleep, and social ratings to the real student's ratings on the same days. If the correlation between agent and real ratings is no better than chance, or if an intervention that increases agent-reported well-being does not move real students' reported well-being in a small prospective study, the central claim fails.","tokens_in":1173,"feed_emoji":"📱","tokens_out":1655,"duration_ms":20011,"temperature":0.7,"pith_summary":"The paper tries to establish that large language model agents can act as believable stand-ins for real students when given two inputs: the student's personality questionnaire answers and their smartphone sensing data. If true, this would let researchers study student mental health in ways the original data cannot support, such as testing interventions, simulating peer influence, and conducting hypothetical interviews. The authors report that one model, GPT-4o-mini, predicts students' self-reported stress, sleep, and social levels better than Gemini-2.5-flash. The value of the claim is that it turns static sensing datasets into interactive laboratories for mental-health research.","feed_headline":"LLM agents simulate student stress from phone sensors","feed_subtitle":"Personality-prompted AI agents predict students' stress, sleep, and social levels across a semester, enabling virtual intervention tests.","key_machinery":"The central machinery is an activity-based mental state management system: each agent maintains a dynamic internal mental state that is updated by the day's sensed activities, and this state is used to generate EMA responses and subsequent behavior predictions. This couples the sensing data to the LLM's language output, letting the agent react to interventions such as altered sleep or social patterns.","core_discovery":"Using the StudentLife Dataset, the paper constructs LLM agents that each represent one Dartmouth College student. Each agent is initialized with the student's Big Five personality questionnaire responses and is then fed the student's smartphone sensing signals day by day across the simulated semester. The agent generates daily predictions of behavior, answers ecological momentary assessment (EMA) questions about stress, sleep, and social state, and completes follow-up personality questionnaires. The paper finds that GPT-4o-mini outperforms Gemini-2.5-flash on these prediction tasks, and that the simulation can go beyond replication by supporting counterfactual manipulations of activities and personality.","pith_inferences":["If agent EMA responses are valid proxies for real student self-reports, then the framework could be used to pre-test digital health interventions before deploying them on students, saving time and reducing participant burden.","The same architecture might transfer to other domains where longitudinal self-reports are paired with passive sensing, such as workplace well-being or chronic disease monitoring, though this would require new validation.","A testable extension is to compare agent-simulated intervention effects against real-world intervention studies on the same population; agreement would strengthen the case for using LLM agents as experimental stand-ins.","The personality questionnaire completion at the end of the semester offers a way to detect whether an agent's simulated personality drifts, which could serve as a stability check for the mental state update mechanism."],"forward_implications":["Researchers could use the simulated agents to run intervention studies, changing activity patterns via sensing signals and observing predicted changes in mental health outcomes.","The framework enables agent-to-agent interactions, allowing exploration of peer influence on well-being within a simulated cohort.","Hypothetical interviews with agents could yield qualitative insights into mental health that are not available from the original EMA responses.","The finding that GPT-4o-mini outperforms Gemini-2.5-flash suggests that model choice matters for behavioral fidelity in such simulations."],"supporting_citations":[],"fun_headline_variants":["Virtual LLM agents simulate student stress from phone data","AI student clones predict mental health from smartphone sensing","Test interventions on virtual students driven by real sensor data","Phone data lets LLM agents explore what-if student scenarios"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM prompted with sensing data produces self-reported mental health responses that are a valid proxy for real students' ecological momentary assessment responses, so that interventions that change agent behavior would also change real student behavior.","fun_headline_variants_meta":{"raw":{"variants":["Virtual LLM agents simulate student stress from phone data","AI student clones predict mental health from smartphone sensing","Test interventions on virtual students driven by real sensor data","Phone data lets LLM agents explore what-if student scenarios"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001149,"raw_usage":{"total_tokens":4745,"prompt_tokens":909,"completion_tokens":3836,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":3773}},"tokens_in":525,"tokens_out":3836,"duration_ms":32236,"temperature":1.0,"reasoning_tokens":3773,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:38:38.105936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out group of students from StudentLife who were not used to build or prompt the agents, and compare each agent's daily EMA stress, sleep, and social ratings to the real student's ratings on the same days. If the correlation between agent and real ratings is no better than chance, or if an intervention that increases agent-reported well-being does not move real students' reported well-being in a small prospective study, the central claim fails.","supporting_citations":[],"review_version":1}