{"id":"1293e6e9-f1d6-4301-b4cf-9afb4a5afbd8","arxiv_id":"1908.04617","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Smartphone-based personality classifiers trained on four countries predict traits in a fifth country with 63-71% accuracy, and adding country-specific data improves accuracy by 3-7% for three of the five traits.","lead":"Researchers collected phone sensor data and personality questionnaires from 166 people in five countries and found that machine learning models can infer personality traits from phone usage, but that models work better when they include data from the country where they will be used. The study suggests personality inference from smartphones is not culture-neutral, which matters for companies and researchers deploying such models globally.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Method 2 adds country dummy variables as features while labels are global-median splits, so the reported 3–7% 'cultural impact' may be a country-label shortcut rather than learned behavioral differences.","rationale":"The reader's weakest assumption concerned cross-country differences in sensor data fidelity (missingness, phone models, opt-out patterns). That is a real concern and is explicitly acknowledged in §3.2. However, the more immediate and load-bearing problem is that Method 2 includes explicit country dummy variables while Method 1 does not, and the outcome variable is a global-median split. This setup allows the 'country-specific data' advantage to be realized by the dummy variables alone, independent of any behavioral signal in the sensor features. The paper does not test this alternative explanation. The limitation section acknowledges low per-country sample sizes and data gaps, but does not mention the country-flag confound or the global-median label issue. A simple ablation would settle whether the central claim is about culture or about a label shortcut. For this reason, the appropriate verdict is CONDITIONAL: the current evidence is not sufficient to accept the cultural-impact interpretation as stated, but the concern is testable and the paper could be strengthened substantially by reporting the proposed control conditions.","tokens_in":21682,"tokens_out":7094,"duration_ms":76474,"concrete_test":"Re-run the Method 1 vs Method 2 comparison under three conditions: (a) Method 2 without the five country dummy features; (b) a label-only model trained solely on the five country dummies predicting the global-median target; (c) Method 2 with labels computed from within-country medians (or country-mean-centered trait scores before median split). If the accuracy gains for Extraversion, Agreeableness, and Conscientiousness largely disappear under (a), or if the label-only model in (b) reproduces the 3–7% improvements, the reported cultural impact is an artifact of the country shortcut. If the gains persist under (a) and (c), the behavioral-data interpretation is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in §5.1 is that including country-specific datasets improves classification of Extraversion, Agreeableness, and Conscientiousness by 3–7%. The comparison in §4.2 contrasts Method 1 (leave-one-country-out, no country features) with Method 2 (leave-one-subset-out). In §3.4, however, only Method 2 'added 5 binary country flags (a dummy variable) to the other features.' Because the classification target in §4.1 is a binary split at the global median of each trait, and the five countries differ in trait means and medians (Table 3), the country flags alone can be highly predictive of the class label. For example, median Neuroticism is 34.5 in Colombia versus 26 in the UK, and median Conscientiousness is 36.5 in Colombia versus 33 in Spain. A Method 2 model can therefore use 'country' as a shortcut to the global-median split, with no sensor data at all. This directly threatens the interpretation that the accuracy gains reflect cultural differences in behavioral manifestations captured by mobile sensing. The paper reports no ablation of the country flags and no control condition showing the gains survive when the flags are removed or when labels are centered within country. As written, the McNemar-significant differences could be driven by the dummy variables rather than by learned behavioral patterns.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether mobile-sensing-based personality classifiers trained in one set of countries generalize to a new country, and whether adding country-specific training data improves accuracy. The authors collected smartphone sensor and usage data for three weeks from 166 participants across the UK, Spain, Peru, Colombia, and Chile, and trained Random Forest classifiers for binary Big Five trait splits at the global median. They compare a leave-one-country-out model (Method 1) with a leave-one-subset-out model (Method 2) and report that the latter, which includes country-specific training data, improves accuracy by 3–7% for Extraversion, Agreeableness, and Conscientiousness. They also analyze which sensing categories are most predictive per country and test robustness to gender and age balancing.","tokens_in":21886,"tokens_out":4922,"duration_ms":50063,"significance":"If the central claim holds, this is a useful contribution: it is among the first studies to test cross-country generalization of mobile-sensing personality models, and it provides a multi-country dataset and a candid discussion of limitations. The reported accuracies (63–71% for country-agnostic models, up to 74% with country-specific data) are comparable to prior single-country work, and the paper's emphasis on cultural context is a timely topic. The authors also deserve credit for transparently describing data-loss mechanisms, the imputation approach, and the small per-country sample sizes. However, the main evidence for 'cultural impact' is threatened by a design confound in the comparison between Method 1 and Method 2, which is load-bearing for the paper's central claim.","major_comments":[{"comment":"The comparison between Method 1 and Method 2 conflates 'including country-specific training data' with 'adding country identity features.' Method 2 adds five binary country flags to the feature set, while Method 1 does not; because the classification labels are binary splits at the global median, and Table 3 shows that country trait medians differ (e.g., Neuroticism median is 34.5 in Colombia versus 26 in the UK), the country flags alone can be highly predictive of the label, with no sensor data. The reported 3–7% accuracy improvements for Extraversion, Agreeableness, and Conscientiousness could therefore reflect a country-label shortcut rather than learned cross-cultural differences in behavioral manifestations. The paper should report an ablation of Method 2 with the country flags removed (same train/test splits) and/or a control in which labels are centered within country, to confirm that the improvement survives without the shortcut. As written, the McNemar test only establishes that the two model variants differ, not that the difference is cultural.","section":"§3.4, §4.2, §5.1"},{"comment":"It is unclear whether Recursive Feature Elimination (RFE) is nested inside each cross-validation fold or performed once on the full dataset before cross-validation. If RFE is applied to the entire dataset, test-fold information leaks into feature selection, which would optimistically bias all reported accuracies in Tables 4–6 and the feature-importance analyses in Figures 2 and 3. The authors should specify the implementation and, if leakage exists, re-run the analysis with feature selection performed inside each training fold; a nested-CV ablation would settle the point.","section":"§3.4"},{"comment":"The 30% missing-data exclusion threshold was chosen by exploring model accuracy as a function of data availability, i.e., by inspecting the outcome variable; this risks selecting a subsample that artificially inflates accuracy. In addition, the paper does not report post-exclusion missingness by country, even though data gaps arose from sensor-specific opt-outs and phone-model differences (e.g., Pedometer present in only 53% of phones). If missingness or hardware characteristics differ systematically by country, the between-country differences in model performance and in feature distributions (Figures 2 and 3) could reflect measurement artifacts rather than cultural differences in behavior. Please report per-country data availability after imputation and a sensitivity analysis around the exclusion threshold.","section":"§3.2"}],"minor_comments":[{"comment":"The text states the country-agnostic model achieved 71% accuracy for Neuroticism, but Table 4 reports 72%; please correct the inconsistency.","section":"§5.1 vs. Table 4"},{"comment":"The Limitations section says 'our per country samples do not exceed N=21', but Table 1 shows UK N=27, Spain N=69, Peru N=25, and Chile N=24; the phrasing should read 'are as low as 21' or similar.","section":"§6, Table 1"},{"comment":"The text in §3.4 (RQ2) states 'We intentionally opted not to explore single country models... due to a relatively low number of participants,' yet §4.4 trains and analyzes a separate model for each country to identify top predictive features; please clarify the distinction between these two statements.","section":"§3.4 and §4.4"},{"comment":"No confidence intervals or other uncertainty measures are reported for the accuracy estimates; for test subsets as small as 21–27 participants, a 3% difference in accuracy may be within sampling variability, so adding binomial or bootstrap intervals would help interpret the magnitude of the reported improvements.","section":"Tables 4–6"},{"comment":"The figure labels contain an odd notation 'X 102' that is difficult to read; please reformat the axis labels for clarity.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper tackles a timely and understudied question, and the dataset is a useful resource. However, the central claim of a cultural impact rests on a comparison that is confounded by the addition of country dummy variables in Method 2, and the RFE nesting issue could substantially affect the absolute accuracies. Both are fixable with additional analyses (an ablation of the country flags, within-country label centering, and a nested-CV RFE check), so I see this as a major-revision case rather than a rejection. The authors were unusually candid about limitations, which is to their credit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: the central claim—that country-specific data improves trait classification by 3–7%—is likely weakened by a methodological confound. The dataset and the leave-one-country-out transfer results are a real contribution, but the headline interpretation needs more work.\n\nWhat's new and useful: this is the first mobile-sensing personality study spanning five countries (UK, Spain, Peru, Colombia, Chile), with 166 participants over three weeks. The leave-one-country-out protocol (Method 1) is a clean way to ask whether a model trained on several countries transfers to a new one; the reported 63–71% accuracies are plausible and above chance. The per-trait, per-country feature analyses (noise, location, unlocks, accelerometer) are descriptive but a reasonable starting point.\n\nThe soft spot is not minor. In Method 2, the authors add five binary country flags to the feature set, and the classification target is a global-median split of each trait. Table 3 shows clear country differences in medians—e.g., Neuroticism median is 34.5 in Colombia vs. 26 in the UK. Under a global median split, country membership alone predicts the label for many participants. The reported McNemar-significant gains for Extraversion, Agreeableness, and Conscientiousness could therefore come from the country flags rather than from sensed behavior. There is no ablation of the flags, no within-country label centering, and no test of a model using only the flags. That omission cuts the ground out from under the headline, not just a secondary analysis.\n\nSmaller concerns: the 30% missing-data threshold was set after inspecting accuracy; it is unclear whether recursive feature elimination is nested inside the cross-validation loop; per-country samples run from 21 to 27; and no confidence intervals are given. The authors are candid about many of these in the limitations section, and they explicitly restrain their cultural conclusions in places—but the abstract and discussion do not carry that caution.\n\nOn citations: the paper covers the relevant prior work, and the two self-citations are not load-bearing.\n\nWho this is for: anyone working on cross-cultural transfer of behavioral sensing models, and anyone teaching at the intersection of ML and social science. It deserves a serious referee, but the decision should be conditional on major revisions: add an ablation of the country dummies, test within-country label splits, and release code/data so the feature-selection pipeline can be inspected.\n\nIf the 3–7% gain disappears once country labels are properly controlled, the paper is a modest descriptive contribution. If it survives, it is a genuinely important finding. Right now, it does not survive the obvious control.","headline":"The new five-country dataset is a useful first step, but the claim that country-specific data improves trait classification is undermined by an unablated country-dummy confound.","tokens_in":22467,"tokens_out":3171,"would_cite":false,"duration_ms":29643,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A smartphone-sensing classifier trained on four countries can predict Big Five personality in a fifth at 63–71% accuracy, and adding country-specific training data significantly improves three of the five traits.","keywords":["personality inference","smartphone sensing","Big Five","cross-cultural","mobile sensing","machine learning","country-specific models","feature importance"],"falsifier":"Take only participants with complete accelerometer, unlock, light, and battery logs (no imputation), rebalance the five countries on phone model and gender, and rerun the four-country-train/fifth-country-test comparison; if the significant 3–7-percentage-point gains for Extraversion, Agreeableness, and Conscientiousness disappear, the cultural explanation is unsupported.","tokens_in":21438,"feed_emoji":"📱","tokens_out":9524,"duration_ms":92986,"temperature":0.7,"pith_summary":"This paper seeks to establish that personality models trained on smartphone sensor data do not travel perfectly across countries, but they travel well enough to be useful. Using three weeks of sensing and Big Five self-reports from 166 participants in the UK, Spain, Peru, Colombia, and Chile, the authors train random-forest classifiers on four countries and test them on the fifth; accuracy lands between 63% (Agreeableness) and 71% (Extraversion). They then show that adding the target country's data to training improves Extraversion, Agreeableness, and Conscientiousness by 3, 7, and 3 percentage points, respectively, while Neuroticism and Openness show no gain. If correct, this matters for anyone deploying personality-based personalization across markets: a single global model is plausible, but local data buys a real accuracy edge for some traits. It also gives social scientists a concrete sign that culture shapes how traits are expressed in everyday sensed behavior.","feed_headline":"Phone data predicts personality abroad; local data improves it","feed_subtitle":"Training on local sensor logs lifts three Big Five traits by 3–7 points.","key_machinery":"The argument is carried by a paired comparison of two random-forest classifiers built from 284 features across eight sensing categories (accelerometer, battery, calls, unlocks, light, location, noise, pedometer). Both predict above/below-median labels for each Big Five trait; the leave-one-country-out classifier is tested on a country it never saw during training, and the leave-one-subset-out classifier is tested on held-out users drawn from the same multi-country pool, with country indicator flags added to the features. The gap between the two isolates the contribution of country-specific data. Recursive feature elimination, repeated random subsampling (about 104 rounds for the cross-country comparison), and trait-by-country feature-importance rankings give the comparison its evidential weight.","core_discovery":"The paper's central claim is that personality inference from phone sensors is partly culture-bound, in a trait-specific way. A classifier trained on four countries and tested on a fifth achieves binary median-split accuracy of 71% for Extraversion, 63% for Agreeableness, 68% for Conscientiousness, 71% for Neuroticism, and 70% for Openness. When the training set also contains data from the test country, accuracy rises significantly by 3 points for Extraversion, 7 for Agreeableness, and 3 for Conscientiousness; Neuroticism and Openness do not improve, indicating those two traits are more robust to cultural context. The authors interpret these results as evidence that the behavioral signatures of personality overlap across countries but are not identical, so models built from culturally diverse data generalize to new countries while still benefiting from country-specific calibration.","pith_inferences":["The authors leave implicit that the 3–7-point country gain sets an upper bound on what local calibration can buy; a service could test whether a small local sample, rather than a full local model, captures most of that gain.","Because weekend features are especially predictive, a data-minimizing design that collects only weekend sensor logs plus a small local calibration sample might retain most of the accuracy at lower privacy cost; this is a direct testable extension of the paper's findings.","The country effect may partly reflect infrastructure rather than culture; a replication that matches countries on phone models and data-completeness rates would show how much of the gain is genuinely cultural.","If the effect is cultural, region-within-country variation may be detectable too, and leave-one-region-out tests in a single large country would extend the authors' design to sub-national cultures."],"forward_implications":["A personality classifier trained on four countries and applied to a fifth can sort users above and below the trait median with 63–71% accuracy, so cross-country deployment is feasible without first collecting local training data.","Adding local training data gives statistically significant accuracy gains of 3, 7, and 3 percentage points for Extraversion, Agreeableness, and Conscientiousness, while Neuroticism and Openness gain nothing, so the value of local calibration is trait-dependent.","Controlling or balancing gender and age in the training set improves prediction by up to 17 percentage points, and gender-separated models predict better still, so demographic composition is as important as country mix.","Noise, location, unlock, and accelerometer features are the most predictive categories across countries, whereas call features are less predictive than earlier studies suggested, pointing to where future sensing pipelines should focus."],"supporting_citations":[{"why":"Establishes the Big Five structure as a human universal, the premise that the same five traits can be measured in all five countries.","marker":"[42]"},{"why":"Shows that behavioral manifestations of traits differ across cultures, motivating the hypothesis that country affects sensor-behavior trait links.","marker":"[12]"},{"why":"Documents country differences in how daily social interaction relates to Big Five traits, grounding the predicted cultural impact.","marker":"[45]"},{"why":"Demonstrates smartphone data can classify Big Five traits at roughly 70%, the benchmark the new models are compared against.","marker":"[9]"},{"why":"Supplies phone-log features and single-country accuracy from an earlier mobile personality study.","marker":"[16]"},{"why":"Provides the within-person variability feature approach used in the feature set.","marker":"[58]"},{"why":"Provides the data-availability threshold method used to decide which participants to keep.","marker":"[47]"},{"why":"Large-sample phone-based personality prediction study used as a sample-size anchor in the technical literature.","marker":"[44]"},{"why":"Supplies the 50-item Big Five questionnaire used to obtain ground-truth trait labels.","marker":"[24]"},{"why":"Shows social-network phone features can classify Big Five traits at 65–80%, a prior accuracy range the study compares against.","marker":"[53]"}],"fun_headline_variants":["Phone data personality models vary by country; local data helps","Culture shapes phone-based personality inference: local data boosts accuracy","Personality from phone sensors: country-specific training lifts 3 of 5 traits","Phone sensing personality: cultural context matters for some traits","Local phone data improves personality predictions in new countries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the phones measured everyone's behavior equally in all five countries; if users in some countries shared less data, or owned different phones, the accuracy differences credited to culture could instead be measurement artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Phone data personality models vary by country; local data helps","Culture shapes phone-based personality inference: local data boosts accuracy","Personality from phone sensors: country-specific training lifts 3 of 5 traits","Phone sensing personality: cultural context matters for some traits","Local phone data improves personality predictions in new countries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1353,"prompt_tokens":1001,"completion_tokens":352,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":269}},"tokens_in":617,"tokens_out":352,"duration_ms":3826,"temperature":1.0,"reasoning_tokens":269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:36:32.973328+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take only participants with complete accelerometer, unlock, light, and battery logs (no imputation), rebalance the five countries on phone model and gender, and rerun the four-country-train/fifth-country-test comparison; if the significant 3–7-percentage-point gains for Extraversion, Agreeableness, and Conscientiousness disappear, the cultural explanation is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the Big Five structure as a human universal, the premise that the same five traits can be measured in all five countries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that behavioral manifestations of traits differ across cultures, motivating the hypothesis that country affects sensor-behavior trait links."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents country differences in how daily social interaction relates to Big Five traits, grounding the predicted cultural impact."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates smartphone data can classify Big Five traits at roughly 70%, the benchmark the new models are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies phone-log features and single-country accuracy from an earlier mobile personality study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the within-person variability feature approach used in the feature set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the data-availability threshold method used to decide which participants to keep."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Large-sample phone-based personality prediction study used as a sample-size anchor in the technical literature."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 50-item Big Five questionnaire used to obtain ground-truth trait labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows social-network phone features can classify Big Five traits at 65–80%, a prior accuracy range the study compares against."}],"review_version":1}