{"id":"7feb4f4c-a883-4d1d-b55a-312235f9e2f8","arxiv_id":"2607.28854","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A simulation benchmark claims semi- and non-parametric models outperform parametric models at recovering discrete choice rules, but flawed data generation and a misapplied BIC weaken the claim.","lead":"This paper benchmarks four machine-learning models on how well they recover five psychological choice rules from simulated two-alternative decisions, and finds semi- and non-parametric models generally beat parametric ones. The comparison, however, relies on a mis-specified choice-probability equation and an incorrect BIC penalty, undermining the quantitative conclusions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1)/(8) makes α a Left bias, not a determinism coefficient: α=0 forces Right, α=10 gives P(L)=10/11 at equal utility. The determinism experiment does not vary randomness-to-determinism, so a central claim is unsupported.","rationale":"I focused on the data-generating logistic because the paper's central claim has two legs: (1) flexible models generally outperform parametric ones, and (2) performance improves with more training sets and with more deterministic choice rules. The second leg depends entirely on the α manipulation. Equation (1) is algebraically not a determinism parameter; it is a prior odds term. At α=0 the DGP produces P(L)=0, so the 'random' baseline in the determinism experiment is not random. At α=10 with equal utilities, P(L)=10/11, so data are Left-biased. Thus the reported improvement as α increases is confounded with an intercept shift and cannot be interpreted as learning under increasing determinism. This is the reader's weakest assumption, and I agree it is load-bearing. I also considered the BIC miscount (k set to number of attributes for every model) and the case-study null log-likelihood issue; these are serious and reinforce rejection, but the DGP error is more fundamental because it undermines the experimental manipulation itself. A corrected simulation of the determinism experiment would settle whether the effect survives. Since this confirms rather than changes the reader's rejection, the REJECT verdict stands.","tokens_in":18771,"tokens_out":10996,"duration_ms":119948,"concrete_test":"Re-run the determinism experiment (Table 1) with the corrected temperature logistic P(L)=e^{αΔu}/(1+e^{αΔu}) for α=0,...,10, keeping all other settings and the same models and BIC computation, then recompute Fig. 9. If the claimed monotone 0–55% BIC improvements are not reproduced, the original determinism conclusion is an artifact of Eq. (1)'s intercept shift rather than of choice-rule determinism. Additionally, report P(L) at α=0 under Eq. (1) with Δu=0: it is 0, not 0.5, which directly falsifies the 'random' baseline.","verdict_should_be":"REJECT","load_bearing_attack":"Equation (1) (and Eq. (8) for MLBA) is written as P(L)=α e^{Δu}/(1+α e^{Δu}), so the odds of L over R are P/(1−P)=α e^{Δu}. Thus α enters as a multiplicative baseline odds for the Left alternative, not as a temperature/determinism parameter. The paper's own Section 4.2.1 defines the experiment as varying α from 0 (random) to 10 (deterministic), but at α=0 the formula gives P(L)=0 regardless of Δu—a deterministic choice of Right, not random choice. At α=10 and equal utilities, P(L)=10/11, a strong Left bias, not a sharpened symmetric sigmoid. The correct sharpening form would be P(L)=e^{αΔu}/(1+e^{αΔu}) (or σ(αΔu)), which gives P(L)=0.5 at α=0 and steepens as α grows. Because all probabilistic choice rules (linear strong utility, monotonic strong utility, ideal point, MLBA) use this equation, the determinism experiment's manipulation is confounded: increasing α shifts the intercept/bias while the slope in Δu remains 1. Consequently the reported '0% to 55%' BIC improvement with determinism, and the conclusion that more deterministic rules are easier to learn, are not supported by the experiments as run. The Limitations section notes that metrics can disagree and significance testing was not performed, but this is a more fundamental problem with the data-generating process.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a Monte Carlo benchmark of four machine-learning models—multinomial logistic regression (MNL), generalized additive model (GAM), twinned neural network (TNN), and Gaussian process (GP)—for recovering five discrete choice rules (linear strong utility, monotonic strong utility, ideal point, lexicographic semiorder, multiattribute linear ballistic accumulator). Three simulation experiments vary the number of attributes, the number of training choice sets, and a 'determinism' coefficient α; performance is reported primarily via BIC, with additional metrics. A case study on real energy-policy preference data is included. The paper claims that semi- and non-parametric models generally outperform parametric models and that performance improves with more training choice sets and greater choice-rule determinism.","tokens_in":19260,"tokens_out":7489,"duration_ms":82252,"significance":"The research question is relevant to policy-oriented preference elicitation, where flexible models could reduce reliance on restrictive parametric assumptions. The paper's strengths include an open-source OSF repository, systematic coverage of five distinct choice rules, and the use of multiple accuracy metrics. However, the validity of the benchmark rests on the correctness of the data-generating process and on the BIC complexity penalty; both are problematic as detailed below. If corrected, the study could provide useful baseline evidence for model selection in discrete choice analysis.","major_comments":[{"comment":"The coefficient α is not a determinism parameter. Because P(L)=α e^{Δu}/(1+α e^{Δu}), the odds are P/(1−P)=α e^{Δu}; α multiplies the baseline odds of choosing Left, not the steepness of the sigmoid. At α=0 the formula gives P(L)=0 for every Δu, which is a deterministic choice of Right, not random choice; at α=10 and Δu=0, P(L)=10/11, a strong Left bias. The intended 'random to deterministic' manipulation in the determinism experiment is therefore confounded with a left-right bias. Since the large- and small-choice-set experiments fix α=10, the data-generating process is a biased logistic rather than the standard Fechnerian strong utility or MLBA rule. Consequently the reported BIC improvements with determinism (Tables 2–4, Figs. 8–9) and the abstract's '0% to 55%' claim are not supported by the experiments as run.","section":"Sec. 3.2.1, Eq. (1); Sec. 3.2.3, Eq. (8); Sec. 4.2.1"},{"comment":"The BIC is computed with k equal to the number of attributes for every model. This is not the number of estimated parameters: MNL has K coefficients, but GAM has smooth terms with effective degrees of freedom, TNN has thousands of weights, and GP has kernel hyperparameters plus nonparametric function estimates. Using the same k for all models removes the complexity penalty that BIC is designed to apply; the 'best model by BIC' comparisons in Figs. 4, 6, and 8 are therefore essentially likelihood rankings with a uniform penalty. The conclusion that flexible models outperform parametric models may be an artifact of this misspecified penalty. The authors should report model-specific k or use an alternative criterion (e.g., cross-validated log-likelihood, AIC with correct parameter count, or effective degrees of freedom).","section":"Sec. 3.5, Eq. (11)"},{"comment":"The BIC is undefined for participants with predicted probabilities of 0 or 1, which the authors report for 642 of 822 participants under MNL. The paper does not state how these null/undefined BIC values were handled when computing the reported mean BIC values (e.g., 13.351 for TNN, 17.491 for MNL). If they were excluded, the comparison is biased toward models that happen not to produce extreme predictions; if they were counted as infinite, the mean is not finite. The case-study ranking is therefore uninterpretable without a stated treatment.","section":"Sec. 4.4, case study"}],"minor_comments":[{"comment":"Numerous subject-verb agreement errors (e.g., 'four machine learning models was trained', 'attributes is') and other grammatical issues need a careful proofread.","section":"Throughout"},{"comment":"The indicator function only defines the choice when the m-th attribute difference exceeds ε; the random tie-breaking when no attribute difference exceeds the threshold is described in the text but not represented in the equation. Clarify the formal definition.","section":"Sec. 3.2.2, Eq. (5)"},{"comment":"The determinism experiment uses attribute levels U(0,2) whereas the other experiments use U(0,1). The text does not explain this change or its potential effect on utility differences.","section":"Table 1"},{"comment":"The figure caption/note mentions 'brier scores' in the shaded-region description, but the figure displays BIC. This appears to be a typo.","section":"Fig. 5 note"},{"comment":"The Limitations section properly acknowledges the absence of significance testing, but the paper still draws strong conclusions from point estimates with overlapping uncertainty bands. Reporting effect sizes with confidence intervals, or a formal significance test, would strengthen the claims.","section":"Sec. 5.1"}],"recommendation":"major_revision","confidential_remarks":"Both major issues are correctable but require substantial rework: the determinism coefficient must be respecified (e.g., P=σ(αΔu)) and all simulations re-run; and the BIC complexity term must use model-appropriate parameter counts or be replaced by a valid comparison criterion. The case study's null-BIC handling must also be clarified. The dataset from reference [18] includes a co-author; this is not necessarily problematic, but the relationship should be disclosed. If the authors can address these points, the paper could serve as a useful benchmark; as submitted, the current conclusions are not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my take. The paper's benchmark grid — four ML models against five choice rules across attribute count, training-set size, and determinism — is genuinely new and well-motivated. The simulations are clearly described, the code appears available, and the case study is a nice touch. I want to give credit where due: the design is sensible overall, and the authors are upfront about limitations like differing metrics and lack of significance testing.\n\nBut there's a load-bearing flaw that the stress-test confirms. Eq. (1) defines P(L)=alpha*e^{Delta u}/(1+alpha*e^{Delta u}). That makes alpha a multiplicative prior odds for the Left alternative, not a temperature/determinism parameter. At alpha=0 the model always chooses Right; at alpha=10 with equal utilities it chooses Left with probability 10/11. The paper's own Section 4.2.1 says alpha=0 is random and higher alpha means more deterministic, so the equation contradicts the verbal definition. This contaminates all three Monte Carlo experiments and completely invalidates the determinism experiment: as alpha increases, you're not sharpening the sigmoid, you're shifting the intercept. The reported 0% to 55% BIC improvement as alpha grows is uninterpretable — models may simply be learning a near-constant left choice.\n\nThere are additional issues. The BIC formula in Eq. (11) uses k equal to the number of attributes for every model, which cannot compare models with different parameter counts. The case study reports finite BIC values while also saying 78% of MNL fits had null log-likelihoods; that needs explanation.\n\nThe central qualitative claim — flexible semi- and non-parametric models often beat parametric MNL — may still be true, but this manuscript doesn't demonstrate it because the data-generating process is misspecified. The fix is straightforward: use P(L)=sigma(alpha*Delta u) and re-run. If the authors do that, the benchmark would be worth taking seriously. As it stands, the main conclusions don't hold.\n\nMy recommendation: this deserves a serious referee, but it needs major revision before it's publishable. I wouldn't cite it in its current form. For a reading group, it's a useful case study in how a subtle DGP bug can sink an otherwise careful empirical project.","headline":"A genuinely useful benchmark grid undermined by a DGP misspecification: alpha in Eq. (1) is a left-bias multiplier, not a determinism parameter, so the determinism experiment doesn't test what it claims.","tokens_in":19648,"tokens_out":4737,"would_cite":false,"duration_ms":49226,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Flexible machine-learning models recover individual discrete-choice rules more accurately than parametric logistic models in simulated preference tasks.","keywords":["discrete choice modeling","preference elicitation","machine learning","generalized additive model","Gaussian process","multinomial logit","choice rules","Monte Carlo simulation"],"falsifier":"Run the paper's own generator at α=0 and at equal utility: Eq. (1) gives P(L≻R)=0 and P=α/(1+α), respectively, so α is not a pure randomness dial. Re-running the determinism experiment with a centered logistic, such as P=σ(α Δu), and checking whether GAM and GP still beat MNL and TNN would settle whether the reported determinism improvements are artifacts of the equation.","tokens_in":18661,"feed_emoji":"🧠","tokens_out":9086,"duration_ms":93383,"temperature":0.7,"pith_summary":"This paper asks whether machine-learning models can recover an individual's decision rule from a small number of two-alternative choices without assuming a linear utility form. Across simulations of five psychologically motivated choice rules, the authors find that flexible semi-parametric (generalized additive model) and non-parametric (Gaussian process) approaches generally predict held-out choices better than the parametric multinomial logit and twinned neural network models. Performance improves as training choice sets grow and as choices become more deterministic, but the best model depends on the choice rule. A real energy-policy preference dataset shows the twinned neural network fitting best, while multinomial logit overfits badly. The paper's central message is that model selection should follow the decision context, and that flexible models are a viable alternative to parametric defaults in preference elicitation.","feed_headline":"Flexible ML beats standard logit at recovering individual choice rules","feed_subtitle":"GAM and Gaussian process models usually outpredict multinomial logit in simulations of five choice rules; a guide for preference surveys.","key_machinery":"The engine is a controlled data-generating process for two-alternative forced choices. It computes a latent score difference for each of five choice rules — weighted utility differences for the strong-utility and ideal-point rules, a weighted valence difference with subjective log-mapping for the MLBA — and turns that difference into a binary left/right choice with the logistic P(L≻R)=α e^{Δ}/(1+α e^{Δ}); the lexicographic semiorder instead decides by thresholded comparisons of ranked attributes. Since the true rule is known in simulation, the performance of each learning model on held-out choices measures how well the model recovers the rule. The four compared learners — parametric multinom","core_discovery":"The paper's central result is a comparative benchmark: given two-alternative choices generated by any of five formal choice rules (linear strong utility, monotonic strong utility, ideal point, lexicographic semiorder, and multiattribute linear ballistic accumulator), the semi-parametric generalized additive model and non-parametric Gaussian process recover the underlying rule more accurately than the parametric multinomial logit or twinned neural network in most contexts. The authors also report regularities that hold across rules: increasing training choice sets improves model fit by roughly 6% to 96%, and increasing the determinism of the choice rule improves fit by 0% to 55%, with the ide","pith_inferences":["Not tested in the paper: the determinism coefficient as written may distort findings, because α=0 gives P=0 rather than the intended random 0.5, and equal utilities give P=α/(1+α); a properly centered logistic could change the determinism-experiment rankings.","Not tested in the paper: the ideal-point rule's consistent difficulty suggests latent ideal values create weak utility gradients; active learning that queries choices near an estimated ideal point could be a testable remedy.","Not tested in the paper: since the winning model changes with context, a decision-support system could switch among GAM, Gaussian process, and TNN per respondent based on early response consistency."],"forward_implications":["Studies that use multinomial logit as a default for preference elicitation may be sacrificing accuracy; switching to a generalized additive model or Gaussian process can improve recovery of the underlying choice rule without requiring a correct parametric form in advance.","Sample size is a first-order lever: moving from 10 to 100 training choice sets reduced BIC by 48% to 96%, so preference-elicitation studies should budget for repeated choices per person whenever possible.","Choice consistency matters: as the simulated choice rule becomes more deterministic, model fit improves by up to 55%, so comparisons between studies should control for how consistently participants choose.","There is no universally best model; context should drive selection — GAM did best for linear strong utility and MLBA in the large-choice-set experiment, GP for ideal point and monotonic strong utility, and TNN for lexicographic semiorder.","In the real energy-policy case, TNN had the lowest BIC, while MNL overfit badly (null BIC in 642 of 822 respondents), suggesting parametric models need careful regularization on real small-sample data."],"fun_headline_variants":["Flexible ML models top parametric in five choice-rule tests","GAM and Gaussian process outperform logit in choice simulations","Choice-rule recovery improves 6–96% with more training sets","Semi-parametric models beat parametric in choice rule recovery","Data-driven choice models gain edge from flexible estimation"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The results rest on the claim that the logistic in Eqs. (1) and (8) correctly implements the strong-utility and MLBA rules with α as a pure determinism dial; as written α=0 yields P=0 rather than random 0.5, so the simulation may not generate choices from the intended random-to-deterministic spectrum.","fun_headline_variants_meta":{"raw":{"variants":["Flexible ML models top parametric in five choice-rule tests","GAM and Gaussian process outperform logit in choice simulations","Choice-rule recovery improves 6–96% with more training sets","Semi-parametric models beat parametric in choice rule recovery","Data-driven choice models gain edge from flexible estimation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000313,"raw_usage":{"total_tokens":1658,"prompt_tokens":826,"completion_tokens":832,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":749}},"tokens_in":570,"tokens_out":832,"duration_ms":8925,"temperature":1.0,"reasoning_tokens":749,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T01:29:21.019808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's own generator at α=0 and at equal utility: Eq. (1) gives P(L≻R)=0 and P=α/(1+α), respectively, so α is not a pure randomness dial. Re-running the determinism experiment with a centered logistic, such as P=σ(α Δu), and checking whether GAM and GP still beat MNL and TNN would settle whether the reported determinism improvements are artifacts of the equation.","supporting_citations":[],"review_version":1}