{"id":"8bc1a710-031c-4314-b41d-f324a31549e9","arxiv_id":"2505.17661","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ASMR couples the Centaur foundation model with Qwen3 to flag and repair a simple cognitive model's failures, reaching Centaur-level AIC after five automated iterations.","lead":"This paper introduces an automated pipeline that uses a foundation model of human behavior and a large language model to find and fix gaps in simple cognitive models. In one decision-making experiment, the pipeline improved simple models until they matched the foundation model's accuracy, suggesting AI can help automate parts of cognitive science.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out evaluation and best-of-30 AIC selection: the 'surpasses Centaur' result likely reflects selection bias rather than discovered noise-ceiling models.","rationale":"The reader's verdict is CONDITIONAL, and that remains appropriate. The reader's weakest assumption focuses on Centaur's validity as an oracle for which trials are 'in principle predictable.' That is a real concern, but the more load-bearing issue for the paper's headline claim is the selection over many stochastic model-revision trajectories with in-sample AIC as the criterion. The strongest operational evidence cited in the abstract is the best-of-30 AIC of 71.73 beating Centaur, and that number is exactly the quantity most distorted by model selection. AIC is an estimate of expected out-of-sample deviance for a pre-specified model, not for the best model found by searching a large space of candidate models on the same data. The paper reports no held-out data, no cross-validation, and no correction for multiple comparisons across simulations. Without such a check, the observed improvement cannot be cleanly attributed to ASMR discovering genuinely better cognitive models; it may simply reflect overfitting to the evaluation set. The proposed concrete test would settle this directly: if the best discovered model retains its advantage on held-out participants, the central claim is substantially supported; if the advantage disappears, the paper should be read as a proof-of-concept whose quantitative headline is not yet established. This does not change the verdict from CONDITIONAL, but it sharpens the condition that must be met.","tokens_in":6524,"tokens_out":2454,"duration_ms":20802,"concrete_test":"Perform a participant-level cross-validation. Split participants into a fitting set and a held-out set (e.g., a 50/50 split, repeated over multiple random seeds). Run the full ASMR pipeline on the fitting set only, freeze the best discovered model according to its fitting-set AIC, and compute its AIC or cross-validated log-likelihood on the held-out participants. Compare this held-out AIC against Centaur's held-out AIC and against the held-out AIC of the initial seed models. If the selected best ASMR model's held-out AIC is not consistently at or below Centaur's, then the 'noise ceiling' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ASMR discovers models predicting at the noise ceiling rests on the observation that the best-performing discovered model reaches AIC 71.73 versus Centaur's 72.5 (Figure 2a). This comparison is not a valid out-of-sample evaluation. The AIC is computed on the same participant data used for parameter fitting and for selecting among models, and the reported best is selected across 10 simulations per model class across 3 initial model classes, i.e., across 30 stochastic revision trajectories. Because AIC is minimized over this search, the selected model's AIC is a downward-biased estimate of its expected predictive performance. Centaur, by contrast, is a fixed reference model that undergoes no selection. The average AIC after five iterations is 80.72, which is still worse than Centaur's 72.5; only the cherry-picked best 'surpasses' Centaur. The abstract's stronger phrase 'at noise ceiling' is therefore unsupported by the reported evidence. The cleanest interpretation is that the headline result is an artifact of selection over many stochastic search paths on the same data, not evidence that ASMR reliably discovers cognitive models that match or beat a strong black-box predictor. The paper's own limitation discussion acknowledges the need for a benchmark but does not address this selection problem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ASMR, an automated loop in which Centaur's per-trial negative log-likelihoods are used to identify trials where an interpretable cognitive model underperforms, and Qwen3-32B rewrites the model code to improve those trials. The method is demonstrated on a multi-attribute decision-making task, with three hand-coded initial models, five iterations, and ten stochastic simulations per initial model class. The central reported results are an aggregate AIC improvement from first to last iteration, a per-participant AIC improvement, and a best discovered model with AIC 71.73 compared to Centaur's 72.5, which the abstract interprets as predicting human behavior at the noise ceiling while retaining interpretability.","tokens_in":6777,"tokens_out":6001,"duration_ms":42956,"significance":"If the central claim were supported, ASMR would be a valuable step toward automated cognitive modeling: the pipeline is end-to-end, the revisions preserve executable model code, and the best discovered model is an interpretable weighted-additive/take-the-best hybrid. The iterative improvement shown in Figure 2a is a genuine demonstration that LLM-suggested revisions can reduce in-sample AIC. However, the significance is contingent on the current evaluation being replaced or supplemented by a held-out, selection-aware assessment; as written, the headline 'noise ceiling' and 'surpasses Centaur' claims are not established. The manuscript also helpfully identifies an open benchmark problem for automated scientific discovery, but it does not solve the evaluation question it raises.","major_comments":[{"comment":"The headline result (AIC_ASMR=71.73 vs. AIC_Centaur=72.5) is not an out-of-sample comparison. AIC is computed on the same participant data used for maximum-likelihood parameter fitting and for selecting among models, and the best model is selected from ten stochastic simulations per initial model class across three classes, i.e., from thirty revision trajectories. Since AIC is minimized during this search, the selected model's AIC is a downward-biased estimate of its expected predictive performance, while Centaur is a fixed reference that undergoes no selection. The average AIC after five iterations is 80.72, which is still worse than Centaur's 72.5; only the single best trajectory 'surpasses' Centaur. The paper should report held-out evaluation (e.g., leave-one-participant-out or repeated cross-validation) and should compare the full distribution of simulation outcomes rather than the best realization.","section":"Results, Figure 2a"},{"comment":"The abstract claims ASMR discovers models that predict human behavior 'at noise ceiling,' but no noise ceiling estimate is provided anywhere in the manuscript. There is no analysis of irreducible choice variability, no repeated-trial consistency measure, and no upper bound on achievable expected log-likelihood; the only reference point is Centaur's AIC, which is a model, not a ceiling. The claim should be removed or supported by a proper ceiling estimate; a 0.77 AIC-unit advantage over a single reference model on the fitting data is not evidence of ceiling-level prediction.","section":"Abstract; Results"},{"comment":"The procedure subtracts cached Centaur negative log-likelihoods from the fitted cognitive model's log-likelihoods and submits trials with Delta >= 0.05 to the reasoning model. This assumes Centaur's per-trial predictions are a valid oracle for which trials are 'in principle predictable.' That assumption is untested. If Centaur is miscalibrated for this task, the revision loop targets noise rather than systematic model error, and the measured AIC improvement may reflect overfitting to idiosyncratic trials. The paper should include a calibration check of Centaur's predicted probabilities (e.g., reliability curves or comparison with empirical choice frequencies) and a sensitivity analysis with respect to the Delta threshold.","section":"Methods, gap selection"},{"comment":"The manuscript reports AIC_Centaur = 72.5 without stating the number of parameters used in the AIC penalty for Centaur. Centaur is a fixed pretrained model whose parameters are not fitted to this dataset, so the standard AIC definition (2k - 2LL with k estimated on the same data) does not directly apply. If k=0 is used, the comparison with ASMR models, which pay a 2k penalty for their fitted parameters, is not a standard AIC comparison. Please report the exact formula used and, ideally, the raw negative log-likelihoods so readers can separate fit quality from model-complexity penalties.","section":"Results, Centaur AIC"}],"minor_comments":[{"comment":"The limitation paragraph correctly notes the need for a benchmark for automated scientific discovery, but it does not address the selection-bias problem in the in-sample AIC comparison; add an explicit acknowledgment that the reported best-model AIC is affected by selection over stochastic revision trajectories.","section":"Discussion"},{"comment":"The figure caption should clarify whether the shaded band is the min/max across simulations, across initial model classes, or across both, and should state how many participants contribute to the AIC values and whether AIC is summed or averaged.","section":"Figure 2a"},{"comment":"The text says 'five iterations were sufficient' but provides no stopping rule; specify how sufficiency was determined and whether the number of iterations was chosen after seeing the results.","section":"Methods"},{"comment":"The prompt instructs the reasoning model to 'structure your answer as follows' and 'not write any text besides that'; please describe how the model output is parsed into executable code and how failures of format compliance are handled.","section":"Methods, prompt template"},{"comment":"No code or data availability statement is included. Releasing the pipeline, the set of discovered models, and the participant data would substantially strengthen reproducibility for a framework paper.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a proof-of-concept from the group that also developed Centaur, and Centaur serves both as the guide and as the benchmark. This is not disqualifying, since the final AIC values are computed on human choices, but the evaluation is not independent in the sense of a neutral reference model. The most pressing fix is a held-out, selection-aware evaluation and an honest noise-ceiling estimate. I would not consider acceptance until those are provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is coupling Centaur with an LLM to automatically revise interpretable cognitive models, iterating on the trials where the cognitive model lags behind the foundation model. That is a plausible roadmap for automating a core part of the modeling pipeline, and the paper shows it can work on a simple task: AIC improves across iterations for every participant, and the Methods contain enough concrete code to reproduce the initial models and the prompt template. Credit where due: the combined system is not in the cited prior work, and the authors are transparent about the proof-of-concept status.\n\nThe soft spots are mostly in the quantitative claims. The headline result—best discovered model AIC 71.73 versus Centaur's 72.5—comes from selecting the best of ten stochastic runs per initial model class, thirty revision trajectories total, with AIC computed on the same participant data used for parameter fitting and model selection. That makes 71.73 a downward-biased estimate of expected predictive performance. The average AIC after five iterations is 80.72, clearly worse than Centaur. So the cherry-picked best does not support the abstract's 'at noise ceiling' claim, and 'surpasses Centaur' is overstated. There is no held-out evaluation, no noise-ceiling estimate, and the gap threshold Δ ≥ 0.05 is not varied to show sensitivity. The circularity concern is real but minor: Centaur picks the gap trials and is also the benchmark, but final AIC is computed on human choices, so the comparison is not purely self-referential.\n\nThe paper's own Discussion explicitly calls for a benchmark and flags future work, but it does not address the selection problem, which is the biggest issue. That matters because the central demonstration is exactly where the selection bias bites. Still, this is a serious proof-of-concept, not a hollow claim. The framework is worth engaging with; it needs cross-validation, a properly described selection procedure, and a more modest abstract before its headline claims stand.\n\nWho gets value: people working on automated cognitive modeling, LLM-guided discovery, and the Centaur line. A serious referee would get a useful picture of both the promise and the pitfalls. My recommendation: send it to peer review, but make clear that the 'surpasses Centaur' and 'noise ceiling' phrasing should be revised, and that out-of-sample evaluation is a requirement rather than a suggestion.","headline":"ASMR is a genuinely new automated model-revision framework, but its 'surpasses Centaur' and 'noise ceiling' claims rest on best-of-thirty in-sample AIC; worth peer review with major revisions.","tokens_in":7299,"tokens_out":2038,"would_cite":false,"duration_ms":22848,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An automated pipeline can take a black-box model of human cognition, use it to flag trials where an interpretable model fails, hand those failures to a reasoning model that rewrites the model, and within five rounds reach an interpretable…","keywords":["automated scientific discovery","cognitive modeling","scientific regret minimization","foundation models of human cognition","large language models","multi-attribute decision-making","interpretable models","Akaike information criterion"],"falsifier":"Run ASMR on the same multi-attribute task but choose the data points shown to the reasoning model at random, or by largest absolute deviation from empirical choice proportions, instead of from Centaur's gap; if the random-selection control matches the Centaur-guided condition in AIC after five iterations, the claim that the oracle's gap signal drives the improvement is falsified.","tokens_in":6313,"feed_emoji":"🧠","tokens_out":8515,"duration_ms":70141,"temperature":0.7,"pith_summary":"The paper tries to establish that scientific regret minimization can be made fully machine-driven: a foundation model of human cognition supplies the target, and a language-based reasoning model supplies the hypotheses. In the case study, ASMR starts from three simple decision heuristics, and after five automatic revisions the discovered models reach an average AIC of 80.72, close to Centaur's 72.5, with the best model at 71.73. The paper argues this approaches the noise ceiling while retaining interpretability, since the final models are short functions whose behavior a researcher can read. This matters because it would replace the manual cycle of collecting large task-specific datasets and staring at failure patterns with a repeatable, automated pipeline.","feed_headline":"Automated loop discovers simple models that rival a black box","feed_subtitle":"A reasoning AI rewrites cognitive models and, within five rounds, matches or beats Centaur's fit to human decisions.","key_machinery":"The load-bearing mechanism is the regret gap. In each iteration, ASMR fits the current cognitive model to each subject by maximum likelihood, then subtracts Centaur's cached per-trial negative log-likelihoods from the fitted model's; trials where this difference is at least 0.05 are the 'regret'---behavior that is in-principle predictable but not yet explained. Those trials, the current model code, and a fixed prompt are sent to Qwen3-32B, which outputs a revised model that is refit, and the loop repeats. The reasoning model is thus not asked to theorize from raw data; it is asked to edit code until the specific failure cases are covered.","core_discovery":"ASMR claims that the two bottlenecks of scientific regret minimization can be removed simultaneously. Centaur, a foundation model trained across many behavioral experiments, supplies per-trial predictions without new data collection; a reasoning model, Qwen3-32B, receives the trials where the current interpretable model loses to Centaur by at least delta = 0.05 in negative log-likelihood, together with the model's Python code, and returns a revised model. Applying this loop to a multi-attribute choice experiment improved every participant's AIC and produced a best model with AIC 71.73, which outperforms Centaur's 72.5, while the discovered code reveals an adaptive upweighting of the highest-validity expert---a mechanism intermediate between take-the-best and weighted-additive strategies.","pith_inferences":["A direct robustness test is to swap Centaur for an independent foundation model or for held-out human data and compare the discovered model families; if the gap set changes drastically, the oracle, not the reasoning model, is doing most of the work.","The discovered 'adaptive upweighting' may be partly caused by the prompt itself stating the expert validities, so an extension would run ASMR on a task without explicit validity information and ask whether the same mechanism still emerges.","Because the reported AIC values come from the same data used to fit the models, out-of-sample evaluation on new participants or trials would be a stricter check of whether the discovered models are reusable theories rather than per-subject fits.","The searchable database the paper envisions could double as a benchmark: recording each proposed model, its gap set, and its AIC trajectory would let future work quantify how much each revision step contributes."],"forward_implications":["Cognitive-model discovery no longer requires a large task-specific training study, because Centaur already embodies cross-task predictions and the reasoning model supplies the revision.","Interpretability does not have to be sacrificed for accuracy: the final models are short functions whose best discovered mechanism---upweighting the most valid expert---can be stated in words.","The iterative loop improves fit for every participant, suggesting the gap signal is informative at the level of individual subjects rather than only in aggregate.","The best discovered model's AIC of 71.73 against Centaur's 72.5 indicates the pipeline can extract more than the oracle it started from, at least on this task.","Five iterations were enough in this paradigm, so the same machinery is a plausible component of a larger automated-science benchmark spanning multiple paradigms."],"supporting_citations":[{"why":"Supplies the oracle: Centaur's cached per-trial negative log-likelihoods define which trials are in-principle predictable; removing it would leave ASMR without a target.","marker":"[Binz et al., 2024]"},{"why":"Defines scientific regret minimization, the framework whose bottlenecks (task-specific training data and manual pattern inspection) ASMR claims to automate.","marker":"[Agrawal et al., 2020]"},{"why":"Provides Qwen3-32B, the language-based reasoning model that receives failure points and returns revised model code at each iteration.","marker":"[Qwen, 2025]"},{"why":"Prior LLM-based cognitive-model automation whose reported performance the best ASMR model matches at AIC 71.73.","marker":"[Rmus et al., 2025]"},{"why":"Defines the multi-attribute decision-making paradigm with four binary expert ratings on which the case study runs.","marker":"[Hilbig and Moshagen, 2014]"},{"why":"Source of the take-the-best heuristic, one of the three model classes used to initialize ASMR.","marker":"[Gigerenzer et al., 2000]"},{"why":"Introduces heuristics from bounded meta-learned inference, another initialization class used in the simulations.","marker":"[Binz et al., 2022]"},{"why":"Provides the Bayesian framing under which the discovered adaptive upweighting is interpreted as interpolating between take-the-best and weighted-additive strategies.","marker":"[Parpart et al., 2018]"}],"fun_headline_variants":["AI rewrites cognitive models to beat black box","Self-improving AI discovers interpretable decision rules","Regret-minimizing AI finds interpretable models","Automated science loop beats black-box predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole loop rests on Centaur's per-trial negative log-likelihoods being a trustworthy oracle for which choices are predictable in principle; if those predictions are miscalibrated or unrepresentative for this task, ASMR flags the wrong trials and ends up revising toward noise.","fun_headline_variants_meta":{"raw":{"variants":["AI rewrites cognitive models to beat black box","Self-improving AI discovers interpretable decision rules","Regret-minimizing AI finds interpretable models","Automated science loop beats black-box predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2378,"prompt_tokens":794,"completion_tokens":1584,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":410,"completion_tokens_details":{"reasoning_tokens":1525}},"tokens_in":410,"tokens_out":1584,"duration_ms":9528,"temperature":1.0,"reasoning_tokens":1525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:42:11.849496+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ASMR on the same multi-attribute task but choose the data points shown to the reasoning model at random, or by largest absolute deviation from empirical choice proportions, instead of from Centaur's gap; if the random-selection control matches the Centaur-guided condition in AIC after five iterations, the claim that the oracle's gap signal drives the improvement is falsified.","supporting_citations":[{"cited_title":"Scaling up psychology via scientific regret minimization","cited_arxiv_id":null,"evidence_quote":"Defines scientific regret minimization, the framework whose bottlenecks (task-specific training data and manual pattern inspection) ASMR claims to automate."},{"cited_title":"Qwen3 technical report","cited_arxiv_id":null,"evidence_quote":"Provides Qwen3-32B, the language-based reasoning model that receives failure points and returns revised model code at each iteration."},{"cited_title":"Generalized outcome-based strategy classification: Comparing deterministic and probabilistic choice models","cited_arxiv_id":null,"evidence_quote":"Defines the multi-attribute decision-making paradigm with four binary expert ratings on which the case study runs."},{"cited_title":"Simple heuristics that make us smart","cited_arxiv_id":null,"evidence_quote":"Source of the take-the-best heuristic, one of the three model classes used to initialize ASMR."},{"cited_title":"Heuristics from bounded meta-learned inference","cited_arxiv_id":null,"evidence_quote":"Introduces heuristics from bounded meta-learned inference, another initialization class used in the simulations."},{"cited_title":"Heuristics as bayesian inference under extreme priors","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian framing under which the discovered adaptive upweighting is interpreted as interpolating between take-the-best and weighted-additive strategies."}],"review_version":1}