{"id":"732fc3a2-0ccc-4c2e-b14a-f5e5ff120b69","arxiv_id":"2505.08049","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Bayesian inference, reformulated as Q-learning, has symmetric decreasing learning rates, and fitting constant-rate Q-learning to such optimal behavior can spuriously produce positivity and confirmation biases.","lead":"This paper argues that common findings of 'positivity bias' and 'confirmation bias' in human reinforcement learning could be statistical artifacts of assuming constant learning rates. It shows that an ideal Bayesian learner, which reduces its update size over time, looks exactly like a biased learner when fit with the standard Q-learning model.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 7's symmetric decreasing learning rate 1/(t+3) is derived from the no-counterfactual update rule but asserted for the counterfactual setting; the per-arm observation count is not the global trial count, so the paper's central equivalence is not rigorously established.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: Eq. 7 is ambiguous about which experimental setting it describes and therefore may attribute a nonzero learning rate to the unchosen arm in the no-counterfactual case or use the wrong update rule in the counterfactual case. My stress-test concurs and adds a second, related inconsistency: Eq. 2 fixes the sum of counts across both arms to t, while Eq. 7 requires each arm's own count to equal t. These are incompatible definitions, so the derivation of the symmetric decreasing rate is not merely under-specified but internally inconsistent as written. This matters because the master-equation analysis in Section 2.2 and the model-comparison argument in Section 3 both rely on the exact functional form α_t=1/(t+3). If the correct per-arm rate is 1/(n_v+3), the time course of Δ_t in Eq. 30 changes, and the quantitative predictions about action-switching rates may shift. That said, the qualitative point—that temporally decreasing learning rates can produce behavioral signatures similar to confirmation bias—is plausible and has independent support in the literature, so the issue is fixable by a corrected derivation. The paper should be held to a conditional standard pending that correction, which is exactly the reader's verdict. I therefore see no reason to move the verdict, but I want the concrete check to be explicit so the authors can settle the ambiguity.","tokens_in":14443,"tokens_out":12192,"duration_ms":123388,"concrete_test":"Re-derive the effective Q-learning rates from the beta-Bernoulli update separately for the two settings. For the no-counterfactual case, verify that the unchosen arm has α_u±=0 and the chosen arm has α_c±=1/(n_c+3) with n_c equal to the number of times that arm has been pulled. For the counterfactual case, derive α_v± from p_{v,t+1}=(α_v+1+r_v)/(n_v+3) and p_{v,t}=(α_v+1)/(n_v+2), and check whether α_v±=1/(n_v+3) or 1/(t+3), where n_v=α_v+β_v is the per-arm observation count and t is the trial index. Then re-run the master-equation predictions in Eqs. 29–30 with the corrected rate and simulate a Bayes-optimal agent with counterfactual feedback to see whether fitting Eq. 1 to that agent still recovers the confirmation-bias pattern shown in Fig. 1(a). If the corrected rates differ materially, the central equivalence and the fitted-bias claim need to be reassessed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the assertion that Bayesian inference maps exactly onto Q-learning with symmetric decreasing learning rates α± = 1/(t+3) for both chosen and unchosen arms. The derivation of Eq. 7, however, uses the update rule in Eq. 3, which is explicitly the no-counterfactual update: only the chosen arm changes, and the unchosen arm's posterior mean is unchanged. In that setting the effective learning rate for the unchosen arm is zero, not 1/(t+3), and for the chosen arm it is 1/(n_c+3), where n_c is that arm's own pull count, not the global trial count. In the counterfactual setting, which the paper says is the focus because both biases are observed there, Eq. 3 is the wrong generative rule: both arms receive an observed reward each trial, so both posterior means update. Moreover, Eq. 2 defines the belief state by α1+β1+α2+β2=t, whereas Eq. 7 requires α_v+β_v=t for each arm individually; these definitions are incompatible. Since the master-equation analysis in Section 2.2 directly substitutes α_t=1/(t+3) into Eqs. 29–30, the theoretical foundation for the claim that decreasing learning rates mimic confirmation bias is not established as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the positivity and confirmation biases reported in human two-armed Bernoulli bandit (TABB) experiments can arise as fitting artifacts: even if an agent updates beliefs through objective Bayesian inference, fitting the standard Q-learning model with constant, asymmetric learning rates recovers both biases. The authors map Bayesian inference onto Q-learning with symmetric, decreasing learning rates, use master equations and a moment-closure approximation to show that both confirmation bias and decreasing learning rates reduce action-switching probabilities, fit Q-learning and Bayesian models to the data of Palminteri et al. (2017), and propose an experimental protocol to distinguish true biases from decreasing-learning-rate artifacts.","tokens_in":14824,"tokens_out":4686,"duration_ms":49493,"significance":"If the central claim is correct, the paper would substantially change the interpretation of a large empirical literature that infers positivity and confirmation biases from TABB data using constant-learning-rate Q-learning models. The master-equation treatment of action-switching dynamics is a useful theoretical contribution, and the proposed new-arm experiment in Sec. 3.2 gives concrete, falsifiable predictions that could separate the two accounts. The analytical derivations, especially Eqs. (28)-(30), are internally plausible and are validated against SDE simulations in Fig. 2. However, the load-bearing mapping in Eq. (7) has a setting inconsistency between the no-counterfactual update rule used in its derivation and the counterfactual setting that the paper says is its focus, and the empirical demonstration that fitting recovers biases is not described with sufficient detail to be checked.","major_comments":[{"comment":"The derivation of α± = 1/(t+3) uses the no-counterfactual update rule Eq. (3), where only the chosen arm's posterior changes, but the paper states that its focus is the counterfactual setting, in which both chosen and unchosen arms are updated. In the no-counterfactual setting the unchosen arm's effective learning rate is zero, not 1/(t+3), and for the chosen arm the denominator should be that arm's own observation count, not the global trial count. In the counterfactual setting, Eq. (3) is the wrong generative rule because both arms receive an observed reward each trial. The paper must state which setting Eq. (7) belongs to, derive the learning rates for both arms in the counterfactual setting, and reconcile Eq. (2), whose constraint α1+β1+α2+β2=t is incompatible with Eq. (7) when both arms are observed on every trial.","section":"§2, Eq. (7)"},{"comment":"The master-equation analysis substitutes α_t = 1/(t+3) for both the chosen and unchosen arms, but this substitution is valid only under specific observation assumptions that are not stated. If the intended setting is counterfactual feedback, then the per-arm observation count is t and the substitution is plausible, but only after a correct counterfactual belief update is written down; if the intended setting is no-counterfactual feedback, the unchosen arm's learning rate is 0 throughout. Because Eqs. (29)-(30) are the theoretical basis for the claim that decreasing learning rates mimic confirmation bias, this ambiguity is load-bearing and must be resolved before the explanation can be accepted.","section":"§2.2, Eqs. (29)-(30)"},{"comment":"The abstract and Fig. 1(a) make the empirical claim that fitting the standard Q-learning model with asymmetric learning rates to data generated by a Bayesian agent recovers both positivity and confirmation biases, but the manuscript gives no simulation protocol: it does not report the number of simulated agents, the number of trials, the fitting procedure, the parameter ranges or initializations, the optimization algorithm, or the variability of the recovered learning rates across simulations. Without this information the central empirical demonstration cannot be reproduced or assessed, and the reader cannot tell whether the recovered asymmetries are robust or an artifact of a particular fitting schedule.","section":"§3.1 and Fig. 1(a)"},{"comment":"The moment-closure approximation truncates the Taylor expansion at second order and assumes that the softmax nonlinearity is well approximated by its local behavior around the mean of the Q-distribution. The paper validates the approximation against SDE simulations for constant learning rates in Fig. 2, but it does not provide the same validation for the time-varying schedule α_t = 1/(t+3) over the trial horizons used in the human data. Since Eq. (30) is derived under this approximation, a direct comparison of the approximate and simulated ⟨K⟩_t for the Bayesian schedule would strengthen the paper's main mechanistic claim.","section":"§2.1, Eq. (20)"}],"minor_comments":[{"comment":"The notation α_i, β_i for beta-distribution counts conflicts with the learning-rate notation α^v_± used throughout; this is confusing in equations such as Eq. (7), where α appears in both roles.","section":"Eq. (2)"},{"comment":"The caption uses K_t before the text defines the ensemble-average action-switching probability; define ⟨K⟩_t in the caption or earlier in the text.","section":"Fig. 1(b)"},{"comment":"The proposed new-arm experiment is described only at a conceptual level; the manuscript should specify how p_3 is chosen for each participant, how many trials are used, how the new arm is trained, and what decision rule will be used to prefer the Bayesian over the Q-learning account.","section":"§3.2"},{"comment":"The shorthand Q^{+,v}_i and Q^{-,v}_i is used before it is defined; define these quantities immediately before Eq. (13).","section":"Eq. (13)"},{"comment":"The table reports mean BIC and NLL across subjects but does not state the number of subjects, the standard error of the mean, or whether the BIC comparison is robust to the choice of fitting procedure; adding these details would strengthen the model-comparison section.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a question of broad interest in cognitive science and reinforcement learning, and the master-equation analysis is a genuinely useful contribution. The main risk is not the algebraic derivation itself but the unresolved mismatch between the no-counterfactual derivation of Eq. (7) and the counterfactual setting that the paper claims to analyze; this mismatch affects the theoretical foundation and must be fixed before the paper can be accepted. I would also ask the editor to require that the simulation protocol behind Fig. 1(a) be made fully explicit, as it is central to the headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nYou should know two things about this paper. First, the central idea is right and worth taking seriously: a Bayesian agent in a two-armed bandit, updated with the rule appropriate to counterfactual feedback, has symmetric learning rates that shrink as 1/t, and fitting a constant-rate Q-learning model to such data can yield apparent positivity and confirmation biases. Second, the paper's own demonstration of that recovery is frustratingly thin—the key figure appears without the simulation details needed to trust it.\n\nWhat's new and good: the explicit mapping of Bayesian posterior means to Q-learning updates with α± = 1/(t+3) is a clean algebraic result, at least for the counterfactual case. The master equation treatment is a genuinely different tool for studying these dynamics, and the result that decreasing learning rates reduce action-switching probabilities in a way similar to confirmation bias is well supported by both the approximate equations and the SDE simulations shown. The proposed experiment (inserting a new arm to separate Bayesian from biased Q-value predictions) is a nice, cheap idea.\n\nThe soft spots are real but mostly fixable. The derivation of Eq. 7 starts from the no-counterfactual update rule (Eq. 3), where only the chosen arm updates, but is then applied to the counterfactual setting without stating that both arms receive updates there. For the counterfactual setting the per-arm count equals the global trial count, so 1/(t+3) is in fact correct; the issue is that the paper never says this explicitly. That's a presentation problem, not a fatal one. More serious is that the central empirical claim—that fitting constant-rate Q-learning to Bayesian-generated data recovers the biases—is essentially asserted. There is no description of how the fits were performed, how many simulations, how parameters were initialized, or what the recovery distributions look like. The human data comparison also lacks paired statistical tests and, oddly, never includes a decreasing-learning-rate Q-learning model as a baseline. Adding that baseline would directly test the paper's thesis.\n\nBottom line: this is a paper for cognitive scientists and decision neuroscientists who use Q-learning fits to infer cognitive biases. The thesis is important and largely supported by the math, but the empirical demonstration needs more rigor. I'd send it to peer review—a good referee will push for the simulation details and a cleaner statement of the update rules, and the core message should survive.","headline":"A smart and important critique of how confirmation bias is inferred from Q-learning fits, with a sound mathematical core that needs empirical support and a cleaner treatment of the update rule settings.","tokens_in":15237,"tokens_out":5097,"would_cite":true,"duration_ms":46459,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","62C10","91E40"],"pacs":[],"model":"deepseek-v4-flash","headline":"An objective Bayesian learner, when fit with standard Q-learning, appears to show the same positivity and confirmation biases found in human data.","keywords":["confirmation bias","positivity bias","Q-learning","Bayesian inference","two-armed Bernoulli bandit","learning rates","action switching","master equation"],"falsifier":"Simulate a Bayesian agent in a counterfactual two-armed bandit where both arms update with $\\alpha_\\pm=1/(t+3)$, fit the standard constant asymmetric Q-learning model to its choices, and check whether the fitted asymmetry grows with trial count; if it does not, the paper's artifact mechanism fails.","tokens_in":14240,"feed_emoji":"🧠","tokens_out":9725,"duration_ms":92000,"temperature":0.7,"pith_summary":"This paper argues that the positivity and confirmation biases reported in human two-armed bandit experiments may be measurement artifacts of the fitting model rather than features of human learning. The reason is that an ideal Bayesian learner, when its choices are fit with the standard Q-learning model, appears to have asymmetric learning rates even though its updates are symmetric. Bayesian inference maps exactly onto Q-learning with learning rates $\\alpha_+ = \\alpha_- = 1/(t+3)$, which are equal but decrease with experience. Using a master-equation analysis of action-switching probabilities, the paper shows that unbiased decreasing learning rates and genuinely biased constant learning rates produce the same behavioral signature. If the argument is right, accounting for decreasing learning rates should shrink the estimated biases in existing data, and the paper's proposed new-arm experiment can separate the two accounts.","feed_headline":"Objective Bayesian agents can look biased in Q-learning fits","feed_subtitle":"Constant learning rates in fitted models turn optimal belief updating into apparent positivity and confirmation bias.","key_machinery":"The central object is the exact mapping from Bayesian belief updating to Q-learning, condensed in the identity $\\alpha_\\pm = 1/(t+3)$: after $t$ pulls of an arm, the Bayesian agent's effective learning rate is the same for positive and negative prediction errors, so the learner is unbiased but its updates shrink over time. The argument is carried by a master equation, a bookkeeping equation for how the probability distribution over Q-values changes in time, closed at second order by Taylor-expanding the softmax choice rule. The resulting moment equations show that the action-switching probability $\\langle K\\rangle_t$ depends on the separation $\\Delta_t = \\langle(Q_1-Q_2)^2\\rangle/2$, which evolves as $\\Delta_{t+1} = (1-\\alpha_t)^2 \\Delta_t + p(1-p)\\alpha_t^2$; the first term is a drift that shrinks the separation, the second is environmental noise that grows it, and decreasing $\\alpha_t$ sustains a low switching rate in the same way that confirmation bias does.","core_discovery":"The paper establishes that Bayesian inference in a two-armed Bernoulli bandit is equivalent to a Q-learning agent with symmetric, decreasing learning rates: $\\alpha_\\pm = 1/(t+3)$. When an objective Bayesian agent's choices are fit with the standard constant-learning-rate Q-learning model, the fitted parameters recover positivity bias ($\\alpha^c_+ > \\alpha^c_-$) and confirmation bias ($\\alpha^c_+ > \\alpha^c_-$, $\\alpha^u_- > \\alpha^u_+$), exactly the patterns reported in human data. The mechanism is a shared reduction in action-switching probability: confirmation bias separates the estimated values of the arms, while decreasing learning rates let early random fluctuations separate the estimates and then lock them in place. The paper therefore concludes that apparent confirmation bias in human behavior can be an artifact of assuming constant learning rates, and it proposes a post-task third-arm choice whose predicted probabilities differ between the Bayesian and biased Q-learning accounts.","pith_inferences":["The paper leaves implicit that the same artifact could arise in any paradigm where a constant-learning-rate model is fit to data from an adaptive process with shrinking updates, such as exponential recency or Bayesian-filter-style learning.","A testable extension is to introduce a change-point after learning has settled: a biased learner should stay overconfident, while a Bayesian learner should re-adapt, giving a behavioral separator beyond the third-arm probe.","The moment-closure analysis could be applied to other generative model pairs to ask systematically which combinations of temporal learning-rate profiles and true biases are statistically distinguishable from constant-rate Q-learning."],"forward_implications":["Re-fitting existing TABB data with a model whose learning rates decrease over time should reduce or remove the fitted positivity and confirmation biases, if the artifact account is correct.","The two mechanisms predict different terminal estimates: Bayesian inference pulls both arms' Q-values toward the true reward probabilities, whereas biased constant-rate Q-learning pushes them apart.","A test with a newly introduced third arm, whose reward rate lies between the two predicted terminal values, can separate the accounts because the Bayesian and biased Q-learning models predict different probabilities of choosing it.","Claims that confirmation bias improves cumulative reward in bandit tasks may actually be capturing the benefit of decreasing learning rates, since both mechanisms lower switching without requiring a true bias."],"supporting_citations":[{"why":"Supplies the human counterfactual TABB dataset and the Q-learning fitting pipeline whose recovered biases the paper re-analyzes.","marker":"[Palminteri et al., 2017]"},{"why":"Reports the positivity/optimistic bias in human reinforcement learning that the paper explains as a fitting artifact.","marker":"[Lefebvre et al., 2017]"},{"why":"Shows that choice autocorrelation alone can generate pseudo-positivity and pseudo-confirmation biases, the prior artifact mechanism this paper extends.","marker":"[Katahira, 2018]"},{"why":"Supplies the master-equation formalism used to derive the moment dynamics of the Q-learning process.","marker":"[Gardiner, 1985]"},{"why":"Frames optimal Bayesian decision-making as a Bayes-adaptive decision process, the policy model used for the Bayesian agent.","marker":"[Duff, 2002]"},{"why":"Gives the steady-state analysis of confirmation bias in Q-learning that the paper generalizes to time-varying learning rates and softmax policies.","marker":"[Lefebvre et al., 2022]"},{"why":"Provides the moment-approximation technique that closes the master-equation system at second order.","marker":"[Benaroya et al., 2005]"}],"fun_headline_variants":["Optimal Bayesians can look biased under constant learning rate fits","The bias illusion: rational agents misread by Q-learning fits","Decreasing learning rates mimic confirmation bias in models","When rational agents look biased: a matter of fit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central equivalence assumes the unchosen arm's estimate also learns at rate $1/(t+3)$, but the update rule used to derive that rate leaves the unchosen arm's estimate unchanged, so the bridge from the derived case to the fitted case is an assumption.","fun_headline_variants_meta":{"raw":{"variants":["Optimal Bayesians can look biased under constant learning rate fits","The bias illusion: rational agents misread by Q-learning fits","Decreasing learning rates mimic confirmation bias in models","When rational agents look biased: a matter of fit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00063,"raw_usage":{"total_tokens":2866,"prompt_tokens":853,"completion_tokens":2013,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":1947}},"tokens_in":469,"tokens_out":2013,"duration_ms":16700,"temperature":1.0,"reasoning_tokens":1947,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:06:48.314076+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate a Bayesian agent in a counterfactual two-armed bandit where both arms update with $\\alpha_\\pm=1/(t+3)$, fit the standard constant asymmetric Q-learning model to its choices, and check whether the fitted asymmetry grows with trial count; if it does not, the paper's artifact mechanism fails.","supporting_citations":[{"cited_title":"The statistical structures of reinforcement learning with asymmetric value updates","cited_arxiv_id":null,"evidence_quote":"Shows that choice autocorrelation alone can generate pseudo-positivity and pseudo-confirmation biases, the prior artifact mechanism this paper extends."},{"cited_title":"Handbook of stochastic methods for physics, chemistry and the natural sciences","cited_arxiv_id":null,"evidence_quote":"Supplies the master-equation formalism used to derive the moment dynamics of the Q-learning process."},{"cited_title":"Probability models in engineering and science, volume 192","cited_arxiv_id":null,"evidence_quote":"Provides the moment-approximation technique that closes the master-equation system at second order."}],"review_version":1}