{"id":"9b8b8d86-1913-423d-8990-84cad433d76d","arxiv_id":"2506.12765","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Using random forests versus Kolmogorov-Arnold networks for nuisance functions in a distributional IV estimator can reverse the estimated effect curve, so model choice is a substantive decision.","lead":"Researchers compared two machine-learning workhorses used inside a statistical procedure for estimating how a financial incentive changes the full distribution of wealth, and found the choice can flip the conclusion. The paper recommends benchmarking multiple learners and validating uncertainty before drawing policy lessons from such distributional instrumental-variable estimates.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) is not a valid moment for D-IV-LATE: its true-nuisance expectation is E[(μ(y,1,X)−μ(y,0,X))(1−(p(1,X)−p(0,X)))], so the RF and KAN curves target a misspecified functional, not the claimed parameter.","rationale":"The reader's verdict is correct, and the problem is more severe than stated: the issue is not merely that Assumption 3.1 fails to imply Y⊥Z|W,X. Even if that conditional independence held, Eq. (5) still has nonzero expectation because the treatment-CDF contrast enters with coefficient 1−(p_1−p_0), not 1. In the 401(k) application the first stage is far from 1, so the bias term is structurally nonzero and can plausibly dominate the estimated curves. This invalidates the paper's central empirical claim that model choice changes the D-IV-LATE: the two estimators are not estimating the D-IV-LATE at all. The impossible Table 1 entries (bias/RMSE around 10 for a parameter bounded in [−1,1]) and the inconsistency between the stated sigmoid CDF and the normal-error outcome equations reinforce that the numerical results cannot be trusted as evidence. No code execution or formal verification was supplied to offset an analytic misspecification. I agree with the reader's rejection but would restate the concern as an exact bias calculation rather than a missing independence condition, so my agreement is partial.","tokens_in":13631,"tokens_out":11372,"duration_ms":135612,"concrete_test":"Analytically recompute E[ψ_α(y)] under Eq. (5) with true nuisances: conditional on X, show E[(Z−π)/(π(1−π))(1{Y≤y}−μ(y,W,X))] = (m_1−m_0) − (μ_1−μ_0)(p_1−p_0), and therefore E[ψ_α] = E[(μ_1−μ_0)(1−(p_1−p_0))] when α is set to E[m_1−m_0]. Then evaluate this expression on the 401(k) application using the repository's fitted p(1,X), p(0,X), μ(y,1,X), and μ(y,0,X). If the implied bias is non-negligible relative to the reported D-IV-LATE curves, Eq. (5) does not identify the claimed parameter.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central identification claim fails at Eq. (5). The paper defines the target numerator as α(y) = E[E[1{Y≤y}|Z=1,X] − E[1{Y≤y}|Z=0,X]], but the score uses μ(y,W,X) = E[1{Y≤y}|W,X], not the instrument-conditional CDF. Conditional on X, with π(X)=P(Z=1|X), p_z(X)=P(W=1|Z=z,X), f_w(X)=μ(y,w,X), and m_z(X)=E[1{Y≤y}|Z=z,X], we have E[(Z−π)/(π(1−π))(1{Y≤y}−μ(y,W,X)) | X] = (m_1−m_0) − (f_1−f_0)(p_1−p_0), because E[(Z−π)/(π(1−π)) W | X] = p_1−p_0. Hence the expectation of Eq. (5) at true nuisances is E[(f_1−f_0)(1−(p_1−p_0))], which is not zero unless f_1=f_0 almost surely or p_1−p_0=1. The implemented estimator in §4.4 solves ¯ψ_α/¯ψ_β, without subtracting α, so its probability limit is [α(y) + E[(f_1−f_0)(1−(p_1−p_0))]]/β, again not D-IV-LATE. Even the extra condition Y⊥Z|W,X would not fix this: it only makes m_1−m_0=(f_1−f_0)(p_1−p_0), leaving the same nonzero bias. Thus the sign reversals and model comparisons in the 401(k) application are comparing estimates of a misspecified functional, not of the D-IV-LATE.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper defines the D-IV-LATE parameter Δ(y) as the complier CDF difference, then proposes cross-fitted DML estimators of the ratio α(y)/β using two choices of nuisance learner (random forests and KANs). It reports simulations and a 401(k) application and argues that the choice of nuisance model can reverse sign conclusions. The paper releases code and data links. The central message is cautionary: model choice matters. However, the moment condition used to estimate the numerator is not a valid orthogonal moment for α(y), and the implementation in §4.4 solves a different equation than the one stated in §4.3, so the reported curves do not estimate the target parameter as defined.","tokens_in":14038,"tokens_out":7222,"duration_ms":76517,"significance":"If the estimator were valid, the paper would make a useful practical point: DML's asymptotic robustness does not immunize applied researchers against model choice, and benchmarking nuisance learners along with overlap and calibration diagnostics is sensible. The manuscript has several genuine strengths: the target is defined cleanly, the simulations are built on explicit monotone principal strata with known complier truth, the code and data are linked, and the interpretation of the KAN results as sensitivity evidence rather than as evidence of dominance is appropriately cautious. These strengths do not compensate for the invalid identification of the reported curves; the sign reversals in the 401(k) application are therefore not evidence about D-IV-LATE.","major_comments":[{"comment":"The score ψ_α uses the residual 1{Y≤y} − μ(y,W,X) with μ(y,W,X)=E[1{Y≤y}|W,X]. Let m_z(x)=E[1{Y≤y}|Z=z,X=x], f_w(x)=μ(y,w,x), p_z(x)=P(W=1|Z=z,X=x), and π(x)=P(Z=1|X=x). A direct calculation gives E[(Z−π)/(π(1−π))(1{Y≤y}−μ(y,W,X))|X] = m_1(x)−m_0(x) − (p_1(x)−p_0(x))(f_1(x)−f_0(x)). Hence the population mean of the score in Eq. (5) at true nuisances is −E[(p_1−p_0)(f_1−f_0)] when α(y) equals the target E[m_1−m_0]. This is not zero under Assumptions 3.1–3.3; it would require p_1=p_0 or f_1=f_0. Adding Y⊥Z|W,X does not help: it only imposes m_1−m_0=(p_1−p_0)(f_1−f_0), making the bias term explicit. In addition, §4.4 defines ψ_α,i without the final −α(y) term and then forms the ratio of sample averages, so the implemented estimator has probability limit [α(y)+E[(1−(p_1−p_0))(f_1−f_0)]]/β rather than α(y)/β. The simulation and 401(k) curves therefore do not estimate the D-IV-LATE defined in Eq. (1).","section":"§4.3, Eq. (5); §4.4"},{"comment":"The text says the average bias was generally small (on the order of 10^-4 or 10^-3), but Table 1 reports average bias −10.0003 at y=10.406 and −9.7141 at y=20.006, with RMSE of the same magnitude. These are not small numbers, and the row for y=28.275 reports exact zeros for both bias and RMSE, which is implausible. The simulation evidence for the RF estimator is therefore not established as written; the table contradicts the accompanying summary.","section":"§6.1.2, Table 1"},{"comment":"Theorem 5.1 is stated without proof; Assumption 5.1 is an assumption about the nuisance estimators, and no argument is given that either the random-forest or the KAN implementation in §4.2.3 satisfies the Donsker and n^{-1/4} rate conditions. More importantly, the abstract's claim that 'KAN pointwise intervals undercover badly under both asymptotic and bootstrap constructions, while RF asymptotic intervals are better calibrated in the validation designs' has no corresponding results in the body: §7 explicitly says that developing confidence intervals is 'an important area for future research.' This is a missing-support problem for a stated empirical claim.","section":"§5, Theorem 5.1; §7; Abstract"}],"minor_comments":[{"comment":"The phrase 'thelayers hidden parameter' should be 'the hidden-layers parameter' or 'the layers parameter'; the sentence is garbled.","section":"§4.2.3"},{"comment":"The list of DGP steps has five items but labels them First, Second, Third, Fourth, and Fourth; the last item should be Fifth.","section":"§6.1.1"},{"comment":"The variable 'net tfa' is used without definition; state that it is net total financial assets.","section":"§7.1"},{"comment":"Figure 1 is described in the text but not displayed in the manuscript; include the figure and axis labels, or state clearly that it is available only in the repository.","section":"§7.3"},{"comment":"The variance formula V(y) uses ψ_α,i from Eq. (5), which includes the −α(y) term, but the estimator implemented in §4.4 uses a score without that term; the asymptotic variance needs to be stated for the score actually used.","section":"§5"},{"comment":"Reference [16] (Kallus and Oprescu) is listed but does not appear to be cited in the text; either cite it in the literature review or remove it.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central identification flaw is decisive: the reported estimator does not target the defined D-IV-LATE parameter, so the main empirical conclusion about model choice is not supported. In addition, the abstract describes results (inference validation, RF-versus-KAN performance rankings) that do not appear in the body, which suggests the manuscript was assembled inconsistently. A revision would need to replace the moment condition with a genuinely orthogonal score for α(y), redo the simulations and application with that estimator, and reconcile the abstract with the actual content. I would not invite a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe load-bearing issue is that Eq. (5) does not identify the D-IV-LATE. The score uses the residual 1{Y≤y} - mu(y,W,X), conditioning on the endogenous W instead of the instrument Z. At true nuisances, the expectation of the alpha-score is E[(mu(y,1,X)-mu(y,0,X))(1-(p(1,X)-p(0,X)))], not zero, so the probability limit of the estimator is not the claimed parameter. The extra condition Y perpendicular Z | W,X does not fix it. This is a genuine algebraic error, not a subtle regularity issue. Consequently, the 401(k) sign reversals and the simulation curves are estimates of a misspecified functional.\n\nThat said, the paper has some virtues. The high-level caveat that nuisance learner choice can matter in DML is reasonable, and the recommendations to benchmark learners, check overlap, and validate calibration are sensible. Comparing random forests and KANs in a distributional IV setting is a legitimate exercise, and the writing is clear. The code is linked, though without a commit hash. The arXiv abstract is properly modest; the full-text abstract is not, and that internal inconsistency needs attention.\n\nThe simulations also have mechanical red flags. Table 1 reports bias and RMSE values around 10 for a parameter that is a difference of CDFs and therefore bounded by [-1,1]. DGP2 specifies a sigmoid CDF for mu but generates outcomes as linear plus sine plus normal noise, which cannot match that CDF. These are not typos; they point to deeper problems in the simulation pipeline.\n\nThe paper could be valuable as a teaching example of how a seemingly plausible DML moment can be misspecified, and the cautionary message might survive a full rewrite. But as it stands, the empirical findings cannot be trusted. I would not send this to peer review in its current form. If the author corrects the moment condition, reruns the analysis, and aligns the abstracts, it might be worth a second look.\n\nRecommendation: reject.","headline":"The paper's central estimator is invalid, so its headline sign-reversal finding is not about the D-IV-LATE; the cautionary message about nuisance-model choice survives, but the empirical evidence does not.","tokens_in":14590,"tokens_out":5229,"would_cite":false,"duration_ms":52717,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that the choice of machine-learning model for nuisance functions can reverse the sign of an estimated distributional instrumental-variable treatment effect, using the 401(k) wealth application as the demonstration.","keywords":["Distributional Treatment Effects","Instrumental Variables","Kolmogorov-Arnold Networks","Local Average Treatment Effect","Double/Debiased Machine Learning","Nuisance Functions","Model Choice","Endogeneity"],"falsifier":"Re-estimate the 401(k) D-IV-LATE curves for both learners with identical folds, seeds, and y-grid, and form a simultaneous band for the RF-minus-KAN curve; the model-risk claim fails empirically if that band contains zero everywhere in the lower tail.","tokens_in":13340,"feed_emoji":"📉","tokens_out":12543,"duration_ms":142961,"temperature":0.7,"pith_summary":"The paper tests a practical question: given that modern causal inference often uses machine learning to estimate nuisance functions, does the choice of machine-learning model change the final causal conclusion? The author builds a double/debiased estimator of the D-IV-LATE, the effect of an endogenous binary treatment on the whole outcome distribution for the subpopulation induced into treatment by an instrument, and estimates it twice: once with random forests and once with Kolmogorov-Arnold Networks (KANs), which are neural networks with learnable spline activations. In simulations with complex nonlinear nuisance functions, the KAN versions run faster, but random forests usually give more accurate D-IV-LATE curves. In the 401(k) application, the two models produce materially different curves, with sign reversals in the lower tail of the wealth distribution, which the paper presents as evidence that nuisance-model choice is a first-order source of model risk. The practical message is to benchmark nuisance learners, inspect overlap, and validate the inference directly instead of assuming the DML framework makes the choice harmless.","feed_headline":"ML model choice flips the 401(k) wealth-effect curve","feed_subtitle":"The chosen machine-learning model can reverse the estimated effect of 401(k) participation on wealth.","key_machinery":"The central object is the ratio $\\Delta(y)=\\alpha(y)/\\beta$, where $\\alpha(y)$ is the covariate-adjusted intent-to-treat effect on the indicator $\\mathbf{1}\\{Y\\le y\\}$ and $\\beta$ is the covariate-adjusted first-stage effect on treatment uptake. The estimator computes $\\hat{\\Delta}(y)=\\frac{1}{n}\\sum_i\\psi_{\\alpha,i}(y)\\big/\\frac{1}{n}\\sum_i\\psi_{\\beta,i}$ from two Neyman-orthogonal scores, and cross-fits the nuisance functions $\\mu(y,w,x)=P(Y\\le y\\mid W=w,X=x)$, $p(z,x)=P(W=1\\mid Z=z,X=x)$, and $\\pi(x)=P(Z=1\\mid X=x)$. The comparison is between random forests and KANs, networks whose edge activations are learnable B-splines; KAN approximation-rate theory is used to argue that KAN nuisance estimators can meet the $n^{-1/4}$ rate and Donsker conditions required by the asymptotic theorem. The scores' orthogonality is what is meant to make the final ratio insensitive to first-order error in the nuisance estimates.","core_discovery":"The paper argues that in DML-based D-IV-LATE estimation, the nuisance learner can be the deciding factor in the estimated sign and shape of the effect curve, not a minor implementation detail. Its evidence is the pair of 401(k) curves: the random-forest version suggests 401(k) participation lowers the chance that compliers end up at the very bottom of the wealth distribution, while the KAN version suggests the opposite at the extreme lower tail and a negative, relatively stable effect at positive wealth levels. The paper also reports that in its validation designs the KAN instrument-propensity estimates concentrate near zero and one, so the KAN curve is best read as sensitivity evidence, and that KAN pointwise intervals undercover under both asymptotic and bootstrap constructions. The author's own framing is a 'caveat emptor': flexibility and speed do not make an ML nuisance learner automatically suitable for causal inference.","pith_inferences":["An inference the paper leaves implicit is that Eq. (5) needs a stronger conditional-independence condition ($Y \\perp Z \\mid W, X$) than Assumption 3.1 states; if that condition fails, the two curves need not be estimating the same target.","A testable extension of the benchmarking advice is a controlled model-risk audit: keep folds, seeds, and the grid fixed, vary the learner family and tuning, and measure how often the lower-tail sign reversal survives.","The same orthogonal-score structure could be exported to other DML targets, such as quantile effects or distributional effects without instruments, to see whether nuisance-model risk is equally severe there.","The paper's own closing limitation about weak instruments suggests a follow-up: a weak-instrument-robust version of the ratio moment, since the denominator is itself estimated and its noise enters every point of the curve."],"forward_implications":["Applied researchers using ML-based distributional IV estimators should treat the nuisance learner as a modelling choice that can change the sign of estimated effects, not as a default implementation detail.","If curves from different learners diverge, the divergence is a signal to investigate overlap, sparse-data regions, and calibration before reporting a single D-IV-LATE curve.","Speed and architectural flexibility do not by themselves justify a learner for causal inference; in the paper's simulations, random forests are usually more accurate than KANs despite KANs being faster.","Confidence intervals built from asymptotic or bootstrap approximations can undercover badly for flexible nuisance learners, so inference calibration should be validated in the specific design rather than assumed from DML theory.","The KAN-based 401(k) results should be read as sensitivity evidence rather than as an estimate of the true complier wealth distribution, because the instrument-propensity estimates concentrate near boundaries."],"supporting_citations":[{"why":"Supplies the double/debiased machine-learning framework, Neyman-orthogonal moments, and cross-fitting that the estimator is built on.","marker":"[10]"},{"why":"Defines the complier outcome-distribution object that D-IV-LATE targets.","marker":"[15]"},{"why":"Gives the LATE identification logic behind the ratio of intent-to-treat contrasts.","marker":"[14]"},{"why":"Provides the distributional treatment-effect-in-IV setting whose bootstrap inference this paper extends to the DML construction.","marker":"[1]"},{"why":"Introduces Kolmogorov-Arnold Networks, the nuisance learner architecture compared against random forests.","marker":"[21]"},{"why":"Supplies approximation-rate guarantees used to argue that KAN nuisance estimators can satisfy the DML rate condition.","marker":"[19]"},{"why":"Provides the 401(k) application and data context in which the model-risk sign reversals are demonstrated.","marker":"[8]"}],"fun_headline_variants":["Model choice flips 401(k) wealth-effect curve","Which ML model? It reverses 401(k) effect sign","KAN and RF give opposite 401(k) wealth-effect signs","ML model choice decides 401(k) effect sign and shape"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the estimating equation measures the same complier contrast as the instrument-based definition even though the treatment is endogenous; the paper does not test this equivalence, and it does not follow automatically from the stated assumptions.","fun_headline_variants_meta":{"raw":{"variants":["Model choice flips 401(k) wealth-effect curve","Which ML model? It reverses 401(k) effect sign","KAN and RF give opposite 401(k) wealth-effect signs","ML model choice decides 401(k) effect sign and shape"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0006,"raw_usage":{"total_tokens":2833,"prompt_tokens":1001,"completion_tokens":1832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":1761}},"tokens_in":617,"tokens_out":1832,"duration_ms":17150,"temperature":1.0,"reasoning_tokens":1761,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:44:18.711550+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-estimate the 401(k) D-IV-LATE curves for both learners with identical folds, seeds, and y-grid, and form a simultaneous band for the RF-minus-KAN curve; the model-risk claim fails empirically if that band contains zero everywhere in the lower tail.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the double/debiased machine-learning framework, Neyman-orthogonal moments, and cross-fitting that the estimator is built on."},{"cited_title":"W., & Rubin, D","cited_arxiv_id":null,"evidence_quote":"Defines the complier outcome-distribution object that D-IV-LATE targets."},{"cited_title":"W., & Angrist, J","cited_arxiv_id":null,"evidence_quote":"Gives the LATE identification logic behind the ratio of intent-to-treat contrasts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the distributional treatment-effect-in-IV setting whose bootstrap inference this paper extends to the DML construction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 401(k) application and data context in which the model-risk sign reversals are demonstrated."}],"review_version":1}