{"id":"c432ac69-1c31-430a-a518-b44638d6d346","arxiv_id":"2510.03013","paper_version":4,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"DistIRL recovers reward distributions and risk-aware policies from offline demonstrations by minimizing first-order stochastic dominance violations between agent and expert returns.","lead":"This paper proposes DistIRL, an offline inverse reinforcement learning method that learns a full distribution over rewards—not just a point estimate—by matching expert and agent return distributions through first-order stochastic dominance. The framework also produces risk-aware policies via distortion risk measures and is tested on gridworld, mouse behavior, and MuJoCo control.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (8)/Prop. B.1's quantile identity has a sign error: for X=δ0, Y=δ−1 the LHS is 0 but the RHS is 1; the correct order is [Q_Y−Q_X]_+, so the FSD reward loss and policy objective are built on the wrong sign.","rationale":"The reader's REJECT is well-supported. I independently re-derived the central identity and found the same sign error. This is not a stylistic or convention issue; the inequality direction in the proof is wrong. The quantile function maps probability level to threshold; F_X(z)≥v is equivalent to Q_X(v)≤z. The paper uses the reverse, which exactly negates the integrand of the FSD violation. Because Eq. (8) is the bridge between the CDF-space loss (Eq. 5/7) and tractable quantile-space computation (Eq. 8) and policy objective (Eq. 9/13), the entire framework's theoretical grounding fails. I also note the abstract's O(ε^{-2}) convergence claim is not supported by the main text—Sec. 4.3 explicitly says rigorous analysis is out of scope—and the skew-normal reward model is unimodal while Sec. 5.2 claims multimodal dopamine recovery; these are aggravating but secondary. The central identity alone is sufficient to reject.","tokens_in":17190,"tokens_out":5496,"duration_ms":43251,"concrete_test":"Check Eq. (14) with X~δ_0, Y~δ_{−1}: compute LHS by hand (0) and RHS by hand (1). Then symbolically re-derive the indicator step in the proof: F_X(z)≥v ⇔ F_X^{-1}(v)≤z, not ≥z; substituting the correct equivalence yields ∫[F_X−F_Y]_+ dz = ∫[F_Y^{-1}−F_X^{-1}]_+ dv, the opposite of Eq. (8). If this corrected identity replaces Eq. (8), the FSD loss and policy objectives in Sec. 4.1–4.2 must be re-written accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing mathematical step is Eq. (8)/Prop. B.1, which asserts ∫[F_X(z)−F_Y(z)]_+ dz = ∫[F_X^{-1}(v)−F_Y^{-1}(v)]_+ dv. This identity is false. Take X≡0 and Y≡−1. Then F_X(z)−F_Y(z) is −1 on [−1,0), so the left side is 0, while F_X^{-1}(v)−F_Y^{-1}(v)=1 for all v, so the right side is 1. The correct identity is ∫[F_X−F_Y]_+ dz = ∫[F_Y^{-1}−F_X^{-1}]_+ dv; the positive part reverses the order of the quantiles. The proof in Appendix B makes the same inversion: F_X(z)≥v implies Q_X(v)≤z, not ≥z. Since Eq. (8) is used in Eq. (7) for the FSD reward gradient and in Eq. (9)/(10) for the policy objective, the sign of the optimized violation is reversed. The central claim that DistIRL learns the full reward distribution 'in a principled manner' therefore rests on an invalid quantile-integral identity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DistIRL, a distributional offline inverse reinforcement learning framework that models the reward as a variational distribution q_phi(r|s,a) and learns a risk-sensitive policy by matching the full return distribution. The method uses a first-order stochastic dominance (FSD) violation between agent and expert return CDFs as the reward-learning loss, a quantile-regression critic, and a distortion risk measure (DRM) for policy updates. Experiments are reported on a gridworld, a mouse dopamine dataset, and MuJoCo control tasks. The central theoretical step is Eq. (8), which claims an equivalence between a positive-part integral of CDF differences and a positive-part integral of quantile differences.","tokens_in":17617,"tokens_out":10387,"duration_ms":98451,"significance":"The problem is relevant: extending IRL to genuinely stochastic rewards and distribution-aware policies could matter for neuroscience and risk-sensitive imitation. The dopamine experiment is a genuine external benchmark, which is a strength. However, the paper's central identity (Eq. 8) is false, the DRM used for policy learning weights the wrong tail of the return distribution, and the abstract's convergence guarantee is explicitly disclaimed in Section 4.3. These are load-bearing issues: the algorithm as written optimizes a different objective from the one claimed, so the empirical results cannot be attributed to the proposed theoretical framework. The contribution is not established in its current form.","major_comments":[{"comment":"Eq. (8) is false as stated. For X=δ_0 and Y=δ_{−1}, F_X−F_Y = −1 on [−1,0), so the LHS is 0; F_X^{-1}(v)−F_Y^{-1}(v)=1 for v∈(0,1), so the RHS is 1. The correct identity is ∫[F_X−F_Y]_+ dz = ∫[F_Y^{-1}−F_X^{-1}]_+ dv. The proof in Appendix B inverts the quantile implication: F_X(z)≥v implies F_X^{-1}(v)≤z, not ≥z. Since Eq. (8) is used to form the FSD reward loss (Eq. 7) and the policy objective (Eq. 9), the sign of the optimized violation is reversed. This invalidates the central theoretical claim.","section":"§4.1, Eq. (8) and Appendix B.1"},{"comment":"The CVaR distortion used in Eq. (13) is not the 'average of worst α-fraction of outcomes' claimed in Table 1. With ξ(v)=min(v/α,1) and dual distortion ξ~(v)=1−ξ(1−v), dξ~ is supported on [1−α,1], so M_ξ(X) = (1/α)∫_{1−α}^1 F_X^{-1}(v)dv, i.e., the conditional expectation of the best α-fraction of returns. Maximizing Eq. (13) is therefore risk-seeking, not risk-averse. This contradicts the text's interpretation and the risk-averse expert descriptions in Secs. 5.1 and 5.3. Additionally, the surrogate-objective argument uses one arbitrary DRM, so Proposition 4.6 does not justify equivalence with Eq. (9).","section":"§4.2, Eq. (13) and Table 1"},{"comment":"The abstract states: 'Theoretical analysis shows that the algorithm converges with O(ε^{-2}) iteration complexity.' Section 4.3 explicitly states: 'Regrettably, a rigorous convergence analysis is beyond the scope of this paper.' No theorem or proof of an O(ε^{-2}) bound appears anywhere in the manuscript. This is an unsupported and internally contradictory claim that must be removed or substantiated.","section":"Abstract and §4.3"},{"comment":"The paper claims the dopamine distributions are 'often right-skewed and multimodal' and that S-DistIRL 'most accurately recovers the shape.' However, S-DistIRL uses Azzalini's skew-normal distribution, which is unimodal by construction and cannot represent multimodal densities. No multimodality test or mixture extension is provided. The quantitative evidence (Fig. 4B) is a single Wasserstein-1 distance without confidence intervals or per-subject statistical comparisons, so the neurobehavioral validation is weaker than claimed.","section":"§5.2"}],"minor_comments":[{"comment":"The table header reads 'IPMD (ours)' while the text and the rest of the paper call the method DistIRL. This appears to be a copy-paste error.","section":"Table 2"},{"comment":"The quantile Huber loss is introduced as ρ^κ_τ but the LQR loss writes ρ_{τ_i} without κ; also θ_j(s',a') notation should be clarified (target quantile index) to match the standard QR loss.","section":"§3.1, Eq. (2)"},{"comment":"The notation 'empirical quantile ... (z_{(-N)},...,z_{(1)})' for order statistics is nonstandard and confusing; standard order-statistic notation would be clearer.","section":"§4.1"},{"comment":"The text says DistIRL achieves 'competitive or superior performance' on deterministic D4RL, but on HalfCheetah DistIRL (7779±228) is clearly below Offline ML-IRL (11231±585). The wording should be adjusted.","section":"§5.3, Table 3"},{"comment":"The ablation reports only 'scaled performance for clarity'; without the actual unscaled returns and standard deviations, the quantitative comparison is difficult to interpret.","section":"Table 4"}],"recommendation":"reject","confidential_remarks":"This is a clear reject: the central identity in Eq. (8) is false, and the CVaR implementation weights the wrong tail. These are not presentation issues but invalidate the theoretical foundation of the proposed algorithm. The authors may be able to repair the sign and risk-tail errors in a future version, but the current manuscript does not support its claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper has a real idea and a real flaw. It is the first IRL formulation I know that tries to learn the full reward distribution with an FSD objective and DRM-based policies, and the mouse dopamine experiment is a genuinely useful external validation. But the mathematical center does not hold. Eq. (8) and Prop. B.1 claim ∫[F_X−F_Y]_+ dz = ∫[Q_X−Q_Y]_+ dv. That's false; the stress-test example is decisive: X≡0, Y≡−1 gives LHS 0, RHS 1. The correct identity swaps the quantile order: ∫[F_X−F_Y]_+ = ∫[Q_Y−Q_X]_+ dv. The proof in Appendix B inverts the quantile condition (F_X(z)≥v means Q_X(v)≤z, not ≥). Because Eq. (7) and Eq. (9) both depend on this identity, the reward loss and the policy objective are optimizing the wrong side of the distribution. This is not a cosmetic typo; it changes which policy is being rewarded.\n\nThere are other soft spots, in descending severity. The abstract promises O(ε^{-2}) convergence while Sec. 4.3 says a convergence analysis is beyond scope. The distributional assumption is skew-normal, which is unimodal, yet the authors describe the target dopamine distributions as multimodal and show better fits with S-DistIRL—that combination does not stand up. No code or data are available, and the main empirical comparison on MuJoCo lacks statistical tests and uses very few baselines. To be fair, the high-level framework is original and the literature engagement is honest; Prop. 4.6 on DRMs is fine as far as it goes. The problems are concentrated exactly where the paper's contribution is claimed.\n\nMy bottom line: this deserves a reject, and I would not spend referee cycles on it in its current form. The dopamine benchmark is the most interesting artifact; if the authors fix the sign, release code, and reframe the convergence claim, the empirical direction could be worth a look. As it stands, the central claim is not supported.","headline":"Real idea, wrong sign: the FSD/DRM combination is original and the dopamine validation is interesting, but Eq. (8)'s quantile identity is false and the main framework rests on it.","tokens_in":18024,"tokens_out":6009,"would_cite":false,"duration_ms":66941,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that inverse reinforcement learning can recover the full distribution of rewards—not just a mean—by aligning return distributions with first-order stochastic dominance, yielding risk-aware policies.","keywords":["distributional inverse reinforcement learning","reward distribution","first-order stochastic dominance","distortion risk measure","offline imitation learning","risk-aware policy","quantile regression","dopamine reward"],"falsifier":"Evaluate Eq. (8) with X constant 0 and Y constant -1: the CDF-side positive integral is 0, while the quantile-side integral is 1. This mismatch shows the identity is false as written, and the paper's own proof in Appendix B.1 contains the step in which the inequality direction is reversed (the 'if and only if' between events $\\{F_X(z) \\ge v, v \\ge F_Y(z)\\}$ and $\\{F^{-1}_X(v) \\ge z, z \\ge F^{-1}_Y(v)\\}$ has the second event's quantile inequalities reversed).","tokens_in":17105,"feed_emoji":"🎲","tokens_out":4862,"duration_ms":40148,"temperature":0.7,"texified_at":"2026-08-05T20:32:50.923469+00:00","pith_summary":"The paper's aim is to lift inverse reinforcement learning (IRL) past point estimates of a reward function. It proposes DistIRL, a framework in which each state-action reward is a random variable with a learnable distribution, and expert demonstrations are matched by their full return distribution rather than by expected return. The matching criterion is first-order stochastic dominance (FSD): the learner minimizes the positive area between the expert's and the agent's return CDFs. Because FSD compares entire distributions, the recovered reward captures variance and higher moments, and the induced policy is risk-sensitive when updated with a distortion risk measure such as CVaR. The paper demonstrates the method on synthetic gridworlds, on mouse dopamine data, and on MuJoCo benchmarks.","texify_model":"deepseek-v4-flash","texify_usage":{"total_tokens":5572,"prompt_tokens":768,"completion_tokens":4804,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":768,"completion_tokens_details":{"reasoning_tokens":4096}},"feed_headline":"Offline IRL recovers full reward distributions, not just means","feed_subtitle":"Matching entire return distributions yields reward variance and risk-aware policies from demonstrations alone.","key_machinery":"The load-bearing devices are (1) the FSD violation loss $L_{\\mathrm{FSD}}(\\pi,r)=\\int [F_{Z_\\pi}(z)-F_{Z_E}(z)]_+ \\, dz$, which the paper rewrites as a positive integral over quantile-function differences via Eq. (8); and (2) the distortion risk measure $M_\\xi(Z_\\pi)=\\int F^{-1}_{Z_\\pi}(v)\\, d\\tilde{\\xi}(v)$, used as a tractable surrogate for the FSD-indicator weighting in the policy objective (Eq. 13). The first connects reward-distribution learning to higher-order moment matching; the second converts FSD-aware policy learning into a standard risk-sensitive RL update.","core_discovery":"The paper's central claim is that the minimax IRL objective can be replaced by a distributional one: minimize over reward distributions and maximize over policies the FSD violation between agent and expert return distributions, plus policy entropy and a reward prior. Solving it yields both a posterior over rewards (via variational inference with a skew-normal family) and a risk-aware policy that optimizes a distortion risk measure (DRM), specifically CVaR in the experiments. The authors claim this is the first principled recovery of the full reward distribution, and they report better imitation performance than baselines on risk-sensitive D4RL tasks and a correlation of about 0.3 between inf","pith_inferences":["If Eq. (8) fails, the actual minimized loss is not the FSD violation; with a corrected sign the algorithm would reward policies for decreasing quantile gaps, reversing the intended risk direction. This can be tested on constant random variables.","The reported correlation between inferred reward means and dopamine suggests a broader path: IRL could infer trial-to-trial variability in reward from behavior, potentially applicable to other neuromodulators.","The skew-normal reward family limits fidelity for multimodal distributions; a mixture-based parameterization would be a natural extension and could be tested on the same dopamine data.","The method's risk-aware policy uses a single DRM (CVaR); because the theory posits all distortion functions are needed for exact FSD, approximating with one DRM is an unverified gap."],"forward_implications":["If the central claim holds, offline IRL can recover not just mean rewards but variance and skewness, making reward inference useful in settings with stochastic or contact-rich outcomes.","Demonstrations from humans or animals can in principle reveal risk preferences, since the recovered policy explicitly optimizes a distortion risk measure.","The method is stated to converge in O(ε^{-2}) iterations, so the algorithmic cost is comparable to standard distributional RL training.","Recovering reward distributions from behavior alone could help decode internal reward signals (e.g., dopamine variability) in neuroscience.","Because the framework accepts any differentiable parametric reward distribution, it is extensible beyond skew-normal to mixtures or heavy-tailed families."],"fun_headline_variants":["Distributional IRL recovers full reward distributions","IRL learns reward distributions, not just expected returns","Beyond means: distributional IRL for risk-aware imitation","New IRL models uncertainty over rewards and returns","Recover reward distributions with distributional inverse RL"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The derivation of both the reward loss and the policy objective assumes the identity $\\int [F_X-F_Y]_+ \\, dz = \\int [F^{-1}_X(v)-F^{-1}_Y(v)]_+ \\, dv$ (Eq. 8); if that quantile-integral identity fails, the algorithm is not minimizing the FSD violation it claims to minimize.","fun_headline_variants_meta":{"raw":{"variants":["Distributional IRL recovers full reward distributions","IRL learns reward distributions, not just expected returns","Beyond means: distributional IRL for risk-aware imitation","New IRL models uncertainty over rewards and returns","Recover reward distributions with distributional inverse RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1140,"prompt_tokens":646,"completion_tokens":494,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":390,"completion_tokens_details":{"reasoning_tokens":434}},"tokens_in":390,"tokens_out":494,"duration_ms":4803,"temperature":1.0,"reasoning_tokens":434,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T12:37:58.317934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate Eq. (8) with X constant 0 and Y constant -1: the CDF-side positive integral is 0, while the quantile-side integral is 1. This mismatch shows the identity is false as written, and the paper's own proof in Appendix B.1 contains the step in which the inequality direction is reversed (the 'if and only if' between events $\\{F_X(z) \\ge v, v \\ge F_Y(z)\\}$ and $\\{F^{-1}_X(v) \\ge z, z \\ge F^{-1}_Y(v)\\}$ has the second event's quantile inequalities reversed).","supporting_citations":[],"review_version":2}