Pith. sign in

REVIEW 4 major objections 4 minor 40 references

A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Quantile matching makes reinforcement learning counterfactually fair.

desk verdict CFSMDM is a genuinely useful sequential extension of distribution-mapping fairness, but Theorem 4 as stated is internally inconsistent and the main performance bounds lean heavily on an unpublished preprint. read the letter →

arxiv 2608.08743 v1 pith:L72WSBUY submitted 2026-08-09 stat.ML cs.CYcs.LGstat.ME

Jianhan Zhang , Jitao Wang , John D. Piette , Donglin Zeng , Chengchun Shi , Zhenke Wu This is my paper · ORCID
classification stat.MLcs.CYcs.LGstat.ME
keywords counterfactualfairnessreinforcementlearningdatapreprocessingquantileregressionsequentialmarginaldistributionmappingofflinecausalinferenceinhealthcare
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a data-preprocessing step—matching each individual’s observed state and reward to the same conditional quantile under every possible sensitive attribute value—can strip sensitive information out of reinforcement-learning trajectories, so that the policy learned afterward treats each person the same whether they are, for example, one ethnicity or another. The method, called CFSMDM, works with existing offline RL algorithms and, unlike the main prior approach, does not require the noise in the environment dynamics to enter additively: strict monotonicity in a scalar noise variable is enough. The paper proves that under this monotonicity assumption, the learned policy is approximately counterfactually fair—its action probabilities change little if an individual’s sensitive attribute were switched—and that its loss in expected reward relative to the optimal policy is bounded. If the claim is right, fairness-sensitive applications such as clinical decision support can correct for implicit bias without redesigning the RL algorithm or assuming a restrictive model of the environment.

What carries the argument

Sequential marginal distribution mapping (SMDM). The paper fits conditional quantile functions for the initial state, the transition distribution, and the reward distribution, all conditional on the sensitive attribute and history. For a given individual it estimates the quantile level of the observed state or reward under the observed attribute, then uses that same quantile level together with the quantile models under every alternative attribute value to reconstruct counterfactual trajectories. The invariance of quantile levels across counterfactual worlds (Theorems 1 and 2) is the load-bearing mechanism that turns observed data into counterfactual targets; the recursion through time is what makes the counterfactual state sequence well defined.

What would settle it

Run CFSMDM on data generated from a known CMDP that satisfies all assumptions except strict monotonicity—for example, make the state depend on noise through a non-monotone function such as a sine or a quadratic U-shaped map—and check whether the empirical CF metric (the rate at which the learned policy changes its action for the same individual under switched sensitive attributes) remains small. Under the paper’s claim this metric should converge to zero as the sample size grows; a persistent large value would falsify the guarantee.

Watch

Extended reading notes

Core claim

On the paper’s own terms, the central claim is that counterfactual fairness in RL is achievable by sequential quantile distribution mapping. For each individual and each time step, one finds the quantile level of the observed state (and reward) within the conditional distribution given the individual’s sensitive attribute and past history; by Theorems 1 and 2 this quantile level is invariant to switching the sensitive attribute when each state dimension and reward is strictly increasing in its own scalar exogenous noise. The counterfactual state under an alternative attribute value is then read off at the same quantile from the conditional distribution under that alternative value, recursively through time. Stacking these counterfactual states into an augmented state, and weighting counterfactual rewards by the population distribution of the sensitive attribute, yields an MDP whose optimal policy is counterfactually fair by construction. Theorems 3 and 4 bound how far a policy learned on estimated augmented states and rewards is from that optimum and how much per-step unfairness remains, with both bounds shrinking as the sample size grows and quantile estimates improve.

Load-bearing premise

The load-bearing premise is that every state dimension and the reward are strictly increasing functions of their own scalar noise variable, so that the quantile level of an observed value is preserved when the sensitive attribute is switched; if that monotonicity fails, the counterfactual reconstructions are biased and the fairness and near-optimality guarantees do not follow.

Editorial extensions

If this is right

  • If the theorems hold, any existing offline RL algorithm can be made approximately counterfactually fair by first running SMDM on the training trajectories and then learning on the augmented states and rewards.
  • The fairness guarantee is individual-level: for each person, the probability of receiving a given action under the learned policy is nearly unchanged if the sensitive attribute were switched, not merely equal across groups.
  • The suboptimality gap bound means fairness is purchased at a controlled price: the policy learned on augmented data is close to the best counterfactually fair policy, and the gap shrinks with more data and better quantile estimation.
  • Because monotonicity is strictly weaker than additivity, the method covers all environments the additive-noise method CFSDP covers and more, including nonadditive dynamics such as the cube-root transitions used in the simulations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One can test the load-bearing quantile-invariance assumption directly in any dataset: for individuals sharing the same observed history but different sensitive attributes, check whether the rank of each state or reward within its own conditional distribution is exchangeable; large deviations would cast doubt on strict monotonicity before deployment.
  • The augmented-state construction removes the sensitive attribute from the information available to the policy, so the same preprocessing should in principle make other objectives—such as fairness-constrained or safety-constrained RL—work on the same counterfactual representation.
  • Because the augmented reward is a weighted average of counterfactual rewards, the approach implicitly assumes the population mix of sensitive attributes is known; a deployment site with a different mix would need to reweight, a point the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CFSMDM, a data-preprocessing method that aims to enforce counterfactual fairness in offline reinforcement learning. The method sequentially estimates counterfactual states and rewards by matching quantile levels under different sensitive-attribute values, relying on strict monotonicity of states and rewards in scalar exogenous noises (Assumptions 1 and 2). The authors prove identifiability results (Theorems 1 and 2), give finite-sample bounds on the suboptimality gap and counterfactual unfairness of the learned policy (Theorems 3 and 4), and support the method with simulations under additive and nonadditive noise plus an application to the PowerED digital-health dataset.

Significance. If the theoretical guarantees are established, the paper makes a useful contribution by extending counterfactually fair RL preprocessing from additive-noise models to a strictly larger class of monotone structural models. The identification argument is natural and the explicit counterexample in Theorem A1 showing that monotonicity is strictly weaker than additivity is a genuine strength. The empirical section is thorough in its baselines, in its use of both additive and nonadditive CMDPs, and in its sensitivity analyses for sample size, effect size, horizon, and quantile-grid size. However, the two main performance bounds are not derived in the manuscript: the key FQI error estimates and the unfairness contraction are imported from the authors' unpublished preprint [31], and Theorem 4 as stated is ill-defined outside a high-probability event. The identification proof in Theorem 1 also contains an apparent typographical error in the t>=1 step. These issues must be resolved before the advertised guarantees can be considered verified.

major comments (4)
  1. [Section 4, Theorem 4] The term E[O(xi_n^alpha)] is not defined on the event {delta_{T,n} > lambda0/(4dL)}. Since xi_n = C1 d L Rmax delta_{T,n}/(lambda0(lambda0 - 4dL delta_{T,n})(1-gamma)^2) + C2 d Rmax kappa log(n)/((1-gamma)^2 lambda0 sqrt(n)) + gamma B Rmax/(1-gamma), the denominator lambda0(lambda0 - 4dL delta_{T,n}) is negative on that event, so xi_n is negative and xi_n^alpha is not a real number for general alpha > 0. The proof applies the bound (4) only under A_n intersect B_n and then, after marginalizing over Xi_n, retains O(xi_n^alpha) over the whole probability space. The theorem should restrict the O(xi_n^alpha) term to B_n, use a truncated denominator such as (delta_{T,n} and lambda0/(8dL)), or state a conditional bound. As written, the statement and proof are internally inconsistent.
  2. [Section I.3.4, proofs of Theorems 3 and 4] The main performance bounds are not self-contained. The proof of Theorem 3 obtains the high-probability control of the sup-norm error of the estimated Q function by 'the same argument as the one in the proof of Theorem 3 in [31]', and inequality (4) in the proof of Theorem 4 is justified only by 'the same argument as the one in the proof of Theorem 4 in [31]'. Reference [31] is an unpublished preprint by the same research group, and no statement of its conditions, lemmas, or derivations is reproduced. Since Theorems 3 and 4 are the central theoretical guarantees advertised in the abstract, the proofs must either be given in full or the precise dependencies on [31] must be stated so that a reader can verify them.
  3. [Remark A1] The assertion that P(delta_{T,n} <= lambda0/(4dL)) converges to 1 exponentially fast for linear quantile regression and quantile-regression neural networks is unproved. This probability appears as a multiplicative factor in Theorem 3 and as an additive term in Theorem 4, so the bounds are not meaningful unless the claim is established. Remark A1 says it can be shown by concentration inequalities but gives no theorem, no conditions, and no citation. Either provide a rigorous statement with explicit regularity conditions on the quantile estimators, or replace Remark A1 by an explicit assumption in A1-A4.
  4. [Section I.1, proof of Theorem 1] In the t>=1 step of the proof of Theorem 1, the third equality is not justified as written. The displayed chain contains P(f^{S(i)}(z^{(j)}, a_{t-1}, s^{z^{(k)}}_{t-1}, U^{S(i)}_t) <= f^{S(i)}(z^{(j)}, a_{t-1}, s^{z^{(j)}}_{t-1}, u^{S(i)}_t)) as an intermediate term, but to equal P(U^{S(i)}_t <= u^{S(i)}_t), both sides of the inequality must use the same state argument s^{z^{(j)}}_{t-1}. With s^{z^{(k)}}_{t-1} on the left and s^{z^{(j)}}_{t-1} on the right, strict monotonicity in the noise alone does not imply the claimed equality. The analogous reward proof in Theorem 2 presumably needs the same correction. I suspect this is a typographical slip, but as it stands the proof of the identification result is incomplete.
minor comments (4)
  1. [Section I.3.3, proof of Lemma A1] In the inductive step there is a typo: the maximum is written as max over i in {t,...,M}, but it should be max over i in {1,...,M}.
  2. [Assumption A4] The notation in Assumption A4 is garbled: the set subtraction A \ arg max_a Qopt(...) needs parentheses, and the roles of Q^* and Qopt should be defined consistently. As printed, the margin condition is hard to parse.
  3. [Algorithm 1] In Step 4 of Algorithm 1, the conditional quantile function W is defined using P(R_t <= x | Z=z, S_{t-1}=s, A_{t-1}=a), which mixes the index of R with the index of (S,A). The same function is used in Step 15 for R_{t-1} and in Section I.3.1 it is defined as P(R_t | Z=z, S_t=s, A_t=a). The time index should be made consistent in the algorithm.
  4. [Section 6, Table 1] The text states that CFSMDM noticeably outperforms CFSDP for education and ethnicity, but the reported mean differences are small relative to the standard deviations, and for sex CFSMDM is slightly worse. A more cautious reading of the table is advisable.

Circularity Check

2 steps flagged · score 5.0 of 10

The advertised unfairness/suboptimality bounds are imported from the authors' own preprint [31], and the PowerED fairness metric is evaluated using the same SMDM estimator on which CFSMDM is built.

  1. self citation load bearing [Section I.3.4, Proofs of Theorems 3 and 4 (Eq. (4), and the FQI error bound preceding Eq. (2))]
    "By the same argument as the one in the proof of Theorem 4 in [31], we have E(˜st,at)∼˜πz′nPt[∥˜πz′t,n(˜ht)−˜πz′′t,n(˜ht)∥2|Ξn,An,Bn]≤O(ξαn)+O((BLδt,n)α)(4) // If δT,n ≤ λ0/(4dL), then, by the same argument as the one in the proof of Theorem 3 in [31], for any κ>0, ∥Q∗−fˆ∥∞≤..."

    The paper's central theoretical guarantees are Theorem 3 and Theorem 4. Their key inequalities are not derived in this manuscript: Inequality (4), which produces the advertised E[O(ξ_n^α)] unfairness term, is asserted to follow 'by the same argument' as Theorem 4 of [31], and the FQI error control that defines ξ_n and drives Theorem 3 is likewise imported 'by the same argument' as Theorem 3 of [31]. Reference [31] is the authors' own unpublished preprint with overlapping authorship (Wang, Shi, Piette, Zeng, Wu). No derivation, external validation, or independent verification of those bounds is supplied, and the paper does not show the assumptions of [31] are satisfied beyond referring to the same setup.

  2. other [Section L, 'Details on training and evaluation in the real data analysis']
    "For the CF metric, we first train an SMDM model on the entire PowerED dataset using quantile regression with a linear specification, and then employ this model to estimate the unobserved counterfactual states of the test dataset."

    The real-data unfairness results in Table 1 are computed from counterfactual states generated by the same SMDM conditional-quantile estimator that defines CFSMDM. Remark 1 concedes that inaccurate quantile models or violations of Assumptions 1-2 would make the reported CF metric inaccurate. Moreover, Algorithm 1 constructs every counterfactual state for an individual from the same estimated quantile level τ̂ and the same fitted quantile functions, so the estimated augmented state is invariant to the observed sensitive attribute by construction. Evaluating CFSMDM on these self-generated counterfactual states can therefore report low unfairness even when the quantile mapping is misspecified.

full rationale

The core methodological contribution, the quantile-matching identifiability result (Theorems 1 and 2), is self-contained: Appendix I.1 derives it directly from strict monotonicity of the structural equations in scalar noise (Assumptions 1 and 2), and no fitted quantity is renamed as a prediction there. The comparison to CFSDP (Theorem A1) is also derived explicitly. However, the paper's headline theoretical claims—the suboptimality gap and counterfactual unfairness bounds of Theorems 3 and 4—are not self-contained. Their decisive steps are imported verbatim from the authors' own preprint [31] without derivation, which is load-bearing self-citation rather than independent support. In addition, the real-data fairness claim is evaluated using the same SMDM estimator on which CFSMDM is built, making that empirical comparison partly circular. There is also a separate correctness concern, distinct from circularity: Theorem 4's RHS is not well-defined on the event {δ_{T,n} > λ0/(4dL)} because ξ_n has a negative denominator there and ξ_n^α is undefined for general α > 0; a valid statement would need to restrict or truncate that term. That defect lowers confidence in the imported bound but is not itself a circularity. Overall, the paper has independent content in its quantile identification and simulation evidence, but the central performance guarantees reduce at key junctures to the authors' own unverified results, and the real-data evaluation is self-referential, giving a partial circularity score of 5.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on strict monotonicity of states and rewards in scalar noise, the root-node status of the sensitive attribute, the augmented-MDP equivalence imported from the self-cited [31], and the FQI regularity conditions A1-A4. No new entities are invented. The two listed free parameters are implementation choices that affect the method's accuracy but not the theoretical identification argument.

free parameters (2)
  • Quantile grid size q = 99 (default; varied in Section J.3)
    CFSMDM approximates conditional quantile functions by fitting q quantile regression models at evenly spaced levels. The method's accuracy depends on this hand-chosen grid; q=9 and q=19 visibly degrade fairness in CMDP 1 (Table A2).
  • Number of FQI iterations B = 200 (simulations), 100 (real data)
    The learned policy uses B iterations of fitted Q iteration. The bounds in Theorems 3 and 4 include B (as a constant growing with iterations) and an initialization bias term; B is chosen by the authors.
assumptions (5)
  • domain assumption For each state dimension i and each t, S_t^(i) is strictly increasing in its scalar exogenous noise U_t^{S(i)} given (Z, A_{t-1}, S_{t-1}), and rewards are strictly increasing in U_t^R given (Z, S_t, A_t).
    Assumptions 1 and 2 in Section 3. This monotonicity is what preserves quantile levels under switching Z in Theorems 1 and 2; without it the counterfactual mapping is not identified by quantile matching.
  • domain assumption The sensitive attribute Z is a root node caused only by exogenous variables, with no unobserved confounders linking Z to states/rewards.
    Section 2.2 states Z 'can only be caused by exogenous variables'. This is the standard causal assumption for counterfactual fairness; it is untestable from data.
  • domain assumption The augmented process (S̃_t, R̃_t, A_t) is a stationary MDP in which every policy is counterfactually fair and the optimal policy is stationary (Theorem 2 of [31]).
    Used in Section 2.3 and in the proofs of Theorems 3 and 4. The result is imported from the authors' self-cited preprint [31] and is not re-derived here.
  • domain assumption Assumptions A1-A4: bounded state/reward spaces; linear Q-function class with completeness and feature coverage; Lipschitz quantile functions and features; margin condition on the optimal Q function.
    Section I.3.2. These regularity conditions are needed for the FQI estimation error bounds and for converting action-value differences into unfairness bounds.
  • ad hoc to paper The conditional quantile estimators are accurate with high probability, i.e., P(δ_{T,n} ≤ λ0/(4dL)) is close to 1.
    Remark A1 asserts this converges to 1 exponentially for linear QR and QR neural networks 'under certain conditions', but gives no theorem or precise conditions. The main bounds in Theorems 3 and 4 are conditional on this event.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning." pith.science (2026). https://pith.science/paper/L72WSBUY

@misc{pith2026260808743,
  author       = {Pith},
  title        = {Pith review of: A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L72WSBUY}},
  note         = {Machine review of arXiv:2608.08743}
}
read the original abstract

Reinforcement learning (RL) seeks to optimize sequential decisions to maximize population-level benefits over time. However, when deployed in high-stakes settings such as healthcare, RL decisions might systematically restrict some subpopulation's access to valuable services in a manner contrary to the values and goals of stakeholders. Counterfactual fairness (CF) offers a promising framework to address this problem based on causal reasoning. This paper develops a data preprocessing algorithm that, when used in tandem with policy learning, enables CF in RL. Our algorithm relies on a novel quantile distribution mapping method for sequentially estimating the counterfactual states and rewards in the data preprocessing step, subsuming common additivity assumptions used for counterfactual prediction as a special case. We theoretically prove that the per-step level of counterfactual unfairness and infinite-horizon suboptimality gap can be bounded under mild regularity conditions. We also empirically test our algorithm in numerical experiments as well as in application to a real-world interventional digital health dataset.

Figures

Figures reproduced from arXiv: 2608.08743 by the authors.

Figure 1
Figure 1. Causal DAG of CMDP [31] assumed to be a deterministic function of Ht. Inspired by this observation, we let AZ←z ′ t (U¯ t(ht)), S Z←z ′ t (U¯ t(ht)), and RZ←z ′ t (U¯ t+1(ht+1)) denote the counterfactual action, state, and reward, re￾spectively, at time t under πt for an individual with observed history ht, had the individual’s sensitive attribute been set to z ′ and the individual followed the observed past action … view at source ↗
Figure 2
Figure 2. Experiment results under nonadditive noise: (a) CF metric vs. Sample size ( [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 34 canonical work pages

  1. [31]

    D., Loftus, J

    Wang, J., Shi, C., Piette, J. D., Loftus, J. R., Zeng, D., and Wu, Z. (2025). Counterfactually fair reinforcement learning via sequential data preprocessing.arXiv preprint arXiv:2501.06366

    arXiv 2025

  2. [1]

    Bian, Z., Wang, L., Shi, C., and Qi, Z. (2026). Double fairness policy learning: Integrating action fairness and outcome fairness in decision-making.arXiv preprint arXiv:2601.19186

    arXiv 2026

  3. [2]

    Chen, H., Lu, W., Song, R., and Ghosh, P. (2024). On learning and testing of counterfactual fair- ness through data preprocessing.Journal of the American Statistical Association, 119(546):1286– 1296

    work page 2024

  4. [3]

    and Jiang, N

    Chen, J. and Jiang, N. (2019). Information-theoretic considerations in batch reinforcement learning. In Chaudhuri, K. and Salakhutdinov, R., editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 1042–1051. PMLR

    work page 2019

  5. [4]

    Chen, J., Tian, J., and Quinn, C. J. (2025). Causal logistic bandits with counterfactual fairness constraints. InForty-second International Conference on Machine Learning

    work page 2025

  6. [5]

    Chernozhukov, V ., Fernández-Val, I., and Galichon, A. (2010). Quantile and probability curves without crossing.Econometrica, 78(3):1093–1125

    work page 2010

  7. [6]

    Chernozhukov, V ., Fernández-Val, I., and Melly, B. (2013). Inference on counterfactual distribu- tions.Econometrica, 81(6):2205–2268

    work page 2013

  8. [7]

    and Hansen, C

    Chernozhukov, V . and Hansen, C. (2005). An iv model of quantile treatment effects.Economet- rica, 73(1):245–261

    work page 2005

Show all 40 references
  1. [8]

    Creager, E., Madras, D., Pitassi, T., and Zemel, R. (2020). Causal modeling for fairness in dynamical systems. In Daumé III, H. and Singh, A., editors,Proceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, p...

    2020

  2. [9]

    D’Amour, A., Srinivasan, H., Atwood, J., Baljekar, P., Sculley, D., and Halpern, Y . (2020). Fairness is not static: deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* ’20...

    2020

  3. [10]

    G., Hickey, J

    Di Stefano, P. G., Hickey, J. M., and Vasileiou, V . (2020). Counterfactual fairness: removing direct effects through regularization.arXiv preprint arXiv:2002.10774

    2020 arXiv

  4. [11]

    Hallak, A., Di Castro, D., and Mannor, S. (2015). Contextual markov decision processes.arXiv preprint arXiv:1502.02259

    2015 arXiv

  5. [12]

    A., Ashburn-Nardo, L., Stewart, J

    Hollingshead, N. A., Ashburn-Nardo, L., Stewart, J. C., and Hirsh, A. T. (2016). The pain experience of hispanic americans: A critical literature review and conceptual model.The Journal of Pain, 17(5):513–528

    2016

  6. [13]

    Hu, Y ., Kallus, N., and Uehara, M. (2023). Fast rates for the regret of offline reinforcement learning.arXiv preprint arXiv:2102.00479

    2023 arXiv

  7. [14]

    Huang, W., Zhang, L., and Wu, X. (2022). Achieving counterfactual fairness for causal bandit. Proceedings of the AAAI Conference on Artificial Intelligence, 36(6):6952–6959

    2022

  8. [15]

    and Langford, J

    Kakade, S. and Langford, J. (2002). Approximately optimal approximate reinforcement learning. InProceedings of the Nineteenth International Conference on Machine Learning, ICML ’02, page 267–274, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc

    2002

  9. [16]

    Kingma, D. P. and Ba, J. (2017). Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980

    2017 arXiv

  10. [17]

    (2005).Quantile Regression

    Koenker, R. (2005).Quantile Regression. Econometric Society Monographs. Cambridge University Press

    2005

  11. [18]

    and Bassett, G

    Koenker, R. and Bassett, G. (1978). Regression quantiles.Econometrica, 46(1):33–50. 11

    1978

  12. [19]

    J., Loftus, J

    Kusner, M. J., Loftus, J. R., Russell, C., and Silva, R. (2018). Counterfactual fairness.arXiv preprint arXiv:1703.06856

    2018 arXiv

  13. [20]

    Le, H., V oloshin, C., and Yue, Y . (2019). Batch policy learning under constraints. In Chaudhuri, K. and Salakhutdinov, R., editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 3703–3712. PMLR

    2019

  14. [21]

    Li, T., Shi, C., Lu, Z., Li, Y ., and Zhu, H. (2024). Evaluating dynamic conditional quantile treatment effects with applications in ridesharing.Journal of the American Statistical Association, 119(547):1736–1750

    2024

  15. [22]

    T., Dean, S., Rolf, E., Simchowitz, M., and Hardt, M

    Liu, L. T., Dean, S., Rolf, E., Simchowitz, M., and Hardt, M. (2018). Delayed impact of fair machine learning. In Dy, J. and Krause, A., editors,Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 31...

    2018

  16. [23]

    Meinshausen, N. (2006). Quantile regression forests.Journal of Machine Learning Research, 7(35):983–999

    2006

  17. [24]

    Padilla, O. H. M., Tansey, W., and Chen, Y . (2022). Quantile regression with relu networks: Estimators and minimax rates.Journal of Machine Learning Research, 23(247):1–42

    2022

  18. [25]

    Pearl, J., Glymour, M., and Jewell, N. P. (2016).Causal inference in statistics: A primer. John Wiley & Sons

    2016

  19. [26]

    C., Krishnamurthy, A., Bartlett, P., and Kakade, S

    Perdomo, J. C., Krishnamurthy, A., Bartlett, P., and Kakade, S. (2023). A complete characteri- zation of linear estimators for offline policy evaluation.Journal of Machine Learning Research, 24(284):1–50

    2023

  20. [27]

    D., Thomas, L., Newman, S., Marinec, N., Krauss, J., Chen, J., Wu, Z., and Bohnert, A

    Piette, J. D., Thomas, L., Newman, S., Marinec, N., Krauss, J., Chen, J., Wu, Z., and Bohnert, A. S. B. (2023). An automatically adaptive digital health intervention to decrease opioid-related risk while conserving counselor time: Quantitative analysis of treatment decisions b...

    2023

  21. [28]

    Riedmiller, M. (2005). Neural fitted q iteration – first experiences with a data efficient neural reinforcement learning method. In Gama, J., Camacho, R., Brazdil, P. B., Jorge, A. M., and Torgo, L., editors,Machine Learning: ECML 2005, pages 317–328, Berlin, Heidelberg. Sprin...

    2005

  22. [29]

    and Perktold, J

    Seabold, S. and Perktold, J. (2010). statsmodels: Econometric and statistical modeling with python. In9th Python in Science Conference

    2010

  23. [30]

    Taylor, J. W. (2000). A quantile regression neural network approach to estimating the conditional density of multiperiod returns.Journal of Forecasting, 19(4):299–311

    2000

  24. [32]

    Wang, Y ., Sridhar, D., and Blei, D. (2023). Adjusting machine learning decisions for equal opportunity and counterfactual fairness.Transactions on Machine Learning Research

    2023

  25. [33]

    offered admission

    Zuo, A., Wei, S., Liu, T., Han, B., Zhang, K., and Gong, M. (2022). Counterfactual fairness with partially known causal graph. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY , USA. Curran Associates Inc. 12 A...

    2022

  26. [34]

    F={w T ϕ(s, a) :w∈R d,∥w∥ 1 ≤ B}, where ϕ is a feature map ˜S × A →Rd with ∥ϕ(s, a)∥∞ ≤1 and component functions {ϕi(s, a)}d i=1

    (Class of the Q Function). F={w T ϕ(s, a) :w∈R d,∥w∥ 1 ≤ B}, where ϕ is a feature map ˜S × A →Rd with ∥ϕ(s, a)∥∞ ≤1 and component functions {ϕi(s, a)}d i=1. Also, assume thatQ ∗ is estimated in FQI using ordinary least squares

  27. [35]

    LetTbe the Bellman optimality operator

    (Completeness). LetTbe the Bellman optimality operator. That is, forf: ˜S × A →R, Tf(s, a) =r(s, a) +γEs′∼P(·|s,a) max a′∈A f(s ′, a′). Assume thatTf∈ Ffor anyf∈ F

  28. [36]

    Full”, “Unaware

    (Feature Coverage). There exists a constant λ0 >0 such that the smallest eigenvalue of Es,a∼µb [ϕ(s, a)ϕ(s, a)T ]is greater than or equal toλ 0. Assumption A3(Lipschitz Continuous Conditional Quantile Functions and Feature Vector).The true conditional quantile functions {Q(i) ...

    2000

  29. [37]

    Specifically, the quantile levels and quantiles for each time step t and dimension i, which are required by FLAP_M, are estimated using the empirical cumulative distribution function (CDF) of the conditional distribution (denoted by ˆP(i) t,n(·|Z)) and its inverse, respectivel...

  30. [38]

    ,[ˆP(M) t,n ]−1(ˆP(M) t,n (s(M) j,t |Z=z j)|Z=z) , where ˆP(i) t,n(·|Z) is the empirical CDF of S(i) t following the same definition as that in FLAP_M

    For eachz∈ Z, find st(z) = [ˆP(1) t,n]−1(ˆP(1) t,n(s(1) j,t |Z=z j)|Z=z), . . . ,[ˆP(M) t,n ]−1(ˆP(M) t,n (s(M) j,t |Z=z j)|Z=z) , where ˆP(i) t,n(·|Z) is the empirical CDF of S(i) t following the same definition as that in FLAP_M

  31. [39]

    Fora∈ A, calculate the estimated counterfactually fair Q value by ˆfcf (sj,t, a) = X z′∈Z p(z′) X z∈Z p(z) ˆf(z, st(z′), a) where p(z) = ˆPn(Z=z) is the empirical population-level probability of the observed sensitive attribute taking the value z, and ˆf:Z × S × A →Ris the Q f...

  32. [40]

    Implementation of CFSDP:Our implementation of CFSDP estimates the environment’s transition kernel using a neural network with hidden layers [64,64]

    The decision (or action) is made by aj,t = arg max a∈A ˆfcf (sj,t, a). Implementation of CFSDP:Our implementation of CFSDP estimates the environment’s transition kernel using a neural network with hidden layers [64,64] . The neural network is trained using a maximum of 1000 ep...

    2000

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.