Pith. sign in

REVIEW 4 major objections 5 minor 46 references

FAST-Q: Fast-track Exploration with Adversarially Balanced State Representations for Counterfactual Action Estimation in Offline Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FAST-Q claims that adversarially balancing state representations across policies enables offline reinforcement learning to estimate counterfactual actions reliably, and reports live-platform gains in returns, lifetime value, engagement…

desk verdict A solid engineering contribution that imports domain-adversarial training into offline recommender RL, but the 'counterfactual estimation' claim is not backed by validation; the paper deserves peer review with a request for a ground-truth check or a softened claim. read the letter →

arxiv 2504.21383 v1 pith:OAGN6RIF submitted 2025-04-30 cs.LG cs.AI

Pulkit Agrawal , Rukma Talwadker , Aditya Pareek , Tridib Mukherjee This is my paper · ORCID
classification cs.LGcs.AI
keywords offlinereinforcementlearningcounterfactualactionestimationgradientreversallayerbalancedstaterepresentationrecommendersystemsmulti-objectiveQ-valuedecompositionpolicyregularizationgamingplatform
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FAST-Q tries to make offline reinforcement learning usable in recommender systems where logged data comes from several competing policies with only partially overlapping state spaces. Its central move is to erase policy-specific bias from player-state representations using gradient-reversal adversarial training, so that actions taken by a different policy than the one that served a player can be evaluated as true counterfactuals. The paper argues this fixes the standard offline-RL failure of clipping such counterfactual actions as out-of-distribution, and it couples the balanced representation with a decomposed Q-value loss and an offline exploration schedule. If this works, a recommender can compare alternative challenge policies on static logs and switch policies without months of online experiments; the paper reports lifts in player returns, lifetime value, engagement, dwell time, and cost on its live gaming platform.

What carries the argument

The Balancing Representation $\Theta(\beta(S_t))$ is the load-bearing object: each policy expert's LSTM hidden state passes through a dense layer, and the result is trained adversarially so that a softmax policy classifier cannot tell which policy produced the state while a critic still predicts outcomes from it. The Gradient Reversal Layer forces the representation to trade off policy invariance against outcome prediction. This representation feeds an actor-critic built on TD3+BC-style behavior-cloning regularization, while a complementary loss decomposes the critic output into per-reward-dimension weights for dwell time, engagement, return time, and an overflow term, which keeps Q-values bounded and makes objective priority explainable. A stepwise discount schedule and an $\epsilon$-greedy sampler of counterfactual actions complete the training loop.

What would settle it

Run a prospective experiment where players are randomly reassigned among the three policies, record the actual rewards for actions that FAST-Q would have judged counterfactual, and compare realized rewards with FAST-Q's predicted Q-values. If the prediction error is systematically larger for actions originating from non-serving policies than for actions from the serving policy, balancing has not removed the policy bias; equivalently, a held-out policy classifier that retains above-chance accuracy on the balanced representation would falsify the invariance.

Watch

Extended reading notes

Core claim

The paper's central claim is that a Balanced Representation, built by training a policy classifier with a Gradient Reversal Layer so that the representation maximizes policy-classification error while minimizing critic loss, makes the state representation invariant to which policy generated it. Under that invariance, the supervised critic estimates $E[R_t \mid \Theta(\beta(S_t)), A_t]$ as the counterfactual outcome of any policy's action, and the actor can be trained on counterfactual actions sampled from other policy experts without the Q-value overestimation that out-of-distribution clipping causes. The paper also claims that decomposing the critic's Q-value into per-objective weighted components yields both explainable objective prioritization and more reliable Q-values, and that removing any of the three components—balanced representation, offline exploration, or Q-value decomposition—measurably lowers estimated returns.

Load-bearing premise

The load-bearing premise is that after gradient-reversal balancing the representation $\Theta(\beta(S_t))$ carries no information about which policy produced the data, so the critic's estimate equals the true counterfactual outcome; the paper assumes this invariance rather than validating it against ground-truth counterfactuals.

Editorial extensions

If this is right

  • Offline counterfactual exploration becomes possible: the actor can sample actions from non-serving policies during training, so learning no longer waits for online experiments on those policies.
  • Recommender policies can be compared and switched on static logs; the paper reports FAST-Q favoring a new mix of the three platform policies rather than any single one.
  • The balanced representation widens the spread of Q-values across counterfactual actions, which is evidence that the critic can actually distinguish alternatives instead of collapsing onto one narrow band.
  • Multi-objective Q-decomposition gives per-player, per-time priority weights, making recommendations explainable while keeping Q-values bounded; the paper reports that ablating it drops maximum achievable Q-values by about 40%.
  • On the live platform, the claims translate to at least 0.15% higher player returns, 2% higher lifetime value, 0.4% higher engagement, 2% higher dwell time, and at least 10% lower recommendation cost over the SOTA baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same balancing trick should transfer to any logged recommender or decision-support setting where experimental path logic biases state coverage, such as personalized pricing, content feeds, or clinical treatment assignment.
  • A direct testable extension is to train the policy classifier on a held-out set of states and measure whether its accuracy is at chance after the gradient-reversal step; residual accuracy would quantify how much policy bias survives.
  • The paper's own report that training stops at a discount factor of 0.7 leaves long-horizon behavior of the decomposed Q-values uncharacterized; whether the decomposition loss permits higher discounts with a slower schedule is an open question.
  • Because the platform's A/B conclusion framework uses permutation testing with support percentages rather than classical normality assumptions, the reported lifts are tied to that inferential choice; re-analyzing the same logs with different tests would show sensitivity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FAST-Q, an offline reinforcement learning method for challenge recommendation on a real-money gaming platform. The architecture combines per-policy LSTM-based "policy experts," a gradient reversal layer that learns a policy-balanced state representation, a TD3-style actor-critic with behavior cloning, a Q-value decomposition into learned weights over reward components, and epsilon-greedy selection of counterfactual actions from other policies. Evaluation consists of retrospective offline analyses, a 21-day online A/B test against TD3+BC, Diffusion-QL, and two in-house policies, and D4RL Gym-MuJoCo benchmarks. The paper reports improvements in engagement, dwell time, LTV, and reduced recommendation cost, and claims that the balanced representation enables counterfactual action estimation.

Significance. If the counterfactual estimation claim were established, this would be a valuable practical advance: it directly targets multi-policy logged data with state-space shift, provides a deployed system with a real A/B test, ships code and hyperparameters, and compares against strong baselines. The D4RL experiments and ablations are also useful for the community. However, the central causal claim is not currently validated: the balancing procedure is not shown to remove confounding, and most offline evidence is based on the model's own Q-values rather than ground-truth counterfactual outcomes. The real-world A/B results are a strength, but they do not by themselves identify which component causes the observed gains.

major comments (4)
  1. [Section 3.4] The balancing condition in Section 3.4 asserts that after adversarial training the critic estimates E[R_t | Θ(β(S_t)), A_t] as a counterfactual outcome, but the condition only enforces invariance of the representation distribution across observed policies conditional on each policy's predicted action. It does not establish unconfoundedness (sequential ignorability), positivity, or consistency; these assumptions are never stated or verified. The text itself identifies player intent as a driver of both actions and rewards, so if intent is not fully captured in S_t, or if state support does not overlap across policies, the counterfactual Q-values remain biased. No experiment compares estimated counterfactual values against ground truth: the higher Q-spread in Figure 13 and the lower dropout variance in Figure 16 are internal consistency checks, not correctness checks. The D4RL experiments in Section 4.4 use single-policy datasets and therefore cannot test cross-policy counterfactual validity. I recommend either adding semi-synthetic validation with known counterfactual outcomes and overlap/positivity diagnostics, or explicitly reframing the contribution as robust out-of-distribution generalization rather than causal counterfactual estimation.
  2. [Section 3.5.2] The Q-value decomposition weights w1..w3 (and overflow w4) are optimized to minimize (R^i_t - w_i Q(s,a))^2, and Figures 10-12 then report these same weights as "objective prioritization." This interpretation is circular: the weights restate the fitted linear relationship between the scalar Q and the reward components; they are not an independent explanation of the agent's trade-offs. Furthermore, because Q is a discounted sum of future rewards rather than the instantaneous reward vector, the loss does not in general identify the contribution of each objective to Q. The paper should validate the decomposition on held-out reward-component prediction or against known objective preferences, or the explainability claim should be softened.
  3. [Sections 4.2.3 and 4.2.4] The claimed offline "time speedup" and the ablation effects are measured in terms of FAST-Q's own Q-values normalized to its own full-data or deployed model (Figures 17 and 18). Faster convergence of internal Q-values under counterfactual exploration does not establish that a better policy is learned faster; a biased critic can converge quickly. These analyses should be reported with a policy-performance measure such as normalized return on a held-out evaluation set or D4RL-style scores, or at minimum the paper should state clearly that only internal Q-value convergence is being measured.
  4. [Section 4.4 and Table 2] The adaptation to single-policy D4RL datasets is under-specified and internally inconsistent. Removing the PE layer and "modifying BR layer training" is not sufficient for reproducibility: gradient reversal requires a domain/policy classifier, but a single-policy medium-v0 dataset has no policy labels. The paper should specify what domains are used, how the GRL is trained, and why this still corresponds to the proposed method. In addition, the text states that FAST-Q "misses on two out of the remaining three by narrow margins," but Table 2 shows three losses, including halfcheetah-medium-expert-v0 with a severe degradation (50.9 vs 97.9). This inconsistency should be corrected and the applicability claim tempered.
minor comments (5)
  1. [Section 3.3] The "weighted MSE loss" for the policy experts does not specify the weights applied to the three action dimensions; please provide the exact weighting.
  2. [Section 3.5.2] Please clarify how the softmax over the four weights interacts with the three reward components and with the cost objective, which the text says is not controlled; the relationship between C=3 and four weights needs a precise statement.
  3. [Section 3.5.3] The inability to increase the discount factor beyond gamma=0.7 is a substantive limitation for long-term objectives and should be acknowledged in the conclusion as well as in the body.
  4. [Section 4.2.1] The retrospective analysis that led to the "policy switch" conclusion is not described: please provide the matching or confounder-adjustment procedure used to define "statistically similar states."
  5. [Throughout] There are typographical and consistency issues, including "Grandient" in Section 1, "Diffuion-QL" in the Conclusion, and inconsistent capitalization of "FAST-Q" versus "Fast-Q" in Table 2 and elsewhere.

Circularity Check

2 steps flagged · score 6.0 of 10

Q-decomposition 'explainability' and offline-exploration 'speedup' reduce to the model's own fitted weights and Q-values; central system claims retain external online/D4RL support.

  1. fitted input called prediction [Section 3.5.2 (Ldecomp) and Section 4.2.1 / Figure 10]
    "Weights w1, w2, w3 are trained to estimate the following objective metrics: Ldecomp = Σ_{i=1}^C [(R_i_t − w_i ∗ Q(S^BR_t, A_t))^2]/C ... Figure 10 shows normalized distribution of the objectives for players with high organic intent to play, FAST-Q puts major focus on the return time followed by players’ engagement to keep them active on the platform."

    The weights are the parameters fitted by minimizing the MSE between w_i*Q and the reward component R_i. The 'objective prioritization' insight in Section 4.2.1 is read directly off those same fitted weights. No external ground truth, human label, or held-out objective-priority measure is used; the explanation is therefore a restatement of the training fit. If the loss had been defined differently, the reported 'focus' would change by construction, so this contribution does not test any hypothesis about the model's priorities.

  2. other [Section 4.2.3 / Figure 17]
    "Figure 17 shows a comparative analysis of the model’s performance, normalized with respect to the Q-values of the FAST-Q model trained on the complete dataset, highlighting the impact of exploration. The graph demonstrates that enabling the exploration component accelerates the learning process. For instance, the return for the model trained with exploration on 2 months of data is equivalent to model trained without exploration on 3 months of data."

    The 'performance' and 'return' used to measure speedup are FAST-Q's own Q-values, normalized by the Q-values of the final FAST-Q model. The exploration component selects actions to maximize this same critic, so the comparison shows convergence to a self-generated target rather than to an external counterfactual return. A model with systematically biased Q-values would exhibit the same 'speedup'; the evaluation cannot distinguish correct counterfactual estimation from self-consistency. Thus the offline-exploration speedup claim is validated only on the model's own output.

full rationale

The clearest circular steps are local. The Q-decomposition 'explainable objective prioritization' is a fitted parameter reported as an insight: w1..w3 are solved to reconstruct reward components, and Section 4.2.1 then presents those weights as the model's discovered priorities. Similarly, the offline-exploration speedup (Figure 17) is measured against FAST-Q's own Q-values rather than any external benchmark, making it a self-consistency check. The paper's central counterfactual-identification claim (Section 3.4) is not an independent derivation: the balancing condition only equalizes representation distributions across policies, and the step to 'estimates the counterfactual outcomes as E[R_t|Θ,A_t]' is an asserted identification assumption, not a consequence of the equations. This is a validity gap rather than an equation-level circularity, but it means the counterfactual component is not externally falsified by Table 2 (single-policy D4RL) or by the aggregate online A/B metrics. The paper's own limitation that γ could not be raised beyond 0.7 (Section 3.5.3) further weakens long-term counterfactual claims. Offsetting these, the paper does contain independent content: the 21-day online A/B test on the real platform and the D4RL normalized scores compare against external baselines and are not forced by the fitted weights, so the system-level improvement claims are not wholly circular. Weighing the self-referential Q-decomposition explanation and the internal speedup metric against this external evidence, the appropriate score is 6: partial circularity in two load-bearing evaluation/explanation steps, while the overall paper retains independent empirical content.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several non-standard modeling choices: the decomposition weights, the gamma/epsilon schedules, and the causal assumption that domain-adversarial balancing removes confounding. None of these are derived from first principles or validated against external counterfactual ground truth.

free parameters (6)
  • critic loss mixing weight alpha = 0.75
    Chosen experimentally in Section 3.5.2 to balance TD loss and decomposition loss.
  • discount factor gamma schedule = 0.1 to 0.7
    Section 3.5.3: gamma is increased stepwise upon stabilization of network losses; training cannot exceed 0.7.
  • exploration factor epsilon = 0.1 to 0.5
    Section 3.6.1: gradually increased during training to select counterfactual actions.
  • Q-decomposition weights w1..w4 = learned (values not reported)
    Section 3.5.2: weights are trained to reconstruct each reward component from the scalar Q; the fourth is an ad hoc overflow weight.
  • actor BC scaling lambda = alpha / mean(|Q|) with alpha = 2.5
    Adopted from TD3+BC (reference [13]) as recommended; affects the behavior-cloning strength.
  • reward normalization functions = Engagement=(6-b)/6, Return=1-sin((6-a)/2)
    Section 3.1: hand-specified transformations of days-to-return, which define the target rewards for the critic.
assumptions (4)
  • domain assumption Adversarially balanced representations remove policy-specific confounding
    Section 3.4 assumes P[Theta(beta_p1(S_t)) | phi_p1(...)] = ... = P[Theta(beta_pn(S_t)) | phi_pn(...)] is achieved by gradient reversal, which is a domain adaptation heuristic not a proven causal identification condition in RL.
  • ad hoc to paper The scalar Q can be decomposed as weighted reward components via learned weights
    Section 3.5.2 introduces w_i such that R_i ~ w_i * Q(S,A) with an extra overflow weight; no justification that such a decomposition exists or that the weights represent causal objective shares.
  • domain assumption Behavior cloning on counterfactual actions does not induce Q overestimation
    Section 3.6.1 uses the TD3+BC penalty with counterfactual actions; this is assumed to keep the actor near the data while exploring, but the OOD behavior of the critic on counterfactual actions is not verified.
  • domain assumption LSTM state history captures time-varying confounders
    Section 3.1 and 3.3: the PE LSTM encodes the player history; the paper assumes this makes the MDP Markovian and that residual confounding is handled by the BR layer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FAST-Q: Fast-track Exploration with Adversarially Balanced State Representations for Counterfactual Action Estimation in Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/OAGN6RIF

@misc{pith2026250421383,
  author       = {Pith},
  title        = {Pith review of: FAST-Q: Fast-track Exploration with Adversarially Balanced State Representations for Counterfactual Action Estimation in Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OAGN6RIF}},
  note         = {Machine review of arXiv:2504.21383}
}
read the original abstract

Recent advancements in state-of-the-art (SOTA) offline reinforcement learning (RL) have primarily focused on addressing function approximation errors, which contribute to the overestimation of Q-values for out-of-distribution actions, a challenge that static datasets exacerbate. However, high stakes applications such as recommendation systems in online gaming, introduce further complexities due to player's psychology (intent) driven by gameplay experiences and the inherent volatility on the platform. These factors create highly sparse, partially overlapping state spaces across policies, further influenced by the experiment path selection logic which biases state spaces towards specific policies. Current SOTA methods constrain learning from such offline data by clipping known counterfactual actions as out-of-distribution due to poor generalization across unobserved states. Further aggravating conservative Q-learning and necessitating more online exploration. FAST-Q introduces a novel approach that (1) leverages Gradient Reversal Learning to construct balanced state representations, regularizing the policy-specific bias between the player's state and action thereby enabling counterfactual estimation; (2) supports offline counterfactual exploration in parallel with static data exploitation; and (3) proposes a Q-value decomposition strategy for multi-objective optimization, facilitating explainable recommendations over short and long-term objectives. These innovations demonstrate superiority of FAST-Q over prior SOTA approaches and demonstrates at least 0.15 percent increase in player returns, 2 percent improvement in lifetime value (LTV), 0.4 percent enhancement in the recommendation driven engagement, 2 percent improvement in the player's platform dwell time and an impressive 10 percent reduction in the costs associated with the recommendation, on our volatile gaming platform.

Figures

Figures reproduced from arXiv: 2504.21383 by the authors.

Figure 2
Figure 2. Higher data sparsity in our Players Data (a) a) Hopper Data (b) b) Player Data [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. High Variability of actions across policies for identical states on our platform compared to Gym-MujoCo Hopper Task Dataset 2. Normalized State Representation for Q-Value Estima￾tion [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 7
Figure 7. FAST-Q Architecture. (a) Representing the PEs and their respective state representations feeding into the Policy Classifier and Actor-Critic (AC) Network, (b) Highlighting the flow of BRs through the AC Networks with Actor using 𝜖 factor to explore counterfactual and exploit exiting actions. 𝑥 ranging from 14, 21, 30 up to 45 days. Since we deal with a continuous action prediction problem we go with a Actor-Critic A… view at source ↗
Figures from the paper (12 more)
Figure 9
Figure 9. Figure 9: Training and loss convergence in FAST-Q with increa￾mental 𝛾 lift 0.75 is represented as: Lcritic = 𝛼 LTD(𝜃𝑞) + (1 − 𝛼) Ldecomp 3.5.3 Stepwise change of the discount factor: Since FAST￾Q learns the individual rewards and associates MSE’s on those as a part of the criti…
Figure 8
Figure 8. Figure 8: Training procedure for building balancing representation 3.5 Critic Network We follow the traditional method of Actor-Critic Training. The op￾timal 𝑄 ∗ (𝑆𝑡 , 𝐴𝑡), having the maximum expected discounted reward achievable by the optimal policy, should follow the optimal …
Figure 11
Figure 11. Figure 11: shows that for an active player, FAST-Q shows good balance of objectives throughout the time under observation vs. a less active player in the [PITH_FULL_IMAGE:figures/full_fig_p006_11.png]
Figure 12
Figure 12. Figure 12: FAST-Q primarily focuses on the return time for a sel￾dom active user [PITH_FULL_IMAGE:figures/full_fig_p006_12.png]
Figure 13
Figure 13. Figure 13: Better spread of Q-values against the actual and coun￾terfactual actions in FAST-Q vs SOTA BR for predictions and this results 2.3x and 6.3x higher spread of Q value estimates against Diffusion-QL and TD3+BC respectively. To further assess the confidence of FAST-Q in …
Figure 14
Figure 14. Figure 14: Well generalized (blended) state representations af￾ter BR [PITH_FULL_IMAGE:figures/full_fig_p006_14.png]
Figure 18
Figure 18. Figure 18: Drop in Q-values w.r.t the deployed model with Ablation on the FAST-Q Network components 4.3 Online Experiment on RummyCircle Dataset To further assess the real-world performance, we conducted an online A/B test by assigning 20% of users to our FAST-Q model and 20% ea…
Figure 16
Figure 16. Figure 16: Uncertainties on Q-value estimation on a state is 2.3x higher without the BR layer [PITH_FULL_IMAGE:figures/full_fig_p007_16.png]
Figure 17
Figure 17. Figure 17: Time speed up with Offline Exploration on Counterfac￾tuals. process. For instance, the return for the model trained with explo￾ration on 2 months of data is equivalent to model trained without exploration on 3 months of data. 4.2.4 Ablation Comparison [PITH_FULL_IMAG…
Figure 20
Figure 20. Figure 20: D21 LTV lift of FAST-Q w.r.t Diffusion, TD3+BC and old policies using our A/B test conclusion framework. The red and blue dotted vertical lines signify 90% and 95% support respectively. the average normalized score over the final 10 evaluations and 5 seeds. The ± capt…
Figure 19
Figure 19. Figure 19: Changes in Δ Engagement, Δ Return Time, Δ Dwell Time, Δ Cost within 21 days FAST-Q shows an impressive reduction of cost by 80% at D21 (21 days) w.r.t the exiting platform policies as well as the TD3+BC. 4.3.3 Results with A/B Experiment Conclusion Framework [PITH_FU…
Figure 21
Figure 21. Figure 21: Volatility in Player Enagement on our platform volatility in Player engagement on our platform over time. We see uncertainties with shifts in the mean values over the entire time period. A.4 Details on Hopper and Player Data State Representation: In the [PITH_FULL_IM…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 28 canonical work pages

  1. [1]

    Pulkit Agrawal, Aditya Pareek, Rukma Talwadker, and Tridib Mukherjee. 2024. ARGO - An AI Based Responsible Gamification Framework for Online Skill Gaming Platform. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3632410.3632455

    arXiv 2024

  2. [2]

    Breunig, Hans-Peter Kriegel, and Jörg Sander

    Mihael Ankerst, Markus M. Breunig, Hans-Peter Kriegel, and Jörg Sander. 1999. OPTICS: ordering points to identify the clustering structure. InProceedings of the 1999 ACM SIGMOD International Conference on Management of Data . Association for Computing Machinery, New York, NY, USA

    work page 1999

  3. [3]

    Leon Barrett and Srini Narayanan. 2008. Learning all optimal policies with multiple criteria. In Proceedings of the 25th International Conference on Machine Learning (ICML ’08)

    work page 2008

  4. [4]

    Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2006. Anal- ysis of Representations for Domain Adaptation. In NIPS

    work page 2006

  5. [5]

    Alaa, James Jordon, and Mihaela van der Schaar

    Ioana Bica, Ahmed M. Alaa, James Jordon, and Mihaela van der Schaar. 2020. Es- timating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representations. arXiv:2002.04083 [cs.LG] https://arxiv.org/abs/2002. 04083

    arXiv 2020

  6. [6]

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. arXiv:arXiv:1606.01540

    arXiv 2016

  7. [7]

    Hasegawa- Johnson, and Thomas S

    Shiyu Chang, Yang Zhang, Jiliang Tang, Dawei Yin, Yi Chang, Mark A. Hasegawa- Johnson, and Thomas S. Huang. 2017. Streaming Recommender Systems. In WWW

    work page 2017

  8. [8]

    Sharanya Eswaran, Mridul Sachdeva, Vikram Vimal, Deepanshi Seth, Suhaas Kalpam, Sanjay Agarwal, Tridib Mukherjee, and Samrat Dattagupta. 2020. Game Action Modeling for Fine Grained Analyses of Player Behavior in Multi-Player Card Games (Rummy as Case Study). In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

    work page 2020

Show all 46 references
  1. [9]

    Sharanya Eswaran, Vikram Vimal, Deepanshi Seth, and Tridib Mukherjee. 2020. GAIM: Game Action Information Mining Framework for Multiplayer Online Card Games (Rummy as Case Study). In Advances in Knowledge Discovery and Data Mining: 24th Pacific-Asia Conference, PAKDD 2020, Sin...

    2020

  2. [10]

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine

  3. [11]

    Scott Fujimoto. 2021. TD3+BC Repository. https://github.com/sfujim/TD3_BC

    2021

  4. [12]

    Scott Fujimoto. 2023. Diffusion-QL git Repo. https://github.com/Zhendong- Wang/Diffusion-Policies-for-Offline-RL

    2023

  5. [13]

    Scott Fujimoto and Shixiang Shane Gu. 2024. A minimalist approach to offline reinforcement learning. In Proceedings of the 35th International Conference on Neural Information Processing Systems (NIPS ’21) . Article 1540, 14 pages

    2024

  6. [14]

    Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function ap- proximation error in actor-critic methods. In International conference on machine learning, ICML. 1587–1596

    2018

  7. [15]

    Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-Policy Deep Re- inforcement Learning without Exploration. arXiv:1812.02900 [cs.LG] https: //arxiv.org/abs/1812.02900

    2019 arXiv

  8. [16]

    Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

    2016

  9. [17]

    Games24x7. 2024. FAST-Q GIT Repo. https://github.com/scarce-user-53/Fast-Q

    2024

  10. [18]

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-Adversarial Training of Neural Networks. arXiv:1505.07818 [stat.ML] https://arxiv.org/abs/1505.07818

    2016 arXiv

  11. [19]

    Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.Neural computation (1997)

    1997

  12. [20]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Opti- mization. arXiv:1412.6980 [cs.LG] https://arxiv.org/abs/1412.6980

    2017 arXiv

  13. [21]

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2021. Offline Reinforcement Learning with Implicit Q-Learning. arXiv:2110.06169 [cs.LG] https://arxiv.org/ abs/2110.06169

    2021 arXiv

  14. [22]

    Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. 2021. DR3: Value-Based Deep Reinforcement Learning Re- quires Explicit Regularization. arXiv:2112.04716 [cs.LG] https://arxiv.org/abs/ 2112.04716

    2021 arXiv

  15. [23]

    Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. 2019. Stabilizing off-policy Q-learning via bootstrapping error reduction. In NIPS

    2019

  16. [24]

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conserva- tive Q-Learning for Offline Reinforcement Learning. arXiv:2006.04779 [cs.LG] https://arxiv.org/abs/2006.04779

    2020 arXiv

  17. [25]

    Riedmiller

    Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. 2012. Batch Reinforce- ment Learning. In Reinforcement Learning . https://api.semanticscholar.org/ CorpusID:18760634

    2012

  18. [26]

    Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, and Jun Ma. 2017. Neural Attentive Session-based Recommendation. In CIKM 2017

    2017

  19. [27]

    Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947)

    1947

  20. [28]

    Leland McInnes, John Healy, and James Melville. 2020. UMAP: Uni- form Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426 [stat.ML] https://arxiv.org/abs/1802.03426

    2020 arXiv

  21. [29]

    Andriy Mnih and Russ R Salakhutdinov. 2007. Probabilistic Matrix Factorization. In NIPS

    2007

  22. [30]

    Sriraam Natarajan and Prasad Tadepalli. 2005. Dynamic preferences in multi- criteria reinforcement learning (ICML ’05)

    2005

  23. [31]

    Robins and Miguel A

    James M. Robins and Miguel A. Hernán. 2008. Estimation of the causal effects of time-varying exposures. https://api.semanticscholar.org/CorpusID:268324361

    2008

  24. [32]

    Wu, and Steve J

    Alice Schoenauer-Sebag, Louise Heinrich, Marc Schoenauer, Michele Sebag, Lani F. Wu, and Steve J. Altschuler. 2019. Multi-Domain Adversarial Learning

    2019

  25. [33]

    Peter Schulam and Suchi Saria. 2018. Reliable Decision Support using Counter- factual Models. arXiv:1703.10651 [stat.ML] https://arxiv.org/abs/1703.10651

    2018 arXiv

  26. [34]

    Kihyuk Sohn, Xinchen Yan, and Honglak Lee. [n. d.]. Learning structured output representation using deep conditional generative models. In NIPS 2015

    2015

  27. [35]

    Hossein Soleimani, Adarsh Subbaswamy, and Suchi Saria. 2017. Treatment- Response Models for Counterfactual Reasoning with Continuous-time, Continuous-valued Interventions. https://arxiv.org/abs/1704.02038

    2017 arXiv

  28. [36]

    Richard S Sutton and Andrew G Barto. 1998. Reinforcement learning: An intro- duction, volume 1. MIT press Cambridge

    1998

  29. [37]

    Rukma Talwadker, Surajit Chakrabarty, Aditya Pareek, Tridib Mukherjee, and Deepak Saini. 2022. CognitionNet: A Collaborative Neural Network for Play Style Discovery in Online Skill Gaming Platform. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

    2022

  30. [38]

    Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. 5026–5033. https://doi.org/10.1109/IROS.2012.6386109

    2012

  31. [39]

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. 2023. Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. arXiv:2208.06193 [cs.LG] https://arxiv.org/abs/2208.06193

    2023 arXiv

  32. [40]

    Zihan Wang, Ziheng Jiang, Zhaochun Ren, Jiliang Tang, and Dawei Yin. 2018. A Path-constrained Framework for Discriminating Substitutable and Complemen- tary Products in E-commerce. In WSDM 2018

    2018

  33. [41]

    Wikipedia contributors. 2024. Rummy — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Rummy&oldid=1194246527. [Online; accessed 10-November-2024]

    2024

  34. [42]

    Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. 2019. A generalized algorithm for multi-objective reinforcement learning and policy adaptation

    2019

  35. [43]

    Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. 2019. A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation. In NIPS

    2019

  36. [44]

    Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin

  37. [2019]

    arXiv:1902.05570 [cs.IR] https://arxiv.org/abs/1902

    Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems. arXiv:1902.05570 [cs.IR] https://arxiv.org/abs/1902. 05570 Pulkit Agrawal, Rukma Talwadker, Aditya Pareek, and Tridib Mukherjee Appendix A ALGORITHM A.1 FAST-Q Algorithm Algorithm 1 Q-learning...

    1902 arXiv

  38. [2021]

    arXiv:2004.07219 [cs.LG] https://arxiv.org/abs/2004.07219

    D4RL: Datasets for Deep Data-Driven Reinforcement Learning. arXiv:2004.07219 [cs.LG] https://arxiv.org/abs/2004.07219

    2004 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.