Pith. sign in

REVIEW 3 major objections 4 minor 23 references

Learning to identify the true action, not the reward, recovers the optimal bandit policy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

CSI converts offline contextual bandit learning into a binary classification problem by comparing the logged action against a counterfactual action sampled from the logging policy, and the argmax of the resulting classifier provably matches the argmax of expected reward.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection CSI is a genuinely new reduction from offline contextual bandits to binary classification, and the proof survives once you restore the missing conditioning on Y=1; the paper needs a corrected lemma and more honest experimental claims, but it deserves referee time. the 3 major comments →

arxiv 2509.10520 v1 pith:ROLQJQ6X submitted 2025-09-03 cs.LG

Offline Contextual Bandit with Counterfactual Sample Identification

Alexandre Gilotte , Otmane Sakhi , Imad Aouali , Benjamin Heymann This is my paper
classification cs.LG
keywords offline contextual banditcounterfactual sample identificationoff-policy learningcounterfactual reasoningconfoundingdirect reward modelinverse propensity scoringlogging policy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Counterfactual Sample Identification (CSI) for offline contextual bandits. Instead of fitting a direct model of expected reward as a function of context and action, CSI keeps only successful outcomes, pairs each one with a counterfactual action drawn from the logging policy, and trains a binary classifier to tell the real action from the resampled one. The paper proves that the classifier's score is a monotone transform of the true expected reward, so picking the action with the highest score is the same as picking the optimal action. On synthetic data with strong context effects, CSI approaches the performance of inverse-propensity-scoring methods and beats the direct method, and in a live banner-design system it improved clicks by 0.5–1% while also working with a reduced feature set where the direct method failed. The method matters because it brings the robustness of IPS-style counterfactual reasoning into standard supervised learning.

Core claim

The central discovery is the identity in Lemma 2.1: after filtering the logged data to positive outcomes and pairing each true action with a counterfactual action sampled from the logging policy π0, the probability that the observed action is the true one, P(Z=1 | X=x, B=a, Y=1), equals σ( log( P(Y=1|X=x,A=a) / P(Y=1|X=x) ) ). Since the denominator does not depend on the action, the classifier score f(x,a) orders actions exactly by their expected reward. The paper therefore claims that argmax_a f(x,a) = argmax_a P(Y=1|X=x,A=a), so the greedy CSI policy is the optimal policy, and this holds without ever modeling the direct effect of context on reward. Empirical results on synthetic environmen

What carries the argument

The load-bearing mechanism is the log-odds ratio identity of Lemma 2.1. By sampling a counterfactual action A' ~ π0(·|x) and labeling the true action Z=1, the paper converts reward estimation into a binary classification problem whose Bayes optimal score is a sigmoid of the log ratio of the action's reward to the context's baseline reward P(Y=1|x). The context baseline cancels out of the argmax, so classification scores rank the actions; the object doing the work is the ratio P(Y=1|x,a)/P(Y=1|x), which the paper calls the multiplicative advantage of an action over the policy's typical outcome.

Load-bearing premise

The reduction holds only when the logging policy assigns a strictly positive probability to every available action in every context, and when the classifier is trained on positive outcomes only; if the logging policy is deterministic for any context, the two labels become indistinguishable and the method outputs a constant 0.5.

What would settle it

On a synthetic environment with a known reward oracle, evaluate the trained CSI classifier and the true reward for a fixed context x across all actions; if the ranking of f(x,a) disagrees with the ranking of P(Y=1|x,a) on any single context, Lemma 2.1 is contradicted. A cheaper check: for one context, compare the classifier logit to the oracle log-odds; a non-monotone scatterplot falsifies the identity.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the classifier score f(x,a) is a monotone transform of the expected reward, then any supervised classifier can be used to build the greedy policy, and its argmax coincides with the optimal action; no context-effect model or IPS variance reduction is needed.
  • CSI removes the need to model main effects of context: a context feature affects the CSI model only when the relative performance of actions changes, which the paper argues reduces the confounding caused by regularization.
  • Replacing the sampled counterfactual by an expectation over all actions weighted by π0(a'|x) removes sampling noise and consistently improves performance, as shown in the synthetic experiments.
  • The method inherits the requirement of IPS: the logging policy must be stochastic and cover all actions; under a deterministic policy, the counterfactual and true actions coincide and the target is uninformative.
  • In production, CSI lifted online clicks by 0.5–1% over a direct reward model, and it recovered nearly all performance of the full-feature model when trained on a reduced feature set, demonstrating that the earlier gap was due to confounding.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The appendix proof cancels the action-sampling probabilities P(A=a|x), so the identity holds for any logging policy with full support, not only the uniform policy named in Lemma 2.1; the paper's uniform assumption is stronger than its proof requires.
  • CSI is effectively a per-context uplift model: it estimates the multiplicative lift of an action over the policy's average outcome. This suggests direct connections to uplift modeling and to off-policy learning of treatment effects beyond the bandit setting.
  • For continuous rewards, CSI could be applied by thresholding outcomes into successes and failures; the choice of threshold would change the target odds and could be tuned as a hyper-parameter, something the paper does not explore.
  • The production observation that IPS improved offline but not online, while CSI improved both, points to a robustness advantage of classification-based identification against small sequence effects; the paper floats this hypothesis, and it is testable in a controlled near-i.i.d. environment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Counterfactual Sample Identification (CSI), a method for offline contextual bandit learning that recasts reward prediction as binary classification on an augmented dataset. The method keeps only positive logged examples, samples a counterfactual action from the known logging policy, and trains a classifier to distinguish the true action from the resampled one. The paper claims a theoretical identity (Lemma 2.1) linking the Bayes-optimal classifier to the true expected reward, so that argmax_a f(x,a) ranks actions by expected reward. Experiments on synthetic data and a large-scale production banner-optimization system are reported, claiming that CSI consistently outperforms direct reward models.

Significance. If the central identity is correct, CSI offers a novel, parameter-free reduction from off-policy learning to supervised classification, with the potential to be more robust to confounding than direct reward models because it focuses on the multiplicative advantage of actions rather than the main effect of context. The theoretical derivation in the appendix is sound and requires only full support of the logging policy. The paper also provides a reproducible synthetic experiment and a rare production deployment study, which are valuable contributions. However, the main-text statement of the key lemma is inaccurate and the empirical claims outrun the reported evidence; these issues need correction before the paper can be accepted.

major comments (3)
  1. [Section 2.2 (Lemma 2.1)] Lemma 2.1 is not well-defined and is false as stated. Z is defined only on the augmented dataset \hat{D} built from positive examples, so the left-hand side should be P(Z=1 | X=x, B=a, Y=1), with B the action in the augmented log, exactly as proved in Appendix A. The main text instead writes P(Z=1 | A=a, X=x), omitting the conditioning on Y=1 and using A rather than B. The assumption 'uniform coverage policy' is also unnecessarily strong; Appendix A only requires full support, pi_0(a|x)>0. Because the argmax identity immediately following the lemma relies on this result, the main text should state and prove the conditioned version. This is not merely cosmetic: a reader could wrongly infer that the method works without the positive-only filter.
  2. [Abstract / Section 3.1 (Table 1)] The abstract claims CSI 'consistently outperforms direct models in both synthetic experiments and real-world deployments.' Table 1 does not support 'consistently' for the synthetic experiments: at 10K samples DM (0.76) outperforms CSI-expect (0.71) and CSI-sampling (0.62); at 100K samples CSI-sampling (0.82) is below DM (0.83). Only means over 100 environments are reported, with no standard errors, confidence intervals, or significance tests. Without dispersion information, the consistency claim is not established. Please report error bars or variance measures and qualify the abstract.
  3. [Section 3.2 (Table 2)] The production experiment reports relative click percentages for online A/B tests, but the table only provides confidence intervals for the IPS estimates, not for the online clicks column. For example, the CSI all-features row reports 101.5% clicks with an IPS interval [100.4%,100.5%], yet the online A/B test column shows the same value without an interval. The claim of a 0.5%-1% consistent online improvement cannot be assessed without uncertainty quantification for the online metric. Please provide confidence intervals or clarify which numbers in the table are online observations.
minor comments (4)
  1. [Section 2.2] Typo in the sentence after Lemma 2.1: 'P(Z=1|A=a, X=a)' should be 'P(Z=1|X=x, A=a)' or 'P(Z=1|X=x, B=a)'.
  2. [Section 2.2] The notation is overloaded: A is used for the online action, A' for the resampled action, and later B for the action observed in the augmented log. Use B consistently in Lemma 2.1 and the displayed argmax.
  3. [Appendix B] Typo in the first sentence: 'Description if the synthetic environment' should be 'Description of the synthetic environment'.
  4. [Table 1] The table heading says 'Click-through rate' but the text says 'mean normalized reward' (Appendix B). Please align the caption with the metric actually reported.

Circularity Check

0 steps flagged

No circularity found: the CSI argmax identity is derived from the data-generating process, not assumed.

full rationale

The paper's central derivation is self-contained and non-circular. The method constructs an augmented dataset \hat{D} from positive logged examples by sampling a counterfactual action A' from the logging policy \pi_0, then trains a classifier to distinguish the true action (Z=1) from the counterfactual (Z=0). The key claim is that the Bayes-optimal classifier score equals sigmoid(log(P(Y=1|X,A)/P(Y=1|X))), so that its argmax over actions matches the argmax of the true expected reward. This is a mathematical identity proved in Appendix A from the definition of the augmented data and the conditional independence of A' from Y given X; it does not assume the conclusion. The main-text Lemma 2.1 is compact and could be read as omitting the conditioning on Y=1 or the full-support requirement, but the appendix proof supplies the correct full-support derivation, and the same identity holds when the probability is interpreted over the augmented dataset. No parameter is fitted to force the ranking: the classifier is fit to the binary Z labels, and the optimal-action claim is a property of the population limit, not a fitted artifact. The deterministic-policy degeneracy is explicitly acknowledged in Section 2.2, not hidden. Self-citations are used only for background and baselines (e.g., Logarithmic Smoothing), not to justify the core identity. The abstract's 'consistently outperforms' is not fully supported by Table 1 at 10K samples, but that is an empirical overstatement, not circular reasoning. Overall, the derivation chain is genuine: definition of Z -> Bayes classifier -> sigmoid reward-odds identity -> argmax equivalence.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 1 invented entities

The central claim rests on standard supervised learning assumptions and the known stochastic logging policy; no new physical entities or ad hoc constants are introduced except the L2 tuning hyperparameter. The method's key identity is derived from the data construction, not assumed.

free parameters (1)
  • L2 regularization strength for CSI logistic regression = not reported
    In Section 3.2 the authors state 'We only re-tuned the L2-regularization of this logistic' for the CSI model, implying a hyperparameter search whose protocol is unspecified. This could affect the comparison if the baseline DM regularization was not similarly tuned.
axioms (4)
  • domain assumption The logging policy pi_0 is known and has full support (pi_0(a|x)>0 for all a,x); the method degenerates if pi_0 is deterministic.
    Section 2.2 'Dependency on pi_0' states the method relies on a stochastic policy that explores well; the appendix assumes for each context x, for all a in A(x), pi_0(a|x) > 0.
  • domain assumption The reward Y is binary (Bernoulli) and depends on x and a via an unknown function.
    Section 2.1 restricts to binary rewards; the identity in Appendix A uses P(Y=1|x,a) and requires the Bernoulli parameter.
  • domain assumption The supervised classifier trained on the reconstructed dataset \hat{D} estimates the population conditional P(Z=1|X,B,Y=1).
    The greedy policy is the argmax of the learned f(x,a); if logistic regression with finite features cannot represent the log-ratio or the training sample is too small, the argmax may not match the true reward argmax. This is a standard supervised learning assumption.
  • domain assumption The contextual bandit i.i.d. assumption holds; residual sequence effects are ignored.
    Section 1 acknowledges 'contextual bandits are only approximations' and that residual sequence effects break the i.i.d. assumption; the paper notes the same question applies to CSI.
invented entities (1)
  • Auxiliary variable Z (true-action indicator) independent evidence
    purpose: Binary target in the reconstructed training set that marks whether the row contains the true logged action or a resampled counterfactual action; used to train the CSI classifier f(x,a).
    Z is a design variable of the method, not a hypothesized physical entity; its distribution is fully controlled by the data construction. It has a clear operational definition and needs no external evidence.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Offline Contextual Bandit with Counterfactual Sample Identification." pith.science (2026). https://pith.science/paper/ROLQJQ6X

@misc{pith2026250910520,
  author       = {Pith},
  title        = {Pith review of: Offline Contextual Bandit with Counterfactual Sample Identification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ROLQJQ6X}},
  note         = {Machine review of arXiv:2509.10520}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In production systems, contextual bandit approaches often rely on direct reward models that take both action and context as input. However, these models can suffer from confounding, making it difficult to isolate the effect of the action from that of the context. We present \emph{Counterfactual Sample Identification}, a new approach that re-frames the problem: rather than predicting reward, it learns to recognize which action led to a successful (binary) outcome by comparing it to a counterfactual action sampled from the logging policy under the same context. The method is theoretically grounded and consistently outperforms direct models in both synthetic experiments and real-world deployments.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages · 1 internal anchor

  1. [1]

    Imad Aouali, Amine Benhalloum, Martin Bompaire, Achraf Ait Sidi Hammou, Sergey Ivanov, Benjamin Heymann, David Rohde, Otmane Sakhi, Flavian Vasile, and Maxime Vono. 2022. Reward Optimizing Recommendation using Deep Learning and Fast Maximum Inner Product Search. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Washi...

    work page arXiv 2022

  2. [2]

    Imad Aouali, Victor-Emmanuel Brunel, David Rohde, and Anna Korba. 2023. Exponential Smoothing for Off-Policy Learning. In Proceedings of the 40th Inter- national Conference on Machine Learning . PMLR, 984–1017

    work page 2023

  3. [3]

    Imad Aouali, Victor-Emmanuel Brunel, David Rohde, and Anna Korba. 2024. Bayesian Off-Policy Evaluation and Learning for Large Action Spaces. arXiv preprint arXiv:2402.14664 (2024)

    Pith/arXiv arXiv 2024

  4. [4]

    Imad Aouali, Victor-Emmanuel Brunel, David Rohde, and Anna Korba. 2024. Unified PAC-Bayesian Study of Pessimism for Offline Policy Learning with Regu- larized Importance Sampling. In The 40th Conference on Uncertainty in Artificial Intelligence. https://openreview.net/forum?id=d7W4H0sTXU

    work page 2024

  5. [5]

    Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. 2013. Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising. Journal of Machine Learning Research 14, 11 (2013)

    2013

  6. [6]

    Miroslav Dudík, John Langford, and Lihong Li. 2011. Doubly Robust Policy Evaluation and Learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning (Bellevue, Washington, USA) (ICML’11). 1097–1104

    work page 2011

  7. [7]

    Dmitri Goldenberg, Javier Albert, Lucas Bernardi, and Pablo Estevez. 2020. Free Lunch! Retrospective Uplift Modeling for Dynamic Promotions Recommendation within ROI Constraints. In Fourteenth ACM Conference on Recommender Systems (RecSys ’20). ACM, 486–491. https://doi.org/10.1145/3383313.3412215

    arXiv 2020

  8. [8]

    Regularization and confounding in linear regression for treatment effect estimation

    P. Richard Hahn, Carlos M. Carvalho, Jingyu He, and David Puelz. 2016. Regu- larization and confounding in linear regression for treatment effect estimation. arXiv:1602.02176 [stat.ME] https://arxiv.org/abs/1602.02176

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  9. [9]

    Daniel G Horvitz and Donovan J Thompson. 1952. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association 47, 260 (1952), 663–685

    1952

  10. [10]

    Olivier Jeunen and Bart Goethals. 2021. Pessimistic reward models for off- policy learning in recommendation. InFifteenth ACM Conference on Recommender Systems. 63–74

    2021

  11. [11]

    Ilja Kuzborskij, Claire Vernade, Andras Gyorgy, and Csaba Szepesvári. 2021. Confident off-policy evaluation and selection through self-normalized importance weighting. In International Conference on Artificial Intelligence and Statistics . PMLR, 640–648

    2021

  12. [12]

    Tor Lattimore and Csaba Szepesvari. 2019. Bandit Algorithms. Cambridge Uni- versity Press

    work page 2019

  13. [13]

    Ben London and Ted Sandler. 2019. Bayesian counterfactual risk minimization. In International Conference on Machine Learning . PMLR, 4125–4133

    work page 2019

  14. [14]

    Alberto Maria Metelli, Alessio Russo, and Marcello Restelli. 2021. Subgaussian and differentiable importance sampling for off-policy evaluation and learning. Advances in Neural Information Processing Systems 34 (2021), 8119–8132

    2021

  15. [15]

    Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. 2017. Elements of causal inference: foundations and learning algorithms . The MIT press

    work page 2017

  16. [16]

    Otmane Sakhi, Pierre Alquier, and Nicolas Chopin. 2023. PAC-Bayesian Offline Contextual Bandits with Guarantees. In International Conference on Machine Learning. PMLR, 29777–29799

    work page 2023

  17. [17]

    Otmane Sakhi, Imad Aouali, Pierre Alquier, and Nicolas Chopin. 2024. Logarith- mic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning. In Advances in Neural Information Processing Systems , A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 80706–80755. https://p...

    2024

  18. [18]

    Otmane Sakhi, Stephen Bonner, David Rohde, and Flavian Vasile. 2020. BLOB: A Probabilistic model for recommendation that combines organic and bandit signals. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 783–793

    work page 2020

  19. [19]

    Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. 2020. Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning. PMLR, 9167–9176

    2020

  20. [20]

    Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. 2019. Cab: Continuous adaptive blending for policy evaluation and learning. InInternational Conference on Machine Learning . PMLR, 6005–6014

    2019

  21. [21]

    Richard Sutton and Andrew Barto. 1998. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA

    work page 1998

  22. [22]

    Adith Swaminathan and Thorsten Joachims. 2015. Batch learning from logged bandit feedback through counterfactual risk minimization.The Journal of Machine Learning Research 16, 1 (2015), 1731–1755

    2015

  23. [23]

    Yinglun Zhu, Dylan J Foster, John Langford, and Paul Mineiro. 2022. Contextual bandits with large action spaces: Made practical. In International Conference on Machine Learning. PMLR, 27428–27453. A PROOF OF THE LEMMA 2.1 We suppose that for each context𝑥,𝜋0 covers the possible action spaceA(𝑥), that is for each context𝑥: ∀𝑎∈A( 𝑥), 𝜋 0(𝑎|𝑥) > 0. Now, to c...

    work page 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.