Pith. sign in

REVIEW 2 major objections 2 minor 39 references

Online Nonsubmodular Optimization with Delayed Feedback in the Bandit Setting

T0 review · 2 major / 2 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read The paper establishes that delayed bandit feedback in nonsubmodular optimization can be priced by average delay rather than maximum delay, and gives a blocking variant whose regret adds the delay cost to the bandit cost.

desk verdict Promising theoretical bounds that improve on prior work, but with only the abstract available the proofs—and the missing analytic assumptions for the one-point estimator—are what a referee must check. read the letter →

arxiv 2508.00523 v1 pith:VHHDWUTU submitted 2025-08-01 cs.LG

classification cs.LG
keywords onlinelearningnonsubmodularoptimizationbanditfeedbackdelayedregretboundDR-submodularone-pointgradientestimatorblockingupdate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies online optimization over losses that are not convex but are only approximately submodular, in the bandit setting where the learner sees only the value of the point it chose and feedback arrives after a delay. The first algorithm, DBGD-NF, uses a one-point gradient estimator and exploits every estimated gradient that has arrived in a round, achieving an $(\alpha,\beta)$-regret bound of $\mathcal{O}(n\bar{d}^{1/3}T^{2/3})$, where $\bar{d}$ is the average delay. The second adds a blocking update mechanism and attains $\mathcal{O}(n(T^{2/3}+\sqrt{dT}))$, separating the delay contribution from the bandit contribution. If these bounds are correct, the price of delayed feedback depends on the typical delay, not the worst spike, and when the maximum delay is small the algorithm recovers the no-delay bandit regret.

What carries the argument

Two mechanisms carry the argument. First, the one-point gradient estimator: the algorithm queries a single function value per round and constructs an unbiased gradient estimate from it, and by using all delayed estimates that have arrived by the current round it accumulates information faster than a method that waits on the latest sample. Second, the blocking update: decisions are held fixed for blocks of rounds, so the randomness from bandit feedback and the randomness from delay can be controlled in separate terms; this is what turns a product of delay and bandit regret into a sum.

What would settle it

Take a problem instance that satisfies the paper's regularity conditions, keep the average delay $\bar{d}$ fixed, and shift all delay into a single round so that the maximum delay $d$ grows; if the empirical regret of DBGD-NF grows with $d$ rather than staying tied to $\bar{d}$, the average-delay bound is false. For the blocking variant, run it under delay patterns with the same $d$ but different block boundaries and check whether the regret separates into an $nT^{2/3}$ bandit term plus an $n\sqrt{dT}$ delay term.

Watch

Extended reading notes

Core claim

The central claim is that delayed feedback in bandit online optimization over $\alpha$-weakly DR-submodular and $\beta$-weakly DR-supermodular losses—continuous functions whose diminishing-returns property holds only approximately—is less costly than earlier bounds suggested. Previous work bounded regret by $\mathcal{O}(nd^{1/3}T^{2/3})$ using the maximum delay $d$ and multiplied the delay penalty with the no-delay bandit regret. DBGD-NF instead averages over all available gradient estimates each round, turning the delay dependence into $\bar{d}^{1/3}$, where $\bar{d}=\frac{1}{T}\sum_{t=1}^T d_t$ is the mean delay. The blocking variant updates the decision only at block boundaries, which changes the joint effect of delay and bandit feedback from a product into a sum, yielding $\mathcal{O}(n(T^{2/3}+\sqrt{dT}))$; for $d=\mathcal{O}(T^{1/3})$ this matches the $\mathcal{O}(nT^{2/3})$ regret achievable without delay.

Load-bearing premise

The bounds rely on the one-point gradient estimator being unbiased with bounded variance, which requires the loss functions to be differentiable and Lipschitz on a bounded convex feasible set; if these regularity conditions fail, the concentration arguments behind the regret bounds collapse.

Editorial extensions

If this is right

  • Regret degrades with the average delay $\bar{d}$ rather than the maximum delay $d$, so rare long delays become nearly harmless under DBGD-NF.
  • The blocking variant achieves an additive $\mathcal{O}(n(T^{2/3}+\sqrt{dT}))$ bound, meaning bandit error and delay error no longer multiply against each other.
  • When $d=\mathcal{O}(T^{1/3})$, the blocking variant reproduces the $\mathcal{O}(nT^{2/3})$ regret bound of the bandit setting without delayed feedback.
  • Comparing the two new bounds, the blocking variant is preferable when the maximum delay satisfies $d=o(\bar{d}^{2/3}T^{1/3})$.
  • Experiments on structured sparse learning illustrate that both methods improve over the previous worst-case-delay algorithm.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A general lesson the paper does not state: for delayed bandit online optimization, the first moment of the delay distribution is often the right object, not the worst case, which suggests other delayed-feedback algorithms could be made delay-robust by averaging over all available queries.
  • The additive decoupling in the blocking variant may transfer to other bandit problems, predicting that any bandit algorithm whose regret is known without delay can be converted to a delayed setting by paying at most an added $\sqrt{dT}$ term.
  • A testable extension would compare DBGD-NF against the previous method under delay distributions with the same maximum delay but different means; if mean-delay dependence is real, the advantage should grow as the mean shrinks.
  • The one-point estimator plus all-available-gradients rule suggests a cheap modification for many online algorithms: keep a queue of delayed gradient estimates and update with all of them, not just the freshest one.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper studies online nonsubmodular optimization with delayed feedback in the bandit setting, assuming the loss functions are α-weakly DR-submodular and β-weakly DR-supermodular. The abstract proposes DBGD-NF, a one-point gradient estimator method that uses all available delayed gradients in each round and claims a regret bound of O(n\bar{d}^{1/3}T^{2/3}) in terms of the average delay \bar{d}, improving on the previous O(nd^{1/3}T^{2/3}) bound that depended on the maximum delay d. A second algorithm uses a blocking update mechanism and claims an O(n(T^{2/3}+\sqrt{dT})) regret bound, which additively decouples the effects of delays and bandit feedback and matches the no-delay bound when d=O(T^{1/3}). The abstract also reports experiments on structured sparse learning. The supplied manuscript contains only the abstract; no full-text proofs or derivations are provided.

Significance. If the claimed bounds are correct, the paper makes a meaningful theoretical contribution: it replaces the maximum delay with the average delay in the first bound and additively decouples delay effects from bandit estimation error in the second, improving known results. The internal arithmetic is consistent: the second bound reduces to the no-delay O(nT^{2/3}) bound when d=O(T^{1/3}), and the comparison condition d=o(\bar{d}^{2/3}T^{1/3}) is arithmetically correct. However, because only the abstract is available, the decisive proof details—especially the concentration arguments and the regularity conditions for the one-point estimator—cannot be checked, so the significance is conditional.

major comments (2)
  1. [Abstract] The abstract assumes only α-weak DR-submodularity and β-weak DR-supermodularity, but the one-point gradient estimator used by both algorithms requires differentiability and L-Lipschitz continuity of the losses on a bounded convex feasible set with a sampling density bounded away from zero. These conditions are not stated, and weak DR conditions do not imply them; the full text must explicitly impose these regularity assumptions and prove the unbiasedness and bounded variance of the estimator, otherwise the concentration arguments behind both regret bounds are unsupported.
  2. [Abstract (DBGD-NF description)] The claim that using all available delayed gradients in each round yields an O(n\bar{d}^{1/3}T^{2/3}) bound relies on a concentration inequality for a sequence in which the current decision depends on past gradients. The abstract gives no indication of how this dependent, delayed sequence is handled; without a proof of such an inequality, the average-delay bound is not established by the available material.
minor comments (2)
  1. [Abstract] The notation \bar{d} is defined as the average delay, but the definition would be clearer if placed in a displayed equation in the full text.
  2. [Abstract] The final sentence mentions experiments on structured sparse learning without naming baselines or metrics; the full text should specify the experimental setup and comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found in the available text; the regret bounds are presented as analytical improvements over an external benchmark with no fitted inputs or self-citation chain visible.

full rationale

Only the abstract is available; the full text is empty. The central claims are theoretical regret bounds for two algorithms, derived against the previously known O(n d^{1/3} T^{2/3}) bound and the no-delay bandit setting. Nothing in the abstract defines a quantity in terms of the target result, fits a parameter and then renames it as a prediction, or imports an unverified uniqueness theorem via self-citation. The skeptical concerns about missing differentiability, Lipschitzness, and variance assumptions are correctness or rigor concerns, not circularity: assuming additional regularity conditions would not make the regret bounds true by construction. Without any equations or proof text, no specific reduction can be exhibited, and the instructions require quoted evidence for any circularity finding. Therefore the honest finding is no significant circularity, with score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the problem-class assumption in the abstract (weak DR conditions) and on standard, unstated regularity conditions for bandit optimization. No free parameters are fitted to data; the algorithm's internal step sizes are not described in the abstract. No invented entities are introduced.

assumptions (3)
  • domain assumption The loss functions are alpha-weakly DR-submodular and beta-weakly DR-supermodular.
    Stated in the abstract; defines the problem class and the (alpha,beta)-regret notion used in all bounds.
  • domain assumption The one-point gradient estimator is unbiased and has bounded variance.
    Not stated in the abstract but required for bandit gradient methods; the abstract says DBGD-NF uses a one-point gradient estimator.
  • domain assumption The feasible set is bounded and convex, and the loss functions are Lipschitz continuous.
    Standard in online bandit optimization proofs; inferred from the need for concentration and projection steps, though not stated in the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Nonsubmodular Optimization with Delayed Feedback in the Bandit Setting." pith.science (2026). https://pith.science/paper/VHHDWUTU

@misc{pith2026250800523,
  author       = {Pith},
  title        = {Pith review of: Online Nonsubmodular Optimization with Delayed Feedback in the Bandit Setting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHHDWUTU}},
  note         = {Machine review of arXiv:2508.00523}
}
abstract

We investigate the online nonsubmodular optimization with delayed feedback in the bandit setting, where the loss function is $\alpha$-weakly DR-submodular and $\beta$-weakly DR-supermodular. Previous work has established an $(\alpha,\beta)$-regret bound of $\mathcal{O}(nd^{1/3}T^{2/3})$, where $n$ is the dimensionality and $d$ is the maximum delay. However, its regret bound relies on the maximum delay and is thus sensitive to irregular delays. Additionally, it couples the effects of delays and bandit feedback as its bound is the product of the delay term and the $\mathcal{O}(nT^{2/3})$ regret bound in the bandit setting without delayed feedback. In this paper, we develop two algorithms to address these limitations, respectively. Firstly, we propose a novel method, namely DBGD-NF, which employs the one-point gradient estimator and utilizes all the available estimated gradients in each round to update the decision. It achieves a better $\mathcal{O}(n\bar{d}^{1/3}T^{2/3})$ regret bound, which is relevant to the average delay $\bar{d} = \frac{1}{T}\sum_{t=1}^T d_t\leq d$. Secondly, we extend DBGD-NF by employing a blocking update mechanism to decouple the joint effect of the delays and bandit feedback, which enjoys an $\mathcal{O}(n(T^{2/3} + \sqrt{dT}))$ regret bound. When $d = \mathcal{O}(T^{1/3})$, our regret bound matches the $\mathcal{O}(nT^{2/3})$ bound in the bandit setting without delayed feedback. Compared to our first $\mathcal{O}(n\bar{d}^{1/3}T^{2/3})$ bound, it is more advantageous when the maximum delay $d = o(\bar{d}^{2/3}T^{1/3})$. Finally, we conduct experiments on structured sparse learning to demonstrate the superiority of our methods.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 36 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bach, F. 2010. Structured sparsity-inducing norms through submodular functions. In Advances in Neural Information Processing Systems 23, 118--126

  4. [4]

    Badanidiyuru, A.; and Vondr \'a k, J. 2014. Fast algorithms for maximizing submodular functions. In Proceedings of the 25th Annual ACM-SIAM Symposium on Discrete Algorithms, 1497--1514

  5. [5]

    P.; Schapire, R

    Cesa-Bianchi, N.; Freund, Y.; Haussler, D.; Helmbold, D. P.; Schapire, R. E.; and Warmuth, M. K. 1997. How to use expert advice. Journal of the ACM, 44(3): 427--485

  6. [6]

    Cesa-Bianchi, N.; and Lugosi, G. 2012. Combinatorial bandits. Journal of Computer and System Sciences, 78(5): 1404--1422

  7. [7]

    Das, A.; Dasgupta, A.; and Kumar, R. 2012. Selecting diverse features via spectral regularization. In Advances in Neural Information Processing Systems 25, 3026--3034

  8. [8]

    Defazio, A.; and Caetano, T. S. 2012. A convex formulation for learning scale-free networks via submodular relaxation. In Advances in Neural Information Processing Systems 25, 1250--1258

Show all 39 references
  1. [9]

    Edmonds, J. 1970. Submodular functions, matroids, and certain polyhedra. In Combinatorial structures and their appli- cations, 69--87

  2. [10]

    El Halabi, M.; and Cevher, V. 2015. A totally unimodular view of structured sparsity. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, 223--231

  3. [11]

    El Halabi, M.; and Jegelka, S. 2020. Optimal approximation for unconstrained non-submodular minimization. In Proceedings of the 37th International Conference on Machine Learning, 3961--3972

  4. [12]

    D.; Kalai, A

    Flaxman, A. D.; Kalai, A. T.; and McMahan, H. B. 2005. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms, 385--394

  5. [13]

    Garber, D.; and Kretzu, B. 2020. Improved regret bounds for projection-free bandit convex optimization. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, 2196--2206

  6. [14]

    Gomez-Rodriguez, M.; and Sch \"o lkopf, B. 2012. Submodular inference of diffusion networks from multiple trees. In Proceedings of the 29th International Conference on Machine Learning, 489--496

  7. [15]

    Gonz \'a lez, J.; Dai, Z.; Hennig, P.; and Lawrence, N. 2016. Batch bayesian optimization via local penalization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, 648--657

  8. [16]

    Hazan, E.; and Kale, S. 2012. Online submodular minimization. Journal of Machine Learning Research, 13(93): 2903--2922

  9. [17]

    Hazan, E.; et al. 2016. Introduction to online convex optimization. Foundations and Trends in Optimization, 2(3-4): 157--325

  10. [18]

    H \'e liou, A.; Mertikopoulos, P.; and Zhou, Z. 2020. Gradient-free online learning in continuous games with delayed rewards. In Proceedings of the 37th International Conference on Machine Learning, 4172--4181

  11. [19]

    Krause, A.; and Golovin, D. 2014. Submodular function maximization. Tractability, 3(71-104): 3

  12. [20]

    E.; Tran-Dinh, Q.; and Cevher, V

    Kyrillidis, A.; Baldassarre, L.; Halabi, M. E.; Tran-Dinh, Q.; and Cevher, V. 2015. Structured sparsity: Discrete and convex approaches. In Compressed Sensing and its Applications: MATHEON Workshop 2013, 341--387

  13. [21]

    Lehmann, B.; Lehmann, D.; and Nisan, N. 2006. Combinatorial auctions with decreasing marginal utilities. Games and Economic Behavior, 55(2): 270--296

  14. [22]

    Liao, Y.; Wan, Y.; Yao, C.; and Song, M. 2023. Improved Projection-free Online Continuous Submodular Maximization. arXiv preprint arXiv:2305.18442

  15. [23]

    Lin, T.; Pacchiano, A.; Yu, Y.; and Jordan, M. 2022. Online nonsubmodular minimization with delayed costs: from full information to bandit feedback. In Proceedings of the 39th International Conference on Machine Learning, 13441--13467

  16. [24]

    Lov \'a sz, L. 1983. Submodular functions and convexity. Mathematical Programming The State of the Art: Bonn 1982, 235--257

  17. [25]

    Quanrud, K.; and Khashabi, D. 2015. Online learning with adversarial delays. In Advances in Neural Information Processing Systems 28, 1270--1278

  18. [26]

    Rapaport, F.; Barillot, E.; and Vert, J.-P. 2008. Classification of arrayCGH data using fused SVM. Bioinformatics, 24(13): i375--i382

  19. [27]

    Sakaue, S. 2019. Greedy and iht algorithms for non-convex optimization with monotone costs of non-zeros. In The 22nd International Conference on Artificial Intelligence and Statistics, 206--215

  20. [28]

    Shalev-Shwartz. 2012. Online learning and online convex optimization. Foundations and Trends in Machine Learning, 4(2): 107--194

  21. [29]

    O.; Girshick, R.; Jegelka, S.; Mairal, J.; Harchaoui, Z.; and Darrell, T

    Song, H. O.; Girshick, R.; Jegelka, S.; Mairal, J.; Harchaoui, Z.; and Darrell, T. 2014. On learning to localize objects with minimal supervision. In Proceedings of the 31th International Conference on Machine Learning, 1611--1619

  22. [30]

    Sviridenko, M.; Vondr \'a k, J.; and Ward, J. 2017. Optimal approximation for submodular and supermodular optimization with bounded curvature. Mathematics of Operations Research, 42(4): 1197--1218

  23. [31]

    Vondr \'a k, J. 2007. Submodularity in combinatorial optimization. Springer

  24. [32]

    Wan, Y.; Wang, Y.; Yao, C.; Tu, W.-W.; and Zhang, L. 2022. Projection-free Online Learning with Arbitrary Delays. arXiv preprint arXiv:2204.04964

  25. [33]

    Wan, Y.; Yao, C.; Song, M.; and Zhang, L. 2023. Non-stationary online convex optimization with arbitrary delays. arXiv preprint arXiv:2305.12131

  26. [34]

    Wan, Y.; Yao, C.; Song, M.; and Zhang, L. 2024. Improved regret for bandit convex optimization with delayed feedback. arXiv preprint arXiv:2402.09152

  27. [35]

    Wang, Y.; Wan, Y.; Zhang, S.; and Zhang, L. 2023. Distributed Projection-Free Online Learning for Smooth and Convex Losses. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, 10226--10234

  28. [36]

    Wang, Y.; Yang, W.; Jiang, W.; Lu, S.; Wang, B.; Tang, H.; Wan, Y.; and Zhang, L. 2024. Non-stationary Projection-Free Online Learning with Dynamic and Adaptive Regret Guarantees. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, 15671--15679

  29. [37]

    Zhang, M.; Chen, L.; Hassani, H.; and Karbasi, A. 2019. Online continuous submodular maximization: From full-information to bandit feedback. In Advances in Neural Information Processing Systems 32, 9210--9221

  30. [38]

    Zheng, J.; Jiang, Z.; Chellappa, R.; and Phillips, J. P. 2014. Submodular attribute selection for action recognition in video. In Advances in Neural Information Processing Systems 27, 481--489

  31. [39]

    Zinkevich, M. 2003. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning, 928--936

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.