REVIEW 2 major objections 2 minor 39 references
Online Nonsubmodular Optimization with Delayed Feedback in the Bandit Setting
T0 review · 2 major / 2 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper establishes that delayed bandit feedback in nonsubmodular optimization can be priced by average delay rather than maximum delay, and gives a blocking variant whose regret adds the delay cost to the bandit cost.
desk verdict Promising theoretical bounds that improve on prior work, but with only the abstract available the proofs—and the missing analytic assumptions for the one-point estimator—are what a referee must check. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two mechanisms carry the argument. First, the one-point gradient estimator: the algorithm queries a single function value per round and constructs an unbiased gradient estimate from it, and by using all delayed estimates that have arrived by the current round it accumulates information faster than a method that waits on the latest sample. Second, the blocking update: decisions are held fixed for blocks of rounds, so the randomness from bandit feedback and the randomness from delay can be controlled in separate terms; this is what turns a product of delay and bandit regret into a sum.
What would settle it
Take a problem instance that satisfies the paper's regularity conditions, keep the average delay $\bar{d}$ fixed, and shift all delay into a single round so that the maximum delay $d$ grows; if the empirical regret of DBGD-NF grows with $d$ rather than staying tied to $\bar{d}$, the average-delay bound is false. For the blocking variant, run it under delay patterns with the same $d$ but different block boundaries and check whether the regret separates into an $nT^{2/3}$ bandit term plus an $n\sqrt{dT}$ delay term.
Extended reading notes
Core claim
The central claim is that delayed feedback in bandit online optimization over $\alpha$-weakly DR-submodular and $\beta$-weakly DR-supermodular losses—continuous functions whose diminishing-returns property holds only approximately—is less costly than earlier bounds suggested. Previous work bounded regret by $\mathcal{O}(nd^{1/3}T^{2/3})$ using the maximum delay $d$ and multiplied the delay penalty with the no-delay bandit regret. DBGD-NF instead averages over all available gradient estimates each round, turning the delay dependence into $\bar{d}^{1/3}$, where $\bar{d}=\frac{1}{T}\sum_{t=1}^T d_t$ is the mean delay. The blocking variant updates the decision only at block boundaries, which changes the joint effect of delay and bandit feedback from a product into a sum, yielding $\mathcal{O}(n(T^{2/3}+\sqrt{dT}))$; for $d=\mathcal{O}(T^{1/3})$ this matches the $\mathcal{O}(nT^{2/3})$ regret achievable without delay.
Load-bearing premise
The bounds rely on the one-point gradient estimator being unbiased with bounded variance, which requires the loss functions to be differentiable and Lipschitz on a bounded convex feasible set; if these regularity conditions fail, the concentration arguments behind the regret bounds collapse.
Editorial extensions
If this is right
- Regret degrades with the average delay $\bar{d}$ rather than the maximum delay $d$, so rare long delays become nearly harmless under DBGD-NF.
- The blocking variant achieves an additive $\mathcal{O}(n(T^{2/3}+\sqrt{dT}))$ bound, meaning bandit error and delay error no longer multiply against each other.
- When $d=\mathcal{O}(T^{1/3})$, the blocking variant reproduces the $\mathcal{O}(nT^{2/3})$ regret bound of the bandit setting without delayed feedback.
- Comparing the two new bounds, the blocking variant is preferable when the maximum delay satisfies $d=o(\bar{d}^{2/3}T^{1/3})$.
- Experiments on structured sparse learning illustrate that both methods improve over the previous worst-case-delay algorithm.
Reading between the lines
- A general lesson the paper does not state: for delayed bandit online optimization, the first moment of the delay distribution is often the right object, not the worst case, which suggests other delayed-feedback algorithms could be made delay-robust by averaging over all available queries.
- The additive decoupling in the blocking variant may transfer to other bandit problems, predicting that any bandit algorithm whose regret is known without delay can be converted to a delayed setting by paying at most an added $\sqrt{dT}$ term.
- A testable extension would compare DBGD-NF against the previous method under delay distributions with the same maximum delay but different means; if mean-delay dependence is real, the advantage should grow as the mean shrinks.
- The one-point estimator plus all-available-gradients rule suggests a cheap modification for many online algorithms: keep a queue of delayed gradient estimates and update with all of them, not just the freshest one.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies online nonsubmodular optimization with delayed feedback in the bandit setting, assuming the loss functions are α-weakly DR-submodular and β-weakly DR-supermodular. The abstract proposes DBGD-NF, a one-point gradient estimator method that uses all available delayed gradients in each round and claims a regret bound of O(n\bar{d}^{1/3}T^{2/3}) in terms of the average delay \bar{d}, improving on the previous O(nd^{1/3}T^{2/3}) bound that depended on the maximum delay d. A second algorithm uses a blocking update mechanism and claims an O(n(T^{2/3}+\sqrt{dT})) regret bound, which additively decouples the effects of delays and bandit feedback and matches the no-delay bound when d=O(T^{1/3}). The abstract also reports experiments on structured sparse learning. The supplied manuscript contains only the abstract; no full-text proofs or derivations are provided.
Significance. If the claimed bounds are correct, the paper makes a meaningful theoretical contribution: it replaces the maximum delay with the average delay in the first bound and additively decouples delay effects from bandit estimation error in the second, improving known results. The internal arithmetic is consistent: the second bound reduces to the no-delay O(nT^{2/3}) bound when d=O(T^{1/3}), and the comparison condition d=o(\bar{d}^{2/3}T^{1/3}) is arithmetically correct. However, because only the abstract is available, the decisive proof details—especially the concentration arguments and the regularity conditions for the one-point estimator—cannot be checked, so the significance is conditional.
major comments (2)
- [Abstract] The abstract assumes only α-weak DR-submodularity and β-weak DR-supermodularity, but the one-point gradient estimator used by both algorithms requires differentiability and L-Lipschitz continuity of the losses on a bounded convex feasible set with a sampling density bounded away from zero. These conditions are not stated, and weak DR conditions do not imply them; the full text must explicitly impose these regularity assumptions and prove the unbiasedness and bounded variance of the estimator, otherwise the concentration arguments behind both regret bounds are unsupported.
- [Abstract (DBGD-NF description)] The claim that using all available delayed gradients in each round yields an O(n\bar{d}^{1/3}T^{2/3}) bound relies on a concentration inequality for a sequence in which the current decision depends on past gradients. The abstract gives no indication of how this dependent, delayed sequence is handled; without a proof of such an inequality, the average-delay bound is not established by the available material.
minor comments (2)
- [Abstract] The notation \bar{d} is defined as the average delay, but the definition would be clearer if placed in a displayed equation in the full text.
- [Abstract] The final sentence mentions experiments on structured sparse learning without naming baselines or metrics; the full text should specify the experimental setup and comparisons.
Circularity Check
No circularity found in the available text; the regret bounds are presented as analytical improvements over an external benchmark with no fitted inputs or self-citation chain visible.
full rationale
Only the abstract is available; the full text is empty. The central claims are theoretical regret bounds for two algorithms, derived against the previously known O(n d^{1/3} T^{2/3}) bound and the no-delay bandit setting. Nothing in the abstract defines a quantity in terms of the target result, fits a parameter and then renames it as a prediction, or imports an unverified uniqueness theorem via self-citation. The skeptical concerns about missing differentiability, Lipschitzness, and variance assumptions are correctness or rigor concerns, not circularity: assuming additional regularity conditions would not make the regret bounds true by construction. Without any equations or proof text, no specific reduction can be exhibited, and the instructions require quoted evidence for any circularity finding. Therefore the honest finding is no significant circularity, with score 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The loss functions are alpha-weakly DR-submodular and beta-weakly DR-supermodular.
- domain assumption The one-point gradient estimator is unbiased and has bounded variance.
- domain assumption The feasible set is bounded and convex, and the loss functions are Lipschitz continuous.
Cite this review
Pith. "Pith review of Online Nonsubmodular Optimization with Delayed Feedback in the Bandit Setting." pith.science (2026). https://pith.science/paper/VHHDWUTU
@misc{pith2026250800523,
author = {Pith},
title = {Pith review of: Online Nonsubmodular Optimization with Delayed Feedback in the Bandit Setting},
year = {2026},
howpublished = {\url{https://pith.science/paper/VHHDWUTU}},
note = {Machine review of arXiv:2508.00523}
}
abstract
We investigate the online nonsubmodular optimization with delayed feedback in the bandit setting, where the loss function is $\alpha$-weakly DR-submodular and $\beta$-weakly DR-supermodular. Previous work has established an $(\alpha,\beta)$-regret bound of $\mathcal{O}(nd^{1/3}T^{2/3})$, where $n$ is the dimensionality and $d$ is the maximum delay. However, its regret bound relies on the maximum delay and is thus sensitive to irregular delays. Additionally, it couples the effects of delays and bandit feedback as its bound is the product of the delay term and the $\mathcal{O}(nT^{2/3})$ regret bound in the bandit setting without delayed feedback. In this paper, we develop two algorithms to address these limitations, respectively. Firstly, we propose a novel method, namely DBGD-NF, which employs the one-point gradient estimator and utilizes all the available estimated gradients in each round to update the decision. It achieves a better $\mathcal{O}(n\bar{d}^{1/3}T^{2/3})$ regret bound, which is relevant to the average delay $\bar{d} = \frac{1}{T}\sum_{t=1}^T d_t\leq d$. Secondly, we extend DBGD-NF by employing a blocking update mechanism to decouple the joint effect of the delays and bandit feedback, which enjoys an $\mathcal{O}(n(T^{2/3} + \sqrt{dT}))$ regret bound. When $d = \mathcal{O}(T^{1/3})$, our regret bound matches the $\mathcal{O}(nT^{2/3})$ bound in the bandit setting without delayed feedback. Compared to our first $\mathcal{O}(n\bar{d}^{1/3}T^{2/3})$ bound, it is more advantageous when the maximum delay $d = o(\bar{d}^{2/3}T^{1/3})$. Finally, we conduct experiments on structured sparse learning to demonstrate the superiority of our methods.
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Bach, F. 2010. Structured sparsity-inducing norms through submodular functions. In Advances in Neural Information Processing Systems 23, 118--126
work page 2010
-
[4]
Badanidiyuru, A.; and Vondr \'a k, J. 2014. Fast algorithms for maximizing submodular functions. In Proceedings of the 25th Annual ACM-SIAM Symposium on Discrete Algorithms, 1497--1514
work page 2014
-
[5]
Cesa-Bianchi, N.; Freund, Y.; Haussler, D.; Helmbold, D. P.; Schapire, R. E.; and Warmuth, M. K. 1997. How to use expert advice. Journal of the ACM, 44(3): 427--485
work page 1997
-
[6]
Cesa-Bianchi, N.; and Lugosi, G. 2012. Combinatorial bandits. Journal of Computer and System Sciences, 78(5): 1404--1422
work page 2012
-
[7]
Das, A.; Dasgupta, A.; and Kumar, R. 2012. Selecting diverse features via spectral regularization. In Advances in Neural Information Processing Systems 25, 3026--3034
work page 2012
-
[8]
Defazio, A.; and Caetano, T. S. 2012. A convex formulation for learning scale-free networks via submodular relaxation. In Advances in Neural Information Processing Systems 25, 1250--1258
work page 2012
Show all 39 references
-
[9]
Edmonds, J. 1970. Submodular functions, matroids, and certain polyhedra. In Combinatorial structures and their appli- cations, 69--87
1970
-
[10]
El Halabi, M.; and Cevher, V. 2015. A totally unimodular view of structured sparsity. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, 223--231
2015
-
[11]
El Halabi, M.; and Jegelka, S. 2020. Optimal approximation for unconstrained non-submodular minimization. In Proceedings of the 37th International Conference on Machine Learning, 3961--3972
2020
-
[12]
D.; Kalai, A
Flaxman, A. D.; Kalai, A. T.; and McMahan, H. B. 2005. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms, 385--394
2005
-
[13]
Garber, D.; and Kretzu, B. 2020. Improved regret bounds for projection-free bandit convex optimization. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, 2196--2206
2020
-
[14]
Gomez-Rodriguez, M.; and Sch \"o lkopf, B. 2012. Submodular inference of diffusion networks from multiple trees. In Proceedings of the 29th International Conference on Machine Learning, 489--496
2012
-
[15]
Gonz \'a lez, J.; Dai, Z.; Hennig, P.; and Lawrence, N. 2016. Batch bayesian optimization via local penalization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, 648--657
2016
-
[16]
Hazan, E.; and Kale, S. 2012. Online submodular minimization. Journal of Machine Learning Research, 13(93): 2903--2922
2012
-
[17]
Hazan, E.; et al. 2016. Introduction to online convex optimization. Foundations and Trends in Optimization, 2(3-4): 157--325
2016
-
[18]
H \'e liou, A.; Mertikopoulos, P.; and Zhou, Z. 2020. Gradient-free online learning in continuous games with delayed rewards. In Proceedings of the 37th International Conference on Machine Learning, 4172--4181
2020
-
[19]
Krause, A.; and Golovin, D. 2014. Submodular function maximization. Tractability, 3(71-104): 3
2014
-
[20]
E.; Tran-Dinh, Q.; and Cevher, V
Kyrillidis, A.; Baldassarre, L.; Halabi, M. E.; Tran-Dinh, Q.; and Cevher, V. 2015. Structured sparsity: Discrete and convex approaches. In Compressed Sensing and its Applications: MATHEON Workshop 2013, 341--387
2015
-
[21]
Lehmann, B.; Lehmann, D.; and Nisan, N. 2006. Combinatorial auctions with decreasing marginal utilities. Games and Economic Behavior, 55(2): 270--296
2006
-
[22]
Liao, Y.; Wan, Y.; Yao, C.; and Song, M. 2023. Improved Projection-free Online Continuous Submodular Maximization. arXiv preprint arXiv:2305.18442
2023 arXiv
-
[23]
Lin, T.; Pacchiano, A.; Yu, Y.; and Jordan, M. 2022. Online nonsubmodular minimization with delayed costs: from full information to bandit feedback. In Proceedings of the 39th International Conference on Machine Learning, 13441--13467
2022
-
[24]
Lov \'a sz, L. 1983. Submodular functions and convexity. Mathematical Programming The State of the Art: Bonn 1982, 235--257
1983
-
[25]
Quanrud, K.; and Khashabi, D. 2015. Online learning with adversarial delays. In Advances in Neural Information Processing Systems 28, 1270--1278
2015
-
[26]
Rapaport, F.; Barillot, E.; and Vert, J.-P. 2008. Classification of arrayCGH data using fused SVM. Bioinformatics, 24(13): i375--i382
2008
-
[27]
Sakaue, S. 2019. Greedy and iht algorithms for non-convex optimization with monotone costs of non-zeros. In The 22nd International Conference on Artificial Intelligence and Statistics, 206--215
2019
-
[28]
Shalev-Shwartz. 2012. Online learning and online convex optimization. Foundations and Trends in Machine Learning, 4(2): 107--194
2012
-
[29]
O.; Girshick, R.; Jegelka, S.; Mairal, J.; Harchaoui, Z.; and Darrell, T
Song, H. O.; Girshick, R.; Jegelka, S.; Mairal, J.; Harchaoui, Z.; and Darrell, T. 2014. On learning to localize objects with minimal supervision. In Proceedings of the 31th International Conference on Machine Learning, 1611--1619
2014
-
[30]
Sviridenko, M.; Vondr \'a k, J.; and Ward, J. 2017. Optimal approximation for submodular and supermodular optimization with bounded curvature. Mathematics of Operations Research, 42(4): 1197--1218
2017
-
[31]
Vondr \'a k, J. 2007. Submodularity in combinatorial optimization. Springer
2007
-
[32]
Wan, Y.; Wang, Y.; Yao, C.; Tu, W.-W.; and Zhang, L. 2022. Projection-free Online Learning with Arbitrary Delays. arXiv preprint arXiv:2204.04964
2022 arXiv
-
[33]
Wan, Y.; Yao, C.; Song, M.; and Zhang, L. 2023. Non-stationary online convex optimization with arbitrary delays. arXiv preprint arXiv:2305.12131
2023
-
[34]
Wan, Y.; Yao, C.; Song, M.; and Zhang, L. 2024. Improved regret for bandit convex optimization with delayed feedback. arXiv preprint arXiv:2402.09152
2024 arXiv
-
[35]
Wang, Y.; Wan, Y.; Zhang, S.; and Zhang, L. 2023. Distributed Projection-Free Online Learning for Smooth and Convex Losses. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, 10226--10234
2023
-
[36]
Wang, Y.; Yang, W.; Jiang, W.; Lu, S.; Wang, B.; Tang, H.; Wan, Y.; and Zhang, L. 2024. Non-stationary Projection-Free Online Learning with Dynamic and Adaptive Regret Guarantees. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, 15671--15679
2024
-
[37]
Zhang, M.; Chen, L.; Hassani, H.; and Karbasi, A. 2019. Online continuous submodular maximization: From full-information to bandit feedback. In Advances in Neural Information Processing Systems 32, 9210--9221
2019
-
[38]
Zheng, J.; Jiang, Z.; Chellappa, R.; and Phillips, J. P. 2014. Submodular attribute selection for action recognition in video. In Advances in Neural Information Processing Systems 27, 481--489
2014
-
[39]
Zinkevich, M. 2003. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning, 928--936
2003
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.