REVIEW 4 major objections 4 minor 35 references
EVO claims that a globally searched cache schedule over the block-timestep lattice can cut diffusion-policy action-generation FLOPs from 15.77G to as low as 1.96G—up to 8.05x speedup—while keeping task success near the uncached policy.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 10:13 UTC pith:ZACJVEXA
load-bearing objection EVO has a genuinely new global cache-scheduling formulation and the ablations support it, but the 8x speedup is a FLOPs ratio (wall-clock is ~3.4x) and the near-full performance claim rests on a single-seed search with an unreported acceptance threshold. the 4 major comments →
Evolving Cache Schedules for Fast Diffusion Policy Inference
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that heterogeneous redundancy across a diffusion policy's transformer blocks makes uniform or per-block cache budgets suboptimal, and that a globally budgeted cache schedule—a subset of the block-timestep lattice of fixed size K—can be found by evolutionary search using rollout success rate as the fitness. The discovered schedule, evaluated in the paper on multiple manipulation benchmarks, preserves near-full success (average 0.78–0.99 across datasets vs 0.79–0.99 for the full model) while reducing FLOPs from 15.77G to about 1.96–2.34G and raising policy inference frequency from 2.23 Hz to 7.57 Hz, without retraining or modifying the diffusion sampler.
What carries the argument
The load-bearing object is the cache schedule: a set S of K positions in the B×T lattice formed by the B cacheable residual computations (self-attention, cross-attention, FFN per layer) and T denoising steps. At inference, positions in S run the original residual branch and refresh a per-module cache; positions outside S reuse the most recent cached residual. EVO's evolutionary search—tournament selection, set-level crossover, mutation, repair, and elitism—treats each schedule as a fixed-size individual and scores it by empirical rollout success, so the budget can move freely from redundant to sensitive modules. Redundancy-aware initialization samples part of the initial population from a di
Load-bearing premise
The single search seed per task, combined with 20-episode quick rollouts as the screening signal, is the load-bearing premise: if that one search trajectory overfits to the quick-evaluation seed or to the simulation distribution, the reported near-full performance on held-out episodes may not replicate.
What would settle it
Take one task from the paper's set and run the EVO search from multiple independent search seeds (e.g., 10) with the same budget and final evaluation protocol; if the mean final success rate across those seeds falls more than the allowed δ_acc below the full model, the claim that EVO reliably finds near-full-performance schedules is refuted. A complementary check: compare the best found schedule against a randomly sampled schedule of the same budget under identical 50-episode evaluation; if random schedules match its success, rollout-driven search is not the source of the result.
If this is right
- If EVO's central claim holds, training-free cache acceleration of diffusion policies can be treated as a global combinatorial search problem rather than a hand-designed refresh rule, and the same search machinery should apply to any transformer-based diffusion policy without retraining.
- A frozen per-task schedule found offline can be deployed directly in the inference loop, so the one-time search cost is amortized over every action query; the paper reports the schedule is reused across episodes from the same environment distribution without online adaptation.
- Allowing the refresh budget to move across blocks improves both performance and search efficiency: the paper's ablation shows global block-timestep allocation raises average success from 0.87 to 0.91 while cutting search cost from 1.48 to 1.00 GPU-days.
- Feature similarity, while a useful starting prior, is too weakly correlated with task sensitivity to serve as the scheduling objective; EVO's results imply rollout-based fitness is needed to find schedules that actually preserve closed-loop behaviour.
- The up-to-8.05x speedup and 1.96G FLOPs at M=8 are measured with 100 denoising steps on the evaluated benchmarks; the same budget settings (M=8, M=10) correspond to fixed total refresh positions K=192 and K=240.
Where Pith is reading between the lines
- Because the search is seeded from activations of the uncached policy on the target task's demonstrations, the found schedules are likely task- and distribution-specific; a natural extension is to test whether a schedule found on one task transfers to a different task or to a real robot without re-searching.
- The weak similarity-sensitivity correlation reported in the paper suggests a cheaper alternative—random schedule search with the same rollout budget—might produce comparable schedules; comparing EVO against such a baseline would clarify how much of the gain comes from the evolutionary search itself versus from global allocation.
- The method's fitness is simulator rollouts; before real-world deployment, robustness to sim-to-real gaps would need checking, since the frozen schedule is optimized under simulation dynamics and a single search seed.
- Since the schedule space grows as binomial(BT, K), the evolutionary approach could also be applied to other discrete inference-optimization problems—e.g., where to skip layers or tokens in large multimodal models—where the bottleneck is the same: a non-differentiable objective defined by downstream performance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. EVO formulates cache-based acceleration of transformer-based diffusion policies as a fixed-budget subset-selection problem over the block–timestep lattice, and solves it with an evolutionary algorithm that uses closed-loop rollout success as fitness. To reduce search cost, EVO seeds part of the population using activation-dissimilarity priors and stops early once a candidate passes a quick screen and then a formal acceptance test. The paper reports that on RoboMimic, Push-T, Block Push, and Kitchen benchmarks, EVO keeps success rates close to the full policy while reducing FLOPs from 15.77G to as low as 1.96G, corresponding to a nominal 8.05x FLOPs reduction, and it compares favorably to EfficientVLA and BAC. The authors also provide wall-clock measurements in the appendix showing a mean inference-time reduction from 448.91 ms to 132.17 ms.
Significance. The core idea—using black-box evolutionary search to allocate a global cache-refresh budget across blocks and denoising steps—is a reasonable and potentially useful extension of per-block caching methods such as BAC. The paper ships source code, and the ablation in Table 4 gives some evidence that global budget allocation and redundancy-aware initialization each contribute to performance and reduce search cost. The motivation analysis in Fig. 1/Fig. 3 is also a useful empirical observation. If the search reliably produces schedules that preserve near-full policy success at one-eighth of the FLOPs, this would be a practical training-free acceleration result. However, the current evidence is materially weakened by (i) a stopping rule that accepts the first schedule clearing a noisy, target-conditioned threshold with an unreported tolerance, (ii) only one search seed per task, and (iii) a headline 'speedup' that conflates FLOPs reduction with wall-clock time. These issues are fixable but are load-bearing for the central claim.
major comments (4)
- [Abstract and Sec. 4.2 / Appendix A.7] The abstract and main text describe 'up to 8.05x action-generation speedup', but the measured wall-clock speedup in Table 6 of Appendix A.7 is 448.91/132.17 ≈ 3.40x for EVO (M=8). The value 8.05 is the FLOPs ratio 15.77/1.96, not an action-generation speedup. The Speed× columns in Tables 1–3 appear to be FLOPs ratios, not wall-clock measurements. Please separate FLOPs reduction from wall-clock speedup and adjust the abstract accordingly; the current wording is misleading.
- [Sec. 3.3, Eq. (9); Sec. 4.1 and Appendix A.1] The acceptance rule accepts only schedules with q_formal(S) ≥ q_base − δ_acc, so 'near-full performance' is partly enforced by the selection criterion rather than independently demonstrated. δ_acc is never reported. In addition, Appendix A.1 states that each task is searched with one search seed. With 50-episode formal evaluations (standard error roughly 0.07 for p≈0.5) and first-pass acceptance, the selected schedule may be a lucky draw whose true success is below the target. Please report δ_acc per task, run the search with multiple seeds (e.g., 5–10) per task, and report the distribution of final success rates over search seeds, including q_base and the accepted q_formal value. Without this, the reliability claim is not established.
- [Sec. 4.3, Table 4] The text states that 'the selected schedule is robust to different initializations', but no experiment varies the search initialization or search seed; every task is searched once. The 'Search Steps to Target' metric is introduced as an average without specifying what it is averaged over. Please either supply multi-seed search results supporting robustness or soften this claim, and clarify the averaging for Search Steps.
- [Sec. 3.3, Eq. (7) and Sec. 4.1] The hyperparameters ε and γ in the initialization sampling distribution and δ_acc in the acceptance rule are never specified. These directly affect the search behavior and the acceptance threshold, so the experiments are not reproducible from the text alone. Please provide concrete values (or a sensitivity study if they are not critical).
minor comments (4)
- [Sec. 4.3] Typo: 'Specificly' should be 'Specifically'.
- [Tables 1–3] The column 'AVG' is inconsistently formatted (sometimes with a space) and the caption says 'speedups are measured with full DP-T inference', which is ambiguous given the FLOPs-vs-wall-clock issue above.
- [Sec. 4.3 / Table 4] The metric 'GPU-days' is attributed to reference [2], which is a differentiable architecture search paper; please cite or define the GPU-days convention directly.
- [Appendix A.5] The full schedule tables (Tables 3–5) are very dense. Consider reporting aggregate statistics (e.g., per-block refresh counts, step histograms) in the main paper and moving the complete schedules to a repository, while keeping one representative schedule as an example.
Circularity Check
No significant circularity: EVO's performance is an empirical search result, not a derivation from its inputs.
full rationale
EVO is an empirical search/optimization paper, not a derivation from first principles. The central claim is that an evolutionary search over the block–timestep lattice, using rollout success as fitness, finds cache schedules that maintain task success under a fixed budget. The fitness (Eq. 4) and optimization objective (Eq. 5) define what is being optimized; Tables 1–3 report the measured success of the selected schedule on held-out episodes. No theoretical quantity is derived from the inputs. The early-stopping rule (Eq. 9) is a selection criterion: it stops search when a schedule passes formal verification at q_base − δ_acc. This is a target for search, not a computation of the reported result; the final evaluation is an independent measurement of the accepted schedule, so the claim is not identical to the stopping rule by construction. Redundancy-aware initialization uses feature dissimilarity only as a soft prior, and the paper explicitly tests and rejects similarity as the final objective (Fig. 3b), so no 'prediction' is hidden in the initialization. The self-citations (BAC [9], Sparse ActionGen [10]) are used as baselines/related work, not as load-bearing justification for EVO's claims. The main limitations—single search seed per task, unreported δ_acc, and possible selection bias from early stopping—are reproducibility/correctness concerns, not circularity. Hence no significant circularity.
Axiom & Free-Parameter Ledger
free parameters (6)
- cache budget M =
8 or 10 refreshes per module (K=192/240)
- epsilon in Eq. (7) =
unspecified
- gamma in Eq. (7) =
unspecified
- delta_acc in Eq. (9) =
unspecified
- quick/formal rollout budgets =
20/50 episodes
- evolutionary hyperparameters =
population 30, elites 10, generations 30
axioms (5)
- domain assumption Residual-cache reuse approximation (Eq. 1)
- domain assumption Temporal redundancy implies reusability
- domain assumption Rollout success rate is a stable fitness signal
- domain assumption Single-search-seed generalization
- domain assumption Simulation fidelity
read the original abstract
Diffusion policies achieve strong visuomotor control by iteratively denoising action chunks, but repeated denoising makes real-time deployment computationally demanding. Cache-based methods reduce inference cost by reusing intermediate activations, but existing training-free schedules typically allocate computation uniformly across blocks, ignoring heterogeneous redundancy across blocks and leading to a suboptimal performance-efficiency trade-off. To bridge this gap, we introduce Evolving Cache Schedules (EVO), a training-free acceleration framework that globally schedules cache refreshes via evolutionary search. EVO represents each candidate as a complete schedule over the block-timestep lattice. Thus, redundant transformer computations during iterative denoising can be skipped through cache reuse while preserving closed-loop rollout performance. To make the search practical, EVO introduces redundancy-aware initialization, which seeds the population with promising schedules, and target-conditioned early stopping, which verifies and terminates once a desired performance target is reached. The offline-optimized schedule can be directly plugged into pretrained diffusion policies without retraining. Extensive manipulation benchmarks show that EVO preserves near-full performance while substantially reducing computation, achieving up to 8.05x action-generation speedup and reducing FLOPs from 15.77G to as low as 1.96G. Source code is available at https://github.com/pillom/EVO.
Figures
Reference graph
Works this paper leans on
-
[1]
Robotics: Science and Systems XIX (2023) 14 S
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakr- ishnan, K., Hausman, K., Herzog, A., Hsu, J., et al.: Rt-1: Robotics transformer for real-world control at scale. Robotics: Science and Systems XIX (2023) 14 S. Wang et al
2023
-
[2]
In: Proceedings of the IEEE/CVF international conference on computer vision
Chen, X., Xie, L., Wu, J., Tian, Q.: Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1294–1303 (2019)
2019
-
[3]
The International Journal of Robotics Research44(10-11), 1684–1704 (2025)
Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., Song, S.: Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research44(10-11), 1684–1704 (2025)
2025
-
[4]
In: 2025 International Joint Conference on Neural Networks (IJCNN)
Guo, J., Zheng, X., Chen, Y., Zhang, X.: Neural architecture search-based meta- reinforcement learning in robotic simulation environments. In: 2025 International Joint Conference on Neural Networks (IJCNN). pp. 1–10. IEEE (2025)
2025
-
[5]
In: 2023 IEEE International Conference on Robotics and Automa- tion (ICRA)
Hegde, S., Sukhatme, G.S.: Efficiently learning small policies for locomotion and manipulation. In: 2023 IEEE International Conference on Robotics and Automa- tion (ICRA). pp. 5909–5915. IEEE (2023)
2023
-
[6]
Advances in neural information processing systems33, 6840–6851 (2020)
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020)
2020
-
[7]
MIT Press (1992)
Holland, J.H.: Adaptation in natural and artificial systems: an introductory anal- ysis with applications to biology, control, and artificial intelligence. MIT Press (1992)
1992
-
[8]
In: International Conference on Machine Learning
Janner, M., Du, Y., Tenenbaum, J., Levine, S.: Planning with diffusion for flexible behavior synthesis. In: International Conference on Machine Learning. pp. 9902–
-
[9]
In: The Fourteenth International Conference on Learning Representations (2026)
Ji, K., Meng, Y., Cui, H., Li, Y., Zhou, J., Hua, S., Chen, L., Wang, Z.: Block-wise adaptive caching for accelerating diffusion policy. In: The Fourteenth International Conference on Learning Representations (2026)
2026
-
[10]
arXiv preprint arXiv:2601.12894 (2026)
Ji, K., Meng, Y., Jianbo, Z., Li, Y., Cui, H., Wang, Z.: Sparse actiongen: Accel- erating diffusion policy with real-time pruning. arXiv preprint arXiv:2601.12894 (2026)
Pith/arXiv arXiv 2026
-
[11]
Transactions on Machine Learning Research (2025)
Liu, H., Zhang, W., Xie, J., Faccio, F., Xu, M., Xiang, T., Shou, M.Z., Perez-Rua, J.M., Schmidhuber, J.: Faster diffusion through temporal attention decomposition. Transactions on Machine Learning Research (2025)
2025
-
[12]
Advances in neural information processing systems35, 5775–5787 (2022)
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., Zhu, J.: Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in neural information processing systems35, 5775–5787 (2022)
2022
-
[13]
Machine Intelligence Research 22(4), 730–751 (2025)
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., Zhu, J.: Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research 22(4), 730–751 (2025)
2025
-
[14]
In: International Conference on Learning Representations
Lyu, Z., Si, C., Song, J., Yang, Z., Qiao, Y., Liu, Z., Wong, K.Y.K.: Fastercache: Training-free video diffusion model acceleration with high quality. In: International Conference on Learning Representations. vol. 2025, pp. 33132–33156 (2025)
2025
-
[15]
Advances in Neural Information Processing Systems 37, 133282–133304 (2024)
Ma, X., Fang, G., Bi Mi, M., Wang, X.: Learning-to-cache: Accelerating diffusion transformer via layer caching. Advances in Neural Information Processing Systems 37, 133282–133304 (2024)
2024
-
[16]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition
Ma, X., Fang, G., Wang, X.: Deepcache: Accelerating diffusion models for free. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 15762–15772 (2024)
2024
-
[17]
In: Proceedings of the 5th Conference on Robot Learning
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., Mart ´ ın-Mart ´ ın, R.: What matters in learning from offline hu- man demonstrations for robot manipulation. In: Proceedings of the 5th Conference on Robot Learning. pp. 1678–1690 (2022)
2022
-
[18]
In: Proceedings of the IEEE/CVF international conference on computer vision
Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4195–4205 (2023) Evolving Cache Schedules for Fast Diffusion Policy Inference 15
2023
-
[19]
In: Proceedings of the AAAI Conference on Artificial Intelligence
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image clas- sifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 4780–4789 (2019)
2019
-
[20]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
2022
-
[21]
In: International Conference on Learning Representations (2022)
Salimans, T., Ho, J.: Progressive distillation for fast sampling of diffusion models. In: International Conference on Learning Representations (2022)
2022
-
[22]
arXiv preprint arXiv:2407.01425 (2024)
Selvaraju, P., Ding, T., Chen, T., Zharkov, I., Liang, L.: Fora: Fast-forward caching in diffusion transformer acceleration. arXiv preprint arXiv:2407.01425 (2024)
Pith/arXiv arXiv 2024
-
[23]
Evolutionary Computation 23(3), 421–449 (2015)
Silva, F., Urbano, P., Correia, L., Christensen, A.L.: odneat: An algorithm for decentralised online evolution of robotic controllers. Evolutionary Computation 23(3), 421–449 (2015)
2015
-
[24]
In: The Fourteenth International Conference on Learning Representations (2026)
Son, B., Jeon, J., Choi, J., Ham, B.: Relational feature caching for accelerating diffusion transformers. In: The Fourteenth International Conference on Learning Representations (2026)
2026
-
[25]
In: International Conference on Learning Representations (2021)
Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: International Conference on Learning Representations (2021)
2021
-
[26]
In: Proceed- ings of the 40th International Conference on Machine Learning
Song, Y., Dhariwal, P., Chen, M., Sutskever, I.: Consistency models. In: Proceed- ings of the 40th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 202, pp. 32211–32252. PMLR (23–29 Jul 2023)
2023
-
[27]
Evolutionary computation10(2), 99–127 (2002)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolutionary computation10(2), 99–127 (2002)
2002
-
[28]
In: International Conference on Machine Learning
Wang, Z., Li, M., Mandlekar, A., Xu, Z., Fan, J., Narang, Y., Fan, L., Zhu, Y., Balaji, Y., Zhou, M., et al.: One-step diffusion policy: Fast visuomotor policies via diffusion distillation. In: International Conference on Machine Learning. pp. 63399–63416. PMLR (2025)
2025
-
[29]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Wimbauer, F., Wu, B., Schoenfeld, E., Dai, X., Hou, J., He, Z., Sanakoyeu, A., Zhang, P., Tsai, S., Kohler, J., et al.: Cache me if you can: Accelerating diffusion models through block caching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6211–6220 (2024)
2024
-
[30]
Advances in Neural Information Processing Systems38, 40891– 40914 (2026)
Yang, Y., Wang, Y., Wen, Z., Zhongwei, L., Zou, C., Zhang, Z., Wen, C., Zhang, L.: Efficientvla: Training-free acceleration and compression for vision-language- action models. Advances in Neural Information Processing Systems38, 40891– 40914 (2026)
2026
-
[31]
Inf Tech J3(1), 301 (1981)
Zames, G.: Genetic algorithms in search, optimization and machine learning. Inf Tech J3(1), 301 (1981)
1981
-
[32]
In: Pro- ceedings of Robotics: Science and Systems
Ze, Y., Zhang, G., Zhang, K., Hu, C., Wang, M., Xu, H.: 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations. In: Pro- ceedings of Robotics: Science and Systems. Delft, Netherlands (July 2024)
2024
-
[33]
Advances in Neural Information Processing Systems36, 49842–49869 (2023)
Zhao, W., Bai, L., Rao, Y., Zhou, J., Lu, J.: Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems36, 49842–49869 (2023)
2023
-
[34]
In: International Conference on Learning Representations
Zhao, X., Jin, X., Wang, K., You, Y.: Real-time video generation with pyramid attention broadcast. In: International Conference on Learning Representations. vol. 2025, pp. 3296–3319 (2025)
2025
-
[35]
Zou, C., Liu, X., Liu, T., Huang, S., Zhang, L.: Accelerating diffusion transformers with token-wise feature caching. In: The Thirteenth International Conference on Learning Representations (2025) 1 A Supplemental Material In this supplemental material, we provide additional experimental details and analyses to support the main conclusions of the paper. I...
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.