REVIEW 5 major objections 5 minor 30 references
HEPPO-GAE: Hardware-Efficient Proximal Policy Optimization with Generalized Advantage Estimation
T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read An FPGA-based accelerator for the advantage-estimation stage of Proximal Policy Optimization claims to cut PPO training time by about 30% while reducing memory use fourfold and improving cumulative rewards by 1.5x.
desk verdict A genuinely useful pipelining trick and a reward standardization scheme, but the headline speedup numbers are estimates against strawman baselines. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The k-step lookahead decomposition of the GAE recurrence: $\hat{A}_t = C^k \hat{A}_{t+k} + \sum_{i=0}^{k-1} C^{(k-1)-i}\delta_{t+i}$, with $C=\gamma\lambda$. Inserting registers into the feedback loop converts a sequential dependency that would otherwise stall the pipeline into a fully pipelined computation, letting the design hit 300 MHz. Around it, dual-port BRAM stacks act as a first-in-last-out memory that feeds 64 rewards and 64 values per cycle, and dynamic standardization updates running mean and variance via Welford's recurrence so the 8-bit quantizer always sees a roughly unit-variance distribution.
What would settle it
Measure the same GAE workload with a vectorized PyTorch implementation that processes all trajectories as one padded tensor on the same 32-core Xeon plus V100 machine, and compare elements per second to the about 9000/s cited from reference [17]; if that vectorized baseline is, say, 100x faster, the claimed 2-million-fold GAE speedup collapses to a more modest figure. Then, to test the hardware claim, synthesize the 64-PE design on the ZCU106, run trajectories of unequal length, and count clock cycles with the write-back handshake; if cycles per element rise above 1 per PE due to unequal-length pipeline drains, the 300 MHz zero-bubble assumption is not sustained in practice.
Extended reading notes
Core claim
The central claim is that HEPPO-GAE, an FPGA-based accelerator on a Zynq UltraScale+ SoC, can compute GAE and rewards-to-go for 64 trajectories in parallel at 300 MHz with fully pipelined processing elements, using a 2-step lookahead transform of the recurrence $\hat{A}_t = \delta_t + (\gamma\lambda)\hat{A}_{t+1}$ to break the feedback loop so the multiplier can be pipelined through DSP blocks. Combined with a FILO block-RAM layout that overwrites rewards and values with advantages and rewards-to-go, and with dynamic reward standardization plus block standardization of values followed by 8-bit uniform quantization, the paper claims a 4x memory reduction, a 1.5x increase in cumulative rewards on MuJoCo and Atari, and a theoretical GAE throughput of 300 million elements per second per processing element — roughly two million times the about 9000 elements per second measured for a CPU-GPU reference implementation.
Load-bearing premise
The headline numbers depend on the CPU-GPU baseline being representative (it is a single Medium-article implementation) and on the FPGA sustaining 300 MHz with no pipeline stalls while feeding 64 rewards and 64 values from BRAM each cycle; the reward gain also depends on treating 'original PPO' without advantage normalization as a fair comparison, even though the paper itself notes normalization is standard practice.
Editorial extensions
If this is right
- If the measured 30% PPO speedup is representative, GAE ceases to be a roughly 30% training-time cost and becomes a negligible stage.
- The 4x memory reduction from quantized on-chip storage means larger batch sizes or longer horizons can fit in the same BRAM budget, which directly helps the memory-bound regime described in Section IV.
- Because the standardization is parameter-free and environment-independent, the reward improvement should transfer across MuJoCo, Atari, and similar RL benchmarks without per-task tuning.
- The design's resource footprint (under 10% of LUTs and FFs and about 30% of DSPs on the ZCU106) leaves room for other PPO phases on the same chip, so a full single-chip PPO is structurally feasible.
- If keeping rewards in standardized form is the key to the 1.5x cumulative-reward gain, then future RL hardware can treat reward preprocessing as a first-class design component rather than a host-side afterthought.
Reading between the lines
- If the claimed 300 MHz, zero-bubble operation is achieved only for equal-length trajectories, the real speedup on variable-length episodes could be much lower; a natural test is to measure the design on trajectories of unequal length, which is exactly the case the CPU-GPU baseline is slower on.
- The k-step lookahead idea is not PPO-specific: any sequential RL update that follows a linear recurrence with a constant discount factor (for instance, other TD(lambda)-style estimators) can use the same transform, so the accelerator could generalize to other policy-gradient methods.
- The reference baseline (~9000 elements/s from a Medium-article PyTorch implementation) is likely far slower than a vectorized batch GAE; if so, the 2-million-fold figure is an artifact of the baseline, not of the hardware, and the true speedup over a well-optimized GPU implementation would be much smaller but still possibly substantial on a single-chip platform.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HEPPO-GAE, an FPGA-based accelerator for the Generalized Advantage Estimation (GAE) stage of Proximal Policy Optimization (PPO), targeting a single AMD-Xilinx Zynq UltraScale+ SoC. The contributions are a dynamic reward standardization scheme, a block standardization for values, 8-bit uniform quantization, a FILO dual-port BRAM memory layout, and a k-step lookahead pipeline that enables pipelined GAE computation. The authors claim a 4x memory reduction, a 1.5x increase in cumulative rewards, a 30% PPO training speedup, and roughly 2,000,000x faster GAE throughput relative to a CPU-GPU baseline. The paper also presents PPO time profiling and resource utilization estimates for the ZCU106 evaluation board.
Significance. The architectural ideas—particularly the k-step lookahead transformation for pipelining the GAE recurrence and the on-chip FILO data layout—are plausible and could be useful for hardware RL accelerators if validated. However, the headline quantitative claims are not supported by measurements against fair baselines. The GAE speedup and 30% PPO speedup rely on a weak baseline from a Medium tutorial, the FPGA throughput is an estimate rather than a measured result, and the reward improvement is compared against an unnormalized PPO baseline despite the paper acknowledging that advantage normalization is standard practice. Internal inconsistencies in the memory and bandwidth analysis further undermine the 4x memory reduction claim. If the claims were substantiated with proper benchmarking and hardware measurements, the work would be a meaningful contribution to single-SoC PPO training, but in its current form the evidence is insufficient.
major comments (5)
- [Section V.D.3] The claimed ~2,000,000x GAE speedup and the resulting 30% PPO speedup are computed against a baseline of about 9000 elements/s taken from a Medium tutorial [17], not from a standard vectorized PPO implementation. The paper itself notes that this baseline processes trajectories one at a time, whereas common RL libraries (e.g., stable-baselines3) vectorize GAE across trajectories in a batch. A fair baseline would likely be orders of magnitude faster, which would correspondingly reduce the claimed speedup and undermine the headline numbers in the abstract and contribution list.
- [Section V.D.1] The 300 MHz operation and 300M elements/s per PE are estimates, as stated in the text ('a single PE is estimated to handle 300 million elements per second') and in Table IV ('estimated for 64 PEs based on our single PE implementation'). No synthesis timing closure, placed-and-routed clock frequency, or end-to-end hardware measurement is reported. The central hardware throughput claim is therefore not empirically validated, and the projected 30% PPO speedup built on this estimate is not supported.
- [Section V.A / Figure 7] The 1.5x cumulative reward improvement is presented as a comparison against 'original PPO' without advantage normalization, even though Section V.A states that advantage normalization is 'widely adopted' in standard implementations [15,16]. Because the proposed modification includes standardization/normalization, the comparison conflates the effect of the proposed dynamic/block standardization with the effect of standard advantage normalization. Without including an advantage-normalized PPO baseline, the claim that the proposed scheme increases cumulative rewards is not established.
- [Section IV.A / Section V.D.2] The claimed 4x memory reduction is inconsistently supported. Section V.D.2 computes memory as 128 bytes per timestep for rewards and values overwritten by advantages and rewards-to-go, and the bandwidth analysis assumes 128 bytes read plus 128 bytes written per cycle, implying 32-bit storage for advantages and rewards-to-go. The abstract's 4x reduction appears to assume 8-bit storage for all four vectors, but the paper never explicitly states the storage width of advantages and rewards-to-go. This inconsistency affects both the memory reduction claim and the BRAM utilization and bandwidth numbers in Section V.D.2.
- [Section I.B / Table I] The time profiling identifying GAE as about 30% of PPO time in CPU-GPU systems is used in Section V.D.3 to justify the '30% PPO speedup' claim. However, the profiling is based on the same weak GAE baseline (Medium tutorial implementation) and is not accompanied by profiling of a vectorized baseline implementation. If GAE is a much smaller fraction of total PPO time in a state-of-the-art implementation, the projected 30% speedup is not credible.
minor comments (5)
- [Section III.B] The text contains a typo: 'revere order' should be 'reverse order'.
- [Equation (6)] The notation 'RunningMeann' mixes a subscript with an appended 'n' in an inconsistent way; using a consistent subscript or superscript notation would improve clarity.
- [Section V.D.2] The bandwidth calculation states that 57 BRAM ports are needed and then says 'this translates to 32 BRAM blocks'. Since each dual-port BRAM provides two ports, 57 ports would require 29 BRAM blocks; the rounding up to 32 is not explained.
- [Section III.B / Figure 4] The text says the 3-step lookahead registers are 'highlighted in yellow', but the Figure 4 caption says 'highlighted in orange'; these should be made consistent.
- [Section II.C.1] The statement that leaving rewards in standardized form 'enhances the cumulative rewards by around 50%' is loosely connected to the 'over 1.5x' claim in Section V.A; the relationship between these two figures should be clarified to avoid apparent inconsistency.
Circularity Check
No circularity found: the headline speedup and reward numbers are unsupported estimates and strawman-baseline comparisons, but no derivation reduces to its own input or to a load-bearing self-citation.
full rationale
The paper's central claims are quantitative estimates and empirical comparisons, not circular derivations. The ~2Mx GAE speedup is an arithmetic ratio between an assumed peak FPGA throughput (300 MHz per PE, 64 PEs) and a measured baseline from a Medium article implementation [17]; this is a weak or unrepresentative baseline and an unverified throughput assumption, but it is not a case where the output is defined in terms of the input. The 30% PPO speedup is simply the GAE share of PPO time taken from the paper's own profiling table, so it is arithmetic, not circular. The 4x memory reduction follows from explicit assumptions: 8-bit storage and in-place overwriting of rewards/values by advantages/rewards-to-go; this is a stated design calculation, not a result smuggled in as a prediction. The 1.5x reward improvement is an empirical comparison against a baseline that the paper itself admits omits widely used advantage normalization; that is a benchmarking/validity concern, not circularity. The k-step lookahead transformation is a mathematically exact algebraic rearrangement of the GAE recurrence (Eqs. 4, 10, 11), and the memory layout and FILO design are described directly. No load-bearing uniqueness theorem, ansatz smuggled in via citation, or self-citation dependence appears: references to the authors' own prior work occur only in the future-work/overclocking discussion (e.g., [18]–[24]) and are not needed to derive any headline result. Thus, while several quantitative claims are not firmly established, the paper's derivation chain does not reduce to its own inputs.
Assumptions & free parameters
free parameters (4)
- Quantization bit width =
8 bits
- Lookahead steps k =
2
- Trajectories and timesteps =
64 and 1024
- Clock frequency =
300 MHz
assumptions (4)
- domain assumption GAE computation consumes about 30% of PPO time in CPU-GPU systems
- domain assumption The CPU-GPU baseline GAE implementation is representative
- domain assumption Standard PPO without advantage normalization is a fair comparison baseline
- domain assumption BRAM can supply 256 bytes/cycle and the systolic array saturates it
Cite this review
Pith. "Pith review of HEPPO-GAE: Hardware-Efficient Proximal Policy Optimization with Generalized Advantage Estimation." pith.science (2026). https://pith.science/paper/ZBJB7JCC
@misc{pith2026250112703,
author = {Pith},
title = {Pith review of: HEPPO-GAE: Hardware-Efficient Proximal Policy Optimization with Generalized Advantage Estimation},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZBJB7JCC}},
note = {Machine review of arXiv:2501.12703}
}
read the original abstract
This paper introduces HEPPO-GAE, an FPGA-based accelerator designed to optimize the Generalized Advantage Estimation (GAE) stage in Proximal Policy Optimization (PPO). Unlike previous approaches that focused on trajectory collection and actor-critic updates, HEPPO-GAE addresses GAE's computational demands with a parallel, pipelined architecture implemented on a single System-on-Chip (SoC). This design allows for the adaptation of various hardware accelerators tailored for different PPO phases. A key innovation is our strategic standardization technique, which combines dynamic reward standardization and block standardization for values, followed by 8-bit uniform quantization. This method stabilizes learning, enhances performance, and manages memory bottlenecks, achieving a 4x reduction in memory usage and a 1.5x increase in cumulative rewards. We propose a solution on a single SoC device with programmable logic and embedded processors, delivering throughput orders of magnitude higher than traditional CPU-GPU systems. Our single-chip solution minimizes communication latency and throughput bottlenecks, significantly boosting PPO training efficiency. Experimental results show a 30% increase in PPO speed and a substantial reduction in memory access time, underscoring HEPPO-GAE's potential for broad applicability in hardware-efficient reinforcement learning algorithms.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[17]
E. Y . Yu. (2023) Coding PPO from Scratch with PyTorch (Part 4/4)[On- line]. Available: https://medium.com/@z4xia/4e21f4a63e5c. [Accessed: 08 April 2024]
work page 2023
-
[1]
Deep Reinforce- ment Learning for Robotic Manipulation—The State of the Art,
S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Deep Reinforce- ment Learning for Robotic Manipulation—The State of the Art,” IEEE Robotics and Automation Letters , vol. 4, no. 2, pp. 319–326, 2018
work page 2018
-
[4]
Trust Region Policy Optimization,
J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust Region Policy Optimization,” in Proceedings of the 32nd International Conference on Machine Learning (ICML) , 2015, pp. 1889–1897
work page 2015
-
[5]
Available: http://arxiv.org/abs/1707.06347
[Online]. Available: http://arxiv.org/abs/1707.06347
-
[6]
High-Dimensional Continuous Control Using Generalized Advantage Estimation,
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High-Dimensional Continuous Control Using Generalized Advantage Estimation,” arXiv preprint arXiv:1506.02438 , 2015. [Online]. Available: https://doi.org/10.48550/arXiv.1506.02438
-
[7]
Adam: A Method for Stochastic Optimization,
D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in Proceedings of the 3rd International Conference on Learning Representations (ICLR) , San Diego, CA, USA, May 7-9 2015, conference Track Proceedings. [Online]. Available: http: //arxiv.org/abs/1412.6980
arXiv 2015
-
[8]
J. Yang, S. Hong, and J.-Y . Kim, “FIXAR: A Fixed-Point Deep Re- inforcement Learning Platform with Quantization-Aware Training and Adaptive Parallelism,” in Proceedings of the 58th ACM/IEEE Design Automation Conference (DAC), 2021, pp. 259–264
work page 2021
-
[9]
QuaRL: Quantization for Fast and Environmentally Sustainable Reinforcement Learning,
S. Krishnan, S. Chitlangia, M. Lam, Z. Wan, A. Faust, and V . J. Reddi, “QuaRL: Quantization for Fast and Environmentally Sustainable Reinforcement Learning,” arXiv preprint arXiv:1910.01055 , 2022. [Online]. Available: http://arxiv.org/abs/1910.01055
arXiv 1910
Show all 30 references
-
[10]
EnvPool: A Highly Parallel Reinforcement Learning En- vironment Execution Engine,
J. Weng et al., “EnvPool: A Highly Parallel Reinforcement Learning En- vironment Execution Engine,” in Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS) , New Orleans, LA, USA, 2022
2022
-
[11]
Accelerating Proximal Policy Optimization on CPU-FPGA Heterogeneous Platforms,
Y . Meng, S. Kuppannagari, and V . Prasanna, “Accelerating Proximal Policy Optimization on CPU-FPGA Heterogeneous Platforms,” in Pro- ceedings of the IEEE 28th Annual International Symposium on Field- Programmable Custom Computing Machines (FCCM), 2020, pp. 19–27
2020
-
[12]
GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning,
J. Liang, V . Makoviychuk, A. Handa, N. Chentanez, M. Macklin, and D. Fox, “GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning,” in Proceedings of the 2nd Conference on Robot Learning (CoRL) , Z ¨urich, Switzerland, 2018
2018
-
[13]
Accelerating Reinforcement Learning through GPU Atari Emulation,
S. Dalton and I. Frosio, “Accelerating Reinforcement Learning through GPU Atari Emulation,” in Proceedings of the 34th International Confer- ence on Neural Information Processing Systems (NeurIPS) , Vancouver, BC, Canada, 2020
2020
-
[14]
D. E. Knuth, The Art of Computer Programming, volume 2: Seminu- merical Algorithms, 3rd edn. Boston: Addison-Wesley, 1998
1998
-
[15]
Note on a Method for Calculating Corrected Sums of Squares and Products,
B. P. Welford, “Note on a Method for Calculating Corrected Sums of Squares and Products,” Technometrics, vol. 4, no. 3, pp. 419–420,
-
[16]
(2018) Understanding Normalization of Advantage Function in PPO [Online]
mbcel. (2018) Understanding Normalization of Advantage Function in PPO [Online]. Available: https://github.com/openai/baselines/issues/544. [Accessed: 08 April 2024]
2018
-
[18]
Z. Yang. (2021) Justifying Advantage Normalization for PPO [On- line]. Available: https://github.com/DLR-RM/stable-baselines3/issues/
2021
-
[19]
Revisiting Deep Learning Paral- lelism: Fine-Grained Inference Engine Utilizing Online Arithmetic,
A. M. Abdelhadi and L. Shannon, “Revisiting Deep Learning Paral- lelism: Fine-Grained Inference Engine Utilizing Online Arithmetic,” in Proceedings of the International Conference on Field-Programmable Technology (ICFPT), 2019, pp. 383–386
2019
-
[20]
Enabling Mixed-Timing NoCs for FPGAs: Reconfigurable Synthesizable Synchronization FIFOs,
A. M. S. Abdelhadi and H. Li, “Enabling Mixed-Timing NoCs for FPGAs: Reconfigurable Synthesizable Synchronization FIFOs,” in Pro- ceedings of the 31st International Conference on Field-Programmable Logic and Applications (FPL) , 2021, pp. 312–318
2021
-
[21]
Reconfigurable Synthesizable Syn- chronization FIFOs,
A. M. S. Abdelhadi and H. Li, “Reconfigurable Synthesizable Syn- chronization FIFOs,” in Proceedings of the IEEE 29th Annual Interna- tional Symposium on Field-Programmable Custom Computing Machines (FCCM), 2021, pp. 272–272
2021
-
[22]
Safe Overclocking of Tightly Coupled CGRAs and Processor Arrays using Razor,
A. Brant et al. , “Safe Overclocking of Tightly Coupled CGRAs and Processor Arrays using Razor,” in Proceedings of the IEEE 21st Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), 2013, pp. 37–44
2013
-
[23]
High-Throughput Synthesizable Synchronization FI- FOs for Mixed-Timing NoCs,
A. M. Abdelhadi, “High-Throughput Synthesizable Synchronization FI- FOs for Mixed-Timing NoCs,” in Proceedings of the 13th International Workshop on Network on Chip Architectures (NoCArc) . Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2020, pp. 1–6
2020
-
[24]
Interleaved Architectures for High-Throughput Synthesizable Synchronization FIFOs,
A. M. S. Abdelhadi and M. R. Greenstreet, “Interleaved Architectures for High-Throughput Synthesizable Synchronization FIFOs,” in Pro- ceedings of the 23rd IEEE International Symposium on Asynchronous Circuits and Systems (ASYNC) . Los Alamitos, CA, USA: IEEE Computer Society,...
2017
-
[25]
Atalanta: A Bit is Worth a “Thousand
A. D. Lascorz et al. , “Atalanta: A Bit is Worth a “Thousand” Tensor Values,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), New York, NY , USA, 2024, p. 85–102
2024
-
[26]
Synthesizable Synchronization FIFOs Utilizing the Asynchronous Pulse-Based Handshake Protocol,
A. M. Abdelhadi, “Synthesizable Synchronization FIFOs Utilizing the Asynchronous Pulse-Based Handshake Protocol,” in Proceedings of the IEEE Nordic Circuits and Systems Conference (NorCAS), 2020, pp. 1–7
2020
-
[27]
Boveda: Building an On-Chip Deep Learning Memory Hierarchy Brick by Brick,
I. Edo Vivancos et al. , “Boveda: Building an On-Chip Deep Learning Memory Hierarchy Brick by Brick,” in Proceedings of Machine Learning and Systems (MLSys) , vol. 3, 2021, pp. 1–
2021
-
[30]
Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models,
A. H. Zadeh et al. , “Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models,” in Proceedings of the 49th Annual International Symposium on Computer Architecture (ISCA), New York, NY , USA, 2022, p. 888–901
2022
-
[32]
Available: https://proceedings.mlsys.org/paper files/paper/ 2021/file/12a304a31e42dfefa21c82431e849124-Paper.pdf 9
[Online]. Available: https://proceedings.mlsys.org/paper files/paper/ 2021/file/12a304a31e42dfefa21c82431e849124-Paper.pdf 9
2021
-
[485]
[Accessed: 08 April 2024]
2024
-
[1962]
Available: https://www.jstor.org/stable/1266577
[Online]. Available: https://www.jstor.org/stable/1266577
-
[2017]
Available: http://arxiv.org/abs/1712.01815
[Online]. Available: http://arxiv.org/abs/1712.01815
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.