{"id":"cb698ac7-b80b-40cf-8299-a03df0ea854c","arxiv_id":"2501.12703","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"HEPPO-GAE is an FPGA design that pipelines GAE with 8-bit quantization and standardization, claiming 30% PPO speedup and 1.5x rewards, with speedups estimated rather than measured.","lead":"This paper proposes an FPGA-based accelerator for the advantage-estimation step of the PPO reinforcement learning algorithm. The authors claim faster training, lower memory use, and higher rewards, but the speedups come from estimates and the reward comparison uses a weak baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline speedup and reward gains rest on strawman baselines: a 9000 elements/s GAE reference and an unnormalized PPO, while FPGA throughput is only estimated; the 30% PPO speedup and 1.5x reward claims are unsupported.","rationale":"Good-faith reading: the paper proposes a real hardware architecture (k-step lookahead pipelining, FILO BRAM stack, systolic array of PEs) and includes a parameterized Verilog model with resource estimates; these are legitimate design artifacts. However, the central claims are quantitative and all of them depend on comparisons whose baselines are either unrepresentative or unmeasured. The most load-bearing assumption is that the ~9000 elements/s GAE baseline from reference [17] is a fair proxy for CPU-GPU GAE. This is contradicted by the existence of standard vectorized GAE implementations and by the paper's own profiling numbers, which make GAE 30% of PPO time partly because the baseline is so slow. The same weakness applies to the 1.5x reward claim: the paper states that advantage normalization is widely adopted, then compares against an 'original PPO' that apparently lacks it. I concur with the reader's REJECT verdict; no adjustment is needed. A revision with actual FPGA measurements (clock frequency, throughput, end-to-end PPO) and fair baselines (vectorized GAE, normalized PPO) could make the paper publishable, but as written the headline numbers are unsupported.","tokens_in":12505,"tokens_out":12008,"duration_ms":113121,"concrete_test":"Implement a vectorized GAE baseline (e.g., Stable-Baselines3 or a PyTorch tensor operation over shape [64, 1024] with an in-place backward loop) and time it on the same 32-core Xeon Silver 4216 + V100 setup used in Table I, using Humanoid trajectories. Also time the Medium reference [17]. If the vectorized baseline exceeds ~1M elements/s, then the 2M x speedup is an artifact of the chosen baseline and the estimated 30% PPO speedup cannot hold against a fair comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central hardware claim—~2M x GAE throughput and ~30% PPO speedup—rests on two unverified numbers. First, the CPU-GPU baseline of ~9000 GAE elements/s is taken from a Medium tutorial implementation [17], not from a standard vectorized library. GAE is a simple backward recurrence over a (trajectory, timestep) batch; a vectorized PyTorch implementation on the same V100 would process 64 trajectories x 1024 timesteps in milliseconds, not ~7s. If the fair baseline is even 100x faster, the claimed 2M x ratio becomes ~20,000x, and the 30% PPO speedup—which is arithmetically derived from their own profiling table where GAE is ~30% of PPO time using that same slow baseline—largely evaporates. Second, the FPGA side is an estimate: Section V.D.1 says a single PE 'is estimated to handle 300M elements/s' and Table IV is extrapolated from a single PE; no synthesis timing closure, measured clock, or end-to-end run is reported. The 4x memory reduction assumes advantages/RTGs are also stored 8-bit, which the paper never explicitly states, and the 1.5x reward gain compares against an 'original PPO' without the advantage normalization that the paper itself admits is standard practice [15,16]. The k-step lookahead and FILO BRAM design are plausible, but none of the headline quantitative claims are measured against fair baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HEPPO-GAE, an FPGA-based accelerator for the Generalized Advantage Estimation (GAE) stage of Proximal Policy Optimization (PPO), targeting a single AMD-Xilinx Zynq UltraScale+ SoC. The contributions are a dynamic reward standardization scheme, a block standardization for values, 8-bit uniform quantization, a FILO dual-port BRAM memory layout, and a k-step lookahead pipeline that enables pipelined GAE computation. The authors claim a 4x memory reduction, a 1.5x increase in cumulative rewards, a 30% PPO training speedup, and roughly 2,000,000x faster GAE throughput relative to a CPU-GPU baseline. The paper also presents PPO time profiling and resource utilization estimates for the ZCU106 evaluation board.","tokens_in":12800,"tokens_out":5126,"duration_ms":43278,"significance":"The architectural ideas—particularly the k-step lookahead transformation for pipelining the GAE recurrence and the on-chip FILO data layout—are plausible and could be useful for hardware RL accelerators if validated. However, the headline quantitative claims are not supported by measurements against fair baselines. The GAE speedup and 30% PPO speedup rely on a weak baseline from a Medium tutorial, the FPGA throughput is an estimate rather than a measured result, and the reward improvement is compared against an unnormalized PPO baseline despite the paper acknowledging that advantage normalization is standard practice. Internal inconsistencies in the memory and bandwidth analysis further undermine the 4x memory reduction claim. If the claims were substantiated with proper benchmarking and hardware measurements, the work would be a meaningful contribution to single-SoC PPO training, but in its current form the evidence is insufficient.","major_comments":[{"comment":"The claimed ~2,000,000x GAE speedup and the resulting 30% PPO speedup are computed against a baseline of about 9000 elements/s taken from a Medium tutorial [17], not from a standard vectorized PPO implementation. The paper itself notes that this baseline processes trajectories one at a time, whereas common RL libraries (e.g., stable-baselines3) vectorize GAE across trajectories in a batch. A fair baseline would likely be orders of magnitude faster, which would correspondingly reduce the claimed speedup and undermine the headline numbers in the abstract and contribution list.","section":"Section V.D.3"},{"comment":"The 300 MHz operation and 300M elements/s per PE are estimates, as stated in the text ('a single PE is estimated to handle 300 million elements per second') and in Table IV ('estimated for 64 PEs based on our single PE implementation'). No synthesis timing closure, placed-and-routed clock frequency, or end-to-end hardware measurement is reported. The central hardware throughput claim is therefore not empirically validated, and the projected 30% PPO speedup built on this estimate is not supported.","section":"Section V.D.1"},{"comment":"The 1.5x cumulative reward improvement is presented as a comparison against 'original PPO' without advantage normalization, even though Section V.A states that advantage normalization is 'widely adopted' in standard implementations [15,16]. Because the proposed modification includes standardization/normalization, the comparison conflates the effect of the proposed dynamic/block standardization with the effect of standard advantage normalization. Without including an advantage-normalized PPO baseline, the claim that the proposed scheme increases cumulative rewards is not established.","section":"Section V.A / Figure 7"},{"comment":"The claimed 4x memory reduction is inconsistently supported. Section V.D.2 computes memory as 128 bytes per timestep for rewards and values overwritten by advantages and rewards-to-go, and the bandwidth analysis assumes 128 bytes read plus 128 bytes written per cycle, implying 32-bit storage for advantages and rewards-to-go. The abstract's 4x reduction appears to assume 8-bit storage for all four vectors, but the paper never explicitly states the storage width of advantages and rewards-to-go. This inconsistency affects both the memory reduction claim and the BRAM utilization and bandwidth numbers in Section V.D.2.","section":"Section IV.A / Section V.D.2"},{"comment":"The time profiling identifying GAE as about 30% of PPO time in CPU-GPU systems is used in Section V.D.3 to justify the '30% PPO speedup' claim. However, the profiling is based on the same weak GAE baseline (Medium tutorial implementation) and is not accompanied by profiling of a vectorized baseline implementation. If GAE is a much smaller fraction of total PPO time in a state-of-the-art implementation, the projected 30% speedup is not credible.","section":"Section I.B / Table I"}],"minor_comments":[{"comment":"The text contains a typo: 'revere order' should be 'reverse order'.","section":"Section III.B"},{"comment":"The notation 'RunningMeann' mixes a subscript with an appended 'n' in an inconsistent way; using a consistent subscript or superscript notation would improve clarity.","section":"Equation (6)"},{"comment":"The bandwidth calculation states that 57 BRAM ports are needed and then says 'this translates to 32 BRAM blocks'. Since each dual-port BRAM provides two ports, 57 ports would require 29 BRAM blocks; the rounding up to 32 is not explained.","section":"Section V.D.2"},{"comment":"The text says the 3-step lookahead registers are 'highlighted in yellow', but the Figure 4 caption says 'highlighted in orange'; these should be made consistent.","section":"Section III.B / Figure 4"},{"comment":"The statement that leaving rewards in standardized form 'enhances the cumulative rewards by around 50%' is loosely connected to the 'over 1.5x' claim in Section V.A; the relationship between these two figures should be clarified to avoid apparent inconsistency.","section":"Section II.C.1"}],"recommendation":"reject","confidential_remarks":"The k-step lookahead pipeline and the FILO BRAM layout are interesting architectural ideas, but the experimental methodology does not support the headline claims. The reliance on a Medium article as the GAE baseline and the use of unnormalized PPO as the reward comparison baseline are serious weaknesses that would require new experiments to address. The paper would need actual synthesis results and benchmarking against a vectorized PPO implementation (e.g., stable-baselines3) to be considered for publication. I also note the unusual acknowledgment thanking reviewers; this is not a substantive issue but is out of place in a formal submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The k-step lookahead transformation for pipelining the GAE feedback loop is the real contribution here. It is a clean, sensible way to break a loop-carried dependency that would otherwise force pipeline bubbles, and I have not seen it applied to GAE before. The systolic array design with FILO BRAM is also plausible and the memory-layout reasoning is careful. The dynamic reward standardization using Welford's algorithm, combined with block standardization for values plus 8-bit quantization, appears to produce a consistent reward improvement in their own plots; the ablation across experiments (Exp 5 outperforming Exp 3/4) is informative and useful.\n\nWhere the paper falls short is in the headline quantitative claims. The ~2M x GAE throughput and the resulting ~30% PPO speedup rest on two unverified pillars. First, the CPU-GPU baseline of ~9000 elements/s comes from a Medium tutorial implementation (ref [17]), not from a standard vectorized library. GAE is a simple backward recurrence; a vectorized PyTorch implementation would almost certainly be hundreds to thousands of times faster, which would crush the claimed ratio. Second, the FPGA side is estimated, not measured: no synthesis timing closure, no measured clock, no end-to-end run. The paper itself says a single PE \"is estimated to handle 300M elements per second\" and Table IV is extrapolated from a single PE. That is not a measurement.\n\nThere are also smaller but real issues. The DSP utilization is reported as 17.7% in Section V.D.1 but Table IV shows 768/2520 = 30.48%; that inconsistency needs fixing. The 1.5x reward improvement compares against \"original PPO\" without advantage normalization, even though the paper itself notes normalization is widely adopted (refs [15,16])—so that baseline is a strawman. No seeds or error bars are reported, so the RL results are hard to evaluate.\n\nNone of this kills the core architectural idea. The k-step lookahead is worth publishing, and the standardization scheme deserves a fair test against a properly normalized baseline. But the current headline numbers overclaim what has actually been demonstrated. This is a revise-and-resubmit situation, not a desk reject. Send it to peer review with the expectation that the authors will need to provide measured FPGA throughput and a fair baseline comparison before the claims can stand.","headline":"A genuinely useful pipelining trick and a reward standardization scheme, but the headline speedup numbers are estimates against strawman baselines.","tokens_in":13361,"tokens_out":1927,"would_cite":false,"duration_ms":20724,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An FPGA-based accelerator for the advantage-estimation stage of Proximal Policy Optimization claims to cut PPO training time by about 30% while reducing memory use fourfold and improving cumulative rewards by 1.5x.","keywords":["FPGA accelerator","Proximal Policy Optimization","Generalized Advantage Estimation","8-bit quantization","dynamic standardization","systolic array","k-step lookahead","reinforcement learning hardware"],"falsifier":"Measure the same GAE workload with a vectorized PyTorch implementation that processes all trajectories as one padded tensor on the same 32-core Xeon plus V100 machine, and compare elements per second to the about 9000/s cited from reference [17]; if that vectorized baseline is, say, 100x faster, the claimed 2-million-fold GAE speedup collapses to a more modest figure. Then, to test the hardware claim, synthesize the 64-PE design on the ZCU106, run trajectories of unequal length, and count clock cycles with the write-back handshake; if cycles per element rise above 1 per PE due to unequal-length pipeline drains, the 300 MHz zero-bubble assumption is not sustained in practice.","tokens_in":12275,"feed_emoji":"⚡","tokens_out":5786,"duration_ms":49481,"temperature":0.7,"pith_summary":"This paper argues that the Generalized Advantage Estimation (GAE) stage, not environment simulation or network updates, is the part of Proximal Policy Optimization (PPO) most worth accelerating with custom hardware, and that a single System-on-Chip can host the whole PPO pipeline. To back that, it introduces an FPGA design whose pipelined processing elements compute advantages with a k-step lookahead trick that removes pipeline bubbles, plus a standardization and quantization scheme that shrinks memory traffic fourfold while improving training reward. If the design works as claimed, GAE ceases to be a bottleneck and PPO training time drops by about 30% compared with a CPU-GPU setup, at a fraction of the memory cost.","feed_headline":"30% faster PPO training from a single-chip GAE accelerator","feed_subtitle":"Custom FPGA pipeline, 8-bit quantization, and on-chip memory cut PPO's advantage bottleneck and lift rewards 1.5x.","key_machinery":"The k-step lookahead decomposition of the GAE recurrence: $\\hat{A}_t = C^k \\hat{A}_{t+k} + \\sum_{i=0}^{k-1} C^{(k-1)-i}\\delta_{t+i}$, with $C=\\gamma\\lambda$. Inserting registers into the feedback loop converts a sequential dependency that would otherwise stall the pipeline into a fully pipelined computation, letting the design hit 300 MHz. Around it, dual-port BRAM stacks act as a first-in-last-out memory that feeds 64 rewards and 64 values per cycle, and dynamic standardization updates running mean and variance via Welford's recurrence so the 8-bit quantizer always sees a roughly unit-variance distribution.","core_discovery":"The central claim is that HEPPO-GAE, an FPGA-based accelerator on a Zynq UltraScale+ SoC, can compute GAE and rewards-to-go for 64 trajectories in parallel at 300 MHz with fully pipelined processing elements, using a 2-step lookahead transform of the recurrence $\\hat{A}_t = \\delta_t + (\\gamma\\lambda)\\hat{A}_{t+1}$ to break the feedback loop so the multiplier can be pipelined through DSP blocks. Combined with a FILO block-RAM layout that overwrites rewards and values with advantages and rewards-to-go, and with dynamic reward standardization plus block standardization of values followed by 8-bit uniform quantization, the paper claims a 4x memory reduction, a 1.5x increase in cumulative rewards on MuJoCo and Atari, and a theoretical GAE throughput of 300 million elements per second per processing element — roughly two million times the about 9000 elements per second measured for a CPU-GPU reference implementation.","pith_inferences":["If the claimed 300 MHz, zero-bubble operation is achieved only for equal-length trajectories, the real speedup on variable-length episodes could be much lower; a natural test is to measure the design on trajectories of unequal length, which is exactly the case the CPU-GPU baseline is slower on.","The k-step lookahead idea is not PPO-specific: any sequential RL update that follows a linear recurrence with a constant discount factor (for instance, other TD(lambda)-style estimators) can use the same transform, so the accelerator could generalize to other policy-gradient methods.","The reference baseline (~9000 elements/s from a Medium-article PyTorch implementation) is likely far slower than a vectorized batch GAE; if so, the 2-million-fold figure is an artifact of the baseline, not of the hardware, and the true speedup over a well-optimized GPU implementation would be much smaller but still possibly substantial on a single-chip platform."],"forward_implications":["If the measured 30% PPO speedup is representative, GAE ceases to be a roughly 30% training-time cost and becomes a negligible stage.","The 4x memory reduction from quantized on-chip storage means larger batch sizes or longer horizons can fit in the same BRAM budget, which directly helps the memory-bound regime described in Section IV.","Because the standardization is parameter-free and environment-independent, the reward improvement should transfer across MuJoCo, Atari, and similar RL benchmarks without per-task tuning.","The design's resource footprint (under 10% of LUTs and FFs and about 30% of DSPs on the ZCU106) leaves room for other PPO phases on the same chip, so a full single-chip PPO is structurally feasible.","If keeping rewards in standardized form is the key to the 1.5x cumulative-reward gain, then future RL hardware can treat reward preprocessing as a first-class design component rather than a host-side afterthought."],"supporting_citations":[{"why":"Supplies the GAE recurrence and the definition of advantage the hardware implements.","marker":"[6]"},{"why":"Defines PPO and the clipped objective whose policy updates consume the advantages the accelerator produces.","marker":"[3]"},{"why":"Provides the systolic-array DNN inference design adapted for actor and critic inference in the programmable logic.","marker":"[9]"},{"why":"Gives Welford's recurrence used for the running variance in dynamic reward standardization.","marker":"[13]"},{"why":"Canonical reference for the running-variance algorithm cited in [13].","marker":"[14]"},{"why":"The CPU-GPU reference implementation from which the about 9000 elements-per-second baseline and the 2-million-fold speedup are derived.","marker":"[17]"},{"why":"Documents that advantage normalization is widely adopted, the practice the paper's reward standardization builds on.","marker":"[15]"},{"why":"Additional discussion of advantage normalization practices that motivates the paper's baseline comparison.","marker":"[16]"}],"fun_headline_variants":["FPGA GAE: 4x memory drop, 1.5x reward gain","Single-chip GAE: PPO speed +30%, memory -4x","On-chip GAE: 8-bit quant, 4x less memory, 1.5x rewards","Pipelined FPGA GAE: 30% faster PPO, 4x memory cut","Hardware-edge GAE: 1.5x rewards, 4x memory savings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers depend on the CPU-GPU baseline being representative (it is a single Medium-article implementation) and on the FPGA sustaining 300 MHz with no pipeline stalls while feeding 64 rewards and 64 values from BRAM each cycle; the reward gain also depends on treating 'original PPO' without advantage normalization as a fair comparison, even though the paper itself notes normalization is standard practice.","fun_headline_variants_meta":{"raw":{"variants":["FPGA GAE: 4x memory drop, 1.5x reward gain","Single-chip GAE: PPO speed +30%, memory -4x","On-chip GAE: 8-bit quant, 4x less memory, 1.5x rewards","Pipelined FPGA GAE: 30% faster PPO, 4x memory cut","Hardware-edge GAE: 1.5x rewards, 4x memory savings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1667,"prompt_tokens":980,"completion_tokens":687,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":569}},"tokens_in":596,"tokens_out":687,"duration_ms":6513,"temperature":1.0,"reasoning_tokens":569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:52:18.963666+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the same GAE workload with a vectorized PyTorch implementation that processes all trajectories as one padded tensor on the same 32-core Xeon plus V100 machine, and compare elements per second to the about 9000/s cited from reference [17]; if that vectorized baseline is, say, 100x faster, the claimed 2-million-fold GAE speedup collapses to a more modest figure. Then, to test the hardware claim, synthesize the 64-PE design on the ZCU106, run trajectories of unequal length, and count clock cycles with the write-back handshake; if cycles per element rise above 1 per PE due to unequal-length pipeline drains, the 300 MHz zero-bubble assumption is not sustained in practice.","supporting_citations":[{"cited_title":"Accelerating Reinforcement Learning through GPU Atari Emulation,","cited_arxiv_id":null,"evidence_quote":"Gives Welford's recurrence used for the running variance in dynamic reward standardization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Canonical reference for the running-variance algorithm cited in [13]."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The CPU-GPU reference implementation from which the about 9000 elements-per-second baseline and the 2-million-fold speedup are derived."},{"cited_title":"Note on a Method for Calculating Corrected Sums of Squares and Products,","cited_arxiv_id":null,"evidence_quote":"Documents that advantage normalization is widely adopted, the practice the paper's reward standardization builds on."},{"cited_title":"(2018) Understanding Normalization of Advantage Function in PPO [Online]","cited_arxiv_id":null,"evidence_quote":"Additional discussion of advantage normalization practices that motivates the paper's baseline comparison."}],"review_version":1}