REVIEW 2 major objections 2 minor 13 references
Quantifying the Energy Floor: Direct Measurement and Replay Buffer Bias in SAC-Based HVAC Control on sbsim
T0 review · 2 major / 2 minor · reviewed 2026-06-28 · grok-4.3
Pith's one-line read Direct measurement shows replay buffer initialization from a schedule policy accounts for 96% of the sub-optimality gap in SAC HVAC control, with equipment minimum power setting the true energy floor at 35.51 USD per day.
desk verdict The paper measures a concrete energy floor via min-action runs on sbsim and shows that schedule-based buffer pre-filling accounts for 96% of the SAC gap, with equipment mins as the real limit. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Direct measurement of the energy floor through minimum-action experiments on sbsim, combined with controlled comparisons of replay buffer initialization strategies.
What would settle it
A feasible control policy on sbsim that achieves a daily cost below 35.51 USD while satisfying all physical and action constraints would falsify the measured energy floor.
Extended reading notes
Core claim
Through minimum-action experiments on the sbsim simulator, the energy floor is directly measured at USD 35.51/day, dominated by continuous electrical loads. The standard SAC baseline initialized with schedule-policy replay buffer transitions converges to USD 37.18/day. Training from an empty buffer reduces cost to USD 35.57/day, eliminating 96% of the gap. Expanding the supply water temperature range by 10 K yields negligible additional savings, and further expansion triggers physical constraint violations. Systematic ablation across planning horizon, reward weights, and observation enrichment shows all pre-filled-buffer configurations cluster within 0.7%, demonstrating that equipment minimu
Load-bearing premise
The minimum-action experiments correctly identify the absolute energy floor without missing any feasible lower-cost actions or unmodeled physical constraints.
Editorial extensions
If this is right
- Training SAC from an empty buffer reaches performance within 0.06 USD of the directly measured floor.
- Widening the supply water temperature range by 10 K produces only 0.03 USD per day in additional savings.
- All configurations that begin with a pre-filled buffer perform within 0.7% of one another.
- The observed discount factor coupling reduces the effective planning horizon from 8.3 hours to 46 minutes.
Reading between the lines
- The same replay buffer initialization bias may limit performance in other reinforcement learning applications that seed buffers with expert or schedule policies.
- The identified discount factor coupling indicates that effective planning horizons should be audited in other RL-based building or process control studies.
- Direct energy floor measurement offers a general method to separate hardware limits from algorithmic performance in simulator-based control research.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims to directly quantify the energy floor for SAC-based HVAC control on the sbsim simulator at USD 35.51/day via minimum-action experiments (dominated by continuous electrical loads at USD 35.44/day). It reports that a standard SAC baseline with schedule-policy replay buffer reaches USD 37.18/day (4.7% above floor), while training from an empty buffer achieves USD 35.57/day and closes 96% of the gap; supply-water-temperature range expansion yields only USD 0.03/day further savings before constraint violations; a discount-factor coupling (gamma_eff=0.891) shrinks the effective horizon from 8.3 h to 46 min; and systematic ablations show all pre-filled-buffer variants cluster within 0.7%, implying equipment minimum power—not algorithmic design—is the binding constraint.
Significance. If the direct-measurement protocol is shown to return the true minimum cost under the given action-space constraints, the work supplies a concrete, falsifiable benchmark for RL HVAC control that isolates the contribution of replay-buffer initialization from other sources of sub-optimality and demonstrates the dominance of physical equipment limits. The explicit ablations across planning horizon, reward weights, and observation enrichment, together with the identification of a discount-factor coupling that may affect other SAC deployments, constitute reproducible experimental protocols that strengthen the result.
major comments (2)
- [Abstract (direct measurement paragraph)] Abstract (paragraph on direct measurement): the claim that minimum-action experiments identify the absolute energy floor at USD 35.51/day rests on the assumption that setting actions to per-step lower bounds exhausts all feasible lower-cost trajectories; without an explicit constrained trajectory optimization or exhaustive verification that no lower-cost sequence exists under the action-space constraints, the reported floor may be loose, which directly undermines the 96% gap-closure interpretation of the empty-buffer result (USD 35.57/day) and the conclusion that equipment minimum power is the sole binding constraint.
- [Abstract (supply water temperature paragraph)] Abstract (supply-water-temperature expansion): the observation that a 10 K expansion yields only USD 0.03/day savings and further expansion triggers violations is presented as supporting evidence that the floor is tight, yet this single-axis perturbation does not rule out other feasible action combinations (e.g., coordinated changes in multiple continuous variables) that could produce lower cost while remaining within constraints.
minor comments (2)
- [Abstract] The derivation or numerical procedure used to obtain gamma_eff = 0.891 and the effective planning horizon reduction (8.3 h to 46 min) is not shown; a short appendix or equation block would allow independent verification.
- The manuscript should state the precise definition of the reward function and the numerical values of all reward weights used in the ablations, as these are required to reproduce the reported cost figures.
Simulated Author's Rebuttal
We thank the referee for the insightful comments on the energy floor measurement. We agree that the minimum-action protocol provides a practical but not formally verified lower bound, and we will revise the manuscript to clarify this distinction while preserving the main empirical findings on replay buffer bias. Point-by-point responses follow.
read point-by-point responses
-
Referee: [Abstract (direct measurement paragraph)] Abstract (paragraph on direct measurement): the claim that minimum-action experiments identify the absolute energy floor at USD 35.51/day rests on the assumption that setting actions to per-step lower bounds exhausts all feasible lower-cost trajectories; without an explicit constrained trajectory optimization or exhaustive verification that no lower-cost sequence exists under the action-space constraints, the reported floor may be loose, which directly undermines the 96% gap-closure interpretation of the empty-buffer result (USD 35.57/day) and the conclusion that equipment minimum power is the sole binding constraint.
Authors: We acknowledge the referee's point: applying per-step lower bounds independently yields an empirical lower bound rather than a provably optimal trajectory, as temporal dependencies in building dynamics could allow coordinated sequences to achieve lower cost. The manuscript presents this as a 'direct measurement' because the protocol exhausts the per-step action bounds in a single rollout, and the 99.8% dominance of fixed electrical loads makes further reduction improbable without constraint violation. The empty-buffer result (35.57) being only 0.06 above supports the interpretation in practice. We will revise the abstract, methods, and discussion to describe the quantity as an 'empirical energy floor via minimum-action rollouts' and explicitly note the absence of constrained trajectory optimization as a limitation. This is a partial revision that tempers the wording without changing the reported numbers or the buffer-bias conclusion. revision: partial
-
Referee: [Abstract (supply water temperature paragraph)] Abstract (supply-water-temperature expansion): the observation that a 10 K expansion yields only USD 0.03/day savings and further expansion triggers violations is presented as supporting evidence that the floor is tight, yet this single-axis perturbation does not rule out other feasible action combinations (e.g., coordinated changes in multiple continuous variables) that could produce lower cost while remaining within constraints.
Authors: We agree that the supply-water-temperature ablation is a single-axis test and does not exhaustively rule out beneficial coordinated changes across multiple variables. It was included as one of several targeted probes (alongside planning horizon, reward weights, and observation enrichment) to check whether relaxing a key continuous constraint could approach lower cost. The negligible savings and subsequent violations indicate the minimum actions already sit near the feasible boundary for that actuator. We will revise the abstract and results section to present the outcome as corroborating rather than conclusive evidence of tightness, and we will add a brief statement that a full multi-variable constrained optimization is left for future work. This is a partial revision that addresses the concern on presentation. revision: partial
Circularity Check
No circularity; energy floor from independent direct measurement, gap closed by explicit ablation
full rationale
The paper defines the energy floor via direct minimum-action experiments on the simulator and measures the SAC baseline and empty-buffer ablation against it with explicit numerical comparisons (35.51, 37.18, 35.57). No step reduces a claimed prediction or uniqueness result to a fitted parameter, self-citation, or ansatz imported from prior work by the same authors. All load-bearing claims rest on simulator runs and ablations whose outputs are not forced by the inputs by construction. The derivation chain is therefore self-contained against external benchmarks.
Assumptions & free parameters
assumptions (2)
- domain assumption The sbsim simulator accurately represents building thermal dynamics, equipment constraints, and energy costs.
- domain assumption The minimum-action policy is feasible and achieves the absolute lowest cost without violating unstated physical limits.
Cite this review
Pith. "Pith review of Quantifying the Energy Floor: Direct Measurement and Replay Buffer Bias in SAC-Based HVAC Control on sbsim." pith.science (2026). https://pith.science/paper/IFUK7QEE
@misc{pith2026260601665,
author = {Pith},
title = {Pith review of: Quantifying the Energy Floor: Direct Measurement and Replay Buffer Bias in SAC-Based HVAC Control on sbsim},
year = {2026},
howpublished = {\url{https://pith.science/paper/IFUK7QEE}},
note = {Machine review of arXiv:2606.01665}
}
read the original abstract
We quantify the energy floor -- the minimum achievable cost given action space constraints -- for Soft Actor-Critic (SAC) HVAC control on the sbsim calibrated building simulator. Through minimum-action experiments, we directly measure this floor at USD 35.51/day, dominated by continuous electrical loads (USD 35.44, 99.8%) with negligible gas consumption. The standard SAC baseline, initialized with schedule-policy replay buffer transitions, converges to USD 37.18/day, 4.7% above the floor. We identify buffer initialization as the dominant source of sub-optimality in this scenario: training from an empty buffer reduces cost to USD 35.57/day, eliminating 96% of the gap. Expanding the supply water temperature range by 10 K yields negligible additional savings (USD 0.03/day), and further expansion triggers physical constraint violations. We additionally uncover a discount factor coupling (gamma_eff = 0.891) shrinking the effective planning horizon from 8.3 h to 46 min -- a benchmark-wide issue warranting audit. Systematic ablation across planning horizon, reward weights, and observation enrichment confirms all pre-filled-buffer configurations cluster within 0.7% (USD 37.18--USD 37.42), demonstrating that equipment minimum power -- not algorithmic design -- imposes the binding constraint.
Figures
Reference graph
Works this paper leans on
-
[1]
Department of Energy
U.S. Department of Energy. Buildings energy data book. https://www.energy. gov/eere/buildings/building-technologies-office, 2011
2011
-
[3]
J. Goldfeder, V. Dean, Z. Jiang, X. Wang, B. Dong, H. Lipson, and J. Sipple. The Smart Buildings Control Suite: A diverse open source benchmark to evaluate and scale HVAC control policies for sustainability.arXiv preprint arXiv:2410.03756, 2025
-
[4]
Haarnoja, A
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InProc. ICML, pages 1861–1870, 2018
2018
-
[5]
T. Wei, Y. Wang, and Q. Zhu. Deep reinforcement learning for building HVAC control. InProc. DAC, pages 1–6, 2017
2017
-
[6]
Zhang and K.P
Z. Zhang and K.P. Lam. Practical implementation and evaluation of deep rein- forcement learning control for a radiant heating system. InProc. BuildSys, pages 148–157, 2018
2018
-
[7]
Biemann, F
M. Biemann, F. Scheller, X. Liu, and L. Huang. Experimental evaluation of model- free reinforcement learning algorithms for continuous HVAC control.Applied Energy, 298:117164, 2021
2021
-
[8]
Drgoňa et al
J. Drgoňa et al. All you need to know about model predictive control for buildings. Annual Reviews in Control, 50:190–232, 2020
2020
-
[9]
B. Chen, Z. Cai, and M. Bergés. Gnu-RL: A precocial reinforcement learning solution for building HVAC control using a differentiable MPC policy. InProc. BuildSys, pages 316–325, 2019
2019
Show all 13 references
-
[10]
Henderson, R
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. InProc. AAAI, pages 3207–3214, 2018
2018
-
[11]
Dulac-Arnold, N
G. Dulac-Arnold, N. Levine, D.J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. Challenges of real-world reinforcement learning: Definitions, benchmarks and analysis.Machine Learning, 110:2419–2468, 2021
2021
-
[12]
de Bruin, J
T. de Bruin, J. Kober, K. Tuyls, and R. Babuska. Experience selection in deep reinforcement learning for control.Journal of Machine Learning Research, 19(9):1- 56, 2018
2018
-
[13]
Hester, M
T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, G. Dulac-Arnold, J. Agapiou, J. Leibo, and A. Gruslys. Deep Q-learning from demonstrations. InProc. AAAI, pages 3223–3230, 2018
2018
-
[14]
Vecerik, T
M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.arXiv preprint arXiv:1707.08817, 2017
2017 arXiv
Reviewed June 28, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.