{"id":"ff6acd46-951e-4e5c-ad57-748e0ca7c30b","arxiv_id":"2505.05796","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A PPO-based HVAC controller that learns from simulated occupant feedback and price signals cuts simulated energy cost versus a rule-based baseline, though occupant comfort is worse in the reported metrics.","lead":"This paper proposes a human-in-the-loop reinforcement learning controller for home HVAC that learns occupant comfort from simulated thermostat overrides and reacts to wholesale electricity prices. The simulation results show lower energy cost than a rule-based controller, but the comfort improvement claim is not supported by the paper's own data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table 1 contradicts the abstract: HITL shows higher comfort violation probability (10.24% vs 3.13%) and MAE (1.82 vs 0.49 °C) than the rule-based baseline, so 'maintaining or enhancing occupant comfort' is not supported.","rationale":"The reader's weakest assumption was the realism of the feedback model, Eqs. (10)-(13). I agree that this is a serious external-validity limitation, but I find an even more direct problem: the paper's own reported comfort metrics contradict the abstract. Table 1 compares HITL, optimization, and rule-based controllers and shows HITL with the highest violation probability and highest MAE to setpoint. Since the abstract explicitly promises 'maintaining or enhancing occupant comfort', the data as reported do not support the central claim. The aggregate metric may hide a comfort-preserving operating point, but the burden is on the authors to show one. My proposed check isolates the comfort-first regime; if it fails, the claim is false under the paper's own simulation. I therefore keep the reader's CONDITIONAL verdict: the energy-cost and occupancy-forecasting contributions are plausible, but the comfort claim requires either a corrected analysis or a revised, more limited claim. Agreement with the reader is partial because the reader located the weakest assumption in the feedback-generating equations, whereas I locate it in the self-reported metrics; both concerns point to the same unsupported comfort claim.","tokens_in":18261,"tokens_out":8424,"duration_ms":86900,"concrete_test":"Recompute the Table 1 comfort metrics separately for each discomfort weight beta in {0.1, 0.3, 0.5, 0.7, 0.9} and each scenario S1-S4 using the 25-run logs, and compare each cell with the rule-based baseline computed in the same evaluation harness. A minimal decisive check is the beta = 0.9 (comfort-first) cell: if HITL's violation probability and MAE in that cell are still worse than the rule-based values (3.13%, 0.49 °C), the abstract's comfort claim is falsified by the authors' own simulation. If the beta = 0.9 cell is not reported, request the per-beta results or reproduce them from the available data before accepting the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires cost reductions with equal or better comfort. The paper's own evaluation metrics in Table 1 show the opposite. HITL's violation probability is 10.24% versus 3.13% for the rule-based controller and 0.00% for the optimization controller; its MAE to setpoint is 1.82 °C versus 0.49 °C (rule-based) and 1.60 °C (optimization). Thus, on the two comfort metrics the paper itself defines, HITL is the worst performer, not a method that 'maintains or enhances' comfort. Because the Table 1 entries average over scenarios and discomfort weights beta, it is possible that some high-beta configuration preserves comfort, but the paper does not report such a configuration; as stated, the headline result is internally contradicted. A secondary issue is that the simulated feedback in Eqs. (10)-(13) and the comfort metrics both use the same T_set = 22 °C, so the simulator and the evaluation share the hidden preference; this further undermines the claim that comfort preferences are learned rather than baked into the reward.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Human-in-the-Loop (HITL) reinforcement-learning framework for residential HVAC control. The controller observes indoor/outdoor temperatures, time-of-day features, forecasted outdoor temperature, occupancy and occupancy forecasts, a buffer of recent occupant overrides, and current/forecasted wholesale electricity prices, and chooses whether to run the HVAC (Secs. 2.2-2.3). Occupant discomfort is simulated as a stochastic override triggered with probability proportional to the squared deviation of indoor temperature from a setpoint T_set = 22°C (Eqs. 10-13), and the reward penalizes these overrides plus energy cost (Eqs. 16-20). PPO is used for learning, with a bidirectional-LSTM occupancy predictor for the realistic scenario (Sec. 3). Experiments over 30 days of May 2023 Melbourne data compare four occupancy-information scenarios against a rule-based controller and a perfect-information optimization controller (Secs. 4-5). The headline claim is that the method cuts energy cost while maintaining or enhancing occupant comfort.","tokens_in":18596,"tokens_out":6448,"duration_ms":54904,"significance":"If the results held, the framework would be a useful step toward demand-response-aware HVAC control that learns occupant preferences from lightweight interaction signals rather than from calibrated thermal-comfort models. The manuscript has concrete strengths: a transparent MDP formulation with an explicit thermal model (Eq. 8), reproducible data sources (ARAS, Visual Crossing, AEMO), a sensitivity analysis of the feedback probability cap (Fig. 3), and an explicit comparison with a rolling-horizon optimization benchmark. However, the evaluation as presented does not support the comfort-preservation claim: the reported comfort metrics are worse than the rule-based baseline, and the comfort reward/evaluation loop is generated from the same setpoint the method is supposed to learn. These issues are load-bearing for the central contribution.","major_comments":[{"comment":"The abstract and Sec. 1 contribution (4) claim the method 'maintains or enhances occupant comfort' and achieves savings 'while maintaining or enhancing occupant comfort when compared to rule-based strategies.' Table 1 directly contradicts this: HITL has a 10.24% temperature violation probability and 1.82°C MAE to setpoint, versus 3.13% and 0.49°C for the rule-based controller and 0.00% violation probability for the optimization controller. Because the table averages over all scenarios and discomfort proportions beta, it is possible that some beta configuration preserves comfort, but the paper does not report such a configuration. The authors should either report a disaggregated table identifying a scenario/beta where HITL meets or improves on the rule-based comfort metrics, or revise the abstract and contribution statements to claim cost savings only.","section":"Table 1 and Sec. 5.3"},{"comment":"The comfort component is self-referential. The simulated feedback probability p_f depends on T_set in Eq. (10), the expected comfort-preserving action a* depends on T_set in Eq. (11), the discomfort reward in Eq. (16) is computed from the resulting feedback buffer, and the evaluation metrics (violation probability and MAE to setpoint) in Sec. 5 are both defined relative to the same T_set = 22°C. Consequently, the agent is rewarded and evaluated against the very preference that generates its feedback signal, which makes the claim in Sec. 2.5.1 that the model 'dynamically adjusts based on the feedback that directly reflects occupant comfort' circular. To support the stated contribution of learning preferences without predefined comfort models, the authors need at least one evaluation against a comfort measure that is not derived from T_set (e.g., a separate occupant model or field data), or they must explicitly frame the contribution as tuning to a known setpoint.","section":"Secs. 2.4.2, 2.5.1, and 5"},{"comment":"The section claims the system learns preferences with 'limited need for user intervention' and that S2 achieves 'a reasonable number of overrides,' but no aggregate override counts or frequencies are reported. Fig. 2(c) shows only a four-day excerpt for S1, S2, S4, and no statistics over the 25 runs or across the seven test days, and there is no comparison with the override rate of the rule-based or optimization controllers. The authors should report quantitative override rates with standard deviations, which is the natural metric for evaluating the claimed reduction in manual interaction.","section":"Sec. 5.1 and Fig. 2(c)"},{"comment":"The optimization benchmark holds indoor temperature within the comfort range as a hard constraint, so its 0% violation probability is guaranteed by construction. Comparing HITL's 10.24% violation probability with this infeasible benchmark does not support the claim in Sec. 1 contribution (4) that HITL 'achieves performance close to the theoretical ceiling.' A more informative comparison would report the optimization controller's energy cost either with the same discomfort weight beta in the objective or with a softened comfort constraint, so the cost-comfort trade-off can be assessed fairly.","section":"Secs. 4.4.2 and 5.3"}],"minor_comments":[{"comment":"The text states that 'longer horizons were considered' but provides no experiment supporting the diminishing-returns claim; the authors should either show the comparison or remove the statement.","section":"Sec. 4.2"},{"comment":"Table 1 reports averages over 25 runs without confidence intervals or standard deviations; given the stochasticity noted in Sec. 5, the authors should provide dispersion measures or box plots for the temperature metrics.","section":"Table 1"},{"comment":"The rule-based controller description does not state the setpoint used; given that Table 1 reports MAE to setpoint, the setpoint should be specified for reproducibility.","section":"Sec. 4.4.1"},{"comment":"These figures omit Scenario 3 and the rule-based controller 'due to page limitations'; since S3 is one of the four central scenarios, the figures should be moved to an appendix or the exclusion criterion should be stated.","section":"Figs. 2(b) and 2(c)"},{"comment":"Equation (33) is described as calculated using 'observable heating and cooling behavior,' but the values T_lower, T_upper, and t_cool are assumed rather than measured in this study; the wording should be adjusted to 'estimated' or 'assumed.'","section":"Eq. (33)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript overclaims comfort preservation in its current form, and the evaluation is self-referential because both feedback generation and comfort metrics are tied to T_set. The core idea is reasonable and the energy-cost results may be salvageable with reframed claims and additional experiments. The paper fits the venue's scope, but the load-bearing contradiction in Table 1 must be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: a careful, clearly-specified simulation study that assembles PPO, an LSTM occupancy predictor, a feedback buffer, and wholesale price forecasts into an HVAC controller. The energy-cost results are plausible. But the paper's own Table 1 contradicts the abstract's comfort claim, and the comfort loop is self-referential. Worth engaging with, but the headline needs to change.\n\nWhat is actually new: putting real-time wholesale price forecasts into the state of a human-in-the-loop HVAC RL controller. That fills a genuine gap in the prior HITL HVAC work (Shuvo and Yilmaz; Chen et al.), which treats comfort and energy but ignores market dynamics. The MDP is written out cleanly, with thermal model, reward, and feedback buffer all explicit. The sensitivity analysis on the feedback probability cap is a reasonable robustness check. The energy-cost reduction against the rule-based controller is believable, and the perfect-prediction optimization controller is a fair ceiling. None of this is a paradigm shift; it is a sensible assembly of existing pieces with one useful extension.\n\nThe soft spots. First, the comfort claim. Table 1 reports HITL violation probability 10.24% versus 3.13% for the rule-based controller and 0.00% for optimization; MAE to setpoint 1.82 versus 0.49 and 1.60. By the paper's own metrics, HITL is the worst comfort performer, and the abstract claims comfort is maintained or enhanced. Averaging over discomfort weights may hide a high-beta configuration that does preserve comfort, but the paper does not report one. The fix is simple: report comfort per beta, and soften the claim. As written, it is unsupported.\n\nSecond, the circularity. The simulated occupant's feedback probability (Eq. 10) is a quadratic function of deviation from T_set = 22 C, and the comfort metrics evaluate deviation from the same 22 C. The agent never observes T_set, so this is a legitimate proof-of-concept for learning a hidden preference. But it does not support the claim that the framework needs no predefined comfort model. The simulator has one, and both the reward signal and the evaluation are generated from it. The claim about learning personalized preferences is therefore weaker than the abstract suggests.\n\nMinor points: no code or data, and the RL results are summarized over 25 runs without seed details, which makes reproduction harder. The description of the optimization baseline's tested comfort ranges is vague. These are secondary after the Table 1 issue.\n\nWho this is for: people building RL-based building energy controllers who want a concrete, well-specified baseline with market-price signals. I would engage with it and give it a serious referee — the energy-side evidence is decent and the framework is explicit — but I would reject the comfort claim as stated and demand the per-beta numbers before trusting the headline.","headline":"Competent HITL HVAC simulation with a plausible energy-cost story, but its own Table 1 contradicts the comfort claim and the comfort evaluation is self-referential; the energy and framework parts deserve peer review, the headline does not.","tokens_in":19052,"tokens_out":4906,"would_cite":true,"duration_ms":49106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims an HVAC controller can learn occupant comfort from overrides and respond to price signals, cutting energy cost without predefined comfort models.","keywords":["HVAC control","human-in-the-loop","reinforcement learning","occupant comfort","feedback buffer","occupancy prediction","electricity price","demand response"],"falsifier":"Replace the simulated occupant in the feedback equations with real logged override data from a deployed thermostat, or with a different discomfort model (for example, one with hysteresis, asymmetric heating and cooling responses, or delayed feedback), and re-run the same PPO training and evaluation. If the feedback-driven policy no longer beats the rule-based baseline in total cost, or if override frequency does not fall as training proceeds, the claim that comfort preferences are genuinely learned from feedback rather than encoded in the simulator is refuted.","tokens_in":18095,"feed_emoji":"⚡","tokens_out":9571,"duration_ms":109347,"temperature":0.7,"pith_summary":"This paper tries to establish that an HVAC controller can learn an occupant's comfort preferences from the occupant's own override actions, rather than from a predefined thermal-comfort model or a fixed setpoint schedule. The proposed human-in-the-loop controller is a reinforcement-learning agent that keeps a buffer of recent overrides, uses that buffer both to correct its immediate action and to define a discomfort cost, and simultaneously optimizes against forecasted wholesale electricity prices and predicted occupancy. In simulation, the authors report significant energy-cost reductions relative to a rule-based occupancy thermostat, with occupant comfort maintained or improved in their discomfort-cost accounting, and performance close to an optimization controller that is given perfect forecasts and explicit comfort constraints. Because HVAC accounts for roughly 38% of building energy use, a controller that personalizes comfort without sensors or manual programming while shifting load to cheap, low-emission hours would be a practical route to demand response.","feed_headline":"HVAC learns comfort from overrides and cuts energy cost","feed_subtitle":"Simulated trials show feedback-driven reinforcement learning beats rule-based thermostats and nears perfect-foresight optimization.","key_machinery":"The central object is the feedback buffer, a fixed-length window of occupant override signals taking values in $\\{-1,0,1\\}$, included in the MDP state and updated each time step. It does three jobs at once: it immediately corrects the applied control action, it enters the state representation so the policy can learn which actions provoke overrides, and it defines the discomfort cost through an exponentially weighted sum of recent override events. The companion mechanism is the simulated feedback generator, which sets the override probability to $p^f_t = \\min\\left(\\left((T_t^{\\mathrm{in}} - T^{\\mathrm{set}})/\\theta_{\\mathrm{range}}\\right)^2, p_{\\max}\\right)$. This quadratic link between temperature deviation and override likelihood is what lets the framework claim that comfort preferences are learned from observed interventions rather than from a predefined comfort model.","core_discovery":"On the paper's own terms, the discovery is that occupant discomfort can be treated as a directly observable feedback signal instead of an unmeasured deviation from a comfort model, and that a policy trained on that signal can simultaneously track electricity prices. The agent's state includes the current indoor and outdoor temperatures, time of day, forecasted outdoor temperature, current and forecasted occupancy, a buffer of recent occupant overrides, and current and forecasted wholesale market rates. The action is simply whether the HVAC is on or off. The reward subtracts an exponentially weighted penalty for recent overrides from the cost of electricity consumed, and the policy is learned with PPO. The paper's simulation generates overrides from a capped quadratic function of the distance between indoor temperature and a comfort setpoint, and its experiments compare four levels of occupancy information, a rule-based thermostat, and a perfect-foresight optimization benchmark. The reported outcome is that the feedback-driven policy cuts energy cost relative to the rule-based controller, that occupancy forecasts help most when comfort is weighted heavily, and that the learned controller stays close to the optimization ideal despite having no explicit comfort constraints.","pith_inferences":["Editorial inference: the simulation generates both the learning signal and the evaluation metrics from the same $T^{\\mathrm{set}}$-anchored override formula, so the strongest validation would be a real deployment or a second simulator with a different occupant model; otherwise the claim that preferences are learned rather than assumed may be circular.","Editorial inference: the same feedback-buffer representation should transfer to other building systems with observable occupant overrides—lighting, window shading, or plug loads—where an explicit comfort model is expensive but a binary corrective signal is cheap.","Editorial inference: the paper's future-work mention of delayed feedback suggests a concrete test: if overrides are shifted by one or more time steps to mimic occupants who react late, the policy must still reduce override frequency and cost for the framework to hold in realistic settings."],"forward_implications":["If the central claim holds, a thermostat could be replaced by a learned policy that treats override events as the only comfort signal, so occupants never need to program schedules or understand a comfort model.","Because wholesale rates and their forecasts are part of the state, the agent learns to pre-cool or pre-heat when energy is cheap and occupancy is expected, shifting HVAC load toward periods of grid surplus and supporting demand response.","The scenario comparison indicates that explicit occupancy information improves the cost–comfort trade-off, but the feedback buffer alone already provides enough signal to operate reasonably when occupancy data are missing or privacy-sensitive.","Sensitivity results across maximum feedback probabilities imply the learned policy remains serviceable when occupants do not always report discomfort, so imperfect human feedback does not by itself break the approach."],"supporting_citations":[{"why":"Supplies the thermal transition model (indoor temperature as a function of current indoor and outdoor temperature plus HVAC power) used in both the RL environment and the optimization benchmark.","marker":"[25]"},{"why":"Provides the multi-resident household occupancy traces that determine who is present in all four scenarios.","marker":"[2]"},{"why":"Fixes the thermal comfort range used in the feedback simulation and in the temperature-violation evaluation.","marker":"[6]"},{"why":"Supplies the wholesale electricity price and demand data that define energy cost and price-responsive behavior.","marker":"[7]"},{"why":"The PPO algorithm used to train the control policy from the MDP reward.","marker":"[32]"},{"why":"Provides the 22°C productivity-based setpoint that anchors the simulated feedback model and the comfort metric.","marker":"[33]"},{"why":"Supplies the outdoor temperature time series that drives the thermal simulation.","marker":"[36]"},{"why":"Defines the MDP formalism in which the human-in-the-loop control problem is cast.","marker":"[28]"}],"fun_headline_variants":["Feedback-driven HVAC AI cuts energy costs, boosts comfort","Human-in-the-loop AI for HVAC reduces energy cost while keeping comfort","Occupant overrides train HVAC AI to save energy and maintain comfort","AI HVAC uses real-time feedback to cut costs and improve comfort"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's simulation assumes an occupant's probability of overriding the controller is a capped quadratic function of how far indoor temperature is from one known setpoint, so both the reward that trains the agent and the comfort metrics used to evaluate it are generated from that same formula; if real occupants override for other reasons or with other patterns, the claimed automatic learning of comfort preferences is not supported.","fun_headline_variants_meta":{"raw":{"variants":["Feedback-driven HVAC AI cuts energy costs, boosts comfort","Human-in-the-loop AI for HVAC reduces energy cost while keeping comfort","Occupant overrides train HVAC AI to save energy and maintain comfort","AI HVAC uses real-time feedback to cut costs and improve comfort"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1543,"prompt_tokens":978,"completion_tokens":565,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":494}},"tokens_in":594,"tokens_out":565,"duration_ms":6584,"temperature":1.0,"reasoning_tokens":494,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:55:42.829809+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the simulated occupant in the feedback equations with real logged override data from a deployed thermostat, or with a different discomfort model (for example, one with hysteresis, asymmetric heating and cooling responses, or delayed feedback), and re-run the same PPO training and evaluation. If the feedback-driven policy no longer beats the rule-based baseline in total cost, or if override frequency does not fall as training proceeds, the claim that comfort preferences are genuinely learned from feedback rather than encoded in the simulator is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the thermal transition model (indoor temperature as a function of current indoor and outdoor temperature plus HVAC power) used in both the RL environment and the optimization benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-resident household occupancy traces that determine who is present in all four scenarios."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Fixes the thermal comfort range used in the feedback simulation and in the temperature-violation evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the wholesale electricity price and demand data that define energy cost and price-responsive behavior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 22°C productivity-based setpoint that anchors the simulated feedback model and the comfort metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MDP formalism in which the human-in-the-loop control problem is cast."}],"review_version":1}