Pith. sign in

REVIEW 2 major objections 6 minor 52 references

Exact Action Values Are Not Enough: Rollout-Verified Reinforcement Fine-Tuning of a Reasoning Model for Multi-Zone VAV Control

T0 review · 2 major / 6 minor · reviewed 2026-07-31 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Exact action values are not enough to teach a reasoning model how VAV commands change the building state.

desk verdict Careful negative result: exact rollout values rank actions but don't teach the model to construct better ones; the critic audit is the cleanest finding, and the transition-test interface is the main soft spot. read the letter →

arxiv 2607.27914 v1 pith:I2U3UEJ5 submitted 2026-07-30 cs.LG cs.SYeess.SY

Takumi Shioda , Kohei Terashima , Tatsuo Nagai This is my paper · ORCID
classification cs.LGcs.SYeess.SY
keywords multi-zoneVAVcontrolreinforcementfine-tuningrolloutverificationactionvaluestransitionmodelreasoningmodelsHVACcontinuous
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests a two-stage idea: first, whether a frontier reasoning model can control a four-zone variable-air-volume HVAC system from text alone without building-specific training, and second, whether TD3-guided reinforcement fine-tuning (RFT) with an exact rollout verifier can transfer that ability into a smaller open-weight model. The first stage works as a capability demonstration: the frontier model cut three-day HVAC electricity by 6.2% relative to a rule-based baseline, though partly by narrowing the CO2 margin. The second stage fails: after 200 fine-tuning steps where every candidate action is scored by an exact emulator rollout, the open-weight model's sampled-action quality does not improve, and its five-minute predictions of how actuator changes affect zone temperatures and CO2 remain at 0?2.1% direction accuracy, no better than chance. The paper's central claim is that exact action values are not enough: scalar rewards rank the actions a model already samples but neither enlarge the set of actions it can propose nor supply the action-conditioned transition knowledge needed to construct better ones. A reader should care because the result separates verifier accuracy from learning-signal completeness in a way that matters for any attempt to use reasoning models in continuous control.

What carries the argument

The central object is the direct rollout verifier: at a saved emulator state, apply one candidate action for one five-minute interval, then let the frozen TD3 teacher control the rest of the day with fixed weather and occupancy; the discounted return is that action's exact score, with zero sampling variance because the emulator is deterministic. This verifier is what exposes the learned critic's failure, and its scores are the only reward used in RFT. The complementary mechanism is the decomposition of the action-value gradient: the direction of improvement in actuator space factors into an immediate-reward gradient plus a preference over next states multiplied by the action-to-next-state Ja

What would settle it

Run the same 12-state, 36-pair counterfactual transition test on the open-weight model with reasoning enabled and with free-form output allowed before the structured predictions; if direction accuracy rises materially above chance, the transition-knowledge failure is an interface artifact rather than a training outcome. Alternatively, run RFT on an open-weight model that demonstrably passes the transition test (for example after transition SFT); if sampled-action return still shows no sustained gain, the missing piece is not transition knowledge.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a negative result with a precise mechanism. A learned critic that tracks true returns almost perfectly across time (r = 0.9998) can still rank actions badly within a fixed state; it selected the rollout-best candidate in only 5 of 10 test states. Switching to an exact rollout verifier removes approximation error but does not fix the learning signal: the gradient of the action value with respect to the action decomposes into an immediate-reward term plus a preference-over-next-states term times an action-to-next-state Jacobian, and the verifier supplies neither factor. In the tested protocol, the open-weight reasoning model could not express

Load-bearing premise

The load-bearing premise is that the counterfactual transition test, which forces numeric-only JSON output with zero reasoning tokens, validly measures whether the open-weight model has usable action-conditioned transition knowledge; if a format or sampling artifact is producing the 0?2.1% direction accuracy, the claim that RFT did not teach local dynamics is not established.

Editorial extensions

If this is right

  • Verifier audits should use within-state ranking of the policy's own sampled actions against ground-truth returns, not across-time correlation; near-perfect marginal correlation can hide unreliable ranking.
  • Group-relative RFT in a continuous action space inherits the quality of the candidate set: an exact verifier can reweight only the actions proposed, and if the model's sampler never proposes coordinated actions, training cannot create them.
  • A frontier reasoning model can coordinate seven coupled continuous actuators from a textual prompt without building-specific weight updates, but in this testbed the achieved energy saving came with a reduced ventilation margin, so constraint handling remains the open issue.
  • Exact scalar reward supervision does not by itself transfer local transition knowledge: direction accuracy stayed at chance after 200 RFT steps, so value-based RFT should be preceded or augmented by transition-prediction supervision.
  • The emulator's save-and-restore interface makes the natural next experiment concrete: compare base model, RFT only, transition SFT only, and transition SFT followed by RFT on transition error and closed-loop control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paper's diagnosis is right, the same rollout-verifier recipe that failed here could succeed once the student model can express transition predictions; the cheapest test is the paper's own proposed four-arm comparison, and success would reframe the negative result as a missing-pretraining problem rather than a flawed reward design.
  • The critic-audit lesson extends to any group-relative reward model: before spending a fine-tuning budget on a learned reward, audit its within-state ranking on a handful of states with ground-truth rollouts; the paper shows that a few hundred generations can settle whether the reward is usable.
  • A broader implication for LLM-based control is that scalar returns are evaluative feedback, not a substitute for a world model; pipelines that pair value-based fine-tuning with an explicit differentiable dynamics model or a transition head have a structural advantage this one lacked.
  • The interface itself may be part of the story: forcing numeric-only output with reasoning disabled could under-measure what the open-weight model knows, so an alternative test that allows free-form reasoning before structured predictions would separate knowledge-access failure from knowledge-acquisition failure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies whether GPT-5, a frontier reasoning model, can perform multi-zone VAV HVAC control from text alone, and whether TD3-guided rollout-verified reinforcement fine-tuning (RFT) can transfer such control ability to a local open-weight model, gpt-oss-20b. The authors first establish a capability reference: GPT-5 achieves slightly lower energy than a Guideline-36 baseline and a trained TD3 policy, although with reduced CO2 compliance. They then audit a TD3 critic as a possible verifier, find that its near-perfect across-time correlation (r=0.9998) masks poor within-state ranking, and select direct emulator rollouts as an exact verifier. RFT with this verifier over 200 fixed-seed steps shows no sustained improvement in sampled-action return gaps, and the open-weight controller remains worse than baseline in energy and comfort after training. A bespoke counterfactual transition test reports that gpt-oss-20b predicts the sign of one-actuator changes with 0–2.1% accuracy before and after RFT, while GPT-5 does so 37.2–97.7% of the time. The paper concludes that exact action values rank sampled actions but do not supply the action-conditioned transition knowledge needed to construct better actions, motivating transition-focused SFT.

Significance. If the findings hold, the paper makes a useful and non-obvious negative contribution: it separates verifier quality from proposal quality in RLVR-style continuous control and provides a concrete, reproducible emulator in which exact rollout scores still fail to improve a sampled policy. The critic audit is a valuable methodological caution — aggregate correlation with returns is the wrong acceptance test for a reward model, and the paper demonstrates this with a concrete example. The work is also carefully self-scoped: Section 6.5 explicitly acknowledges the single fixed-seed RFT run and the protocol-dependence of the transition test. The principal value is the clean empirical demonstration that a zeroth-order exact-value verifier, even when error-free, does not automatically transfer control knowledge when the student lacks a differentiable model or a good proposal distribution. However, the strength of the motivational conclusion depends on the validity of the transition-knowledge probe and on whether the negative RFT result is more than an artifact of one hyperparameter configuration.

major comments (2)
  1. [Section 5, Appendix F] The transition-knowledge diagnosis rests on a single restrictive interface: numeric JSON output, thinking disabled, and instructions to return only structured values. The reported 0.0–2.1% strict direction accuracy is consistent with the model emitting (nearly) identical predictions for the low and high branches — i.e., output degeneration rather than wrong transition knowledge. The paper does not report how often the model's low/high predictions differed, nor does it provide an interface-robustness variant (e.g., allowing a thought process, free-form sign judgment, or a forced-choice format). The limitation in §6.5 correctly says the model 'did not express' accurate predictions 'under this protocol,' but §6.4 states the conclusion without that qualifier and uses it to motivate transition SFT. Please add the distribution of prediction differences and at least one alternate interface chec
  2. [Sections 3.7 and 4.3] The title and abstract generalize to 'Exact Action Values Are Not Enough,' but the supporting RFT evidence is a single 200-step fixed-seed run with one reward temperature (τ=0.05) and specific Dr. GRPO settings. The limitation in §6.5 scopes the negative result, yet the conclusion in §6.4 is categorical. Please either add seed/config sensitivity (at least 2–3 seeds) or restrict the claim to 'under the reported protocol.' In addition, the claim that the update 'does not make high-return actions consistently more probable' is inferred only from return-gap curves (Fig. 10); report a direct measure of the policy's probability mass on high-return candidates (e.g., group-level selection frequency of the best action as a function of training step) to rule out a purely optimizer-related failure.
minor comments (6)
  1. [Table 3] The row labels 'gpt-osspre-RFT' and 'gpt-osspost-RFT' are missing spaces; should be 'gpt-oss pre-RFT' and 'gpt-oss post-RFT'.
  2. [Eq. (15)] The derivative expression combines a scalar V with a vector ∂s'/∂a; make the transpose/dot-product structure explicit for clarity.
  3. [Figure 2(b)] The legend entries 'candidate' and 'critic top-1' are cryptic; clarify in the caption that orange denotes the action ranked first by the critic in each state.
  4. [Section 4.3] The caption of Figure 10 says 'thin lines show the four-state means'; specify that these are means over the four prompt groups for the current optimizer step, not over training states.
  5. [Section 5] The statement that 'all three evaluated conditions produced zero reasoning tokens' is surprising given GPT-5 used minimal reasoning effort; state how this was verified (e.g., API token counts, no thought_process field).
  6. [Data availability] Given the paper's emphasis on exactness and reproducibility, consider releasing the emulator code and RFT/audit scripts alongside the data rather than 'on request'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the exact-verifier limitation follows from the verifier definition, while the negative result rests on independent empirical measurements; self-cited TES work is motivation only.

full rationale

The paper's central formal step (Sec. 6.4, Eq. 15) is a standard one-step expansion of the action-value gradient, and the claim that the rollout verifier 'supplies neither factor' follows directly from the verifier's definition (Eqs. 1-2) as a scalar return evaluated at sampled actions. This is an explicit derivation, not a hidden equivalence between input and output. The empirical support is independent of that definition: the 200-step return-gap curves (Sec. 4.3, Fig. 10), the critic audit (Sec. 2.2), and the counterfactual transition test (Sec. 5, Fig. 11) are all measured outcomes not fitted to the conclusion. The self-cited TES paper [32] is used as motivation and as a contrast case; the present negative result does not reduce to it, and no uniqueness theorem or ansatz is imported from the authors' prior work. The acknowledged limitations (Sec. 6.5; single 200-step fixed-seed run, protocol-specific transition test with zero reasoning tokens in Appendix F) are validity caveats--the transition interface may artifactually suppress gpt-oss-20b's predictions--but an interface artifact is a correctness/interpretation risk, not circularity. I find no step in which a prediction is, by the paper's own equations or a self-citation chain, identical to its input.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on hand-chosen reward and baseline settings, a deterministic uncalibrated emulator, and an ad hoc transition test. No new physical entities are introduced; the rollout verifier is a procedure, not an entity.

free parameters (5)
  • Shaped reward weights (omega_T, omega_B, omega_C, omega_V, omega_E) = 1.0, 2.0, 0.5, 10.0, 0.2
    Hand-chosen weights in Eq. (13) define the TD3 teacher, the verifier return, and therefore what RFT optimizes; the authors state the reward is not equivalent to the constrained objective.
  • Rollout softmax reward temperature tau = 0.05
    Eq. (5) converts exact returns into group-relative rewards; tau concentrates nearly all weight on the top sampled action and is fixed without sensitivity analysis.
  • G36 baseline ventilation settings = 950 ppm CO2 setpoint, 50 ppm deadband
    The rule baseline's deliberately conservative ventilation margin (Section 3.4) shifts both its energy use and CO2 compliance, affecting all relative comparisons.
  • RFT protocol = 200 optimizer steps, G=16, K=4, LoRA r=4, beta=0, LR 5e-5, seed 42
    The negative transfer result is conditional on this exact protocol and seed; no hyperparameter search or multi-seed variance is reported.
  • Emulator equipment constants = COP=4.0, fan/motor efficiency 0.62/0.88, pump efficiency 0.80
    Uncalibrated equipment-model constants set the electricity split in Eq. (11); the absolute energy savings numbers scale with them.
assumptions (5)
  • domain assumption The emulator is deterministic and state restoration reproduces the original trajectory to floating-point precision (Section 3.2, Eq. 9), so one rollout is an exact Q-value under Eq. (1).
    Exactness of the verifier rests on this; any hidden stochasticity or restoration error would break Eq. (2).
  • ad hoc to paper The shaped reward in Eq. (13) is an acceptable surrogate for the constrained objective in Eq. (7); the paper explicitly says it is 'not mathematically equivalent'.
    All improvement and violation trade-offs in TD3 and the verifier inherit these hand-chosen weights.
  • domain assumption Controllers observe only o(s), not the full state x(s); the problem is treated as a POMDP (Section 2.2).
    The partial-observability premise frames why critic extrapolation can be problematic and why exact rollouts use restored full states.
  • standard math Standard finite-horizon policy improvement (Eq. 4) justifies TD3-continuation scores; the paper notes finite samples cannot guarantee it holds.
    This is the basis for interpreting rollout returns as baseline-policy action values.
  • ad hoc to paper The five-minute numeric JSON transition test (Section 5, Appendix F) is a valid probe of transition knowledge rather than an output-format artifact.
    The conclusion that RFT did not teach local dynamics depends on this measurement instrument.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exact Action Values Are Not Enough: Rollout-Verified Reinforcement Fine-Tuning of a Reasoning Model for Multi-Zone VAV Control." pith.science (2026). https://pith.science/paper/I2U3UEJ5

@misc{pith2026260727914,
  author       = {Pith},
  title        = {Pith review of: Exact Action Values Are Not Enough: Rollout-Verified Reinforcement Fine-Tuning of a Reasoning Model for Multi-Zone VAV Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I2U3UEJ5}},
  note         = {Machine review of arXiv:2607.27914}
}
abstract

Multi-zone variable-air-volume control must balance thermal comfort, indoor air quality, and electricity use across several continuous actuators. Model predictive control and reinforcement learning are widely studied, but deployment typically requires building-specific modeling or training, limiting scalability. We first test whether a frontier reasoning model (an LLM trained to use additional inference-time computation) can achieve competitive VAV control from text without building-specific training. With that capability established, we then test whether TD3-guided reinforcement fine-tuning (RFT) can transfer control knowledge into a locally deployable open-weight model. Five controllers are evaluated over three summer days in a physics-based four-zone emulator. Relative to a Guideline 36-based baseline, TD3 reduced HVAC electricity by 4.5% while improving temperature and CO$_2$ compliance. Without building-specific training, GPT-5 achieved the largest reduction (6.2%) but reduced the ventilation margin. For RFT, deterministic rollouts restore a saved state, apply one candidate, and follow TD3 to score each action. Auditing a learned critic against these rollouts exposed a failure hidden by its near-perfect across-time correlation ($r=0.9998$): within-state ranking was unreliable; the critic selected the rollout-best candidate in only 5 of 10 states. Even with the rollout verifier, 200 RFT steps produced no sustained improvement in sampled-action return; the open-weight controller used more electricity than the baseline before and after training, and its five-minute predictions remained worse than persistence. GPT-5 predicted transitions far better. Exact rollout scores rank sampled actions but reveal neither next-state effects nor an improvement direction. The unchanged transition errors motivate transition-focused supervised fine-tuning before value-based RFT.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 10 canonical work pages

  1. [1]

    G. Shim, L. Song, and G. Wang, Comparison of different fan control strategies on a variable air volume systems through simulationsandexperiments, Build.Environ.72(2014)212–22. https://doi.org/10.1016/j.buildenv.2013.11.003

    work page doi:10.1016/j.buildenv.2013.11.003 2014

  2. [2]

    X. Lu, Z. Pang, Y. Fu, and Z. O’Neill, The nexus of the indoor CO2 concentration and ventilation demands underlying CO2-based demand-controlled ventilation in commercial buildings: A critical review, Build. Environ. 218 (2022) 109116. https://doi.org/10.1016/j.buildenv.2022.109116

    arXiv 2022

  3. [3]

    Xu and S

    X. Xu and S. Wang, An adaptive demand-controlled ventilation strategy with zone temperature reset for multi-zone air-conditioning systems, Indoor Built Environ. 16(5) (2007) 426–37.https://doi.org/10.1177/1420326X07082744

    work page doi:10.1177/1420326x07082744 2007

  4. [4]

    S. Shi, S. Miyata, and Y. Akashi, A hybrid multi-agent distributed optimal control strategy of multizone VAV systems for edge computing in smart buildings, Energy Build. 345 (2025) 116089.https://doi.org/10.1016/j.enbuild.2025.116089

    arXiv 2025

  5. [5]

    B. C. Ahn and J. W. Mitchell, Optimal control development for chilled water plants using a quadratic representation, Energy Build. 33 (2001) 371–8.https://doi.org/10.1016/S0378-7788(00)00119-5

    work page doi:10.1016/s0378-7788(00)00119-5 2001

  6. [6]

    Jetté, M

    I. Jetté, M. Zaheer-Uddin, and P. Fazio, PI-control of dual duct systems: Manual tuning and control loop interaction, Energy Convers. Manag. 39(14) (1998) 1471–82.https://doi.org/10.1016/S0196-8904(98)00020-X

    work page doi:10.1016/s0196-8904(98)00020-x 1998

  7. [7]

    5850–6.https://doi.org/10.24963/ijcai.2019/811

    M.Fiducioso,S.Curi,B.Schumacher,M.Gwerder,andA.Krause, SafecontextualBayesianoptimizationforsustainableroom temperature PID control tuning, in: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI 2019), 2019, pp. 5850–6.https://doi.org/10.24963/ijcai.2019/811

    doi:10.24963/ijcai.2019/811 2019

  8. [8]

    https://doi.org/10.3390/en11030495

    F.Behrooz,N.Mariun,M.H.Marhaban,M.A.MohdRadzi,andA.R.Ramli, ReviewofcontroltechniquesforHVACsystems— nonlinearityapproachesbasedonfuzzycognitivemaps, Energies11(2018)495. https://doi.org/10.3390/en11030495. 21 Rollout-Verified RFT for Multi-Zone VAV Control

    work page doi:10.3390/en11030495 2018

Show all 52 references
  1. [9]

    Yamamoto, S

    S. Yamamoto, S. Miyata, Y. Akashi, T. Sawachi, and M. Momota, Energy-saving effect of VAV, VWV and CO2 concentration control of air conditioning system considering automatic control logic and parameters, Part 1—Simulation construction and energy saving effect of VAV/VWV contro...

    2021 doi

  2. [10]

    ASHRAE, ASHRAEGuideline36-2024: High-PerformanceSequencesofOperationforHVACSystems, ASHRAE,Peachtree Corners, GA, 2024

    2024

  3. [11]

    Drgoňa, J

    J. Drgoňa, J. Arroyo, I. C. Figueroa, D. Blum, K. Arendt, D. Kim, et al., All you need to know about model predictive control for buildings, Annu. Rev. Control 50 (2020) 190–232.https://doi.org/10.1016/j.arcontrol.2020.09.001

    2020 doi

  4. [12]

    Saloux, J

    E. Saloux, J. A. Candanedo, C. Vallianos, N. Morovat, and K. Zhang, From theory to practice: A critical review of model predictive control field implementations in the built environment, Appl. Energy 393 (2025) 126091.https: //doi.org/10.1016/j.apenergy.2025.126091

    2025

  5. [13]

    Wang and T

    Z. Wang and T. Hong, Reinforcement learning for building controls: The opportunities and challenges, Appl. Energy 269 (2020) 115036.https://doi.org/10.1016/j.apenergy.2020.115036

    2020

  6. [14]

    Savino, G

    S. Savino, G. Razzano, M. Pagone, C. Novara, and A. Capozzoli, Deploying deep reinforcement learning for low-level HVAC control in multi-zone buildings: A comparative study with ASHRAE G36 sequences, Energy Build. 348 (2025) 116456. https://doi.org/10.1016/j.enbuild.2025.116456

    2025

  7. [15]

    Heidari, L

    A. Heidari, L. Girardin, C. Dorsaz, and F. Maréchal, A trustworthy reinforcement learning framework for autonomous control of a large-scale complex heating system: Simulation and field implementation, Appl. Energy 378 (2025) 124815. https://doi.org/10.1016/j.apenergy.2024.124815

    2025

  8. [16]

    Z. An, X. Ding, and W. Du, Go beyond black-box policies: Rethinking the design of learning agent for interpretable and verifiable HVAC control, in: Proceedings of the 61st ACM/IEEE Design Automation Conference (DAC 2024), 2024, Article 86, pp. 1–6.https://doi.org/10.1145/36493...

    2024

  9. [17]

    National Institute of Standards and Technology, AI-optimized building controls, https://www.nist.gov/ programs-projects/ai-optimized-building-controls(accessed 2026-07-24)

    2026

  10. [18]

    U.S. Department of Energy, Energy management and control systems workforce development roundtable, Building Technologies Office, April 2023, https://www.energy.gov/sites/default/files/2023-04/ bto-emcs-workforce-roundtable-040423.pdf(accessed 2026-07-24)

    2023

  11. [19]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, et al., Attention is all you need, in: Advances in Neural Information Processing Systems 30 (NeurIPS 2017), 2017, pp. 5998–6008

    2017

  12. [20]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, et al., Language models are few-shot learners, in: Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020, pp. 1877–1901

    2020

  13. [21]

    D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, et al., DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning, Nature 645 (2025) 633–8.https://doi.org/10.1038/s41586-025-09422-z

    2025 doi

  14. [22]

    Zhang and Z

    L. Zhang and Z. Chen, Large language model-based interpretable machine learning control in building energy systems, Energy Build. 313 (2024) 114278.https://doi.org/10.1016/j.enbuild.2024.114278

    2024

  15. [23]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, et al., DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, arXiv preprint arXiv:2402.03300 (2024).https://doi.org/10.48550/arXiv.2402.03300

    2024 doi arXiv

  16. [24]

    Mirshekali, M

    H. Mirshekali, M. R. Shadi, F. Ghanadi Ladani, and H. R. Shaker, A review of large language models for energy systems: Applications, challenges, and future prospects, IEEE Access 13 (2025) 163162–88.https://doi.org/10.1109/ACCESS. 2025.3610994

    2025

  17. [25]

    L. Song, C. Zhang, L. Zhao, and J. Bian, Pre-trained large language models for industrial control, arXiv preprint arXiv:2308.03028 (2023).https://doi.org/10.48550/arXiv.2308.03028

    2023 doi arXiv

  18. [26]

    K. U. Ahn, D.-W. Kim, H. M. Cho, and C.-U. Chae, Alternative approaches to HVAC control of Chat Generative Pre- Trained Transformer (ChatGPT) for autonomous building system operations, Buildings 13(11) (2023) 2680.https: //doi.org/10.3390/buildings13112680

    2023 doi

  19. [27]

    Sawada, M

    T. Sawada, M. Mizuno, T. Hasegawa, K. Yokoyama, and M. Kono, Office-in-the-loop: An investigation into agentic AI for advancedbuildingHVACcontrolsystems, Data-CentricEng.6(2025)e31. https://doi.org/10.1017/dce.2025.10010

    2025

  20. [28]

    K.Bhatt,X.Lin,M.Amos,F.D.Salim,andW.Hu, ThermoLLM:Thermodynamics-awareHVACcontrolwithspatial-semantic knowledge graph, arXiv preprint arXiv:2606.22911 (2026).https://doi.org/10.48550/arXiv.2606.22911

    2026 doi arXiv

  21. [29]

    Ko and R

    Y.-D. Ko and R. K. Jain, DARLIN: Domain-guided augmented retrieval for LLM-based interpretable HVAC control, in: Proceedings of the 12th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys ’25), 2025, pp. 440–3.https://d...

    2025

  22. [30]

    Zhong, T

    D. Zhong, T. Xing, K. Sun, X. Yang, H. Huang, I. Qaisar, et al., Hierarchical control framework integrating LLMs with RL for decarbonized HVAC operation, arXiv preprint arXiv:2603.26050 (2026).https://doi.org/10.48550/arXiv.2603. 26050. 22 Rollout-Verified RFT for Multi-Zone V...

    2026 doi

  23. [31]

    Z.Liu,C.Chen, W.Li,P.Qi, T.Pang, C.Du, etal., UnderstandingR1-Zero-liketraining: Acriticalperspective, arXivpreprint arXiv:2503.20783 (2025).https://doi.org/10.48550/arXiv.2503.20783

    2025 doi arXiv

  24. [32]

    T.Shioda,K.Terashima, andT.Nagai, Verifier-basedreinforcementfine-tuningofreasoningmodelsforthermalenergystorage control, arXiv preprint arXiv:2607.12856 (2026).https://doi.org/10.48550/arXiv.2607.12856

    2026 doi arXiv

  25. [33]

    Y. Mroueh, Reinforcement learning with verifiable rewards: GRPO’s effective loss, dynamics, and success amplification, arXiv preprint arXiv:2503.06639 (2025).https://doi.org/10.48550/arXiv.2503.06639

    2025 doi

  26. [34]

    Ichter, A

    B. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, et al., Do as I can, not as I say: Grounding language in robotic affordances, in: Proceedings of the 6th Conference on Robot Learning, PMLR 205, 2023, pp. 287–318

    2023

  27. [35]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, et al., LoRA: Low-rank adaptation of large language models, in: International Conference on Learning Representations (ICLR 2022), 2022

    2022

  28. [36]

    Fujimoto, D

    S. Fujimoto, D. Meger, and D. Precup, Off-policy deep reinforcement learning without exploration, in: Proceedings of the 36th International Conference on Machine Learning (ICML 2019), PMLR 97, 2019, pp. 2052–62

    2019

  29. [37]

    Kumar, A

    A. Kumar, A. Zhou, G. Tucker, and S. Levine, Conservative Q-learning for offline reinforcement learning, in: Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020, pp. 1179–91

    2020

  30. [38]

    Tesauro and G

    G. Tesauro and G. R. Galperin, On-line policy improvement using Monte-Carlo search, in: Advances in Neural Information Processing Systems 9 (NIPS 1996), 1997, pp. 1068–74

    1996

  31. [39]

    R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, second ed., MIT Press, Cambridge, MA, 2018

    2018

  32. [40]

    Ministry of Health, Labour and Welfare, Japan, Building environmental hygiene management standards (in Japanese), https://www.mhlw.go.jp/bunya/kenkou/seikatsu-eisei10/index.html(accessed 2026-07-23)

    2026

  33. [41]

    Miyata, S

    S. Miyata, S. Shi, Y. Akashi, and T. Sawachi, Phyvac: A Python module for highly flexible HVAC system simulation, and fault dataset generation as an application example, in: Proceedings of the 18th IBPSA Conference (Building Simulation 2023), Shanghai, China, 2023, pp. 915–22....

    2023

  34. [42]

    Japan Meteorological Agency, Past Weather Data Search: 10-minute observations at Tokyo station 47662, 28–30 July 2025, https://www.data.jma.go.jp/stats/etrn/(accessed 2026-07-18)

    2025

  35. [43]

    Fujimoto, H

    S. Fujimoto, H. van Hoof, and D. Meger, Addressing function approximation error in actor-critic methods, in: Proceedings of the 35th International Conference on Machine Learning (ICML 2018), PMLR 80, 2018, pp. 1587–96

    2018

  36. [44]

    OpenAI, GPT-5 system card, 2025,https://openai.com/index/gpt-5-system-card/(accessed 2026-07-30)

    2025

  37. [45]

    OpenAI, gpt-oss-120b & gpt-oss-20b model card, arXiv preprint arXiv:2508.10925 (2025).https://doi.org/10.48550/ arXiv.2508.10925

    2025 doi arXiv

  38. [46]

    Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, et al., Reasoning models don’t always say what they think, arXiv preprint arXiv:2505.05410 (2025).https://doi.org/10.48550/arXiv.2505.05410

    2025 doi arXiv

  39. [47]

    S. Hao, Y. Gu, H. Ma, J. Hong, Z. Wang, D. Wang, et al., Reasoning with language model is planning with world model, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 8154–73. https://doi.org/10.18653/v1/2023.emnlp-main.507

    2023 doi

  40. [48]

    Silver, G

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, Deterministic policy gradient algorithms, in: Proceedings of the 31st International Conference on Machine Learning (ICML 2014), PMLR 32, 2014, pp. 387–95

    2014

  41. [49]

    Hafner, T

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, Dream to control: Learning behaviors by latent imagination, in: International Conference on Learning Representations (ICLR 2020), 2020

    2020

  42. [50]

    P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg, DayDreamer: World models for physical robot learning, in: Proceedings of the 6th Conference on Robot Learning (CoRL 2022), PMLR 205, 2023, pp. 2226–40

    2022

  43. [51]

    Xiang, T

    J. Xiang, T. Tao, Y. Gu, T. Shu, Z. Wang, Z. Yang, et al., Language models meet world models: Embodied experiences enhance language models, in: Advances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023, pp. 75392–412. https://doi.org/10.52202/075280-3295

    2023 doi

  44. [52]

    we don’t have dynamic model. We’ll assume it’s okay

    K. Xie, I. Yang, J. Gunerli, and M. Riedl, Making large language models into world models with precondition and effect knowledge, in: Proceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 7532–45. 23 Rollout-Verified RFT for Multi-Zone VAV Co...

    2025

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.