{"id":"03715011-578a-4752-a1a8-09c9214caaa4","arxiv_id":"2607.27914","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Exact rollout-based action scores did not improve an open-weight LLM controller on a VAV HVAC simulator because the model lacked action-to-next-state prediction knowledge.","lead":"This paper compares five ways to control a multi-room air-conditioning system, including a frontier language model and an open-weight model trained with exact action scores. The frontier model saved some electricity while cutting ventilation margins, and score-based fine-tuning failed because the smaller model could not predict what its commands would do next.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transition test interface may artifactually suppress gpt-oss-20b's transition knowledge; central claim overreaches without an interface-robustness check.","rationale":"The reader's weakest assumption identified the Section 5 transition test as the load-bearing premise: the forced numeric-JSON, zero-reasoning-token interface may artifactually suppress gpt-oss-20b's transition knowledge. My reading agrees. The strongest claim ('exact action values are not enough') is well-supported by the Section 4.3 return-gap curves and the critic audit (Section 2.2), which are sound as far as they go: the negative RFT result is honestly limited to one 200-step fixed-seed run (Section 3.7, acknowledged in Section 6.5). However, the paper's central conceptual conclusion—that exact values lack directional content because the student lacks action-conditioned transition knowledge—rests on the transition test. The test's interface is coercive: it explicitly forbids analysis, disables thinking for gpt-oss-20b, demands pure numeric JSON, and the paper notes zero reasoning tokens. This is exactly the setup where an open-weight model trained with instruction-following may produce degenerate or default outputs rather than its best prediction. The evidence in Appendix D shows gpt-oss-20b does attempt transition reasoning in its control channel ('But would that maintain temperature? Hard to know')—it is uncertain but not silent. The near-zero direction accuracy (0.0-2.1%) is so extreme that an output-format artifact is a plausible alternative explanation. The paper's own limitation language ('did not express accurate local transition predictions' under this protocol) concedes the gap. A simple interface-robustness check—allowing the model's native reasoning or a verbal-response variant—would settle whether the knowledge is absent or merely suppressed. If the concern lands, the central claim should be weakened to 'under this protocol, exact values did not transfer transition knowledge' rather than the broader 'exact action values are not enough' claim. If the concern is resolved, the paper's CONDITIONAL verdict could be upgraded to ACCEPT for the empirical finding, though the single-seed RFT run remains a separate scoped limitation. The reader's verdict of CONDITIONAL is appropriate; I would keep it because the concern is specific and testable, not because the paper is flawed in its stated scope.","tokens_in":31657,"tokens_out":2036,"duration_ms":18238,"concrete_test":"Run the Section 5 counterfactual test with gpt-oss-20b in its normal control-style interface: allow a thought_process/analysis channel, or use the same numeric JSON but with thinking enabled and intermediate reasoning excluded from scoring, or add a verbal-prediction variant ('will temperature increase or decrease?') alongside the numeric JSON. If direction accuracy rises from near-zero to chance-level or above (e.g., >30%) in any variant, the original numeric-only test is an interface artifact and the paper's central claim needs to be re-scoped. If accuracy remains near zero across all variants, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that exact rollout values are not enough because they supply no transition knowledge—depends on Section 5's counterfactual transition test showing that gpt-oss-20b 'did not express useful local transition predictions' (0.0–2.1% direction accuracy pre/post-RFT). The test forces the model to output only numeric JSON with thinking disabled and strict instructions to return only structured values. The paper notes all three conditions produced zero reasoning tokens. This is an unusually strong interface constraint: for gpt-oss-20b, the control task normally permits a thought_process channel, and its qualitative logs in Section 4.2 show it articulating physical reasoning (albeit with 'hard to know' hedges). The near-zero accuracy is suspiciously extreme—below chance for a 2-way sign test on 36-48 comparisons, which would require systematic anti-correlation or output degeneracy. The paper's own limitation (Section 6.5) says the test shows the model 'did not express accurate local transition predictions' under this protocol, not that it lacks the knowledge. If the failure is an interface artifact—e.g., the model emits a default or degenerate JSON when deprived of its reasoning channel—then the conclusion that RFT 'did not teach local dynamics' is under-supported, and the central claim (Section 6.4) that exact action values are 'not enough' is weakened because a different training signal or interface might expose the transition knowledge. The claim as stated is about what exact values can supply, and the empirical support hinges on this single, possibly confounded measurement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether GPT-5, a frontier reasoning model, can perform multi-zone VAV HVAC control from text alone, and whether TD3-guided rollout-verified reinforcement fine-tuning (RFT) can transfer such control ability to a local open-weight model, gpt-oss-20b. The authors first establish a capability reference: GPT-5 achieves slightly lower energy than a Guideline-36 baseline and a trained TD3 policy, although with reduced CO2 compliance. They then audit a TD3 critic as a possible verifier, find that its near-perfect across-time correlation (r=0.9998) masks poor within-state ranking, and select direct emulator rollouts as an exact verifier. RFT with this verifier over 200 fixed-seed steps shows no sustained improvement in sampled-action return gaps, and the open-weight controller remains worse than baseline in energy and comfort after training. A bespoke counterfactual transition test reports that gpt-oss-20b predicts the sign of one-actuator changes with 0–2.1% accuracy before and after RFT, while GPT-5 does so 37.2–97.7% of the time. The paper concludes that exact action values rank sampled actions but do not supply the action-conditioned transition knowledge needed to construct better actions, motivating transition-focused SFT.","tokens_in":32057,"tokens_out":7967,"duration_ms":80980,"significance":"If the findings hold, the paper makes a useful and non-obvious negative contribution: it separates verifier quality from proposal quality in RLVR-style continuous control and provides a concrete, reproducible emulator in which exact rollout scores still fail to improve a sampled policy. The critic audit is a valuable methodological caution — aggregate correlation with returns is the wrong acceptance test for a reward model, and the paper demonstrates this with a concrete example. The work is also carefully self-scoped: Section 6.5 explicitly acknowledges the single fixed-seed RFT run and the protocol-dependence of the transition test. The principal value is the clean empirical demonstration that a zeroth-order exact-value verifier, even when error-free, does not automatically transfer control knowledge when the student lacks a differentiable model or a good proposal distribution. However, the strength of the motivational conclusion depends on the validity of the transition-knowledge probe and on whether the negative RFT result is more than an artifact of one hyperparameter configuration.","major_comments":[{"comment":"The transition-knowledge diagnosis rests on a single restrictive interface: numeric JSON output, thinking disabled, and instructions to return only structured values. The reported 0.0–2.1% strict direction accuracy is consistent with the model emitting (nearly) identical predictions for the low and high branches — i.e., output degeneration rather than wrong transition knowledge. The paper does not report how often the model's low/high predictions differed, nor does it provide an interface-robustness variant (e.g., allowing a thought process, free-form sign judgment, or a forced-choice format). The limitation in §6.5 correctly says the model 'did not express' accurate predictions 'under this protocol,' but §6.4 states the conclusion without that qualifier and uses it to motivate transition SFT. Please add the distribution of prediction differences and at least one alternate interface chec","section":"Section 5, Appendix F"},{"comment":"The title and abstract generalize to 'Exact Action Values Are Not Enough,' but the supporting RFT evidence is a single 200-step fixed-seed run with one reward temperature (τ=0.05) and specific Dr. GRPO settings. The limitation in §6.5 scopes the negative result, yet the conclusion in §6.4 is categorical. Please either add seed/config sensitivity (at least 2–3 seeds) or restrict the claim to 'under the reported protocol.' In addition, the claim that the update 'does not make high-return actions consistently more probable' is inferred only from return-gap curves (Fig. 10); report a direct measure of the policy's probability mass on high-return candidates (e.g., group-level selection frequency of the best action as a function of training step) to rule out a purely optimizer-related failure.","section":"Sections 3.7 and 4.3"}],"minor_comments":[{"comment":"The row labels 'gpt-osspre-RFT' and 'gpt-osspost-RFT' are missing spaces; should be 'gpt-oss pre-RFT' and 'gpt-oss post-RFT'.","section":"Table 3"},{"comment":"The derivative expression combines a scalar V with a vector ∂s'/∂a; make the transpose/dot-product structure explicit for clarity.","section":"Eq. (15)"},{"comment":"The legend entries 'candidate' and 'critic top-1' are cryptic; clarify in the caption that orange denotes the action ranked first by the critic in each state.","section":"Figure 2(b)"},{"comment":"The caption of Figure 10 says 'thin lines show the four-state means'; specify that these are means over the four prompt groups for the current optimizer step, not over training states.","section":"Section 4.3"},{"comment":"The statement that 'all three evaluated conditions produced zero reasoning tokens' is surprising given GPT-5 used minimal reasoning effort; state how this was verified (e.g., API token counts, no thought_process field).","section":"Section 5"},{"comment":"Given the paper's emphasis on exactness and reproducibility, consider releasing the emulator code and RFT/audit scripts alongside the data rather than 'on request'.","section":"Data availability"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and unusually well-scoped for a negative result, and the critic audit is a genuine methodological contribution. The main risk is the transition-test interface: if the near-zero direction accuracy is a degenerate-output artifact, the 'transition knowledge' diagnosis and the SFT recommendation lose their empirical footing. The authors can address this with a relatively small additional analysis (prediction-difference distribution plus an alternate interface). The single-seed RFT is a second, acknowledged fragility; adding seeds would materially strengthen the general claim made in the title. I would not reject the paper on these grounds, but the revision should either provide the requested diagnostics or explicitly narrow the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. The paper is a careful negative result about rollout-verified RFT for LLM-based HVAC control, and the sharpest contribution is the critic audit: a learned value function with across-time correlation r=0.9998 can still pick the rollout-best action in only 5 of 10 states when you look within-state. That is a clean, transferable warning for anyone using a critic as a reward model in group-relative RL. The rest of the study—GPT-5 doing decent zero-shot VAV control, gpt-oss-20b failing to improve after 200 steps of exact-rollout RFT, and a counterfactual transition probe showing the model can't predict next-state effects—hangs together as evidence that exact scalar values rank sampled actions without supplying action-space direction. Eq. (15) formalizes that point nicely.\n\nThe transparency is a real plus: the paper states its single fixed-seed run, uncalibrated emulator, and three-day evaluation as limitations, and the negative result is reported with return-gap curves rather than a single aggregate number.\n\nSoft spots, in order of softness. First, the central claim rests on one 200-step RFT run. That is acknowledged, but it means \"exact action values are not enough\" is really \"this configuration did not improve within 200 steps.\" A second seed or a slightly different softmax temperature would help a lot. Second, the transition test forces pure numeric JSON with thinking off; gpt-oss-20b scores near 0% direction accuracy, which is exactly what a model that outputs the same values for both branches would do. The paper's own phrasing (\"did not express\") is honest, but the discussion sometimes reads as if the model lacks the knowledge, when the test shows only that it can't express it in that format. The persistence baseline being better than the model's MAE suggests it is actively guessing wrong rather than staying put, so the result is not purely an artifact, but an interface-robustness check would settle it. Third, the GPT-5 vs gpt-oss comparison is illustrative, not controlled, and the paper says so.\n\nWho it's for: people building RLVR pipelines for continuous physical control, LLM-as-agent folks, and the buildings/HVAC community. It deserves a serious referee. I would send it out and in revision ask for either a multi-seed RFT or a clear hyperparameter sensitivity note, and a transition-probe variant that lets the model produce reasoning or an alternative output schema. The critic audit alone is worth preserving in the record.","headline":"Careful negative result: exact rollout values rank actions but don't teach the model to construct better ones; the critic audit is the cleanest finding, and the transition-test interface is the main soft spot.","tokens_in":32519,"tokens_out":5138,"would_cite":true,"duration_ms":44889,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Exact action values are not enough to teach a reasoning model how VAV commands change the building state.","keywords":["multi-zone VAV control","reinforcement fine-tuning","rollout verification","action values","transition model","reasoning models","HVAC control","continuous control"],"falsifier":"Run the same 12-state, 36-pair counterfactual transition test on the open-weight model with reasoning enabled and with free-form output allowed before the structured predictions; if direction accuracy rises materially above chance, the transition-knowledge failure is an interface artifact rather than a training outcome. Alternatively, run RFT on an open-weight model that demonstrably passes the transition test (for example after transition SFT); if sampled-action return still shows no sustained gain, the missing piece is not transition knowledge.","tokens_in":1614,"feed_emoji":"🏢","tokens_out":2418,"duration_ms":81970,"temperature":0.7,"pith_summary":"The paper tests a two-stage idea: first, whether a frontier reasoning model can control a four-zone variable-air-volume HVAC system from text alone without building-specific training, and second, whether TD3-guided reinforcement fine-tuning (RFT) with an exact rollout verifier can transfer that ability into a smaller open-weight model. The first stage works as a capability demonstration: the frontier model cut three-day HVAC electricity by 6.2% relative to a rule-based baseline, though partly by narrowing the CO2 margin. The second stage fails: after 200 fine-tuning steps where every candidate action is scored by an exact emulator rollout, the open-weight model's sampled-action quality does not improve, and its five-minute predictions of how actuator changes affect zone temperatures and CO2 remain at 0?2.1% direction accuracy, no better than chance. The paper's central claim is that exact action values are not enough: scalar rewards rank the actions a model already samples but neither enlarge the set of actions it can propose nor supply the action-conditioned transition knowledge needed to construct better ones. A reader should care because the result separates verifier accuracy from learning-signal completeness in a way that matters for any attempt to use reasoning models in continuous control.","feed_headline":"Exact action scores can't teach an open-weight VAV controller","feed_subtitle":"Rollout-verified training left 5-minute transition forecasts at chance; exact scores rank actions but don't explain next states.","key_machinery":"The central object is the direct rollout verifier: at a saved emulator state, apply one candidate action for one five-minute interval, then let the frozen TD3 teacher control the rest of the day with fixed weather and occupancy; the discounted return is that action's exact score, with zero sampling variance because the emulator is deterministic. This verifier is what exposes the learned critic's failure, and its scores are the only reward used in RFT. The complementary mechanism is the decomposition of the action-value gradient: the direction of improvement in actuator space factors into an immediate-reward gradient plus a preference over next states multiplied by the action-to-next-state Ja","core_discovery":"On the paper's own terms, the central discovery is a negative result with a precise mechanism. A learned critic that tracks true returns almost perfectly across time (r = 0.9998) can still rank actions badly within a fixed state; it selected the rollout-best candidate in only 5 of 10 test states. Switching to an exact rollout verifier removes approximation error but does not fix the learning signal: the gradient of the action value with respect to the action decomposes into an immediate-reward term plus a preference-over-next-states term times an action-to-next-state Jacobian, and the verifier supplies neither factor. In the tested protocol, the open-weight reasoning model could not express","pith_inferences":["If the paper's diagnosis is right, the same rollout-verifier recipe that failed here could succeed once the student model can express transition predictions; the cheapest test is the paper's own proposed four-arm comparison, and success would reframe the negative result as a missing-pretraining problem rather than a flawed reward design.","The critic-audit lesson extends to any group-relative reward model: before spending a fine-tuning budget on a learned reward, audit its within-state ranking on a handful of states with ground-truth rollouts; the paper shows that a few hundred generations can settle whether the reward is usable.","A broader implication for LLM-based control is that scalar returns are evaluative feedback, not a substitute for a world model; pipelines that pair value-based fine-tuning with an explicit differentiable dynamics model or a transition head have a structural advantage this one lacked.","The interface itself may be part of the story: forcing numeric-only output with reasoning disabled could under-measure what the open-weight model knows, so an alternative test that allows free-form reasoning before structured predictions would separate knowledge-access failure from knowledge-acquisition failure."],"forward_implications":["Verifier audits should use within-state ranking of the policy's own sampled actions against ground-truth returns, not across-time correlation; near-perfect marginal correlation can hide unreliable ranking.","Group-relative RFT in a continuous action space inherits the quality of the candidate set: an exact verifier can reweight only the actions proposed, and if the model's sampler never proposes coordinated actions, training cannot create them.","A frontier reasoning model can coordinate seven coupled continuous actuators from a textual prompt without building-specific weight updates, but in this testbed the achieved energy saving came with a reduced ventilation margin, so constraint handling remains the open issue.","Exact scalar reward supervision does not by itself transfer local transition knowledge: direction accuracy stayed at chance after 200 RFT steps, so value-based RFT should be preceded or augmented by transition-prediction supervision.","The emulator's save-and-restore interface makes the natural next experiment concrete: compare base model, RFT only, transition SFT only, and transition SFT followed by RFT on transition error and closed-loop control."],"fun_headline_variants":["Exact action scores can't fix within-state ranking in VAV","A near-perfect critic still ranks actions wrong in VAV","Rollout-verified RL leaves VAV transition forecasts at chance","Open-weight VAV controller gets no benefit from exact rollouts","Exact rollout values don't explain next states in VAV control"],"cache_read_input_tokens":33792,"weakest_assumption_plain":"The load-bearing premise is that the counterfactual transition test, which forces numeric-only JSON output with zero reasoning tokens, validly measures whether the open-weight model has usable action-conditioned transition knowledge; if a format or sampling artifact is producing the 0?2.1% direction accuracy, the claim that RFT did not teach local dynamics is not established.","fun_headline_variants_meta":{"raw":{"variants":["Exact action scores can't fix within-state ranking in VAV","A near-perfect critic still ranks actions wrong in VAV","Rollout-verified RL leaves VAV transition forecasts at chance","Open-weight VAV controller gets no benefit from exact rollouts","Exact rollout values don't explain next states in VAV control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000301,"raw_usage":{"total_tokens":1632,"prompt_tokens":863,"completion_tokens":769,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":691}},"tokens_in":607,"tokens_out":769,"duration_ms":7238,"temperature":1.0,"reasoning_tokens":691,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:06:09.001185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 12-state, 36-pair counterfactual transition test on the open-weight model with reasoning enabled and with free-form output allowed before the structured predictions; if direction accuracy rises materially above chance, the transition-knowledge failure is an interface artifact rather than a training outcome. Alternatively, run RFT on an open-weight model that demonstrably passes the transition test (for example after transition SFT); if sampled-action return still shows no sustained gain, the missing piece is not transition knowledge.","supporting_citations":[],"review_version":1}