{"id":"7bb862ed-6f04-4c13-a8e2-447e78abb324","arxiv_id":"2505.08988","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"In monitored MDPs, a deep reward model plus Q-learning can generalize to unmonitored states and reach near-optimal behavior, but can also overgeneralize; ensemble-based cautious policies reduce that overgeneralization.","lead":"This paper tests whether function approximation lets reinforcement learning agents act well when rewards are only sometimes observable. It shows a learned reward model can generalize across monitored and unmonitored regions, but can also over-extrapolate, and that robust policy optimization reduces that overgeneralization.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Near-optimal claim lacks an optimality baseline; the half-room result only compares Zone 1 and Zone 2, so the 'unsolvable environments' claim is an inductive guess rather than a demonstrated solution.","rationale":"The paper's headline result is that function approximation lets agents 'achieve near-optimal policies in environments formally defined as unsolvable.' In Section 4.2, the only evidence is that the reward-model policy earns comparable rewards in Zone 1 and Zone 2; there is no oracle, dynamic-programming, or hand-coded optimal baseline. Because the half-room Mon-MDP is unsolvable with respect to the agent's information - multiple reward functions agree on the monitored zone and differ in the unmonitored zone - any policy the agent learns is necessarily one possible guess. The experimenter chose the true reward, so a policy that happens to match it does not demonstrate that the unsolvability barrier is overcome; it demonstrates that the CNN's inductive bias selected the right reward function for this particular environment. The paper's own cacti experiment shows the same mechanism can select the wrong reward function when the true reward differs. The proposed flipped-reward variant would settle whether the agent truly generalized the correct reward structure or merely imposed a visual-similarity prior. I do not think this concern overturns the paper's empirical demonstrations of generalization and overgeneralization, so conditional acceptance remains appropriate; the reader's chosen weakest assumption, visual similarity, is related but not identical to the missing optimality baseline, which is why I partially agree.","tokens_in":16860,"tokens_out":5695,"duration_ms":62524,"concrete_test":"Run the half-room environment with the true reward for watering Zone 2 plants flipped to -1, keeping all other rewards and Zone 1 identical, and train the reward-model agent under the paper's protocol; if the agent continues to water Zone 2 plants at the end of training, the generalization is imposing a visual-similarity prior rather than solving the unsolvable Mon-MDP, and the near-optimal claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that function approximation plus a learned reward model 'achieves near-optimal policies in environments formally defined as unsolvable.' In the half-room experiment (Section 4.2), the only evidence is that the reward-model policy accumulates comparable discounted rewards in Zone 1 and Zone 2, and that the agent waters plants in both zones. No comparison is made to the true optimal return of the underlying MDP, which the experimenter knows exactly (watering dry plants gives +1, watering wet plants gives -1, watering empty cells gives -0.2). Without such a baseline, 'near-optimal' is an interpretation, not a measurement. This is load-bearing because the half-room environment is formally unsolvable: there exist two reward functions that are identical on monitored Zone 1 but assign opposite rewards to watering plants in Zone 2. The agent cannot distinguish these from its observations, so any policy that performs well in the experimenter's chosen environment is selecting one of many indistinguishable reward functions by way of the CNN's inductive bias. The cacti experiment (Section 4.3) demonstrates exactly the failure mode: the same generalization mechanism waters cacti despite their distinct representation and the -1 penalty they incur in the unmonitored zone. Thus the central claim needs either an explicit optimality gap or an adversarial indistinguishability check; otherwise it conflates successful inductive guessing with overcoming the unsolvability barrier.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies monitored Markov decision processes (Mon-MDPs) beyond the tabular case. It trains a convolutional reward model on observed rewards and a DQN on predicted rewards, and compares against baselines that treat unobserved rewards as zero or discard those transitions. Experiments cover a binary monitor, a half-room with one monitored half, a plant/cactus environment, and a botanical garden where an ensemble reward model with k-of-N CFR is used for cautious policy optimization. The paper claims that function approximation plus a learned reward model generalizes from monitored to unmonitored states, achieves near-optimal policies in some formally unsolvable Mon-MDPs, and that cautious policy optimization mitigates the overgeneralization exhibited by non-robust function approximation.","tokens_in":17111,"tokens_out":4998,"duration_ms":49718,"significance":"The question is significant: whether function approximation can overcome the tabular indistinguishability barrier in Mon-MDPs is an important step toward practical applications. The paper has concrete strengths: experiments use multiple seeds, hyperparameter tuning is reported, an explicit failure case is studied, and code is included with the submission. The cautious-policy experiments are a useful first step. However, the headline claims on near-optimality and on overcoming formal unsolvability are not backed by the reported measurements, and an appendix result appears to contradict the main-text convergence claim. If the near-optimality evidence is supplied, the contribution would be valuable.","major_comments":[{"comment":"The claim that the reward model achieves near-optimal policies in the half-room environment is not measured against any optimality baseline. The environment reward structure is fully known: watering dry or partially dry plants gives +1, watering wet plants gives -1, watering empty cells gives -0.2, and there is no monitoring penalty in this environment. An optimal return or a strong upper bound is therefore computable. The figure only shows that discounted rewards in Zone 1 and Zone 2 are comparable, which demonstrates consistency across zones but not proximity to optimal. This is load-bearing because the abstract and Contribution 2 assert near-optimality in formally unsolvable Mon-MDPs.","section":"Section 4.2, Figure 4a"},{"comment":"The formal definition of an unsolvable Mon-MDP is that no policy is optimal for all indistinguishable reward functions. The half-room experiment does not test this definition: it fixes one reward function for Zone 2 and shows that the CNN propagates Zone 1 rewards to Zone 2. The claim that FA enables agents to navigate some unsolvable environments conflates successful inductive guessing on one reward function with overcoming the formal barrier. The paper should either include an adversarial indistinguishability check, varying the Zone 2 reward functions and reporting regret against the optimal policy for each variant, or explicitly reframe the contribution as showing that FA can generalize on some instances of unsolvable Mon-MDPs.","section":"Section 4.2 and Section 2.2"},{"comment":"Appendix A.4 states that in the binary environment, after 6M timesteps the agent's performance deteriorates and it starts requesting monitoring again, as shown in Figure 8. The main text in Section 4.1 and Figure 3a shows training to 10M timesteps with stable high discounted reward and claims that the reward model empirically converges to the optimal policy. These two presentations are inconsistent. If performance collapses with longer training, the near-optimal claim is only a stopping-time artifact, and the manuscript must report the exact training duration used in the main curves and explain the discrepancy.","section":"Appendix A.4, Figure 8, and Section 4.1, Figure 3a"},{"comment":"The cautious-policy result is partly by construction: k-of-N CFR optimizes against the worst k of N reward models, so a policy trained this way is expected to reduce actions in states where the ensemble is uncertain. The interesting finding is that robust policies maintain or increase watering of familiar plants while reducing novel-plant watering, but the main text reports only ratios relative to the reward model. Please present the absolute watering frequencies in the main text (Table 5 in the appendix already contains them) and include the ensemble-mean baseline without worst-case selection (the 10-of-10 result in the appendix) so that the effect of ensembling can be separated from the effect of worst-case robustness.","section":"Section 4.4, Table 1"}],"minor_comments":[{"comment":"The phrase 'cautious police optimization' should be 'cautious policy optimization'.","section":"Abstract"},{"comment":"The sentence 'the success of reward model generalization depends critically on the: i) state representation' contains a misplaced colon before the list; rephrase for clarity.","section":"Section 3"},{"comment":"The environment is called 'plant-cactus' in the text and 'plants & cacti' in the figure caption; use one consistent name.","section":"Section 4.3 and Figure 4b"},{"comment":"The table reports area under the training curve for hyperparameter tuning, but the main text never defines what a 'training curve' is for the half-room environment; please add a one-sentence definition.","section":"Appendix A.1, Table 4"},{"comment":"The description of the novel-plant representations and the reporting in Table 1 are clear, but the sentence in Section 4.4 about novel plants being 'sampled from {[0,0,1], ...}' is not followed by the distribution used for the experiments; please state whether each novel plant type is sampled uniformly at the start of each episode.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is competently executed and the negative result with cacti is a strength. The main risk is overclaiming: the headline contributions are not supported by the reported metrics. I recommend requesting the optimality baseline and the indistinguishability experiment before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is empirical: the paper shows that a learned reward model with a CNN can transfer reward predictions from monitored to unmonitored zones of a grid-world, and it documents a concrete failure mode where that transfer misfires on visually distinct objects (cacti). That combination—both the promise and the failure of FA in Mon-MDPs—is new relative to the tabular Mon-MDP literature. The cautious-policy mitigation via k-of-N CFR is sensible and does reduce undesirable watering of novel plants. The authors run multiple seeds, tune hyperparameters, report confidence intervals, and ship code. That is solid, reproducible empirical work.\n\nThe soft spot is the central claim. The paper says the reward model achieves 'near-optimal policies in environments formally defined as unsolvable.' In the half-room experiment, the evidence is that rewards in Zone 2 are comparable to Zone 1, and the agent waters plants in both zones. But the true optimal return of that MDP is known to the experimenter: watering dry plants gives +1, wet plants -1, empty floor -0.2. Without a comparison to that optimal baseline, 'near-optimal' is an interpretation, not a measurement. The cacti experiment undercuts the claim further: the same generalization mechanism waters cacti in the unmonitored zone, which is clearly suboptimal. So the 'unsolvable environments' claim should be restated as something like 'FA can pick out a plausible reward function, and sometimes it is the right one, sometimes it is not.'\n\nTwo secondary concerns. First, the appendix documents a performance collapse after 6M timesteps in the binary environment (plasticity loss), which contradicts the earlier statement that the reward model 'empirically converges to the optimal policy.' The authors flag it as future work, so they are aware, but it needs to be acknowledged in the main text as a stability limitation. Second, the cautious-policy result is partly by construction: optimizing against the worst k-of-N reward models will reduce actions on uncertain states by design. The effect is real, but it is less surprising than the paper implies.\n\nOverall, this is a competent first step for a niche framework. It will interest researchers working on partial reward observability or reward learning, but it is not a breakthrough. It deserves peer review—an editor should send it out—but the referees should ask for an optimality baseline in the half-room environment and a more measured statement of what 'unsolvable' means when the inductive bias does the work.","headline":"A first empirical look at function approximation for Mon-MDPs with a real finding about overgeneralization, but the main 'near-optimal' claim outruns the evidence.","tokens_in":17669,"tokens_out":1504,"would_cite":false,"duration_ms":16106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that coupling a learned reward model with function approximation lets reinforcement-learning agents generalize from states with visible rewards to states where rewards are hidden, achieving near-optimal behavior in…","keywords":["monitored Markov decision processes","unobservable rewards","function approximation","reward model generalization","overgeneralization","robust policy optimization","epistemic uncertainty","reinforcement learning"],"falsifier":"Run the half-room experiment while gradually increasing the visual divergence between Zone 1 and Zone 2 (changing colors, shapes, or adding objects) and measure the gap between the reward-model policy's return and an oracle that knows true rewards in Zone 2; if the gap stays near zero even for large divergence, the generalization claim is stronger than the similarity assumption predicts, while a growing gap would confirm that the claim is bounded by feature transfer.","tokens_in":16604,"feed_emoji":"🤖","tokens_out":9379,"duration_ms":84073,"temperature":0.7,"pith_summary":"This paper asks whether reinforcement learning can act sensibly when rewards are only sometimes observable, such as a plant-watering robot that receives feedback only while the homeowner is watching. It extends the Mon-MDP framework, previously limited to small tabular environments, to high-dimensional state spaces by pairing a learned reward model with function approximation. The central claim is that the reward model can generalize from monitored states to unmonitored states, yielding near-optimal policies in environments that were formally unsolvable in tabular Mon-MDPs. The paper also shows the same generalization can go wrong, and that a cautious policy optimization based on reward uncertainty reduces the harm.","feed_headline":"RL agents can infer unobserved rewards—and sometimes overgeneralize","feed_subtitle":"Monitored-MDP agents generalize from observed to unobserved states, but unchecked extrapolation can water the cacti.","key_machinery":"The load-bearing object is the learned reward model $\\hat{R}(s^E, a^E; \\theta)$, a convolutional neural network trained only on transitions whose proxy reward equals the true environment reward. It feeds predicted rewards into a deep Q-network that also receives the monitor state, so generalization across the monitored and unmonitored boundary is carried by the CNN's feature-sharing, which maps visually similar states to similar reward predictions. To counter overgeneralization, the paper adapts a cautious optimizer: an ensemble of 500 reward models quantifies epistemic uncertainty, and $k$-of-$N$ counterfactual regret minimization optimizes the average of the $k$ worst models, interpolating between risk-neutral and highly risk-averse behavior.","core_discovery":"The paper demonstrates that a reward model trained with function approximation on states where rewards are observable can predict rewards in states where rewards are not observable, allowing the agent to learn near-optimal policies even in Mon-MDPs that are formally unsolvable in the tabular setting because the function approximator's inductive bias effectively selects one reward function among indistinguishable ones. In a half-room environment with monitoring only on the left, the agent waters plants in the unmonitored right zone at a comparable rate to the monitored zone. The paper also shows that this same generalization mechanism overreaches: an agent that only observed rewards for standard plants waters cacti in the unmonitored zone despite their distinct representation and negative true reward. To mitigate this, the paper adapts a cautious policy optimization method using an ensemble of reward models and $k$-of-$N$ counterfactual regret minimization, which reduces watering of novel plants while preserving watering of familiar ones.","pith_inferences":["If the visual-similarity assumption is the real carrier of generalization, then the formal 'unsolvable' status of a Mon-MDP is representation-dependent: a tabular agent cannot distinguish reward functions, but a CNN-based reward model can, by effectively selecting one reward function through its inductive bias.","A practical design suggested by the results is to treat reward-model uncertainty as a signal for when to request monitoring or abstain: the cacti episode is exactly a case where the model is confident but wrong, so confidence alone may be insufficient.","The cautious $k$-of-$N$ method likely trades off sensitivity: it will also reduce watering of genuinely rewarding novel plants that share few features with familiar plants, so deployment would need a separate mechanism to discover new reward sources rather than simply avoiding them.","The half-room and botanical-garden experiments together imply a testable scaling relation between visual divergence of the unmonitored zone and performance loss; measuring that curve would place a quantitative bound on when the reward-model approach applies."],"forward_implications":["In the binary-monitor plant-watering environment, the reward-model policy converges to near-optimal behavior and stops requesting monitoring, while treating hidden rewards as zero or discarding them leaves the agent stuck with suboptimal monitoring behavior.","In the half-room environment, the agent waters plants in the unmonitored zone about as often as in the monitored zone, which the paper takes as evidence that reward knowledge transfers across the monitored boundary even though the environment is formally unsolvable in the tabular setting.","Overgeneralization is real: an agent that only ever observed rewards for standard plants also waters cacti in the unmonitored zone, despite their distinct appearance and negative true reward.","Cautious policies built from a reward-model ensemble and $k$-of-$N$ CFR reduce watering of novel plants by up to five times while keeping watering of familiar plants, so robustness tuning can partially offset overgeneralization.","The paper's results position function approximation as the ingredient that lets Mon-MDPs move beyond tabular environments, and its appendix points to plasticity loss as a separate obstacle for deep Mon-MDP training."],"supporting_citations":[{"why":"Introduces Mon-MDPs, defines solvable versus unsolvable and indistinguishable reward cases, and proposes the tabular reward model that this work extends to function approximation.","marker":"Parisi et al., 2024b"},{"why":"Prior tabular Mon-MDP exploration work that motivates moving beyond tabular environments.","marker":"Parisi et al., 2024a"},{"why":"Tabular model-based exploration in Mon-MDPs, another predecessor for the non-tabular extension.","marker":"Kazemipour et al., 2025"},{"why":"Supplies the learning-to-be-cautious ensemble and CVaR machinery adapted here for Mon-MDPs.","marker":"Mohammedalamen et al., 2021"},{"why":"Provides the $k$-of-$N$ counterfactual regret minimization objective used to optimize the cautious policy.","marker":"Chen & Bowling, 2012"},{"why":"Provides the deep Q-network architecture with target network and replay buffer used for the reward model and Q-model.","marker":"Mnih et al., 2013"},{"why":"Deep ensembles, the uncertainty-estimation technique behind the reward-model ensemble.","marker":"Lakshminarayanan et al., 2017"}],"fun_headline_variants":["RL agents infer hidden rewards, but overgeneralization is a risk","Mon-MDP RL: generalizing rewards to unmonitored states, with caution","Reward model lets RL infer unobserved rewards—sometimes too much","RL in Mon-MDPs: bridging observed to unobserved rewards, safely"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole transfer result rests on unmonitored states looking and behaving enough like monitored states that the features learned in the monitored zone stay predictive in the unmonitored zone.","fun_headline_variants_meta":{"raw":{"variants":["RL agents infer hidden rewards, but overgeneralization is a risk","Mon-MDP RL: generalizing rewards to unmonitored states, with caution","Reward model lets RL infer unobserved rewards—sometimes too much","RL in Mon-MDPs: bridging observed to unobserved rewards, safely"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1212,"prompt_tokens":940,"completion_tokens":272,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":192}},"tokens_in":556,"tokens_out":272,"duration_ms":3523,"temperature":1.0,"reasoning_tokens":192,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:42:43.033570+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the half-room experiment while gradually increasing the visual divergence between Zone 1 and Zone 2 (changing colors, shapes, or adding objects) and measure the gap between the reward-model policy's return and an oracle that knows true rewards in Zone 2; if the gap stays near zero even for large divergence, the generalization claim is stronger than the similarity assumption predicts, while a growing gap would confirm that the claim is bounded by feature transfer.","supporting_citations":[{"cited_title":"E., and Bowling, M","cited_arxiv_id":null,"evidence_quote":"Tabular model-based exploration in Mon-MDPs, another predecessor for the non-tabular extension."},{"cited_title":"Learning to be cautious","cited_arxiv_id":null,"evidence_quote":"Supplies the learning-to-be-cautious ensemble and CVaR machinery adapted here for Mon-MDPs."},{"cited_title":"and Bowling, M","cited_arxiv_id":null,"evidence_quote":"Provides the $k$-of-$N$ counterfactual regret minimization objective used to optimize the cautious policy."},{"cited_title":"Playing A tari with deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the deep Q-network architecture with target network and replay buffer used for the reward model and Q-model."},{"cited_title":"Simple and scalable predictive uncertainty estimation using deep ensembles","cited_arxiv_id":null,"evidence_quote":"Deep ensembles, the uncertainty-estimation technique behind the reward-model ensemble."}],"review_version":1}