{"id":"25be5439-e9e3-4fb6-b99f-48331399c9b5","arxiv_id":"2607.08647","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"A hierarchical teaching algorithm selects complementary environments and feedback modalities to learn reward functions that generalize across unseen MDPs, proving that single-environment teaching leaves structural reward ambiguity.","lead":"This paper shows that teaching robots the right reward function requires picking the right environments and the right types of human feedback, not just more feedback in one setting. A smart generalist would read this because it offers a concrete algorithm for making AI agents generalize better across different situations without needing more data.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Experiments use 2D/4D reward spaces where the constraint universe is trivially spanned by 50 training MDPs, so the i.i.d. generalization assumption is never actually stress-tested.","rationale":"The reader correctly identified the i.i.d. assumption as load-bearing, but I think the deeper issue is that this assumption is not even being stress-tested in the current experimental design. The 2D/4D feature spaces make it nearly impossible for held-out environments to expose genuinely new constraint directions, so the generalization result is almost guaranteed by construction rather than by HSCOT's environment selection. That said, the theoretical contributions (Theorem 1, Propositions 1-3) are sound, the algorithm is well-specified, and the code is available. The paper explicitly acknowledges the limitation to simple domains in its conclusion. The verdict of CONDITIONAL is appropriate: the core theoretical insight is valid and novel, but the empirical evidence does not yet demonstrate that the approach works in the regime where the problem actually arises. My concern does not change the verdict—it reinforces why CONDITIONAL rather than ACCEPT is the right call. The reader's assessment of novelty (6.0) and the three identified limitations (simple domains, i.i.d. assumption, expert teacher) are all well-placed. I would add that the missing budget specification in the main experiments is a reporting gap that should be addressed.","tokens_in":25281,"tokens_out":4948,"duration_ms":284899,"concrete_test":"Run the LavaMiniGrid experiments with a 16- or 32-dimensional feature space (e.g., by using tile-coded or radial-basis features instead of 4 hand-designed ones), keeping 50 training MDPs and 20% held-out. Measure: (1) what fraction of the constraint universe U each individual MDP covers (if >50%, the problem is too easy); (2) whether held-out MDPs expose constraint directions not in U (count new directions); (3) whether HSCOT still achieves near-zero held-out regret. If held-out environments expose directions outside U and HSCOT's regret increases, the i.i.d. assumption is genuinely load-bearing and the current results don't test it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that HSCOT achieves 'substantially lower regret and stronger generalization to held-out environments.' The theoretical motivation (Theorem 1) is that single-MDP teaching leaves residual ambiguity because individual environments don't span all reward-relevant constraint directions. However, the experiments use 2D (GridWorld) and 4D (LavaMiniGrid) linear reward spaces with 50 training environments. In such low-dimensional settings, the constraint universe U—the set of all distinct feature-difference directions inducible across training MDPs—is almost certainly the full feature space, regardless of which environments are selected. This means held-out environments will expose constraint directions already in U, making HSCOT's complete coverage guarantee trivially transfer. The i.i.d. assumption (Section 7.1) that the reader correctly identifies as load-bearing is not actually being tested: it would only become binding when held-out environments expose genuinely new constraint directions not covered by training MDPs, which is nearly impossible when the feature space has only 2 or 4 dimensions and 50 training environments are available. The regime where environment-dependent identifiability is practically problematic—high-dimensional rewards where each MDP exposes only a small subset of constraint directions—is never tested. Additionally, the paper does not report the actual feedback budget used in the main experiments (only the number of activated environments in Table 1), making it impossible to assess whether HSCOT's advantage would persist at larger budgets where uniform teaching might also achieve complete coverage.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper studies machine teaching for inverse reinforcement learning across multiple environments with heterogeneous feedback modalities. The authors make three main contributions: (1) a theoretical analysis showing that comparisons impose strictly stronger global constraints than demonstrations, corrections, and E-stops in the unlimited-data regime, while demonstrations are more constraint-efficient per query under tight budgets; (2) a formal characterization of environment-dependent reward identifiability (Theorem 1), proving that even unlimited feedback in a single MDP can leave residual reward ambiguity; and (3) HSCOT, a hierarchical greedy set-cover algorithm that first selects informative environments exposing complementary reward constraints, then selects feedback atoms within those environments. Experiments on GridWorld (2D rewards) and LavaMiniGrid (4D rewards) show HSCOT achieves near-zero held-out regret and complete constraint coverage, substantially outperforming uniform teaching baselines under identical feedback budgets.","tokens_in":26034,"tokens_out":1369,"duration_ms":381615,"significance":"The paper addresses a genuine gap in the machine teaching for IRL literature: prior work (Brown and Niekum, 2019; Buning et al., 2022) operates within single environments using demonstrations only, and the environment-dependent identifiability limitation identified here is both formally novel and practically relevant. The theoretical results (Propositions 1-3, Theorem 1) are clean and correctly proven under an explicitly stated genericity assumption. The generalized BEC framework unifying heterogeneous feedback across multiple MDPs is a natural and useful extension. The HSCOT algorithm is well-motivated by the theory and the hierarchical decomposition into environment selection and atom selection is principled. Reproducible code is provided. The main limitation is that the experimental validation operates in regimes where the theoretical motivation is not strongly stress-tested (see major comments).","major_comments":[{"comment":"The experimental regime does not adequately stress-test the central theoretical motivation. Theorem 1 establishes that single-MDP teaching leaves residual ambiguity because individual environments do not span all reward-relevant constraint directions. However, the experiments use 2D (GridWorld) and 4D (LavaMiniGrid) reward spaces with 50 training environments. In such low-dimensional settings, the constraint universe U is almost certainly the full feature space regardless of which environments are selected, making HSCOT's complete coverage guarantee (Figure 6) trivially achievable and the i.i.d. generalization assumption (Section 7.1) non-binding. The regime where environment-dependent identifiability is practically problematic—higher-dimensional rewards where each MDP exposes only a small subset of constraint directions—is never tested. The authors should either (a) include experiments,","section":null},{"comment":"The feedback budget used in the main experiments (Figures 5-6, Table 1) is not reported anywhere in the paper. The number of feedback queries per environment is a critical experimental condition that determines whether HSCOT's advantage comes from intelligent environment selection, intelligent atom selection, or simply having more queries per environment. Without this information, the comparison to uniform teaching under 'identical feedback budgets' cannot be fully evaluated. The per-budget analysis in Appendix 10.2 is conducted only for a single MDP and does not address this gap. Please report the budget explicitly and ideally show regret/coverage as a function of budget for the multi-environment setting.","section":null},{"comment":"The i.i.d. assumption (Section 7.1: 'Under the assumption that training and evaluation environments are drawn i.i.d. from the same distribution, minimizing loss on the training MDPs provides a principled surrogate for minimizing loss on unseen environments') is load-bearing for the generalization claim but is not tested. The 20% held-out split (10 of 50 MDPs) drawn from the same generation process does not constitute a meaningful distribution shift. To assess robustness, the authors should evaluate on held-out environments with systematically different structure (e.g., different grid sizes, transition dynamics, or feature distributions) and report whether HSCOT's coverage advantage transfers. Without this, the claim of 'stronger generalization' is limited to in-distribution extrapolation.","section":null}],"minor_comments":[{"comment":"Section 3.2: The E-stop definition uses notation xi^halted = xi_{0:t_R} xi_{t_R} ... xi_{t_R} which is unclear. Clarify whether this denotes repeating the state at t_R or concatenating segments, and define the notation precisely.","section":null},{"comment":"Figure 2 caption refers to 'the layout in Figure 4a' but the connection between the heatmap axes (w1, w2) and the MDP features is not explained. A brief note on how the 2D reward space maps to the gridworld features would help readers.","section":null},{"comment":"Table 1 reports 'Average environments activated' but does not specify the feedback budget under which these counts were obtained. Since the number of activated environments likely depends on the budget, this context is needed.","section":null},{"comment":"Section 7.1, Eq. (2): the loss function uses V^{pi*_k(w)}_k(w*) but the subscript/superscript notation is dense. A clearer breakdown of what each term denotes would improve readability.","section":null},{"comment":"The paper mentions 'low-cost feedback' in the abstract and 'strategically queries low-cost feedback' but does not formalize or discuss feedback costs. If different modalities have different costs, this should be formalized; if not, the term 'low-cost' should be removed or clarified.","section":null},{"comment":"Appendix 10.3: the 2x3 gridworld layouts in Figure 9 are very small and hard to read. Consider enlarging or providing a higher-resolution version.","section":null},{"comment":"The related work section could better position the contribution relative to meta-IRL (Yu et al., 2019; Chen et al., 2024), since meta-IRL also addresses cross-environment reward transfer. The distinction that meta-IRL learns priors for fast adaptation while HSCOT teaches a single transferable reward is mentioned but could be sharpened.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about low-dimensional reward spaces is valid and is the primary reason for the major revision recommendation. The theory is sound and the algorithm is well-motivated, but the experiments as currently presented do not demonstrate that the theoretical contribution (environment-dependent identifiability) is practically consequential. A single experiment in a higher-dimensional reward setting (e.g., 8-16 features) with environments that expose only partial constraint directions would substantially strengthen the paper. The missing feedback budget is also a notable oversight that should be straightforward to fix."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"Two things matter here. First, Theorem 1 — that reward identifiability is environment-dependent and unlimited feedback in a single MDP can leave residual ambiguity — is a genuine structural insight for IRL. Second, the experiments don't actually test the regime where this matters, because the reward spaces are 2D and 4D with 50 training environments, making the constraint universe trivially spanned regardless of which environments you pick. The theory is the real contribution; the experiments are illustrative at best and misleading at worst. The paper does several things well. The generalized BEC framework unifying heterogeneous feedback across multiple MDPs is a clean extension of Brown and Niekum (2019). Propositions 1–3 are correct: comparisons impose strictly stronger global constraints than demonstrations, corrections, or E-stops in the unlimited-data regime, while demonstrations are more constraint-efficient per query under tight budgets. The proofs are straightforward linear algebra and geometry, nothing fancy but nothing wrong. The HSCOT algorithm — greedy environment selection then greedy atom selection — is well-specified and the code is public. The stress-test concern about low-dimensional experiments lands hard. With 2D rewards and 50 training MDPs, the constraint universe U is almost certainly the full feature space, so HSCOT's complete coverage guarantee transfers trivially to held-out environments. The i.i.d. assumption between train and test environments is load-bearing but never actually stressed — it would only bind when held-out environments expose genuinely new constraint directions, which is nearly impossible in these settings. The regime where environment-dependent identifiability is practically problematic — high-dimensional rewards where each MDP exposes only a small subset of constraint directions — is never tested. The expert-teacher assumption (knowing w*) also limits practical relevance, though the paper acknowledges this. The reader's CONDITIONAL verdict is about right, maybe slightly generous on significance. The theory is sound and novel within the reward learning subfield. But the empirical gap between the theoretical motivation and the experimental validation is real and should be flagged. This paper deserves a serious referee who can evaluate whether the theoretical contribution stands on its own and whether the experimental limitations are acceptable for the venue. I'd lean toward accept with major revision requiring experiments in higher-dimensional reward spaces where the constraint coverage problem is non-trivial.","headline":"Clean theory on environment-dependent reward identifiability, but experiments are too low-dimensional to stress-test the central claim.","tokens_in":26256,"tokens_out":533,"would_cite":true,"duration_ms":115438,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"No single environment pins down a reward function","keywords":[],"falsifier":"If one could construct two MDPs sharing the same reward features where the first MDP's feasible trajectory space spans all reward-relevant feature-difference directions (i.e., V1 = span(V1 ∪ V2)), then Theorem 1's strict inclusion would fail and unlimited feedback in that single MDP would suffice to identify the reward, contradicting the claim that multi-environment teaching is necessary.","tokens_in":25486,"feed_emoji":"🧭","tokens_out":1016,"duration_ms":307303,"temperature":0.7,"pith_summary":"The paper argues that reward functions learned from human feedback in one environment are fundamentally ambiguous—no amount of feedback within a single Markov decision process can fully identify the underlying reward, because the environment's dynamics constrain which reward-relevant feature differences any trajectory can expose. This ambiguity is structural, not data-driven: it persists even with unlimited idealized feedback. The resolution the paper proposes is to teach across multiple environments whose dynamics expose complementary constraint directions, using a hierarchical strategy that first selects environments whose combined trajectory spaces span the full reward-feature space, then selects feedback instances within those environments to cover the remaining constraint directions. The paper also shows that different feedback modalities constrain rewards in qualitatively different ways: under tight budgets, demonstrations are the most constraint-efficient per query because each one implicitly enforces many optimality constraints at once; with unlimited data, comparisons dominate because they can enforce global ordering constraints across arbitrary trajectory pairs.","feed_headline":"No single environment pins down a reward function","feed_subtitle":"Even unlimited human feedback in one setting leaves reward ambiguity; teaching across diverse environments resolves it.","key_machinery":"The generalized behavioral equivalence class (gBEC) extends the classical single-MDP behavioral equivalence class to arbitrary feedback datasets across multiple environments. Each feedback instance—demonstration, comparison, correction, or emergency stop—induces one or more linear half-space constraints on the reward weight vector. The gBEC is the intersection of all such half-spaces, and its volume measures residual reward ambiguity. The hierarchical set-cover structure exploits the fact that environment dynamics determine which constraint directions are reachable while individual feedback instances instantiate those constraints: the outer stage greedily selects environments by marginal new","core_discovery":"The central result is Theorem 1: if two MDPs share the same reward features and ground-truth reward, but one MDP's feasible trajectory space does not span all reward-relevant feature-difference directions (i.e., its constraint span is a proper subspace of the union of both MDPs' spans), then there exists a reward vector distinct from the ground truth that satisfies all feedback constraints in the first MDP but violates constraints only visible in the second. This makes reward identifiability environment-dependent—a property of the MDP's dynamics and layout, not of the feedback quantity. The paper operationalizes this insight through HSCOT, a greedy two-stage set-cover algorithm that selects,","pith_inferences":["If the i.i.d. assumption between training and test environments fails—e.g., test environments have qualitatively different transition structures—then constraint coverage on training MDPs may not translate to low regret on test MDPs, and the near-zero regret results could degrade substantially.","The analysis assumes linear rewards over known features. If the feature representation is misspecified or the true reward is nonlinear, the gBEC framework would need extension, and the environment-dependent ambiguity result may not directly apply.","The greedy set-cover approach is efficient but may be suboptimal when environments have overlapping constraint contributions; an optimal combinatorial solution could potentially use fewer environments or queries.","For continuous state-action spaces, the constraint universe becomes infinite and the set-cover formulation would require discretization or function approximation, which could introduce its own ambiguity."],"forward_implications":["Reward learning systems that collect feedback in only one environment risk systematic misalignment when deployed elsewhere, even with perfect, unlimited feedback.","Environment diversity matters more than feedback quantity for resolving reward ambiguity; a small set of well-chosen environments can outperform a large volume of feedback in a single environment.","The constraint-coverage framework could be applied to curriculum design for human learners: if different problem contexts expose different skill-relevant feature directions, a teacher should select complementary contexts rather than drilling in one.","The hierarchy between environment selection and feedback selection suggests a natural division of labor in human-robot interaction: a system designer chooses deployment contexts, and within each context, the human provides whichever feedback is cheapest or most natural."],"fun_headline_variants":["Reward identifiability depends on MDP dynamics, not feedback quantity","Comparisons constrain rewards more than demonstrations in unlimited-data regime","Multi-environment teaching resolves reward ambiguity single environments cannot","Greedy environment selection exposes complementary reward constraints","HSCOT: hierarchical set-cover algorithm for dynamics-robust reward learning"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The approach assumes that the training environments and the held-out evaluation environments are drawn from the same distribution, so that covering all constraint directions in the training set is a reliable proxy for covering the directions that will matter in deployment.","fun_headline_variants_meta":{"raw":{"variants":["Reward identifiability depends on MDP dynamics, not feedback quantity","Comparisons constrain rewards more than demonstrations in unlimited-data regime","Multi-environment teaching resolves reward ambiguity single environments cannot","Greedy environment selection exposes complementary reward constraints","HSCOT: hierarchical set-cover algorithm for dynamics-robust reward learning"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":671,"prompt_tokens":589,"completion_tokens":82,"prompt_tokens_details":null},"tokens_in":589,"tokens_out":82,"duration_ms":137705,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T03:38:02.072055+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If one could construct two MDPs sharing the same reward features where the first MDP's feasible trajectory space spans all reward-relevant feature-difference directions (i.e., V1 = span(V1 ∪ V2)), then Theorem 1's strict inclusion would fail and unlimited feedback in that single MDP would suffice to identify the reward, contradicting the claim that multi-environment teaching is necessary.","supporting_citations":[],"review_version":1}