{"id":"4453c5f6-c8b5-44ad-ac03-5b8863ebff99","arxiv_id":"2501.00381","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"EIG-based query selection for full-trajectory demonstrations reduces the number of human demonstrations needed to learn a reward in tabular gridworlds compared to random and entropy-based baselines.","lead":"This paper adapts the expected information gain acquisition function to active inverse reinforcement learning, where a human provides full trajectories rather than single state annotations. It shows in small gridworld experiments that this query selection outperforms random and heuristic baselines, but the benefit depends on how trajectory collection is costed relative to single states.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EIG's superiority is asserted from mean curves with 10/16 seeds and no error bars; the central empirical claim needs a seed-robustness check before it can be accepted.","rationale":"The reader's conditionality is appropriate, but the specific weakest assumption identified (fixed beta=1) is not the most load-bearing threat to the claimed empirical result. Within the experiments, demonstrations are generated with beta=1, so the Boltzmann-rational model is correctly specified; beta misspecification would matter for transferring the method to real human data, not for validating the method on these synthetic benchmarks. The more direct threat is statistical: the Section 4 claim is based on mean curves without error bars or significance tests, from 10 and 16 seeds, and the paper's own cost-scaling comparison shows that part of the apparent trajectory advantage is due to query length rather than information per state. A seed-robustness check would settle whether the remaining EIG advantage is real. This does not change the reader's CONDITIONAL verdict, but it shifts the justification: the paper should be accepted only conditionally on stronger empirical evidence, not primarily because of beta sensitivity. No internal circularity or mathematical invalidity was found; the EIG derivation is standard and the baselines are reasonable, though the action-entropy baseline's jail-state failure is a known failure mode of that heuristic rather than a flaw in the comparison.","tokens_in":9532,"tokens_out":8192,"duration_ms":89529,"concrete_test":"Rerun both gridworld environments with at least 50 seeds per method (identical seed sets across methods) and report per-step means with 95% bootstrap confidence intervals for posterior entropy and apprentice regret. Additionally, run a paired test across seeds at the final active step (200 for entropy, max plotted for regret) comparing EIG against random, single-state EIG, and the two entropy baselines. If EIG curves overlap the baseline intervals at most active steps, the superiority claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the Section 4 statement that EIG 'is superior to the baselines' in both environments. The load-bearing evidence is Fig. 2, which plots means from only 10 (structured) or 16 (fully random) reward/terminal assignments, with no error bars, no confidence intervals, and no significance tests. With that sample size, a few high-variance runs can change the ordering, especially on the regret metric where the plotted x-range is short. The paper's own scaling comparison (single-state EIG with x/8.8) shows that the full-trajectory advantage is largely a matter of query cost, so the remaining EIG-vs-baseline advantage is exactly the part that most needs variance information. The reader's beta=1 concern is real for external generalization, but not for the internal claim: demonstrations are generated with beta=1, so the model is correctly specified in-sample. The more pressing concern is whether the reported means are statistically distinguishable from random or single-state EIG. The EIG estimates themselves also use only 20 reward samples and 2 trajectories per reward, and the BO-EIG variant is an approximation, so if the seed-sensitivity check fails, this is the likely source.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an active inverse reinforcement learning (IRL) framework in which the learner selects the initial state of a full expert trajectory rather than querying a single state-action pair. The acquisition function is the expected information gain (EIG) under a Bayesian posterior over rewards, approximated by a nested Monte Carlo estimator, and made computationally cheaper by a Bayesian-optimization/UCB allocation scheme. The method is evaluated on two tabular gridworlds against random sampling, Q-entropy, posterior predictive action entropy from Kweon et al. [5], and single-state EIG variants. The paper claims that EIG-based acquisition is superior to baselines in both environments in terms of posterior reward entropy and apprentice regret, and that it exposes a failure mode in the prior trajectory-based acquisition function.","tokens_in":9685,"tokens_out":3125,"duration_ms":34681,"significance":"The formulation is a natural and principled adaptation of Bayesian experimental design to trajectory-level active IRL, and the nested Monte Carlo estimator in Eq. (2) is mathematically sound. The Bayesian-optimization approximation is a reasonable practical contribution. If the empirical superiority claim survives a robustness check, the paper would be a useful step toward reducing demonstration cost in IRL. At present, however, the central claim rests on mean curves from 10 and 16 random reward assignments with no error bars or significance tests, so the significance is conditional on additional validation.","major_comments":[{"comment":"The central claim that EIG 'is superior to the baselines' in both environments is supported only by mean curves computed from 10 (structured) and 16 (fully random) reward/terminal assignments, with no error bars, confidence intervals, or significance tests. With these sample sizes, a few high-variance seeds could change the ordering, particularly on the regret metric. Please report per-seed results or confidence intervals, and run paired statistical tests (e.g., Wilcoxon signed-rank) at the relevant active-learning steps. This is load-bearing because the remaining EIG-vs-baseline advantage, after accounting for the query-cost comparison in the 'Single-st. EIG (x/8.8)' curves, is exactly the part that needs variance information.","section":"Section 4, Fig. 2"},{"comment":"The EIG estimates use only Nr=20 reward samples and Ntau=2 trajectories per reward, and the BO-EIG variant adds an approximation layer on top of these noisy observations. The manuscript does not report the variance or bias of the EIG estimator, nor any convergence check as Nr or Ntau increases. Since the BO-UCB acquisition rule is driven by these estimates, the reported advantage over baselines could in principle be an artifact of estimator noise rather than of the EIG objective. Please validate the estimator (e.g., bootstrap confidence intervals, or a small sensitivity study increasing Nr and Ntau) and state the resulting uncertainty in the Fig. 2 curves.","section":"Section 3, Eq. (2); Appendix C.3"},{"comment":"There is a direct inconsistency in the prior used for the structured environment: Section 4 states that the unknown rewards ('water', 'mud', 'lava') have an independent uniform prior in the interval [-100, 0], while Appendix C.3.1 says the reward was drawn from 'independent Uniform[0, 100]'. One of these is a typo, but it matters for reproducibility and for the interpretation of the entropy/regret results. Please correct this and confirm that the experiments used the intended negative-reward prior.","section":"Appendix C.3.1 vs. Section 4"}],"minor_comments":[{"comment":"The expert rationality beta is fixed to 1 in all experiments, and the demonstrations are generated under the same beta=1 model. This makes the model correctly specified in-sample, but the paper would be stronger if it acknowledged that beta misspecification is a limitation for real human demonstrations and ideally reported at least one sensitivity run.","section":"Section 2, Eq. (1); Appendix C.3"},{"comment":"The scaling experiments report '3 repeated trials' but show only means with no variability information. Please add error bars or state the observed spread.","section":"Appendix B, Fig. 3"},{"comment":"The caption describes 'Single st. EIG (x / 8.8)' as scaling by the mean trajectory length, but the text should also clarify whether this is an upper-bound comparison or a matched-query-cost comparison. The sentence 'the latter baseline serves as an upper bound' in Appendix C.2 is helpful; consider moving some of that explanation to the main text.","section":"Section 4, Fig. 2 caption"},{"comment":"The claim that BO-EIG incurs 'less than a 10% increase in computation time' is based on average timings of 6.6 vs 6.2 seconds. Please report the number of runs and the variance of these timings, since the difference is small.","section":"Appendix C.4"},{"comment":"Minor notation issues: Eq. (2) uses a hat over EIG on the left but not in the text; and the definition of the apprentice policy pi_A in Section 2 would benefit from a precise statement of the expectation over the posterior p(r|D_N).","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core formulation is sound and the paper is clearly written, but the empirical evaluation is currently too thin for a journal-level claim of superiority. The lack of error bars and the tiny number of reward assignments are fixable within the scope of the paper by adding seed-level results, confidence intervals, and simple significance tests. I would not reject the paper, but I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clean workshop-style paper that takes a standard acquisition function—expected information gain—and applies it to active IRL where the query is a whole expert trajectory. That is a legitimate new application, and the paper also shows that the one prior full-trajectory heuristic [5] degenerates to querying a jail state. The x/8.8 comparison is honest: it shows the trajectory advantage is mostly about query cost, not information per state.\n\nNew and good: the EIG formulation (Eq. 2) is correctly derived, the BO approximation is a reasonable way to allocate samples, and the authors explicitly flag the limitations of their own study. The empirical finding that action entropy can completely fail is a useful caution for people building on [5]. No circularity: EIG is computed from the model and judged against external baselines.\n\nSoft spots: the central claim—'EIG is superior to the baselines'—rests on mean curves from 10 and 16 random reward assignments, with no error bars, no confidence intervals, and no significance tests. With that sample size, a few high-variance runs can flip ordering, especially on regret at short horizons. The EIG estimator itself uses only 20 reward samples and 2 trajectories per reward, and the BO variant adds hand-chosen priors and κ=3. The β=1 issue is real for deployment but not for the internal claim: demonstrations are generated under the same model, so it is correctly specified in-sample. The bigger problem is that the reported means are not shown to be statistically distinguishable from random or from single-state EIG at equal query cost.\n\nAlso no code is released, which makes it hard to verify the BO efficiency claim. The paper's own scaling suggests the BO method matches NMC with about double budget, but that's a single observation without variance.\n\nWho it's for: people working on active IRL or experimental design for sequential queries. It's not a breakthrough, but it is a legitimate initial study with a clear negative result about a prior baseline.\n\nRecommendation: I would send it to peer review, but the referee should require error bars or seed-level results, significance tests, and ideally code release. Without those, the empirical superiority claim is unsupported.","headline":"A sensible, modest extension of EIG to whole-trajectory active IRL; the core claim is plausible but the empirical support lacks error bars and needs a seed-robustness check.","tokens_in":10320,"tokens_out":1980,"would_cite":false,"duration_ms":20178,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Active inverse IRL should query full trajectories by expected information gain, not by action entropy.","keywords":["active inverse reinforcement learning","expected information gain","Bayesian experimental design","Bayesian optimization","apprenticeship learning","Boltzmann-rational expert","full-trajectory queries","gridworld"],"falsifier":"Run the paper's two gridworld experiments with expert demonstrations generated under a Boltzmann policy with $\\beta = 0.2$ or $\\beta = 5$ while the acquisition continues to assume $\\beta = 1$; if the EIG-based method no longer achieves lower posterior entropy than random sampling over the same demonstration budget, then the central claim depends on the expert model being exactly right. Alternatively, generate demonstrations from a non-Boltzmann expert policy and check whether EIG-based query selection degrades to or below random baseline performance.","tokens_in":9242,"feed_emoji":"🎯","tokens_out":6781,"duration_ms":62657,"temperature":0.7,"pith_summary":"This paper argues that when an AI system must learn human preferences from demonstrations, the scarce expert queries should be chosen by how much information they are expected to reveal about the reward, and that the right unit for a query is often a whole trajectory rather than a single-state action annotation. It adapts the expected information gain (EIG) acquisition function from Bayesian experimental design to active inverse reinforcement learning, where the choice variable is the environment setup—in the experiments, the starting state of a demonstration. Because the EIG has no closed form, the paper approximates it with nested Monte Carlo sampling and accelerates that by allocating hypothetical trajectory samples through Bayesian optimization with an upper confidence bound rule. In two tabular gridworlds, the EIG-based method reaches lower posterior reward entropy and lower apprentice regret than random sampling, Q-entropy, and a posterior predictive action entropy baseline, and it exposes a failure mode in the only prior full-trajectory acquisition function it is aware of.","feed_headline":"EIG-based active IRL beats baselines in gridworlds","feed_subtitle":"Choosing demonstration starting states by expected information gain lowers reward uncertainty and apprentice regret.","key_machinery":"The engine of the method is the expected information gain (EIG) acquisition function, defined as the mutual information between the unknown reward and a hypothetical expert trajectory from a given environment setup. Trajectories are modeled by a Boltzmann-rational expert policy, Eq. (1), with a known rationality coefficient $\\beta$, which gives a closed-form likelihood for any trajectory under any reward sample. Because the EIG expectations are intractable, the paper uses a nested Monte Carlo estimator, Eq. (2), and an adaptive sampling scheme built on Bayesian optimization: each hypothetical trajectory yields a noisy Gaussian observation of the true EIG, and an upper confidence bound rule ($\\kappa = 3$) decides where to spend additional samples so the best starting state can be identified with a limited budget. The reward posterior is sampled by MCMC, using PolicyWalk in the structured environment and ValueWalk in the fully random environment.","core_discovery":"The central claim, stated in Section 4, is that in both environments the performance of EIG in terms of posterior entropy and apprentice performance is superior to the baselines. Concretely, an active IRL agent that selects each next demonstration starting state by maximizing the expected information gain about the unknown reward ends up with a tighter posterior over reward parameters and a better apprenticeship policy after the same demonstration budget than an agent using random initial states, maximum Q-entropy, or maximum posterior predictive action entropy. The paper also claims that whole trajectories are a natural query unit in high-frequency domains, that the Bayesian-optimization approximation makes EIG practical with less than ten percent extra computation time, and that the posterior predictive action entropy baseline fails completely in the structured environment because it keeps requesting demonstrations from a jail state whose action entropy is irreducible.","pith_inferences":["The paper fixes the expert rationality coefficient $\\beta$ at 1 and does not report sensitivity to it; a natural extension is to treat $\\beta$ as an unknown hyperparameter and marginalize it out, which should make the acquisition more robust to realistic experts.","Because the EIG is defined over the environment setup parameter $\\xi$, not just the starting state, the same machinery could choose transition dynamics or other environment properties; the jail-state failure shows why information-theoretic querying matters there too.","The trajectory-versus-single-state comparison suggests a cost model for human effort; future work could let the acquisition function choose query type (trajectory length or number of single-state annotations) under an explicit budget.","Scaling to continuous control will likely require replacing MCMC sampling with a variational posterior over rewards, since the paper identifies Bayesian IRL sampling as the computational bottleneck."],"forward_implications":["With the same budget of demonstrations, an EIG-based active learner should produce a more accurate reward estimate and a better apprentice policy, because it avoids queries whose information gain is zero or redundant.","Whole trajectories are more informative than single-state action labels per query, but the experimental comparison indicates that the advantage depends on cost: an equal number of single-state queries matches or beats one full trajectory, so the right choice depends on the relative cost of asking for a trajectory versus a single action.","The failure of the posterior predictive action entropy baseline implies that action-entropy heuristics are unreliable for trajectory-level active IRL, because high action entropy can reflect irreducible randomness in the expert rather than uncertainty about the reward.","The Bayesian-optimization approximation reaches the performance of naive nested Monte Carlo with roughly half the hypothetical trajectory budget and less than ten percent overhead, making EIG-based active IRL practical in tabular environments."],"supporting_citations":[{"why":"Supplies the only prior full-trajectory active IRL acquisition function, whose failure on the jail state motivates the information-theoretic alternative.","marker":"[5]"},{"why":"Provides the Bayesian IRL formulation and the PolicyWalk MCMC sampler used to estimate the reward posterior.","marker":"[14]"},{"why":"Supplies the ValueWalk sampler used in the fully random environment to obtain posterior reward samples.","marker":"[20]"},{"why":"Supplies the Bayesian experimental design framework and the EIG acquisition function that the method adapts to active IRL.","marker":"[6]"},{"why":"Provides the upper confidence bound rule used to allocate hypothetical trajectory samples in the Bayesian-optimization approximation.","marker":"[9]"},{"why":"Represents the prior single-state active IRL approach that the paper contrasts with full-trajectory queries.","marker":"[2]"}],"fun_headline_variants":["EIG active IRL beats random starts in gridworlds","Info gain picks demos that crush IRL baselines","Smart demo starts shrink reward uncertainty fast","Active IRL: EIG queries outperform action-entropy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The expert's demonstrations are generated by a Boltzmann-rational policy with a known, fixed rationality coefficient $\\beta$ (set to 1 in the experiments); if the expert deviates from this model or $\\beta$ is misspecified, the expected information gain can misrank demonstrations and the acquisition may lose its advantage.","fun_headline_variants_meta":{"raw":{"variants":["EIG active IRL beats random starts in gridworlds","Info gain picks demos that crush IRL baselines","Smart demo starts shrink reward uncertainty fast","Active IRL: EIG queries outperform action-entropy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000126,"raw_usage":{"total_tokens":1052,"prompt_tokens":827,"completion_tokens":225,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":161}},"tokens_in":443,"tokens_out":225,"duration_ms":3165,"temperature":1.0,"reasoning_tokens":161,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:52:50.662220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's two gridworld experiments with expert demonstrations generated under a Boltzmann policy with $\\beta = 0.2$ or $\\beta = 5$ while the acquisition continues to assume $\\beta = 1$; if the EIG-based method no longer achieves lower posterior entropy than random sampling over the same demonstration budget, then the central claim depends on the expert model being exactly right. Alternatively, generate demonstrations from a non-Boltzmann expert policy and check whether EIG-based query selection degrades to or below random baseline performance.","supporting_citations":[{"cited_title":"Trajectory-based active inverse reinforce- ment learning for learning from demonstration","cited_arxiv_id":null,"evidence_quote":"Supplies the only prior full-trajectory active IRL acquisition function, whose failure on the jail state motivates the information-theoretic alternative."},{"cited_title":"Bayesian Inverse Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian IRL formulation and the PolicyWalk MCMC sampler used to estimate the reward posterior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ValueWalk sampler used in the fully random environment to obtain posterior reward samples."},{"cited_title":"A versatile stochastic model of a function of unknown and time varying form","cited_arxiv_id":null,"evidence_quote":"Provides the upper confidence bound rule used to allocate hypothetical trajectory samples in the Bayesian-optimization approximation."},{"cited_title":"Active learning for reward estimation in inverse reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Represents the prior single-state active IRL approach that the paper contrasts with full-trajectory queries."}],"review_version":1}