Pith. sign in

REVIEW 3 major objections 5 minor 24 references

Toward Information Theoretic Active Inverse Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Active inverse IRL should query full trajectories by expected information gain, not by action entropy.

desk verdict A sensible, modest extension of EIG to whole-trajectory active IRL; the core claim is plausible but the empirical support lacks error bars and needs a seed-robustness check. read the letter →

arxiv 2501.00381 v1 pith:FVHDBY34 submitted 2024-12-31 cs.LG stat.ML

Ondrej Bajgar , Sid William Gould , Rohan Narayan Langford Mitta , Jonathon Liu , Oliver Newcombe , Jack Golden This is my paper · ORCID
classification cs.LGstat.ML
keywords activeinversereinforcementlearningexpectedinformationgainBayesianexperimentaldesignoptimizationapprenticeshipBoltzmann-rationalexpertfull-trajectoryqueriesgridworld
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when an AI system must learn human preferences from demonstrations, the scarce expert queries should be chosen by how much information they are expected to reveal about the reward, and that the right unit for a query is often a whole trajectory rather than a single-state action annotation. It adapts the expected information gain (EIG) acquisition function from Bayesian experimental design to active inverse reinforcement learning, where the choice variable is the environment setup—in the experiments, the starting state of a demonstration. Because the EIG has no closed form, the paper approximates it with nested Monte Carlo sampling and accelerates that by allocating hypothetical trajectory samples through Bayesian optimization with an upper confidence bound rule. In two tabular gridworlds, the EIG-based method reaches lower posterior reward entropy and lower apprentice regret than random sampling, Q-entropy, and a posterior predictive action entropy baseline, and it exposes a failure mode in the only prior full-trajectory acquisition function it is aware of.

What carries the argument

The engine of the method is the expected information gain (EIG) acquisition function, defined as the mutual information between the unknown reward and a hypothetical expert trajectory from a given environment setup. Trajectories are modeled by a Boltzmann-rational expert policy, Eq. (1), with a known rationality coefficient $\beta$, which gives a closed-form likelihood for any trajectory under any reward sample. Because the EIG expectations are intractable, the paper uses a nested Monte Carlo estimator, Eq. (2), and an adaptive sampling scheme built on Bayesian optimization: each hypothetical trajectory yields a noisy Gaussian observation of the true EIG, and an upper confidence bound rule ($\kappa = 3$) decides where to spend additional samples so the best starting state can be identified with a limited budget. The reward posterior is sampled by MCMC, using PolicyWalk in the structured environment and ValueWalk in the fully random environment.

What would settle it

Run the paper's two gridworld experiments with expert demonstrations generated under a Boltzmann policy with $\beta = 0.2$ or $\beta = 5$ while the acquisition continues to assume $\beta = 1$; if the EIG-based method no longer achieves lower posterior entropy than random sampling over the same demonstration budget, then the central claim depends on the expert model being exactly right. Alternatively, generate demonstrations from a non-Boltzmann expert policy and check whether EIG-based query selection degrades to or below random baseline performance.

Watch

Extended reading notes

Core claim

The central claim, stated in Section 4, is that in both environments the performance of EIG in terms of posterior entropy and apprentice performance is superior to the baselines. Concretely, an active IRL agent that selects each next demonstration starting state by maximizing the expected information gain about the unknown reward ends up with a tighter posterior over reward parameters and a better apprenticeship policy after the same demonstration budget than an agent using random initial states, maximum Q-entropy, or maximum posterior predictive action entropy. The paper also claims that whole trajectories are a natural query unit in high-frequency domains, that the Bayesian-optimization approximation makes EIG practical with less than ten percent extra computation time, and that the posterior predictive action entropy baseline fails completely in the structured environment because it keeps requesting demonstrations from a jail state whose action entropy is irreducible.

Load-bearing premise

The expert's demonstrations are generated by a Boltzmann-rational policy with a known, fixed rationality coefficient $\beta$ (set to 1 in the experiments); if the expert deviates from this model or $\beta$ is misspecified, the expected information gain can misrank demonstrations and the acquisition may lose its advantage.

Editorial extensions

If this is right

  • With the same budget of demonstrations, an EIG-based active learner should produce a more accurate reward estimate and a better apprentice policy, because it avoids queries whose information gain is zero or redundant.
  • Whole trajectories are more informative than single-state action labels per query, but the experimental comparison indicates that the advantage depends on cost: an equal number of single-state queries matches or beats one full trajectory, so the right choice depends on the relative cost of asking for a trajectory versus a single action.
  • The failure of the posterior predictive action entropy baseline implies that action-entropy heuristics are unreliable for trajectory-level active IRL, because high action entropy can reflect irreducible randomness in the expert rather than uncertainty about the reward.
  • The Bayesian-optimization approximation reaches the performance of naive nested Monte Carlo with roughly half the hypothetical trajectory budget and less than ten percent overhead, making EIG-based active IRL practical in tabular environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper fixes the expert rationality coefficient $\beta$ at 1 and does not report sensitivity to it; a natural extension is to treat $\beta$ as an unknown hyperparameter and marginalize it out, which should make the acquisition more robust to realistic experts.
  • Because the EIG is defined over the environment setup parameter $\xi$, not just the starting state, the same machinery could choose transition dynamics or other environment properties; the jail-state failure shows why information-theoretic querying matters there too.
  • The trajectory-versus-single-state comparison suggests a cost model for human effort; future work could let the acquisition function choose query type (trajectory length or number of single-state annotations) under an explicit budget.
  • Scaling to continuous control will likely require replacing MCMC sampling with a variational posterior over rewards, since the paper identifies Bayesian IRL sampling as the computational bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an active inverse reinforcement learning (IRL) framework in which the learner selects the initial state of a full expert trajectory rather than querying a single state-action pair. The acquisition function is the expected information gain (EIG) under a Bayesian posterior over rewards, approximated by a nested Monte Carlo estimator, and made computationally cheaper by a Bayesian-optimization/UCB allocation scheme. The method is evaluated on two tabular gridworlds against random sampling, Q-entropy, posterior predictive action entropy from Kweon et al. [5], and single-state EIG variants. The paper claims that EIG-based acquisition is superior to baselines in both environments in terms of posterior reward entropy and apprentice regret, and that it exposes a failure mode in the prior trajectory-based acquisition function.

Significance. The formulation is a natural and principled adaptation of Bayesian experimental design to trajectory-level active IRL, and the nested Monte Carlo estimator in Eq. (2) is mathematically sound. The Bayesian-optimization approximation is a reasonable practical contribution. If the empirical superiority claim survives a robustness check, the paper would be a useful step toward reducing demonstration cost in IRL. At present, however, the central claim rests on mean curves from 10 and 16 random reward assignments with no error bars or significance tests, so the significance is conditional on additional validation.

major comments (3)
  1. [Section 4, Fig. 2] The central claim that EIG 'is superior to the baselines' in both environments is supported only by mean curves computed from 10 (structured) and 16 (fully random) reward/terminal assignments, with no error bars, confidence intervals, or significance tests. With these sample sizes, a few high-variance seeds could change the ordering, particularly on the regret metric. Please report per-seed results or confidence intervals, and run paired statistical tests (e.g., Wilcoxon signed-rank) at the relevant active-learning steps. This is load-bearing because the remaining EIG-vs-baseline advantage, after accounting for the query-cost comparison in the 'Single-st. EIG (x/8.8)' curves, is exactly the part that needs variance information.
  2. [Section 3, Eq. (2); Appendix C.3] The EIG estimates use only Nr=20 reward samples and Ntau=2 trajectories per reward, and the BO-EIG variant adds an approximation layer on top of these noisy observations. The manuscript does not report the variance or bias of the EIG estimator, nor any convergence check as Nr or Ntau increases. Since the BO-UCB acquisition rule is driven by these estimates, the reported advantage over baselines could in principle be an artifact of estimator noise rather than of the EIG objective. Please validate the estimator (e.g., bootstrap confidence intervals, or a small sensitivity study increasing Nr and Ntau) and state the resulting uncertainty in the Fig. 2 curves.
  3. [Appendix C.3.1 vs. Section 4] There is a direct inconsistency in the prior used for the structured environment: Section 4 states that the unknown rewards ('water', 'mud', 'lava') have an independent uniform prior in the interval [-100, 0], while Appendix C.3.1 says the reward was drawn from 'independent Uniform[0, 100]'. One of these is a typo, but it matters for reproducibility and for the interpretation of the entropy/regret results. Please correct this and confirm that the experiments used the intended negative-reward prior.
minor comments (5)
  1. [Section 2, Eq. (1); Appendix C.3] The expert rationality beta is fixed to 1 in all experiments, and the demonstrations are generated under the same beta=1 model. This makes the model correctly specified in-sample, but the paper would be stronger if it acknowledged that beta misspecification is a limitation for real human demonstrations and ideally reported at least one sensitivity run.
  2. [Appendix B, Fig. 3] The scaling experiments report '3 repeated trials' but show only means with no variability information. Please add error bars or state the observed spread.
  3. [Section 4, Fig. 2 caption] The caption describes 'Single st. EIG (x / 8.8)' as scaling by the mean trajectory length, but the text should also clarify whether this is an upper-bound comparison or a matched-query-cost comparison. The sentence 'the latter baseline serves as an upper bound' in Appendix C.2 is helpful; consider moving some of that explanation to the main text.
  4. [Appendix C.4] The claim that BO-EIG incurs 'less than a 10% increase in computation time' is based on average timings of 6.6 vs 6.2 seconds. Please report the number of runs and the variance of these timings, since the difference is small.
  5. [Throughout] Minor notation issues: Eq. (2) uses a hat over EIG on the left but not in the text; and the definition of the apprentice policy pi_A in Section 2 would benefit from a precise statement of the expectation over the posterior p(r|D_N).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the EIG acquisition function is defined from the Bayesian model and evaluated against external baselines.

full rationale

The paper defines EIG in Section 3 directly from the current posterior over rewards and the Boltzmann-rational expert model (Eq. 1), then estimates it by nested Monte Carlo (Eq. 2) with Bayesian optimization (Eqs. 3-7). The reported comparisons are against external baselines (random sampling, Q-entropy, posterior predictive action entropy from Kweon et al. [5], and single-state variants) on two gridworlds, with rewards and terminal states generated from the same prior used by the Bayesian IRL method. No parameter is fitted to the experimental outcome: the BO approximation uses hand-chosen priors and kappa=3, and the NMC estimator uses 20 reward samples and 2 trajectories per reward, none of which encode the measured entropy or regret values. The only self-citation is the ValueWalk sampler [20] used as an MCMC engine in the fully random environment; this is a computational subroutine, not a premise that forces the EIG comparison, and no uniqueness or optimality claim for EIG is imported from that citation. The manuscript itself flags limitations (small tabular experiments, deferred scaling analysis, truncation needed for the posterior predictive entropy baseline), but none of these indicate that the central claim reduces by definition to the method's inputs. The absence of error bars or significance tests in Figure 2 is a statistical robustness concern, not a circularity concern under the review rules. There is therefore no significant circularity; the correct score is 0.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The ledger reflects the method's dependence on a correct expert model and known environment dynamics. The free parameters are experimental/hyperparameter choices made by the authors, not fitted to a ground-truth target. No new physical or conceptual entities are introduced beyond the EIG acquisition function and the BO approximation, which are algorithmic constructs rather than invented entities.

free parameters (6)
  • UCB exploration parameter κ = 3
    Hand-chosen in Eq. (7) for the Bayesian optimization allocation; affects the trade-off between exploring initial states and exploiting the current EIG estimate.
  • Expert rationality β = 1
    Assumed known in Eq. (1); all experiments fix β=1 with no sensitivity analysis. The posterior and EIG depend critically on this assumption.
  • Noise prior p_ϕ for EIG estimates = not specified
    Eq. (4) assumes a prior on the noise ε_s but the text never specifies p_ϕ or its parameters, which the MAP update in Eq. (6) depends on.
  • Trajectory length caps = 15 (structured), 10 (random)
    Samples are capped to bound computation; affects EIG estimates and the mean trajectory length (8.8).
  • EIG Monte Carlo sample counts = 20 rewards, 2 trajectories per reward
    Used in the NMC estimator; the small numbers may yield noisy EIG estimates.
  • MCMC warmup and samples = 100 warmup, 200 samples
    Bayesian IRL posterior approximation; limited samples may not fully converge, affecting EIG accuracy.
assumptions (6)
  • domain assumption Expert demonstrations follow the Boltzmann-rational policy π with fixed β (Eq. 1).
    The posterior and EIG depend on this exact likelihood model; if the expert is not Boltzmann-rational or β is unknown, the EIG estimates can mislead.
  • domain assumption The prior over rewards p(r) is known and correctly specified (e.g., Uniform[-100,0] or N(0,3)).
    Used to initialize the posterior and compute EIG; a misspecified prior biases the acquisition.
  • domain assumption Environment transition dynamics p_ξ are known and used to simulate hypothetical trajectories.
    The EIG estimator relies on sampling trajectories from p_ξ; unknown dynamics would break the method.
  • domain assumption The posterior p(r|D_n) is approximately represented by the MCMC samples.
    All EIG estimates and the apprentice policy use these samples; non-convergence would invalidate the acquisition.
  • domain assumption State types are known a priori in the structured environment (three unknown rewards shared across states).
    The structured experiment exploits this factorization; the method would need modification if reward structure is unknown.
  • standard math Bayes' rule and Monte Carlo sampling are valid for the nested EIG estimator.
    Standard probability; no special justification needed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Toward Information Theoretic Active Inverse Reinforcement Learning." pith.science (2026). https://pith.science/paper/FVHDBY34

@misc{pith2026250100381,
  author       = {Pith},
  title        = {Pith review of: Toward Information Theoretic Active Inverse Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FVHDBY34}},
  note         = {Machine review of arXiv:2501.00381}
}
read the original abstract

As AI systems become increasingly autonomous, aligning their decision-making to human preferences is essential. In domains like autonomous driving or robotics, it is impossible to write down the reward function representing these preferences by hand. Inverse reinforcement learning (IRL) offers a promising approach to infer the unknown reward from demonstrations. However, obtaining human demonstrations can be costly. Active IRL addresses this challenge by strategically selecting the most informative scenarios for human demonstration, reducing the amount of required human effort. Where most prior work allowed querying the human for an action at one state at a time, we motivate and analyse scenarios where we collect longer trajectories. We provide an information-theoretic acquisition function, propose an efficient approximation scheme, and illustrate its performance through a set of gridworld experiments as groundwork for future work expanding to more general settings.

Figures

Figures reproduced from arXiv: 2501.00381 by the authors.

Figure 1
Figure 1. (a) shows an illustrative gridworld and its true rewards. The lower left corner has a "jail" [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Results on two gridworld environments comparing EIG-based methods with baselines. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. These plots show that the EIG calculation step scales approximately quadratically in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 15 canonical work pages

  1. [5]

    Trajectory-based active inverse reinforce- ment learning for learning from demonstration

    Sehee Kweon, Himchan Hwang, and Frank C Park. Trajectory-based active inverse reinforce- ment learning for learning from demonstration. In 2023 23rd International Conference on Control, Automation and Systems (ICCAS), pages 1807–1812. IEEE, 2023

    work page 2023

  2. [1]

    Human Compatible: Artificial Intelligence and the Problem of Control

    Stuart Russell. Human Compatible: Artificial Intelligence and the Problem of Control. Penguin Random House, 2019

    work page 2019

  3. [2]

    Active learning for reward estimation in inverse reinforcement learning

    Manuel Lopes, Francisco Melo, and Luis Montesano. Active learning for reward estimation in inverse reinforcement learning. In Wray Buntine, Marko Grobelnik, Dunja Mladeni ´c, and John Shawe-Taylor, editors,Machine learning and knowledge discovery in databases, pages 31–46, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. ISBN 978-3-642-04174-7

    work page 2009

  4. [3]

    Brown, Yuchen Cui, and Scott Niekum

    Daniel S. Brown, Yuchen Cui, and Scott Niekum. Risk-Aware Active Inverse Reinforcement Learning. In Proceedings of The 2nd Conference on Robot Learning, pages 362–372. PMLR, Oc- tober 2018. URL https://proceedings.mlr.press/v87/brown18a.html. ISSN: 2640- 3498

    work page 2018

  5. [4]

    Provably Efficient Learning of Transferable Rewards

    Alberto Maria Metelli, Giorgia Ramponi, Alessandro Concetti, and Marcello Restelli. Provably Efficient Learning of Transferable Rewards. InProceedings of the 38th International Conference on Machine Learning, pages 7665–7676. PMLR, July 2021. URL https://proceedings. mlr.press/v139/metelli21a.html. ISSN: 2640-3498

    work page 2021

  6. [6]

    Ivanova, and Freddie Bickford Smith

    Tom Rainforth, Adam Foster, Desi R. Ivanova, and Freddie Bickford Smith. Modern Bayesian Experimental Design, February 2023. URL http://arxiv.org/abs/2302.14545. arXiv:2302.14545 [cs, stat]

    arXiv 2023

  7. [7]

    Peter I. Frazier. A Tutorial on Bayesian Optimization. arXiv:1807.02811, July 2018. URL http://arxiv.org/abs/1807.02811. arXiv: 1807.02811

    arXiv 2018

  8. [8]

    Bayesian optimization

    Roman Garnett. Bayesian optimization. Cambridge University Press, 2023

    2023

Show all 24 references
  1. [9]

    A versatile stochastic model of a function of unknown and time varying form

    Harold J Kushner. A versatile stochastic model of a function of unknown and time varying form. Journal of Mathematical Analysis and Applications, 5(1):150–167, 1962. Publisher: Elsevier. 6

    1962

  2. [10]

    Learning agents for uncertain environments (extended abstract)

    Stuart Russell. Learning agents for uncertain environments (extended abstract). In Proceedings of the eleventh annual conference on Computational learning theory, pages 101–103, Madison Wisconsin USA, July 1998. ACM. ISBN 978-1-58113-057-7. doi: 10.1145/279943.279964. URL http...

    1998

  3. [11]

    R. E. Kalman. When Is a Linear Control System Optimal? Journal of Basic Engineering, 86 (1):51–60, March 1964. ISSN 0021-9223. doi: 10.1115/1.3653115. URL https://doi.org/ 10.1115/1.3653115

    1964 doi

  4. [12]

    A survey of inverse reinforcement learning: Challenges, methods and progress

    Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, August 2021. ISSN 00043702. doi: 10.1016/j.artint.2021.103500. URL https://linkinghub.elsevier.com/retrieve/ pii/S0004370221000515

    2021

  5. [13]

    Stephen Adams, Tyler Cody, and Peter A. Beling. A survey of inverse reinforce- ment learning. Artificial Intelligence Review , February 2022. ISSN 0269-2821, 1573-

    2022

  6. [14]

    Bayesian Inverse Reinforcement Learning

    Deepak Ramachandran and Eyal Amir. Bayesian Inverse Reinforcement Learning. In Proceed- ings of the Twentieth International Joint Conference on Artificial Intelligence, 2007

    2007

  7. [15]

    Environment Design for Inverse Reinforcement Learning

    Thomas Kleine Buening, Victor Villin, and Christos Dimitrakakis. Environment Design for Inverse Reinforcement Learning. In Proceedings of the 41st International Conference on Machine Learning, pages 24808–24828. PMLR, July 2024. URL https://proceedings. mlr.press/v235/kleine-b...

    2024

  8. [16]

    Interactive Inverse Reinforcement Learning for Cooperative Games

    Thomas Kleine Büning, Anne-Marie George, and Christos Dimitrakakis. Interactive Inverse Reinforcement Learning for Cooperative Games. In Proceedings of the 39th International Conference on Machine Learning , pages 2393–2413. PMLR, June 2022. URL https:// proceedings.mlr.press/...

    2022

  9. [17]

    Active Preference-Based Learning of Reward Functions

    Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active Preference-Based Learning of Reward Functions. In Robotics: Science and Systems XIII. Robotics: Science and Systems Foundation, July 2017. ISBN 978-0-9923747-3-0. doi: 10.15607/RSS.2017.XIII.053. URL http://w...

    2017 doi

  10. [18]

    Losey and Marcia K

    Dylan P. Losey and Marcia K. O’Malley. Including Uncertainty when Learning from Human Cor- rections. In Proceedings of The 2nd Conference on Robot Learning, pages 123–132. PMLR, Oc- tober 2018. URL https://proceedings.mlr.press/v87/losey18a.html. ISSN: 2640- 3498

    2018

  11. [19]

    Active Exploration for Inverse Reinforcement Learning

    David Lindner, Andreas Krause, and Giorgia Ramponi. Active Exploration for Inverse Reinforcement Learning. Advances in Neural Information Processing Systems , 35:5843– 5853, December 2022. URL https://proceedings.neurips.cc/paper/2022/hash/ 26d01e5ed42d8dcedd6aa0e3e99cffc4-Abs...

    2022

  12. [20]

    Ondrej Bajgar, Konstantinos Gatsis, Alessandro Abate, and Michael A. Osborne. Walking the Values in Bayesian Inverse Reinforcement Learning. In Proceedings of the 40th Conference on Uncertainty in Artificial Intelligence, 2024

    2024

  13. [21]

    Scalable Bayesian Inverse Reinforcement Learning

    Alex J Chan and Mihaela van der Schaar. Scalable Bayesian Inverse Reinforcement Learning. ICLR 2021, 2021

    2021

  14. [22]

    Simon Duane, A. D. Kennedy, Brian J. Pendleton, and Duncan Roweth. Hybrid Monte Carlo. Physics Letters B, 195(2):216–222, September 1987. doi: 10.1016/0370-2693(87)91197-X

    1987 doi

  15. [23]

    upper bound

    Matthew D Hoffman and Andrew Gelman. The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15:1593–1623, 2014. 7 A Related work Our work builds on two strands of work: inverse reinforcement learning (IRL) and a...

    2014

  16. [7462]

    URL https://link.springer.com/10.1007/ s10462-021-10108-x

    doi: 10.1007/s10462-021-10108-x. URL https://link.springer.com/10.1007/ s10462-021-10108-x

    doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.