Inverse Reinforcement Learning using Revealed Preferences and Passive Stochastic Optimization

Vikram Krishnamurthy

REVIEW 3 major objections 6 minor 23 references

Inverse Reinforcement Learning using Revealed Preferences and Passive Stochastic Optimization

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An inverse learner can recover a hidden reward function just by watching a learner's noisy gradient updates.

desk verdict The passive Langevin IRL algorithm in Chapter 3 is genuinely new and worth engaging, but its main convergence claim is cited to the author's prior paper rather than proved here; Chapters 1-2 are a clean restatement of known revealed-preference results. read the letter →

arxiv 2507.04396 v1 pith:XWHXRFHS submitted 2025-07-06 cs.LG eess.SP

Vikram Krishnamurthy This is my paper · ORCID

classification cs.LGeess.SP

keywords inversereinforcementlearningrevealedpreferencesAfriat'stheorempassiveLangevindynamicsstochasticgradientalgorithmsBayesianIRLcognitiveradarGibbsmeasure

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Inverse reinforcement learning (IRL) aims to recover an agent's reward function from its observed decisions; this monograph argues that two passive, nonparametric routes can do it. The first route, taken from revealed-preference theory, checks whether a finite dataset of probe–response pairs could have been produced by a constrained utility maximizer, and if so returns the whole family of utility functions that rationalize the data, not a single point estimate. The second route is an adaptive algorithm that runs while reinforcement learners perform stochastic gradient ascent on a possibly time-varying reward: an inverse learner passively observes their noisy gradient evaluations and runs a kernel-weighted Langevin recursion whose samples asymptotically follow the Gibbs measure $p(\alpha)\propto \exp(\beta R(\alpha))$. Since the reward is then the logarithm of the sample density (up to the scale factor $\beta$), the hidden reward is reconstructed in real time without the inverse learner ever choosing where gradients are evaluated. The same machinery is applied to detect cognitive radar, to test whether a sensor is a Bayes-optimal sequential detector, and to mask a utility from an adversary's IRL.

What carries the argument

The load-bearing identity is the reversible-diffusion stationary measure: a diffusion with drift $(\beta/2)\pi(\alpha)\nabla R(\alpha)+\nabla\pi(\alpha)$ multiplied by $\pi(\alpha)$ and unit diffusion coefficient has stationary density $p(\alpha)\propto \exp(\beta R(\alpha))$. The paper's algorithm (3.2) is an Euler–Maruyama discretization of this diffusion, made passive by inserting the kernel $K_\Delta(\theta-\alpha)$ so that a noisy gradient evaluated at the wrong point $\theta_k$ contributes to the update at $\alpha_k$ with weight proportional to its proximity. Two averaging steps then complete the mechanism: stochastic averaging replaces the fast RL iterates by their stationary distribution $\pi$, and the kernel-width limit $\Delta\to 0$ turns the kernel into a Dirac delta, delivering the reversible diffusion above. On the revealed-preference side, the corresponding identity is Afriat's theorem—GARP (1.4) is equivalent to feasibility of the linear inequalities (1.2), whose solutions parametrize the concave piecewise-linear utilities (1.3) that rationalize the dataset—and its Bayesian analogue NIAS/NIAC (2.11)–(2.12) for rationally inattentive agents.

What would settle it

For a reward $R(\theta)=-\frac12\|\theta\|^2$ the claimed stationary measure is a Gaussian $\mathcal{N}(0,I/\beta)$; run algorithm (3.2) while several learners run gradient ascent on this reward, collect a long sample path $\{\alpha_k\}$, and test whether the empirical log-density is quadratic with the predicted curvature $-(\beta/2)\|\alpha\|^2$ plus a constant. A persistent mismatch that does not shrink as $\mu$ and $\Delta$ go to zero would refute the Gibbs-measure claim.

Watch

Extended reading notes

Core claim

The central claim of Chapter 3 is that the passive Langevin IRL algorithm (3.2) asymptotically generates samples $\{\alpha_k\}$ from the Gibbs measure $p(\alpha)\propto \exp(\beta R(\alpha))$ with $\beta=\varepsilon/\mu$, where $\varepsilon$ is the step size of the reinforcement learners' gradient algorithm (3.1) and $\mu$ is the step size of the inverse learner's recursion. The recursion weights each observed noisy gradient $\nabla_\theta r_k(\theta_k)$ by a kernel $K_\Delta(\theta_k-\alpha_k)$ that decays with the distance between the learner's evaluation point and the inverse learner's current estimate, incorporates the learners' initialization density $\pi$, and injects Gaussian noise; under stochastic averaging and the $\Delta\to 0$ kernel limit, the interpolated trajectory converges weakly to a reversible diffusion whose stationary measure is exactly $\exp(\beta R)$. Consequently the expected reward $R(\theta)=\mathbb{E}\{r_k(\theta)\}$ can be recovered nonparametrically as the logarithm of the empirical density of $\{\alpha_k\}$. Chapters 1 and 2 make the complementary claim that a dataset of probes and responses is consistent with (Bayesian) utility maximization if and only if a set of linear inequalities—Afriat's inequalities (1.2) for the classical case, NIAS and NIAC (2.11)–(2.12) for rationally inattentive Bayesian agents—is feasible, and every feasible solution defines a set of utility functions that rationalize the data.

Load-bearing premise

The load-bearing premise is that the points where the reinforcement learners evaluate their gradients move around on a much faster time scale than the inverse learner's own estimate updates, so the kernel-weighted noisy gradient can be replaced by its expectation over the learners' typical behavior; if the learners evolve too slowly, the samples generated by the inverse algorithm will not follow $\exp(\beta R)$.

Editorial extensions

If this is right

The inverse learner can reconstruct the entire reward function from passively observed gradient streams, without ever specifying where gradients should be evaluated; the log-empirical-density estimator recovers $R$ up to the scale $\beta$.
Revealed-preference IRL yields set-valued utility estimates: the feasible polytope of Afriat's inequalities (or NIAS/NIAC) contains every utility that rationalizes the dataset, so predictions for future probes come as intervals or confidence regions rather than single points.
The same linear-inequality test applies to Bayesian stopping-time problems, so an analyst can decide whether a sequential detector's decisions are Bayes-optimal and reconstruct its stopping and continue costs from observed action frequencies alone.
Because the inverse recursion uses a constant step size, it can track time-varying rewards; when the reward jumps according to a Markov chain, the interpolated IRL trajectory converges to a Markov-switched diffusion whose limit behavior is analyzed in three time-scale regimes.
A cognitive sensor that knows its own utility can deliberately shrink the feasibility margin of Afriat's test (by solving (1.27)), hiding its cognition from an adversary's IRL at a small performance cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the passive-Langevin claim extends to arbitrary stochastic-gradient logs, reward models could be extracted from recorded SGD training histories of deep networks, a setting the paper does not discuss but whose ingredients (random evaluation points, noisy gradients) match its assumptions.
The multikernel variant (3.20) is offered as a high-dimensional remedy; a natural follow-up is to measure empirically how its self-normalized importance-weighting bias trades off against variance reduction, and how the optimal pool size $L$ scales with dimension $N$.
The paper's Type-I error bound for the noisy-detection test suggests that the choice of probe vectors in active IRL could be formulated as a Bayesian design problem; the paper optimizes probes via SPSA, but a closed-form characterization of the optimal probe remains open.
The tracking analysis predicts that when the environment changes at the same rate as the IRL updates ($\eta=O(\mu)$), the reconstructed reward is a mixture over recent rewards; an explicit simulation test of this mixing behaviour would cleanly separate the paper's contribution from classical stochastic-gradient tracking.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague—

The genuinely new piece here is Chapter 3: a passive Langevin algorithm that reconstructs a reward function from noisy gradients observed from RL agents, plus a multi-kernel variance-reduced variant. Chapters 1 and 2 are a clean, correct restatement of Afriat’s theorem and the Caplin–Dean NIAS/NIAC machinery, applied to cognitive radar and Bayesian stopping problems. That material is not new, but it is well organized and useful.

The passive Langevin idea is worth attention. The algorithm (3.2) uses a kernel to weight gradients evaluated at θ_k when the IRL iterate is α_k, and the heuristic in Sec. 3.3.1 shows why the stationary measure should be exp(βR(α)). The numerics comparing against classical Langevin and Metropolis–Hastings are reassuring, and the multi-kernel version (3.20) that avoids knowing the initialization density π is a genuine practical addition.

The soft spot is exactly where the reader put it: the central claim (3.4) is not proved in this manuscript. The informal proof does state the bounded-run assumption τ_{n+1}−τ_n < M, which is the load-bearing condition for the stochastic averaging step, but the actual theorem is said to be in Sec. 3.5 and then cited to the author’s earlier paper [KY21]. A reader cannot verify the claim from the monograph alone. The numerics also use short runs (100 iterations, ε = 1e−3), so they stay close to the initialization and do not probe the regime where the bounded-run condition fails. I don’t read the self-citation as a problem — the proof belongs in the earlier paper — but it does make this document non-self-contained on its main new result.

If the formal proof in [KY21] is sound, this is a solid contribution to IRL and controlled sensing. Who gets value: researchers working on inverse reinforcement learning from passive observations, and anyone wanting a compact summary of revealed-preference tests with radar examples. It deserves a serious referee; the referee should insist that the main theorem be stated in the paper with its hypotheses, or at least that the bounded-run condition be explicit in the statement of (3.4).

Referee Report

3 major / 6 minor

Summary. The manuscript is a three-chapter monograph on inverse reinforcement learning (IRL). Chapter 1 recasts IRL as revealed-preference analysis: it restates Afriat's theorem, extends it to nonlinear budgets, applies it to cognitive-radar waveform and beam allocation, and develops statistical detection and utility-masking procedures. Chapter 2 develops Bayesian IRL for rationally inattentive agents, presenting NIAS/NIAC feasibility tests for one-step decisions and for Bayesian stopping-time problems, with applications to inverse sequential detection, search, and quickest detection. Chapter 3 proposes adaptive IRL by passive Langevin dynamics: observing noisy gradients produced by multiple stochastic-gradient RL agents, the IRL algorithm (3.2) is claimed to generate samples asymptotically from the Gibbs measure p(alpha) proportional to exp(beta R(alpha)), so that R can be reconstructed nonparametrically from the empirical density of the samples. The chapter also presents multi-kernel, active, and non-reversible variants and gives numerical comparisons against classical Langevin dynamics and Metropolis-Hastings.

Significance. If the main convergence claim (3.4) holds, the paper offers a genuinely new capability: nonparametric reconstruction of a reward function from passively observed, misspecified noisy gradients, in real time and with a constant step size that permits tracking of time-varying rewards. This would be useful in adversarial controlled sensing, adaptive Bayesian learning, and inverse stochastic control. The first two chapters are careful, largely correct restatements of published revealed-preference results, and the manuscript is explicit about the set-valued and ordinal nature of reconstructed utilities, which is a strength. The numerical work in Chapter 3 is honest in comparing against classical Langevin sampling and Metropolis-Hastings, and reports quantitative variational distances. The central contribution, however, is conditional on a formal weak-convergence theorem that is not present in the submitted text: the proof is deferred to Section 3.5, which is not included, and is cited to the author's own prior work [KY21]. The load-bearing stochastic-averaging assumption is stated only informally and is not reflected in the main claim as written.

major comments (3)

[Sec. 3.3.1, Eqs. (3.11)-(3.13), and claim (3.4)] The Gibbs-measure claim (3.4) rests on the stochastic-averaging step in Eq. (3.11), which replaces the kernel-weighted noisy gradient at theta_k by its expectation over theta~pi. This replacement is valid only if the RL iterates theta_k are a fast variable with stationary distribution pi. The model (3.1) does not provide such mixing: with epsilon = beta mu -> 0, a bounded-length run keeps theta_k within O(epsilon) of its pi-initialization, whereas an unbounded run drives theta_k toward local maxima of R. The bounded-run condition tau_{n+1}-tau_n < M is introduced only in the informal proof (Sec. 3.3.1, item 2) and is absent from the statement of (3.4) and from the algorithm description. The formal theorem in Sec. 3.5 is not included in the manuscript and is cited to [KY21]. The formal statement must be included and must state the bounded-run or reset conditions; otherwise (3.4) is not a theorem about algorithm (3.2) with the RL protocol (3.1) as described.
[Sec. 3.4.1] The numerical validation of (3.4) operates entirely in the favorable regime of the bounded-run assumption: each RL agent runs only 100 iterations with epsilon = 10^{-3}, so theta_k remains close to its initialization and the empirical distribution of theta is close to pi. This experiment does not probe the regime where the RL agents run until convergence, which is precisely the regime where the averaging in (3.11) fails and the stationary measure of the IRL algorithm is not exp(beta R). Please add experiments with large or unbounded run lengths, or alternatively restrict the formal claim explicitly to the bounded-run case, since this assumption is load-bearing for the central result.
[Sec. 3.3.2, Eq. (3.20)] The multi-kernel IRL algorithm (3.20) is claimed to converge to the same Gibbs measure, with the formal proof again cited to [KY21] rather than included. The informal argument uses the self-normalized importance-sampling limit (3.23), which requires L -> infinity for fixed alpha, followed by a small-variance Bernstein-von Mises limit so that the conditional density acts as a Dirac measure. The relative rates of L, mu, and sigma^2 are not specified, so the order of limits in (3.24)-(3.25) is not justified in the manuscript. If the multi-kernel algorithm is advertised as a contribution, the missing theorem and rate conditions need to be supplied.

minor comments (6)

[Sec. 3.4.1] The parameters state epsilon = 10^{-3} and mu = 5 x 10^{-4}, which gives beta = epsilon/mu = 2, but the text says 'beta = 1'; please correct this inconsistency or state the actual values used in the simulations.
[Sec. 3.1, after Eq. (3.1)] The observer's reconstruction of the gradient as (theta_{k+1} - theta_k)/mu yields beta grad r_k(theta_k), not grad r_k(theta_k), since theta_{k+1} - theta_k = epsilon grad r_k(theta_k); although the scale factor beta is later absorbed, the presentation should make this explicit so the reader can see how the algorithm is implemented without knowing epsilon.
[Sec. 3.3.1, Eqs. (3.12)-(3.13)] The informal proof takes the double limit mu -> 0 for fixed Delta and then Delta -> 0; the claim (3.4) should state this order of limits explicitly, since for any fixed Delta > 0 the stationary distribution of the averaged diffusion (3.12) is not exactly exp(beta R).
[Contents and Chapter 3] The table of contents lists Sec. 3.5 (Weak Convergence Analysis) and Sec. 3.6 (Tracking Analysis), but the submitted text breaks off inside Sec. 3.4.2; these sections contain the formal statements that underlie the main claims of Chapter 3 and must be supplied.
[Sec. 1.8, Theorem 1.11] The proof uses the fact that the complementary CDF ar F_M(M) is uniform in [0,1]; this holds only when the random variable M has a continuous distribution, and this regularity condition on the noise sequence should be stated explicitly.
[Summary and Sec. 3.2] The Summary states that the monograph is work in progress and that Chapter 3 draws heavily on [KY21]; for a journal submission, the relationship to [KY21] should be clarified so that the reader can identify which results are new in this manuscript.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular reduction found: the central Langevin-IRL claim is a designed sampler with independent numerical benchmarks, and the self-citations to [KY21] are not load-bearing.

full rationale

The claimed convergence (3.4) is not circular. Algorithm (3.2) is deliberately constructed as a passive sampler: its drift and diffusion terms are chosen so that the Fokker–Planck calculation in (3.13)-(3.14) has p(α) ∝ exp(βR(α)) as the stationary measure, and the kernel-weighted gradient terms are then shown by stochastic averaging (3.11)-(3.12) to converge to that diffusion in the limits μ→0 and Δ→0. The target R is not used inside the update; the update uses only the observed noisy gradients ∇θ r_k(θ_k) and the known initialization density π. The numerics in Sec.3.4.1 benchmark the reconstruction against classical Langevin dynamics and a Metropolis–Hastings ground truth on a known multimodal R, which is genuine external validation rather than fitting or renaming. The manuscript explicitly states that the formal weak-convergence proof appears in Sec.3.5 and that the multikernel variant's formal proof is in the author's own [KY21]; this is a self-citation, but it is not load-bearing because the central claim is independently supported by the informal derivation in Sec.3.3.1, the stated formal section, and the reproducible numerical comparisons. The bounded-run condition τ_{n+1}−τ_n < M is an explicit modeling assumption used to justify the stochastic-averaging step (Sec.3.3.1, item 2), not a hidden reuse of the conclusion; if that mixing condition fails, the invariant measure would differ, which is a correctness caveat rather than circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The free parameters are standard algorithm hyperparameters rather than quantities fitted to reproduce the target result. The main load-bearing assumptions are the utility-maximization model of cognitive radar and the stochastic averaging premises behind the Langevin convergence.

free parameters (6)

kernel bandwidth Δ = 0.02 (numerical example)
Chosen by hand for the simulation; the convergence result requires Δ→0.
IRL step size μ = 5e-4 (numerical example)
Constant step size; weak convergence as μ→0.
RL step size ε = 1e-3 (numerical example)
Step size of the forward stochastic gradient agents; β=ε/μ=1 in the example.
inverse temperature β = 1
Ratio ε/μ; the stationary measure is exp(βR), so β scales the reconstructed reward and is not identifiable from the samples alone.
number of gradient samples L = 10^5 agents
Monte Carlo pool for the multi-kernel algorithm; large L approximates the conditional expectation.
noise variance σ² = 0.02 kernel variance
Controls the kernel width and the Gaussian smoothing in the active and multi-kernel algorithms.

assumptions (5)

standard math Afriat's theorem: a finite dataset is consistent with utility maximization iff GARP holds iff Afriat's inequalities are feasible.
Classical result used throughout Chapter 1; the paper includes a proof.
domain assumption The cognitive radar is modeled as a constrained utility maximizer with linear budget α_k' β ≤ 1, justified by the information Kalman filter formulation (Lemma 1.5).
Assumes the radar's waveform and beam choices can be encoded as utility maximization under an SNR bound; this is a modeling assumption, not a theorem.
standard math Caplin and Dean's BRP test: NIAS and NIAC linear inequalities are necessary and sufficient for rational inattention utility maximization.
Used in Chapter 2; the paper proves necessity and cites sufficiency.
domain assumption Stochastic averaging and weak convergence of constant step-size stochastic approximations.
The informal proof of the passive Langevin convergence (Sec 3.3.1) relies on the standard averaging theory; the formal proof is cited to [KY21].
ad hoc to paper The RL agents' iterates θ_k have stationary distribution π and the noisy gradients average to ∇R(θ).
Assumed for the passive Langevin IRL; if the forward learners do not mix to π on the fast time scale, the stationary measure is not exp(βR).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Inverse Reinforcement Learning using Revealed Preferences and Passive Stochastic Optimization." pith.science (2026). https://pith.science/paper/XWHXRFHS

@misc{pith2026250704396,
  author       = {Pith},
  title        = {Pith review of: Inverse Reinforcement Learning using Revealed Preferences and Passive Stochastic Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XWHXRFHS}},
  note         = {Machine review of arXiv:2507.04396}
}

read the original abstract

This monograph, spanning three chapters, explores Inverse Reinforcement Learning (IRL). The first two chapters view inverse reinforcement learning (IRL) through the lens of revealed preferences from microeconomics while the third chapter studies adaptive IRL via Langevin dynamics stochastic gradient algorithms. Chapter uses classical revealed preference theory (Afriat's theorem and extensions) to identify constrained utility maximizers based on observed agent actions. This allows for the reconstruction of set-valued estimates of an agent's utility. We illustrate this procedure by identifying the presence of a cognitive radar and reconstructing its utility function. The chapter also addresses the construction of a statistical detector for utility maximization behavior when agent actions are corrupted by noise. Chapter 2 studies Bayesian IRL. It investigates how an analyst can determine if an observed agent is a rationally inattentive Bayesian utility maximizer (i.e., simultaneously optimizing its utility and observation likelihood). The chapter discusses inverse stopping-time problems, focusing on reconstructing the continuation and stopping costs of a Bayesian agent operating over a random horizon. We then apply this IRL methodology to identify the presence of a Bayes-optimal sequential detector. Additionally, Chapter 2 provides a concise overview of discrete choice models, inverse Bayesian filtering, and inverse stochastic gradient algorithms for adaptive IRL. Finally, Chapter 3 introduces an adaptive IRL approach utilizing passive Langevin dynamics. This method aims to track time-varying utility functions given noisy and misspecified gradients. In essence, the adaptive IRL algorithms presented in Chapter 3 can be conceptualized as inverse stochastic gradient algorithms, as they learn the utility function in real-time while a stochastic gradient algorithm is in operation.

Figures

Figures reproduced from arXiv: 2507.04396 by the authors.

**Figure 1.** [PITH_FULL_IMAGE:figures/full_fig_p018_1.png] view at source ↗

**Figure 1.2.** Hiding utility from adversary. The convex polytopes are specified by Afriat’s inequalities. We design the perturbed dataset D˜ so that MD˜ < M𝑈. Feasibility Margin for Afriat’s Test To discuss utility masking, we first define the feasibility margin for Afriat’s test. Recall that, given probe signals 𝛼1:𝑁 , a utility maximizer generates its response as 𝛽𝑘 ∈ argmax 𝛼 ′ 𝑘 𝛽≤1 𝑈(𝛽), 𝑘 = 1, . . . , 𝑁. (1.24) This yields … view at source ↗

Figures from the paper (9 more)

**Figure 1.** Figure 1 [PITH_FULL_IMAGE:figures/full_fig_p028_1.png]

**Figure 2.** Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p031_2.png]

**Figure 2.** Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p049_2.png]

**Figure 3.** Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p055_3.png]

**Figure 3.** Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p067_3.png]

**Figure 3.** Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p068_3.png]

**Figure 3.** Figure 3: (a) displays the cumulative reward [PITH_FULL_IMAGE:figures/full_fig_p073_3.png]

**Figure 3.** Figure 3: (b) displays the penalized cumulative reward [PITH_FULL_IMAGE:figures/full_fig_p074_3.png]

**Figure 3.** Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p076_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 18 canonical work pages

[1]

The construction of utility functions from expenditure data

[Afr67] S. Afriat. “The construction of utility functions from expenditure data”. In: International Eco- nomic Review 8.1 (1967), pp. 67–77. [Afr87] S. Afriat. Logic of Choice and Economic Theory. Clarendon Press,

work page 1967
[3]

Finite-sample bounds for adaptive inverse reinforcement learn- ing using passive langevin dynamics

Elsevier, 2010, pp. 155–181. [SK25] L. Snow and V. Krishnamurthy. “Finite-sample bounds for adaptive inverse reinforcement learn- ing using passive langevin dynamics”. In: IEEE Transactions on Information Theory 71.6 (2025), pp. 4637–4670. [SKS23] L. Snow, V. Krishnamurthy, and B. M. Sadler. “Identifying coordination in a cognitive radar net- work – A mul...

work page 2025
[4]

The strong ergodic theorem for densities: generalized Shannon-McMillan- Breiman theorem

[Bar85] A. R. Barron. “The strong ergodic theorem for densities: generalized Shannon-McMillan- Breiman theorem”. In: The Annals of Probability 13.4 (1985), pp. 1292–1303. [Bil99] P. Billingsley. Convergence of Probability Measures. 2nd ed. Wiley,

work page 1985
[8]

Fine-tuning language models from human preferences

Chicago, IL, USA. 2008, pp. 1433–1438. [Zie+20] D. M. Ziegler et al. “Fine-tuning language models from human preferences”. In:arXiv preprint arXiv:1909.08593v2 (2020). [ZYZ21] F. Zou, G. G. Yen, and C. Zhao. “Dynamic multiobjective optimization driven by inverse reinforcement learning”. In: Information Sciences 575 (2021), pp. 468–484

arXiv 2020
[15]

Unifying Revealed Preference and Revealed Rational Inattention

[PK23a] K. Pattanayak and V. Krishnamurthy. “Necessary and sufficient conditions for inverse reinforce- ment learning of Bayesian stopping time problems”. In: Journal of Machine Learning Research 24.52 (2023), pp. 1–64. [PK23b] K. Pattanayak and V. Krishnamurthy. “Unifying revealed preference and revealed rational inattention”. In: (2023). eprint: arXiv:2...

work page Pith review arXiv 2023
[16]

Analysis of recursive stochastic algorithms

2020, pp. 1–10. [Lju77] L. Ljung. “Analysis of recursive stochastic algorithms”. In: IEEE Transactions on Automatic Control AC-22.4 (1977), pp. 551–575. [LK12] S. Levine and V. Koltun. “Continuous inverse optimal control with locally optimal examples”. In: arXiv preprint arXiv:1206.4617 (2012). [LY16] D. Luenberger and Y. Ye. Linear and Nonlinear Programm...

arXiv 1977
[21]

Langevin-type models I: diffusions with given stationary distri- butions and their discretizations

[ST99] O. Stramer and R. L. Tweedie. “Langevin-type models I: diffusions with given stationary distri- butions and their discretizations”. In: Methodology and Computing in Applied Probability 1.3 (1999), pp. 283–306. [TTV16] Y. W. Teh, A. H. Thiery, and S. J. Vollmer. “Consistency and fluctuations for stochastic gradient Langevin dynamics”. In: The Journa...

work page 1999
[500]

Regularized Inverse Reinforcement Learning

[JE09] B. E. Jones and D. L. Edgerton. “Testing utility maximization with measurement errors in the data”. In: Measurement Error: Consequences, Applications and Solutions. Emerald Group Publishing Limited, 2009, pp. 199–236. [Jeo+20] W. Jeon et al. “Regularized inverse reinforcement learning”. In:arXiv preprint arXiv:2010.03691 (2020). [JK25] A. Jain and ...

work page Pith review arXiv 2020

Show all 23 references

[1979]

Apprenticeship learning via inverse reinforcement learning

[AN04] P. Abbeel and A. Y. Ng. “Apprenticeship learning via inverse reinforcement learning”. In: International Conference on Machine Learning. 2004, p

2004
[1983]

Non-convex learning via stochastic gradient Langevin dynamics: a nonasymptotic analysis

[RRT17] M. Raginsky, A. Rakhlin, and M. Telgarsky. “Non-convex learning via stochastic gradient Langevin dynamics: a nonasymptotic analysis”. In: arXiv preprint arXiv:1702.03849 (2017). [Rus94] J. Rust. “Structural estimation of Markov decision processes”. In: Handbook of Econ...

2017 arXiv
[1984]

Real-time reinforcement learning of constrained Markov decision processes with weak derivatives

[KV18] V. Krishnamurthy and F. Vazquez Abad. “Real-time reinforcement learning of constrained Markov decision processes with weak derivatives”. In:arXiv preprint arXiv:1110.4946 (2018). [Kwo+20] M. Kwon, S. Daptardar, P. Schrater, and X. Pitkow. “Inverse rational control with ...

2018 arXiv
[1986]

Privacy preserving classification on local differential privacy in data centers

[Fan+20] W. Fan et al. “Privacy preserving classification on local differential privacy in data centers”. In: Journal of Parallel and Distributed Computing 135 (2020), pp. 70–82. [FBP19] X. Fontaine, Q. Berthet, and V. Perchet. “Regularized contextual bandits”. In: Internation...

2020 arXiv
[1987]

Thompson sampling for contextual bandits with linear payoffs

[AG13] S. Agrawal and N. Goyal. “Thompson sampling for contextual bandits with linear payoffs”. In: International Conference on Machine Learning. 2013, pp. 127–135. [AK03] F. V. Abad and V. Krishnamurthy. “Constrained stochastic approximation algorithms for adaptive control of...

2013
[1994]

Maximum margin planning

[RBZ06] N. D. Ratliff, J. A. Bagnell, and M. A. Zinkevich. “Maximum margin planning”. In: Interna- tional Conference on Machine Learning. 2006, pp. 729–736. [RC13] C. P. Robert and G. Casella. Monte Carlo Statistical Methods. Springer-Verlag,

2006
[1996]

Identifiability in inverse reinforcement learning

[CCS21] H. Cao, S. Cohen, and L. Szpruch. “Identifiability in inverse reinforcement learning”. In: Advances in Neural Information Processing Systems34 (2021), pp. 12362–12373. [CD15] A. Caplin and M. Dean. “Revealed preference, rational inattention, and costly information ac- ...

2021
[1999]

Risk-constrained markov decision processes

[BJ10] V. Borkar and R. Jain. “Risk-constrained markov decision processes”. In: 49th IEEE Conference on Decision and Control (CDC). IEEE. 2010, pp. 2664–2669. [BKP07] M. Babaioff, R. Kleinberg, and C. Papadimitriou. “Congestion games with malicious players”. In: ACM Conference...

2010
[2000]

Revealed preference and its applications

[Var12] H. Varian. “Revealed preference and its applications”. In:The Economic Journal122.560 (2012), pp. 332–338. [Var82] H. Varian. “The nonparametric approach to demand analysis”. In: Econometrica 50.1 (1982), pp. 945–973. [Var83] H. Varian. “Non-parametric tests of consume...

2012 arXiv
[2003]

Langevin dynamics for adaptive inverse reinforcement learning of stochastic gradient algorithms

[KY21] V. Krishnamurthy and G. Yin. “Langevin dynamics for adaptive inverse reinforcement learning of stochastic gradient algorithms”. In: Journal of Machine Learning Research 22 (2021), pp. 1–49. [KY22] V. Krishnamurthy and G. Yin. “Multikernel passive stochastic gradient alg...

2021
[2008]

A testable model of consumption with externalities

[Deb09] R. Deb. “A testable model of consumption with externalities”. In: Journal of Economic Theory 144.4 (2009), pp. 1804–1816. [Die12] W. Diewert. “Afriat’s theorem and some extensions to choice under uncertainty”. In: The Eco- nomic Journal 122.560 (2012), pp. 305–331. [Di...

2009
[2013]

A characterization of rationalizable consumer behavior

[Ren15] P. J. Reny. “A characterization of rationalizable consumer behavior”. In: Econometrica 83.1 (2015), pp. 175–192. Bibliography 81 [R´ev77] P. R ´ev´esz. “How to apply the method of stochastic approximation in the non-parametric estima- tion of a regression function”. In...

2015
[2016]

On revealed preference analysis

[Mas78] A. Mas-Colell. “On revealed preference analysis”. In: The Review of Economic Studies (1978), pp. 121–131. [Mat+20] R. Mattila, C. R. Rojas, V. Krishnamurthy, and B. Wahlberg. “Inverse filtering for hidden Markov models with applications to counter-adversarial autonomou...

1978 arXiv
[2018]

Implications of rational inattention

[Sim03] C. A. Sims. “Implications of rational inattention”. In: Journal of Monetary Economics 50.3 (2003), pp. 665–690. [Sim10] C. A. Sims. “Rational inattention and monetary economics”. In: Handbook of Monetary Eco- nomics. Vol

2003
[2025]

Inverse game theory: learning utilities in succinct games

[KS15] V. Kuleshov and O. Schrijvers. “Inverse game theory: learning utilities in succinct games”. In: International Conference on Web and Internet Economics. Springer. 2015, pp. 413–427. [KS91] I. Karatzas and S. Shreve. Brownian Motion and Stochastic Calculus. 2nd ed. Springer,

2015

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

The construction of utility functions from expenditure data

[Afr67] S. Afriat. “The construction of utility functions from expenditure data”. In: International Eco- nomic Review 8.1 (1967), pp. 67–77. [Afr87] S. Afriat. Logic of Choice and Economic Theory. Clarendon Press,

work page 1967

[2] [3]

Finite-sample bounds for adaptive inverse reinforcement learn- ing using passive langevin dynamics

Elsevier, 2010, pp. 155–181. [SK25] L. Snow and V. Krishnamurthy. “Finite-sample bounds for adaptive inverse reinforcement learn- ing using passive langevin dynamics”. In: IEEE Transactions on Information Theory 71.6 (2025), pp. 4637–4670. [SKS23] L. Snow, V. Krishnamurthy, and B. M. Sadler. “Identifying coordination in a cognitive radar net- work – A mul...

work page 2025

[3] [4]

The strong ergodic theorem for densities: generalized Shannon-McMillan- Breiman theorem

[Bar85] A. R. Barron. “The strong ergodic theorem for densities: generalized Shannon-McMillan- Breiman theorem”. In: The Annals of Probability 13.4 (1985), pp. 1292–1303. [Bil99] P. Billingsley. Convergence of Probability Measures. 2nd ed. Wiley,

work page 1985

[4] [8]

Fine-tuning language models from human preferences

Chicago, IL, USA. 2008, pp. 1433–1438. [Zie+20] D. M. Ziegler et al. “Fine-tuning language models from human preferences”. In:arXiv preprint arXiv:1909.08593v2 (2020). [ZYZ21] F. Zou, G. G. Yen, and C. Zhao. “Dynamic multiobjective optimization driven by inverse reinforcement learning”. In: Information Sciences 575 (2021), pp. 468–484

arXiv 2020

[5] [15]

Unifying Revealed Preference and Revealed Rational Inattention

[PK23a] K. Pattanayak and V. Krishnamurthy. “Necessary and sufficient conditions for inverse reinforce- ment learning of Bayesian stopping time problems”. In: Journal of Machine Learning Research 24.52 (2023), pp. 1–64. [PK23b] K. Pattanayak and V. Krishnamurthy. “Unifying revealed preference and revealed rational inattention”. In: (2023). eprint: arXiv:2...

work page Pith review arXiv 2023

[6] [16]

Analysis of recursive stochastic algorithms

2020, pp. 1–10. [Lju77] L. Ljung. “Analysis of recursive stochastic algorithms”. In: IEEE Transactions on Automatic Control AC-22.4 (1977), pp. 551–575. [LK12] S. Levine and V. Koltun. “Continuous inverse optimal control with locally optimal examples”. In: arXiv preprint arXiv:1206.4617 (2012). [LY16] D. Luenberger and Y. Ye. Linear and Nonlinear Programm...

arXiv 1977

[7] [21]

Langevin-type models I: diffusions with given stationary distri- butions and their discretizations

[ST99] O. Stramer and R. L. Tweedie. “Langevin-type models I: diffusions with given stationary distri- butions and their discretizations”. In: Methodology and Computing in Applied Probability 1.3 (1999), pp. 283–306. [TTV16] Y. W. Teh, A. H. Thiery, and S. J. Vollmer. “Consistency and fluctuations for stochastic gradient Langevin dynamics”. In: The Journa...

work page 1999

[8] [500]

Regularized Inverse Reinforcement Learning

[JE09] B. E. Jones and D. L. Edgerton. “Testing utility maximization with measurement errors in the data”. In: Measurement Error: Consequences, Applications and Solutions. Emerald Group Publishing Limited, 2009, pp. 199–236. [Jeo+20] W. Jeon et al. “Regularized inverse reinforcement learning”. In:arXiv preprint arXiv:2010.03691 (2020). [JK25] A. Jain and ...

work page Pith review arXiv 2020