REVIEW 3 major objections 5 minor 13 references
Is Optimal Transport Necessary for Inverse Reinforcement Learning?
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that optimal transport is not necessary for inverse reinforcement learning: two optimization-free, nearest-state reward heuristics match or beat OT-based rewards across 32 benchmarks, so the field should credit proximity…
desk verdict Useful baseline study, but the central claim is overreaching: offline comparisons are confounded by unequal reward squashing, and the online results favor TemporalOT. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the replacement of an OT coupling-weighted reward with a negative minimum-distance query. Where OT computes $r(s_i) = -\sum_j c(s_i, s_j^e) \mu^*_{ij}$ from an optimal transport plan $\mu^*$, the simple methods compute $r(s_t) = -\min_{s^e \in W(t)} \operatorname{Dist}(s_t, s^e)$, with $W(t)$ either the whole expert trajectory or a segment. The appendix subsumes both in a window formulation $W(t) = \{ s^e_j : j \in [\lfloor bt \rfloor - a, \lfloor bt \rfloor + c] \cap [1,T_e] \cap \mathbb{Z}\}$, so the empirical comparison isolates what the OT plan adds beyond nearest-neighbor proximity.
What would settle it
A reader could rerun the OT baselines under the same reward scaling ($e^{r}$ with $\alpha=1,\beta=1$) and the same per-method hyperparameter search as the simple methods on all 32 tasks; if OT then outperforms the simple methods, the paper's central claim is refuted. A complementary test would intentionally vary expert versus non-expert speed to see whether Segment-Matching fails while OT retains an edge, which would show that temporal optimal coupling carries information beyond proximity.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that optimal transport is not needed for the reward-labeling step of IRL. The Minimum-Distance heuristic, $r_{\min}(s_t) = -\min_{s^e \in \tau^e} \operatorname{Dist}(s_t, s^e)$, and the Segment-Matching heuristic, which restricts the minimum to a contiguous temporally aligned segment of the expert trajectory, reproduce or exceed the scores of Optimal Transport Reward and TemporalOT when either IQL or ReBRAC is used offline and DrQ-v2 online. Because the simple rewards avoid the coupling matrix, the Sinkhorn iterations, and the OT-specific tuning, the paper reads this as evidence that proximity alignment, not the optimal coupling, carried most of the earlier methods' benefit.
Load-bearing premise
The load-bearing premise is that the experiments were fair: the simple rewards use a gentler squashing ($\exp(r)$ with $\alpha=1,\beta=1$) than the OT baselines ($5\exp(5r)$), ReBRAC is tuned per method and dataset, and the Segment-Matching method also assumes the non-expert moves at roughly the expert's speed; if those choices, rather than proximity alignment itself, produced the gains, the central conclusion would not follow.
Editorial extensions
If this is right
- Reward labeling for IRL can be reduced to nearest-neighbor queries: no coupling matrices, Sinkhorn iterations, or OT-specific tunable parameters are needed to match OT on the tested benchmarks.
- Per-trajectory cost drops from $O(T^2 d)$ to $O(Td)$, which makes the approach attractive for long trajectories and large unlabeled datasets.
- Temporal alignment still contributes: Segment-Matching and Segment-Window outperform order-free Minimum-Distance on tasks needing temporal coordination, meaning a light temporal inductive bias is sufficient.
- Future IRL proposals should be compared against these simple baselines; otherwise gains attributed to optimization machinery may be illusory.
Reading between the lines
- A direct prediction the paper leaves implicit: on tasks where expert and non-expert trajectories intentionally differ in speed or traverse states in different order, the gap between OT and Segment-Matching should stay small; if it grows large, the optimal-coupling component is carrying real signal.
- The same proximity story should transfer to learned representation spaces, where distances reflect task-relevant structure; one testable extension is to replace raw-state distance with a pretrained latent distance and check whether the gap between simple and OT rewards narrows further.
- The benchmark evidence is concentrated in locomotion, dexterous manipulation, and short-horizon tabletop tasks; extending Min-Dist and Seg-match to long-horizon multi-modal demonstrations would stress whether a single temporal alignment is enough.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper challenges the necessity of optimal transport (OT) in inverse reinforcement learning (IRL) by proposing two optimization-free reward-labeling heuristics: Minimum-Distance Reward (Min-Dist) and Segment-Matching Reward (Seg-match), plus a sliding-window variant (Seg-window) for online settings. The reward functions are compared against OT and TemporalOT on 23 offline D4RL datasets with IQL and ReBRAC and on nine online MetaWorld tasks with DrQ-v2. The paper reports that the simple methods match or outperform the OT baselines across the tested benchmarks and concludes that the benefits of OT in IRL may stem from basic proximity alignment rather than from the optimal coupling formulation.
Significance. If the comparison were properly controlled, this would be a valuable negative result: it would give future IRL papers a strong, simple, and linear-time baseline and would redirect attention from OT-specific machinery to proximity-based reward design. The empirical effort is substantial and should be credited: 23 D4RL datasets, nine MetaWorld tasks, multiple downstream algorithms, repeated seeds with standard deviations, an ablation on distance metric, and an ablation on window size. I also found no circularity: the proposed rewards have no fitted parameters, and the comparison is external. However, the central claim is not currently established because the offline comparisons vary reward post-processing between the two families, and the online aggregate results do not support the 'consistently match or exceed' wording in the abstract.
major comments (3)
- [Appendix A.1 / Table 4] The offline comparison is confounded by unequal reward post-processing. Appendix A.1 fixes α=5, β=5 for OT and TemporalOT but α=1, β=1 for Seg-match and Min-Dist, and Table 4 makes the mismatch larger: OT/TemporalOT are squashed as 5.0·exp(5.0·T·r/|S|) while the simple methods use exp(r). With episode length T=1000 and state dimensions from 17 to 56 in the D4RL domains, the factor 5T/|S| is between roughly 89 and 294, so the two reward families have vastly different exponential sharpness before the affine rescaling by dataset statistics. An affine rescaling cannot compensate for this nonlinear difference, so the higher IQL scores for Seg-match in Table 1 could be caused by the squashing function rather than by the absence of an OT coupling. To support the central claim, the two families should be compared under the same squashing function, or the authors should ablate α and β within each family; otherwise the offline results only show competitiveness under different default post-processing.
- [Table 3] The online results partially contradict the abstract's claim that the simple rewards 'match or outperform' OT-based approaches. In Table 3, TemporalOT has the highest total success rate (449.61), followed by Seg-window (437.95) and Seg-match (413.40), with standard OT last (391.90). Thus the simple methods beat the basic OT baseline but do not consistently beat TemporalOT in the aggregate. The claim should be narrowed (for example, 'competitive with TemporalOT') or the conclusion should acknowledge that TemporalOT retains an advantage when fine-grained temporal coordination is required. The omission of Min-Dist because it 'fails in these challenging online tasks' also weakens the generality of the simple-methods message.
- [Appendix A.3 / Table 2] The ReBRAC results in Table 2 rely on regularization coefficients that are tuned per method and per dataset on the same evaluation benchmarks, with the best choices reported in Tables 9 and 10. This is selection on the test set: there is no held-out validation, and the tuning grids are not identical across methods. The paper includes default-untuned results in Table 6, which is helpful, but the main robustness claim is based on the tuned Table 2, and per-method tuning can mask or create differences between reward-labeling approaches. The authors should either use a single shared tuning protocol for all methods, add a validation split, or provide pairwise significance tests for the comparisons that drive the conclusion.
minor comments (5)
- [Appendix A.3] The text refers to 'Tabel 2' where Table 2 is meant; this typo should be corrected.
- [Appendix B.4] The text refers to 'Table B.4' but the table is numbered Table 12; the cross-reference should be updated.
- [Tables 2 and 3] There are small formatting typos: 'Min-Dis t' in the header of Table 2 and inconsistent capitalization of 'Metaworld-total' in Table 3.
- [Appendix A.1 / Table 4] The squashing formula in Table 4, 5.0·exp(5.0·T·r/|S|), is not derived from or even mentioned in Appendix A.1's α=5, β=5 description; the notation and the role of T and |S| should be defined and reconciled.
- [Section 4.2] The speed-matching assumption behind Segment-Matching is stated explicitly, but its failure modes are not discussed; a sentence on when this assumption is violated would help readers interpret the Antmaze and MetaWorld results.
Circularity Check
No meaningful circularity: the proposed rewards are parameter-free heuristics, the OT baselines are external, and the single self-citation supports only a symmetric evaluation detail.
-
other
[Section 5.1, IRL + IQL paragraph (Table 1 discussion)]
"Instead of reporting the performance from the very last evaluation during the training process, we report the average performance over the last four evaluations to offset the fluctuation of evaluation results over timesteps [Wang et al., 2024b]."
Wang et al. 2024b shares two authors (Z. Dong, K. Ross) with the present paper, making this the paper's only self-citation. It is cited solely to justify the evaluation-window averaging convention. That convention is applied identically to the proposed simple rewards and to the external OT baselines (OT and TemporalOT from Luo et al. 2023 and Fu et al. 2024), and the headline differences in Table 1 are far larger than evaluation-window noise (e.g., Seg-match 87.69 vs OT 65.59 on hopper-medium-replay). The citation does not supply any premise from which a conclusion is derived, so it is a minor, non-load-bearing self-citation rather than a circular step.
full rationale
The paper's derivation chain is empirical and self-contained against external benchmarks. The proposed Min-Dist reward (r_min(s_t) = -min_{s^e in tau^e} Dist(s_t, s^e)) and Seg-match reward (r_seg(s_t) = -min_{s^e in Gamma_t} Dist(s_t, s^e)) are closed-form, parameter-free functions of the data; no parameter is fitted to a subset of data and later renamed as a prediction. The OT and TemporalOT rewards are imported verbatim from Luo et al. 2023 and Fu et al. 2024, and the comparison is carried out against those external methods on D4RL and MetaWorld with IQL, ReBRAC, TD3+BC, and DrQ-v2. The only self-citation (Wang et al. 2024b, sharing authors Dong and Ross) supports the 'average of last four evaluations' reporting detail, which is symmetric across all methods and does not load-bear on the central claim. The headline conclusion that OT's benefits 'may arise from basic proximity alignment rather than its optimal coupling formulation' is an abductive interpretation of the score comparisons, not a quantity derived from its own assumptions, so it does not reduce to its inputs by construction. Two fairness risks are noted but are not circularity: (1) offline reward post-processing differs between methods (Appendix A.1/Table 4: OT and TemporalOT use 5.0*exp(5.0*T*r/|S|) while Seg-match and Min-Dist use exp(r)), creating a nonlinear reward-shaping confound; and (2) ReBRAC coefficients are tuned per method per dataset on the evaluation benchmarks (Appendix A.3), with no held-out validation. These concerns weaken the strength of the empirical conclusion but do not constitute a reduction of the paper's outputs to its inputs, and per the reviewing rules they belong under correctness risk rather than circularity. Overall, no significant circularity is present; the score reflects the single minor, non-load-bearing self-citation.
Assumptions & free parameters
free parameters (3)
- segment_window_size_kw =
10
- reward_squash_parameters =
simple rewards: alpha=1, beta=1; OT baselines: alpha=5, beta=5
- rebrac_regularization_coefficients =
per-dataset tuples for actor and critic, reported in Tables 9 and 10
assumptions (4)
- standard math Standard finite-horizon MDP and reinforcement learning background as defined in Section 2.
- domain assumption The non-expert agent has a similar movement speed to the expert for Segment-Matching to make sense.
- domain assumption Cosine distance is a suitable state similarity metric for reward labeling.
- domain assumption The single trajectory with the highest ground-truth return in each offline dataset is a valid expert demonstration.
Cite this review
Pith. "Pith review of Is Optimal Transport Necessary for Inverse Reinforcement Learning?." pith.science (2026). https://pith.science/paper/TKMT4HHT
@misc{pith2026250606793,
author = {Pith},
title = {Pith review of: Is Optimal Transport Necessary for Inverse Reinforcement Learning?},
year = {2026},
howpublished = {\url{https://pith.science/paper/TKMT4HHT}},
note = {Machine review of arXiv:2506.06793}
}
read the original abstract
Inverse Reinforcement Learning (IRL) aims to recover a reward function from expert demonstrations. Recently, Optimal Transport (OT) methods have been successfully deployed to align trajectories and infer rewards. While OT-based methods have shown strong empirical results, they introduce algorithmic complexity, hyperparameter sensitivity, and require solving the OT optimization problems. In this work, we challenge the necessity of OT in IRL by proposing two simple, heuristic alternatives: (1) Minimum-Distance Reward, which assigns rewards based on the nearest expert state regardless of temporal order; and (2) Segment-Matching Reward, which incorporates lightweight temporal alignment by matching agent states to corresponding segments in the expert trajectory. These methods avoid optimization, exhibit linear-time complexity, and are easy to implement. Through extensive evaluations across 32 online and offline benchmarks with three reinforcement learning algorithms, we show that our simple rewards match or outperform recent OT-based approaches. Our findings suggest that the core benefits of OT may arise from basic proximity alignment rather than its optimal coupling formulation, advocating for reevaluation of complexity in future IRL design.
Reference graph
Works this paper leans on
-
[1]
CNN padding V ALID Buffer size 1.5e5 DrQ-specific Settings Action repeat 2 Frame stack 3 Image resolution 84× 84× 3 Number of expert demonstrations 2 Reward Labeling Context length kc 3 Temporal mask width km (TemporalOT) 10 Segment window size (Seg-window) 10 GPU model NVIDIA A100 8358 80GB Parallel tasks per GPU 5 Computation CPU workers per task 2 Memor...
work page 2020
-
[6]
Ilya Kostrikov, Denis Y arats, and Rob Fergus
URL https://arxiv.org/abs/2110.06169. Ilya Kostrikov, Denis Y arats, and Rob Fergus. DrQ-v2: Data- efficient reinforcement learning for visual control. In International Conference on Machine Learning ,
-
[7]
Y uyang Liu, Weijun Dong, Yingdong Hu, Chuan Wen, Zhao-Heng Yin, Chongjie Zhang, and Y ang Gao
URL https://arxiv.org/abs/2005.01643. Y uyang Liu, Weijun Dong, Yingdong Hu, Chuan Wen, Zhao-Heng Yin, Chongjie Zhang, and Y ang Gao. Imitation learning from observation with automatic di scount scheduling. In International Conference on Learning Representations ,
arXiv 2005
-
[8]
URL https://arxiv.org/abs/2303.13971. Jiafei Lyu, Mengbei Y an, Zhongjian Qiao, Runze Liu, Xiaoten g Ma, Deheng Y e, Jing-Wen Y ang, Zongqing Lu, and Xiu Li. Cross-domain offline policy adaptat ion with optimal transport and dataset constraint. In International Conference on Learning Representations ,
-
[10]
12 A Offline RL Evaluations A.1 Reward Post-processing To prevent instability during policy optimization, especi ally for value-based RL algorithms, we first apply a standard exponential squashing function to all rewards produced by IRL. This transformation helps mitigate the implicit penalization of longer traject ories and controls the variance of reward ...
work page 2021
-
[11]
Eventually, their total scores match each other
After hyperpara meter tuning, we can see that the overall performance on Antmaze is much better than the untun ed results for both OT and Segment- matching. Eventually, their total scores match each other. However, we still observe a large gap the comparing them with the oracle results. On the one hand, t his may suffer from a large variance when running ...
work page 2021
-
[12]
To address this, we adopt the same reward scaling proce- dure used in Temporal OT [Fu et al., 2024]
(0.002, 0.1) antmaze-large-diverse (0.001, 0.001) (0.002, 0.001) B Online RL Evaluations B.1 Reward Post-processing Unlike in the offline setting, global reward normalization b ased on known return bounds is not feasi- ble in the online case, as new trajectories are collected con tinuously and the maximum or minimum achievable returns are unknown a priori....
work page 2024
-
[2019]
Wasserstein Adversarial Imitation Learning
URL https://arxiv.org/abs/1906.08113. Tianhe Y u, Deirdre Quillen, Rohin Shah, Chelsea Finn, and Se rgey Levine. Meta-World: A bench- mark and evaluation for multi-task and meta reinforcement l earning. In Conference on Robot Learning,
work page Pith review arXiv 1906
Show all 13 references
-
[2020]
Arip Asadulaev, Rostislav Korst, Alexander Korotin, V age E giazarian, Andrey Filchenkov, and Evgeny Burnaev
URL https://arxiv.org/abs/1806.06877. Arip Asadulaev, Rostislav Korst, Alexander Korotin, V age E giazarian, Andrey Filchenkov, and Evgeny Burnaev. Rethinking optimal transport in offline rei nforcement learning. In Advances in Neural Information Processing Systems ,
-
[2021]
Y uwei Fu, Haichao Zhang, Di Wu, Wei Xu, and Benoit Boulet
URL https://arxiv.org/abs/2004.07219. Y uwei Fu, Haichao Zhang, Di Wu, Wei Xu, and Benoit Boulet. Robot policy learning with temporal optimal transport reward. In Advances in Neural Information Processing Systems ,
2004 arXiv
-
[2022]
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Serg ey Levine
URL https://arxiv.org/abs/2110.03684. Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Serg ey Levine. D4RL: Datasets for deep data-driven reinforcement learning,
-
[2023]
Ilya Kostrikov, Ashvin Nair, and Sergey Levine
URL https://arxiv.org/abs/2303.01497. Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline rein forcement learning with implicit q- learning,
-
[2024]
Marco Cuturi
URL https://arxiv.org/abs/2402.13037. Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.