Pith. sign in

REVIEW 3 major objections 3 minor 41 references

Action-Constrained Imitation Learning

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces DTWIL, an imitation-learning method for agents whose action space is smaller than an expert's: it replaces the expert demos with feasible surrogate trajectories aligned to the expert's states via dynamic time warping a

desk verdict The ACIL framing is a genuine and useful reframing, but as submitted here I can't verify the central empirical claim—and the DTW-alignment idea has a load-bearing gap that the missing experiments may or may not close. read the letter →

arxiv 2508.14379 v1 pith:C3UFE5VZ submitted 2025-08-20 cs.RO cs.LG

Chia-Han Yeh , Tse-Sheng Nan , Risto Vuorio , Wei Hung , Hung-Yen Wu , Shao-Hua Sun , Ping-Chun Hsieh This is my paper · ORCID
classification cs.ROcs.LG
keywords Action-ConstrainedImitationLearningdynamictimewarpingmodelpredictivecontrolsurrogatedemonstrationsoccupancymeasureactionconstraintsrobot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a new problem, Action-Constrained Imitation Learning (ACIL): an agent with limited action capabilities must learn from demonstrations produced by an expert with a larger action space. It argues that simply projecting expert actions into the feasible set distorts the occupancy measure, so the imitator can never match the expert's behavior. Its proposed method, DTWIL, instead builds a surrogate dataset of feasible trajectories that visit the same states as the expert, using dynamic time warping to decide which expert states matter and model predictive control to plan constraint-respecting paths. Learning from this surrogate data, standard imitation algorithms perform better across several robot-control tasks and do so more sample-efficiently than the benchmark IL methods compared in the paper. If this holds, imitation under action constraints becomes a matter of aligning states rather than copying actions.

What carries the argument

Dynamic Time Warping (DTW) distance is the core object: it measures how cheaply one time series can be warped onto another, so a constrained trajectory can match expert states at shifted timestamps. Model Predictive Control (MPC) is the planner that produces surrogate trajectories: given a dynamics model, it searches over feasible action sequences to minimize the DTW distance to the expert demonstration while respecting the action constraints. The output is a surrogate dataset that replaces the original expert demos.

What would settle it

Run a task whose success requires a precise high-force action at a specific moment (for example, striking or pushing a stuck object), train a DTWIL imitator under a torque limit, and check whether the surrogate trajectories are feasible but fail to accomplish the task. If they are feasible yet task-incompetent, the sufficiency of state-only alignment is refuted.

Watch

Extended reading notes

Core claim

The central claim is that ACIL fails when the imitator tries to match the expert's occupancy measure directly, because a restricted action set makes the required state-action distribution unreachable. DTWIL sidesteps this by changing the target: instead of imitating the expert's actions, it imitates a feasible path through the expert's states. Each expert demonstration is turned into a surrogate trajectory through MPC, with the DTW distance as the alignment objective, and the resulting surrogate set is used to train any downstream imitation learner. The paper reports that this approach yields better performance and sample efficiency on multiple robot-control benchmarks than projection-based

Load-bearing premise

The method assumes that a feasible trajectory that passes through roughly the same states as the expert, even with different timing and forces, is enough to teach the constrained agent the task, and that the MPC planner's dynamics model is accurate enough to find such trajectories.

Editorial extensions

If this is right

  • ACIL is tractable: a constrained agent can learn useful policies from a less-constrained expert without ever seeing feasible expert actions.
  • Because DTWIL outputs a standard demonstration dataset, existing imitation-learning algorithms can be applied directly to the constrained setting.
  • Projection-based action-constrained methods are not a sufficient recipe for imitation; their occupancy-measure distortion is an identified failure mode.
  • DTWIL's sample-efficiency gains suggest that state-level alignment carries more usable information than action-level projection when constraints bind.
  • The DTW-MPC pipeline gives a general way to compare experts and imitators whose action spaces differ.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If state alignment is the right level of abstraction, DTWIL's surrogate datasets could also seed offline reinforcement learning or serve as a pretraining source for related constrained tasks.
  • The MPC step depends on an accurate dynamics model; replacing it with an online-learned model or a model-free planner is a natural extension, and the method's success will hinge on how well the planner covers the constrained region.
  • Tasks where success depends on precise timing or force profiles rather than visited states would stress-test state-only alignment; a constrained pushing or insertion task with a force threshold is a concrete probe.
  • Augmenting the DTW cost with action- or torque-weighted penalties could preserve more of the expert's timing intent while still respecting constraints.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces a new problem setting, Action-Constrained Imitation Learning (ACIL), in which an imitator with a restricted action space must learn from demonstrations of a constraint-free expert. The authors argue that existing IL methods fail under action constraints because the constraint-induced mismatch in occupancy measures prevents the imitator from matching the expert. They propose DTWIL, a method that replaces the expert demonstrations with a surrogate dataset: an MPC planner generates trajectories that are feasible under the imitator's action constraints and that minimize Dynamic Time Warping (DTW) distance to the expert's state trajectories. A standard IL algorithm is then trained on this surrogate dataset. The abstract claims that DTWIL significantly improves performance across multiple robot control tasks and achieves better sample efficiency than several benchmark IL algorithms. The manuscript as provided contains an introduction, references, and brief closing matter, but no method section, no experiments, no tables, and no figures.

Significance. If the claims are correct, ACIL is a genuinely novel and useful problem formulation: it addresses a common practical gap where an agent must imitate an expert with a larger action set, e.g., a lower-torque robot learning from a higher-torque demonstrator. The proposed approach of using MPC and DTW to generate action-feasible surrogate trajectories is creative and clearly distinguishes DTWIL from projection-based adaptations of existing constrained-RL methods. The paper also publicly releases code, which is a positive step for reproducibility. However, because the submitted text omits the method and experiments, the central empirical claim is not auditable and the significance cannot be verified from the manuscript itself. The DTW-based surrogate generation also raises a correctness-risk concern: minimizing state-trajectory distance does not by itself guarantee task competence, especially for contact-rich or time-critical tasks. The paper needs both a full description of the method and targeted experiments or analysis to substantiate the central claim.

major comments (3)
  1. [Overall structure (between Introduction and References)] The manuscript as submitted is incomplete: after the Introduction, the text jumps directly to a closing fragment, the Impact Statement, acknowledgements, and references. There is no method section describing the DTWIL algorithm, no MPC formulation, no action-constraint handling, and no experimental section. The abstract's claim of 'extensive experiments' and 'outperforms various benchmark imitation learning algorithms' is therefore unsupported. This is the most load-bearing issue: without the full method and results, the paper's central contribution cannot be evaluated.
  2. [Abstract and Introduction] DTWIL's stated objective is to generate surrogate trajectories that follow 'similar state trajectories' to the expert while adhering to action constraints. DTW distance is invariant to time warping, and state trajectories do not encode contact forces, action profiles, or timing information. For contact-rich or time-critical tasks, a feasible surrogate can shadow the expert's state path while failing the actual task (e.g., approaching a peg without inserting it). The manuscript provides no theorem or analysis bounding task suboptimality in terms of DTW distance, and no experimental evidence that surrogate demonstrations achieve expert-level returns. This gap undermines the claim that learning from the surrogate dataset yields a competent constrained policy.
  3. [MPC / dynamics model (implicit in method)] DTWIL relies on an MPC planner to generate surrogate trajectories. MPC requires a dynamics model, but the manuscript does not describe the model, its accuracy, or how model error is handled. If the model is approximate, the generated surrogate trajectories may be feasible only under the model, not in the true environment, which would violate the premise that the dataset adheres to the action constraints and resembles feasible expert behavior. The paper should specify the model class, its source, and any robustness measures taken.
minor comments (3)
  1. [References] The reference 'Pomerleau & A, 1991' should be formatted correctly (the author is Dean A. Pomerleau). Please check all reference entries for consistency.
  2. [Text fragment after Introduction] The sentence fragment 'lenges that previous methods have not effectively addressed' appears immediately after the Introduction, suggesting a missing paragraph or a broken page boundary. Please proofread the full text.
  3. [Repository name] The code repository is named 'ACRL-Baselines'. Since ACIL is presented as a distinct problem from ACRL, a different repository name would reduce confusion about the relationship between the two lines of work.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified; the DTWIL pipeline is an empirical surrogate-data-generation and evaluation procedure, and the cited self-work is background rather than load-bearing.

full rationale

The paper's central claim is an empirical pipeline: expert demonstrations are fed into an MPC planner that outputs surrogate trajectories minimizing DTW distance to the expert state sequence while respecting action constraints; a downstream IL method is trained on these surrogates; and performance is then evaluated on the task. None of these stages reduces by construction to its input. The DTW objective is a state-space matching criterion, not a renamed task-reward objective, and the paper does not claim that matching expert states is definitionally equivalent to task success. The only potentially self-referential citation is Hung et al. (2025), the authors' own ACRL paper, but it is cited as general background for action-constrained RL and is not used to justify DTWIL's correctness or to forbid alternative approaches. No uniqueness theorem, fitted quantity renamed as prediction, or ansatz smuggled in via self-citation is present in the available text. The version under review omits the method and experiment sections, so the surviving evidence is limited; however, absence of evidence for task competence of the surrogate data is a correctness/empirical-validity concern, not a circularity. Per the stated rules, circularity must be exhibited by quoting a specific equation-level or definitional reduction, and none can be identified here. Therefore the honest finding is no significant circularity, score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities are visible from the abstract; the method will contain tuned hyperparameters (MPC horizon, DTW bandwidth, dataset size) but their values are not reported in the text provided. The three axioms above are assumptions the central claim depends on.

assumptions (3)
  • domain assumption Expert and imitator share the same state space and transition dynamics; only the action set differs.
    The ACIL problem definition requires this; otherwise matching expert state trajectories under imitator constraints is not meaningful.
  • ad hoc to paper Matching expert state trajectories under DTW distance is sufficient for learning a good constrained policy.
    The entire method converts expert demonstrations into surrogate data by state-trajectory alignment; this design hypothesis is load-bearing and not derived.
  • domain assumption An accurate dynamics model is available for the MPC planner.
    Trajectory alignment as planning needs a predictive model; if it is inaccurate, surrogate trajectories will not track expert states.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Action-Constrained Imitation Learning." pith.science (2026). https://pith.science/paper/C3UFE5VZ

@misc{pith2026250814379,
  author       = {Pith},
  title        = {Pith review of: Action-Constrained Imitation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C3UFE5VZ}},
  note         = {Machine review of arXiv:2508.14379}
}
read the original abstract

Policy learning under action constraints plays a central role in ensuring safe behaviors in various robot control and resource allocation applications. In this paper, we study a new problem setting termed Action-Constrained Imitation Learning (ACIL), where an action-constrained imitator aims to learn from a demonstrative expert with larger action space. The fundamental challenge of ACIL lies in the unavoidable mismatch of occupancy measure between the expert and the imitator caused by the action constraints. We tackle this mismatch through \textit{trajectory alignment} and propose DTWIL, which replaces the original expert demonstrations with a surrogate dataset that follows similar state trajectories while adhering to the action constraints. Specifically, we recast trajectory alignment as a planning problem and solve it via Model Predictive Control, which aligns the surrogate trajectories with the expert trajectories based on the Dynamic Time Warping (DTW) distance. Through extensive experiments, we demonstrate that learning from the dataset generated by DTWIL significantly enhances performance across multiple robot control tasks and outperforms various benchmark imitation learning algorithms in terms of sample efficiency. Our code is publicly available at https://github.com/NYCU-RL-Bandits-Lab/ACRL-Baselines.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 34 canonical work pages

  1. [1]

    Resource constrained deep reinforcement learning

    Bhatia, A., Varakantham, P., and Kumar, A. Resource constrained deep reinforcement learning. In International Conference on Automated Planning and Scheduling, 2019

    work page 2019

  2. [2]

    S., and Mikelsons, L

    Boborzi, D., Straehle, C.-N., Buchner, J. S., and Mikelsons, L. Imitation learning by state-only distribution matching. Applied Intelligence, 53 0 (24): 0 30865--30886, 2023

    work page 2023

  3. [3]

    FlowPG: Action-constrained Policy Gradient with Normalizing Flows

    Brahmanage, J., Ling, J., and Kumar, A. FlowPG: Action-constrained Policy Gradient with Normalizing Flows . Advances in Neural Information Processing Systems, 2023

    work page 2023

  4. [4]

    Openai gym

    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv:1606.01540, 2016

    arXiv 2016

  5. [5]

    Generative modelling of stochastic actions with arbitrary constraints in reinforcement learning

    Chen, C., Karunasena, R., Nguyen, T., Sinha, A., and Varakantham, P. Generative modelling of stochastic actions with arbitrary constraints in reinforcement learning. Advances in Neural Information Processing Systems, 2024

    work page 2024

  6. [6]

    A Lyapunov-based approach to safe reinforcement learning

    Chow, Y., Nachum, O., Duenez-Guzman, E., and Ghavamzadeh, M. A Lyapunov-based approach to safe reinforcement learning. In Advances in Neural Information Processing Systems, 2018

    work page 2018

  7. [7]

    Deep reinforcement learning in a handful of trials using probabilistic dynamics models

    Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 2018

    work page 2018

  8. [8]

    Safe exploration in continuous action spaces

    Dalal, G., Dvijotham, K., Vecerik, M., Hester, T., Paduraru, C., and Tassa, Y. Safe exploration in continuous action spaces. arXiv:1801.08757, 2018

    arXiv 2018

Show all 41 references
  1. [9]

    J., Sarafian, E., and Kraus, S

    Freund, G. J., Sarafian, E., and Kraus, S. A coupled flow approach to imitation learning. In International Conference on Machine Learning, 2023

    2023

  2. [10]

    D4rl: Datasets for deep data-driven reinforcement learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv:2004.07219, 2020

    2004 arXiv

  3. [11]

    and Peng, J

    Gangwani, T. and Peng, J. State-only imitation with transition dynamics mismatch. arXiv preprint arXiv:2002.11879, 2020

    2002 arXiv

  4. [12]

    Y., and Jin, H

    Gu, L., Zeng, D., Li, W., Guo, S., Zomaya, A. Y., and Jin, H. Intelligent VNF orchestration and flow scheduling via model-assisted deep reinforcement learning . IEEE Journal on Selected Areas in Communications, 2019

    2019

  5. [13]

    Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates

    Gu, S., Holly, E., Lillicrap, T., and Levine, S. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In IEEE International Conference on Robotics and Automation, 2017

    2017

  6. [14]

    A., Su, H., and Wang, X

    Hansen, N. A., Su, H., and Wang, X. Temporal difference learning for model predictive control. In International Conference on Machine Learning, 2022

    2022

  7. [15]

    and Chiba, S

    Hiroaki, S. and Chiba, S. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1978

    1978

  8. [16]

    and Ermon, S

    Ho, J. and Ermon, S. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2016

    2016

  9. [17]

    Diffusion imitation from observation

    Huang, B.-R., Yang, C.-K., Lai, C.-M., Wu, D.-J., and Sun, S.-H. Diffusion imitation from observation. In Neural Information Processing Systems, 2024

    2024

  10. [18]

    Efficient action-constrained reinforcement learning via acceptance-rejection method and augmented mdps

    Hung, W., Sun, S.-H., and Hsieh, P.-C. Efficient action-constrained reinforcement learning via acceptance-rejection method and augmented mdps. In International Conference on Learning Representations, 2025

    2025

  11. [19]

    and Porta, J

    Jaillet, L. and Porta, J. M. Path planning under kinematic constraints by rapidly exploring manifolds. IEEE Transactions on Robotics, 2012

    2012

  12. [20]

    Benchmarking actor-critic deep reinforcement learning algorithms for robotics control with action constraints

    Kasaura, K., Miura, S., Kozuno, T., Yonetani, R., Hoshino, K., and Hosoe, Y. Benchmarking actor-critic deep reinforcement learning algorithms for robotics control with action constraints. Robotics and Automation Letters, 2023

    2023

  13. [21]

    K., Dwibedi, D., Levine, S., and Tompson, J

    Kostrikov, I., Agrawal, K. K., Dwibedi, D., Levine, S., and Tompson, J. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019 a

    2019

  14. [22]

    Imitation learning via off-policy distribution matching

    Kostrikov, I., Nachum, O., and Tompson, J. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2019 b

    2019

  15. [23]

    F., Chen, M.-H., and Sun, S.-H

    Lai, C.-M., Wang, H.-C., Hsieh, P.-C., Wang, Y.-C. F., Chen, M.-H., and Sun, S.-H. Diffusion-reward adversarial imitation learning. In Neural Information Processing Systems, 2024

    2024

  16. [24]

    Lee, Y., Szot, A., Sun, S.-H., and Lim, J. J. Generalizable imitation learning from observation via inferring goal proximity. In Neural Information Processing Systems, 2021

    2021

  17. [25]

    Escaping from zero gradient: Revisiting action-constrained reinforcement learning via Frank-Wolfe policy optimization

    Lin, J.-L., Hung, W., Yang, S.-H., Hsieh, P.-C., and Liu, X. Escaping from zero gradient: Revisiting action-constrained reinforcement learning via Frank-Wolfe policy optimization . In Uncertainty in Artificial Intelligence, 2021

    2021

  18. [26]

    Robust regression for safe exploration in control

    Liu, A., Shi, G., Chung, S.-J., Anandkumar, A., and Yue, Y. Robust regression for safe exploration in control. In Learning for Dynamics and Control, 2020

    2020

  19. [27]

    State alignment-based imitation learning

    Liu, F., Ling, Z., Mu, T., and Su, H. State alignment-based imitation learning. arXiv preprint arXiv:1911.10947, 2019

    1911 arXiv

  20. [28]

    Optlayer - Practical constrained optimization for deep reinforcement learning in the real world

    Pham, T.-H., De Magistris, G., and Tachibana, R. Optlayer - Practical constrained optimization for deep reinforcement learning in the real world . In IEEE International Conference on Robotics and Automation, 2018

    2018

  21. [29]

    Efficient training of artificial neural networks for autonomous navigation

    Pomerleau and A, D. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 1991

    1991

  22. [30]

    State-only imitation learning for dexterous manipulation

    Radosavovic, I., Wang, X., Pinto, L., and Malik, J. State-only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 7865--7871. IEEE, 2021

    2021

  23. [31]

    Learning off-policy with online planning

    Sikchi, H., Zhou, W., and Held, D. Learning off-policy with online planning. In Conference of Robot Learning, 2021

    2021

  24. [32]

    Behavioral cloning from observation

    Torabi, F., Warnell, G., and Stone, P. Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence, 2018 a

    2018

  25. [33]

    Generative adversarial imitation from observation

    Torabi, F., Warnell, G., and Stone, P. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158, 2018 b

    2018 arXiv

  26. [34]

    Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning

    Tsounis, V., Alge, M., Lee, J., Farshidian, F., and Hutter, M. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning. IEEE Robotics and Automation Letters, 2020

    2020

  27. [35]

    H., and Yang, D

    Xu, Z., Tang, J., Meng, J., Zhang, W., Wang, Y., Liu, C. H., and Yang, D. Experience-driven networking: A deep reinforcement learning based approach. In IEEE Conference on Computer Communications, 2018

    2018

  28. [36]

    Imitation learning from observations by minimizing inverse dynamics disagreement

    Yang, C., Ma, X., Huang, W., Sun, F., Liu, H., Huang, J., and Gan, C. Imitation learning from observations by minimizing inverse dynamics disagreement. Advances in Neural Information Processing Systems, 2019

    2019

  29. [37]

    M., Khosravi, A., and Nahavandi, S

    Zare, M., Kebria, P. M., Khosravi, A., and Nahavandi, S. A survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions on Cybernetics, 2024

    2024

  30. [38]

    Zhang, J., Ye, M., Guo, Z., Yen, C.-Y., and Chao, H. J. CFR-RL: Traffic engineering with reinforcement learning in SDN . IEEE Journal on Selected Areas in Communications, 2020

    2020

  31. [39]

    robosuite: A modular simulation framework and benchmark for robot learning, 2025

    Zhu, Y., Wong, J., Mandlekar, A., Martín-Martín, R., Joshi, A., Lin, K., Maddukuri, A., Nasiriany, S., and Zhu, Y. robosuite: A modular simulation framework and benchmark for robot learning, 2025

    2025

  32. [40]

    Off-policy imitation learning from observations

    Zhu, Z., Lin, K., Dai, B., and Zhou, J. Off-policy imitation learning from observations. Advances in Neural Information Processing Systems, 2020

    2020

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.