REVIEW 3 major objections 4 minor 1 cited by
Data Retrieval with Importance Weights for Few-Shot Imitation Learning
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper shows that scoring prior robot demonstrations by the ratio of Gaussian kernel density estimates of the target and prior distributions, instead of by minimum L2 distance, consistently improves few-shot imitation learning in simula
desk verdict The method is simple and works, but the paper's headline theoretical claim—that nearest-neighbor retrieval is the zero-bandwidth limit of KDE—is wrong as written; with that fixed, it's a useful but modest empirical upgrade. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the importance weight p_KDE_t / p_KDE_prior, the ratio of two Gaussian kernel density estimates over latent embeddings of state-action pairs. The paper shows the classic min-L2 rule is the h->0 limit of a KDE of the target density alone, so replacing it with a smoothed density ratio corrects both the high variance of nearest-neighbor estimates and the bias from ignoring the prior distribution. The KDE bandwidth follows a multiplicative factor of Scott's rule, and the covariance is the sample covariance of the embeddings, which lets the estimate smooth over all data and account for feature correlations.
What would settle it
Rank the same prior samples with IWR and L2 on a fixed task; train policies on the top-IWR set and, as a control, on the bottom-IWR set. If the bottom set performs as well or better on held-out task success, the density-ratio ranking is not capturing usefulness in that latent space.
Extended reading notes
Core claim
The central claim is that retrieval should be viewed probabilistically: the goal is to approximate an expectation under the target task distribution using samples from a large prior dataset, which is exactly the importance-sampling identity E_prior[(p_t/p_prior) log pi] = E_t[log pi]. Existing retrieval rules only approximate the numerator p_t, by a nearest-neighbor kernel that emerges as the bandwidth of a Gaussian KDE goes to zero; they ignore the prior's own distribution and use high-variance estimates. IWR instead estimates both densities with Gaussian KDEs using Scott's rule and the sample covariance, scores prior points by p_KDE_t / p_KDE_prior, and thresholds the top-scoring points fo
Load-bearing premise
The method's gains depend on Gaussian density estimates in the learned latent space actually tracking which prior samples help the target task; when the latent space lacks smoothness, as the authors report for BYOL, the importance weights no longer help.
Editorial extensions
If this is right
- IWR can be dropped into any retrieval method that produces a smooth latent space, turning L2 retrieval into density-ratio retrieval without retraining the representation.
- Retrieval quality shifts from matching isolated target points toward selecting samples that sit in dense, target-relevant regions of the prior—visible as more relevant tasks and more balanced coverage across demonstration phases.
- Even when the target task has a missing critical subtask in the prior, retrieval gains are capped; IWR cannot invent absent behaviors.
- Threshold choice still matters: retrieving too much of the prior reintroduces harmful data and hurts performance.
Reading between the lines
- Going beyond the paper's experiments, IWR's density-ratio viewpoint suggests replacing Gaussian KDE with a scalable density-ratio estimator, which could remove the low-latent-dimension restriction the authors flag as a limitation.
- The same probabilistic reading implies other retrieval metrics, like dynamic-time-warping distance, may also be limit cases of a smoothed density and could be upgraded by the same ratio correction.
- A practical takeaway not stated by the paper: when choosing a latent space for retrieval, one should test whether small perturbations in latent space move retrieved samples in semantically meaningful ways, because KDE-based weights inherit the space's smoothness.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Importance Weighted Retrieval (IWR), a modification of the scoring rule used in retrieval-based few-shot imitation learning. Instead of selecting prior data by minimum L2 distance to target demonstrations in a learned latent space, IWR scores prior points by a Gaussian KDE estimate of the density ratio p_t / p_prior, computed with a bandwidth from Scott's rule. The authors claim that the standard nearest-neighbor rule is the zero-bandwidth limit of a KDE density estimate, and that replacing it with smoothed KDE ratios addresses high variance and bias. They evaluate IWR on Robomimic Square, five LIBERO tasks, and three real Bridge V2 tasks, using Behavior Retrieval, Flow Retrieval, and SAILOR embeddings, and report improved success rates over the base retrieval methods. The paper includes ablations on the denominator p_prior, bandwidth, and retrieval threshold, as well as an analysis of retrieved data composition.
Significance. If the empirical findings hold, IWR is a simple, broadly compatible drop-in replacement for the nearest-neighbor scoring used in several existing retrieval methods, and the paper's release of code and real-robot evaluations strengthens this contribution. The paper is also candid about limitations, notably that IWR does not help in the non-smooth BYOL latent space. However, the theoretical motivation is a central part of the paper's framing, and the claimed equivalence in Section 4.1 is not mathematically correct as written. The empirical results also do not uniformly support the abstract's claim that IWR 'consistently improves' all retrieval methods. The corrected version of the paper should be valuable, but the current manuscript overstates both the theoretical derivation and the universality of the empirical gains.
major comments (3)
- [Section 4.1, Eq. (4)] The claimed equivalence between the nearest-neighbor retrieval rule (Eq. 2) and the zero-bandwidth limit of the KDE is not established. In Eq. (4), the expression is h^{-2} log Σ exp(-||z-z'||^2/h^2). As h→0, log-sum-exp is dominated by the nearest neighbor, giving -d_min^2/h^2, and after multiplying by 1/h^2 this becomes -d_min^2/h^4. Thresholding by -ζ therefore forces d_min^2 < ζ h^4, which converges to d_min = 0, not the fixed-radius ball d_min^2 < ζ of Eq. (2). To recover Eq. (2), one would need the soft-max scaled by h^2, i.e., h^2 log Σ exp(-d^2/h^2), or an explicitly h-dependent KDE log-density threshold. As written, Eq. (4) is a different scaling and its limit is degenerate. The authors should either correct the scaling and present Eq. (4) as a soft relaxation rather than an equivalence, or remove the claim of recovering the original retrieval rule.
- [Sections 4.2-4.3, Eq. (6)] The paper justifies IWR via the importance-sampling identity E_{p_prior}[p_t/p_prior log π] = E_{p_t}[log π], but the algorithm does not perform importance sampling or resampling; it thresholds the estimated density ratio and trains on retrieved data with uniform weighting, as stated in Section 4.3 ('IWR is still a biased estimate'). Thus the theoretical benefit of the importance-weight thresholding is not proven, and the claim that IWR 'mitigates the bias of previous selection rules' is an empirical heuristic. The authors should temper the theoretical language: the contribution is a smoothed density-ratio-based scoring rule, not an unbiased estimator of the target expectation.
- [Abstract and Table 2] The abstract states that IWR 'consistently improves performance of existing retrieval-based methods.' This is contradicted by Table 2. For Flow Retrieval, FR-IWR is worse than FR on Square (67±1.9 vs 79±5.0) and Soup-Sauce (42±3.3 vs 45±5.5). For SAILOR, SR-IWR is worse on Soup-Sauce (48±3.4 vs 51±2.2) and on the real Corn task (9/20 vs 12/20). The average improvement may be positive, and the authors later use 'generally' in Section 5.3, but the word 'consistently' in the abstract overstates the results. Please revise the claim to reflect the actual distribution of outcomes across tasks and embeddings.
minor comments (4)
- [Section 4.1] The sentence 'recovering the original retrieval rule from Eq. (3)' should refer to Eq. (2), not Eq. (3).
- [Eqs. (4) and (5)] The Gaussian exponent in Eq. (4) is exp(-||·||^2/h^2), while Eq. (5) has the standard 1/2 factor and covariance h^2 Σ. The relationship between the two bandwidth/scale conventions should be stated explicitly to avoid confusion.
- [Appendix D, note after Fig. 10] The appendix note acknowledges that the main-paper Figure 4 contains labeling errors ('Others' marked as Misc instead of Harmful, and similar ending configurations misclassified). This should be corrected in the main text before publication; a note saying 'We plan to update the main paper when possible' is not acceptable in a submitted version.
- [Section 4.3] The threshold η in Section 4.3 is introduced without a formal definition, whereas ζ is used in Eq. (2). The authors should define η consistently with the importance-weight score and explain how it relates to the retrieval percentage used in experiments.
Circularity Check
No load-bearing circularity; central empirical claim is externally benchmarked. The KDE-equivalence derivation in Eq. (4) has a scaling flaw, but that is a correctness gap, not circularity.
full rationale
The central claim of IWR—that importance-weight retrieval improves few-shot imitation—is tested against external baselines (BC, BR, FR, SR, STRAP) on Robomimic, LIBERO, and real Bridge tasks. The method computes KDE estimates of p_t and p_prior and thresholds by p_KDE_t / p_KDE_prior; these scores are not fitted to the reported success rates, and the evaluations are externally benchmarked. The paper's self-citations (Behavior Retrieval, Flow Retrieval, Remix, robot-data-curation) are used as baselines or representation sources, not as the justification for IWR's correctness; IWR is measured against these prior works and often improves on them. The motivating theoretical claim that the L2 nearest-neighbor rule is recovered as h→0 from Eq. (4) is not supported as written: with the (1/h^2) scaling, the log-sum-exp limit gives -d_min^2/h^4, so the threshold degenerates to d_min=0 rather than the fixed-radius rule of Eq. (2). This is a derivation error, not a circular reduction. Appendix D also contains an unusual note admitting corrections to Figure 4 but stating the conclusions are unchanged; that is an internal-consistency concern, not circularity. Overall, no prediction or first-principles result reduces to its inputs by construction, so the circularity score is low; a small score reflects minor non-load-bearing self-citation rather than any circular step.
Assumptions & free parameters
free parameters (2)
- KDE bandwidth multiplier c =
4 (ablated to 2)
- Retrieval threshold eta / retrieval percentage =
determined per experiment (e.g., 30% for Square, 2.5% for LIBERO)
assumptions (3)
- domain assumption Importance sampling identity requires the support of the target distribution to be within the support of the prior distribution over state-action pairs.
- domain assumption Gaussian KDEs on the learned latent space provide reliable density estimates for both target and prior data.
- standard math Min-distance retrieval rule in the limit h -> 0 equals the KDE density estimate.
Cite this review
Pith. "Pith review of Data Retrieval with Importance Weights for Few-Shot Imitation Learning." pith.science (2026). https://pith.science/paper/A6HPDCYF
@misc{pith2026250901657,
author = {Pith},
title = {Pith review of: Data Retrieval with Importance Weights for Few-Shot Imitation Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/A6HPDCYF}},
note = {Machine review of arXiv:2509.01657}
}
read the original abstract
While large-scale robot datasets have propelled recent progress in imitation learning, learning from smaller task specific datasets remains critical for deployment in new environments and unseen tasks. One such approach to few-shot imitation learning is retrieval-based imitation learning, which extracts relevant samples from large, widely available prior datasets to augment a limited demonstration dataset. To determine the relevant data from prior datasets, retrieval-based approaches most commonly calculate a prior data point's minimum distance to a point in the target dataset in latent space. While retrieval-based methods have shown success using this metric for data selection, we demonstrate its equivalence to the limit of a Gaussian kernel density (KDE) estimate of the target data distribution. This reveals two shortcomings of the retrieval rule used in prior work. First, it relies on high-variance nearest neighbor estimates that are susceptible to noise. Second, it does not account for the distribution of prior data when retrieving data. To address these issues, we introduce Importance Weighted Retrieval (IWR), which estimates importance weights, or the ratio between the target and prior data distributions for retrieval, using Gaussian KDEs. By considering the probability ratio, IWR seeks to mitigate the bias of previous selection rules, and by using reasonable modeling parameters, IWR effectively smooths estimates using all data points. Across both simulation environments and real-world evaluations on the Bridge dataset we find that our method, IWR, consistently improves performance of existing retrieval-based methods, despite only requiring minor modifications.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 1 Pith paper
-
Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF
Two plug-and-play strategies — per-timestep advantage weighting and advantage-based trajectory replay — improve diffusion RLHF sample efficiency up to 6× across five reward functions.
Reference graph
Works this paper leans on
-
[1]
A. Radford, J. W . Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P . Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PMLR, 2021
work page 2021
-
[2]
S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-maron, M. Gim´enez, Y . Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y . Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URLhttps://openrevie...
work page 2022
-
[3]
T. Z. Zhao, J. Tompson, D. Driess, P . Florence, S. K. S. Ghasemipour, C. Finn, and A. Wahid. ALOHA unleashed: A simple recipe for robot dexterity. In8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id=gvdXE7ikHI
work page 2024
-
[4]
O’Neill, A
A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024
2024
-
[5]
M. Du, S. Nair, D. Sadigh, and C. Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets.arXiv preprint arXiv:2304.08742, 2023
arXiv 2023
-
[6]
L.-H. Lin, Y . Cui, A. Xie, T. Hua, and D. Sadigh. Flowretrieval: Flow-guided data retrieval for few-shot imitation learning. In8th Annual Conference on Robot Learning
-
[7]
S. Nasiriany, T. Gao, A. Mandlekar, and Y . Zhu. Learning and retrieval from prior data for skill-based imitation learning. InConference on Robot Learning (CoRL), 2022
work page 2022
- [8]
Show all 38 references
-
[9]
Khazatsky, K
A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024
2024 arXiv
-
[10]
Memmel, J
M. Memmel, J. Berg, B. Chen, A. Gupta, and J. Francis. STRAP: Robot sub-trajectory retrieval for augmented policy learning. InThe Thirteenth International Conference on Learning Representations,
-
[11]
S. Y ue, J. Liu, X. Hua, J. Ren, S. Lin, J. Zhang, and Y . Zhang. How to leverage diverse demonstrations in offline imitation learning.arXiv preprint arXiv:2405.17476, 2024
2024 arXiv
-
[12]
Hanna, S
J. Hanna, S. Niekum, and P . Stone. Importance sampling policy evaluation with an estimated behavior policy. InInternational Conference on Machine Learning, pages 2605–2613. PMLR, 2019. 9
2019
-
[13]
Sinha, J
S. Sinha, J. Song, A. Garg, and S. Ermon. Experience replay with likelihood-free importance weights. In Learning for Dynamics and Control Conference, pages 110–123. PMLR, 2022
2022
-
[14]
S. M. Xie, S. Santurkar, T. Ma, and P . S. Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201–34227, 2023
2023
-
[15]
C. H. Bennett. Efficient estimation of free energy differences from monte carlo data. Jour- nal of Computational Physics, 22(2):245–268, 1976. ISSN 0021-9991. doi:https://doi.org/10. 1016/0021-9991(76)90078-4. URL https://www.sciencedirect.com/science/article/ pii/0021999176900784
1976
-
[16]
Rhodes, K
B. Rhodes, K. Xu, and M. U. Gutmann. Telescoping density-ratio estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 4905–4916. Curran Associates, Inc., 2020. URL https://proce...
2020
-
[17]
K. Choi, C. Meng, Y . Song, and S. Ermon. Density ratio estimation via infinitesimal classification. In International Conference on Artificial Intelligence and Statistics, pages 2552–2573. PMLR, 2022
2022
-
[18]
Maddukuri, Z
A. Maddukuri, Z. Jiang, L. Y . Chen, S. Nasiriany, Y . Xie, Y . Fang, W. Huang, Z. Wang, Z. Xu, N. Chernyadev, S. Reed, K. Goldberg, A. Mandlekar, L. Fan, and Y . Zhu. Sim-and-real co-training: A simple recipe for vision-based robotic manipulation, 2025. URLhttps://arxiv.org/a...
2025
-
[19]
Hejna, C
J. Hejna, C. A. Bhateja, Y . Jiang, K. Pertsch, and D. Sadigh. Remix: Optimizing data mixtures for large scale imitation learning. In 8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id=fIj88Tn3fc
2024
-
[20]
Belkhale, Y
S. Belkhale, Y . Cui, and D. Sadigh. Data quality in imitation learning.Advances in Neural Information Processing Systems, 36, 2024
2024
-
[21]
Hejna, S
J. Hejna, S. Mirchandani, A. Balakrishna, A. Xie, A. Wahid, J. Tompson, P . Sanketi, D. Shah, C. Devin, and D. Sadigh. Robot data curation with mutual information estimators.arXiv preprint arXiv:2502.08623, 2025
2025 arXiv
-
[22]
Kuhar, S
S. Kuhar, S. Cheng, S. Chopra, M. Bronars, and D. Xu. Learning to discern: Imitating heterogeneous human demonstrations with preference and representation learning. In7th Annual Conference on Robot Learning, 2023
2023
-
[23]
Beliaev, A
M. Beliaev, A. Shih, S. Ermon, D. Sadigh, and R. Pedarsani. Imitation learning by estimating expertise of demonstrators. InInternational Conference on Machine Learning, pages 1732–1748. PMLR, 2022
2022
-
[24]
H. Ha, P . Florence, and S. Song. Scaling up and distilling down: Language-guided robot skill acquisition. InProceedings of the 2023 Conference on Robot Learning, 2023
2023
-
[25]
Mandlekar, S
A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y . Narang, L. Fan, Y . Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In7th Annual Conference on Robot Learning, 2023
2023
-
[26]
D. W . Scott.Multivariate density estimation: theory, practice, and visualization. John Wiley & Sons, 2015
2015
-
[27]
Gelman and X.-L
A. Gelman and X.-L. Meng.Applied Bayesian modeling and causal inference from incomplete-data perspectives. John Wiley & Sons, 2004
2004
-
[28]
Mandlekar, D
A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ın-Mart´ın. What matters in learning from offline human demonstrations for robot manipulation. InConference on Robot Learning (CoRL), 2021. 10
2021
-
[29]
B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P . Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning.arXiv preprint arXiv:2306.03310, 2023
2023 arXiv
-
[30]
C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, page 02783649241273668, 2023
2023
-
[31]
J. Pari, N. M. Shafiullah, S. P . Arunachalam, and L. Pinto. The surprising effectiveness of representa- tion learning for visual imitation, 2021
2021
-
[32]
Relevant
J.-B. Grill, F. Strub, F. Altch´e, C. Tallec, P . H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. V alko. Bootstrap your own latent: A new approach to self-supervised learning, 2020. URLhttps://arxiv.org/a...
2020 arXiv
-
[34]
(Left)Retrieval distribution across tasks.(Right)Retrieval distribution across timesteps
Pick up salad dressing and put it in tray 7: Put white mug on left plate 8: Pick up chocolate pudding and put it in tray 9: Others 10: Put chocolate pudding to right of plate 11: Put white Mug on Plate MUG MICROWA VE Figure 5: Mug-Microwave LIBERO T ask. (Left)Retrieval distri...
-
[35]
(Left)Retrieval distribution across tasks.(Right)Retrieval distribution across timesteps
Pick up salad dressing and put it in tray 7: Put white mug on left plate 8: Pick up chocolate pudding and put it in tray 9: Others 10: Put chocolate pudding to right of plate 11: Put white Mug on Plate MUG MUG Figure 6: Mug-Mug LIBERO T ask. (Left)Retrieval distribution across...
-
[36]
(Left)Retrieval distribution across tasks.(Right)Retrieval distribution across timesteps
Pick up salad dressing and put it in tray 7: Put white mug on left plate 8: Pick up chocolate pudding and put it in tray 9: Others 10: Put chocolate pudding to right of plate 11: Put white Mug on Plate MUG PUDDING NEW Figure 7: Mug-Pudding LIBERO T ask. (Left)Retrieval distrib...
-
[37]
(Left)Retrieval distribution across tasks.(Right)Retrieval distribution across timesteps
Pick up salad dressing and put it in tray 7: Put white mug on left plate 8: Pick up chocolate pudding and put it in tray 9: Others 10: Put chocolate pudding to right of plate 11: Put white Mug on Plate SOUP CHEESE Figure 8: Soup-Cheese LIBERO T ask. (Left)Retrieval distributio...
-
[38]
(Left)Retrieval distribution across tasks.(Right)Retrieval distribution across timesteps
Pick up salad dressing and put it in tray 7: Put white mug on left plate 8: Pick up chocolate pudding and put it in tray 9: Others 10: Put chocolate pudding to right of plate 11: Put white Mug on Plate SOUP SAUCE Figure 9: Soup-Sauce LIBERO T ask. (Left)Retrieval distribution ...
-
[2025]
URL https://openreview.net/forum?id=4VHiptx7xe
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.