{"id":"a9c9239a-9a57-407e-abb0-72f98e0e3913","arxiv_id":"2510.17059","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"CIRL trains an agent with no rewards or demonstrations by having it propose and reach its own goals, then imitates a single test-time demonstration by inferring and reaching the demonstrated goal.","lead":"This paper presents CIRL, a self-supervised method that trains agents to imitate a single demonstration using no rewards and no expert data during training. The agent invents its own goals during pretraining, then decides which goal a new demonstration is aiming for and reaches it.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CIRL's consistency proof only holds when pretraining goals are sampled from the expert's goal prior p(g); GoalKDE violates this, so Lemma 1 does not apply to the algorithm as implemented.","rationale":"The reader's weakest assumption correctly identifies that CIRL's consistency theorem requires the pretraining goal prior to equal the expert's goal prior, while GoalKDE explores a different, policy-dependent distribution. The paper itself concedes this distribution shift in Appendix D, and the proof in Appendix A.1 explicitly relies on the same prior p(g) being used in data collection and modeling. I considered other potential concerns, such as the mean-field 'WLOG' claim and the transition from reverse-KL ELBO to forward amortized variational inference, but the prior mismatch is the most load-bearing because it invalidates the main theoretical contribution for the actual algorithm. The proposed concrete test would settle whether the concern lands by checking whether the variational posterior trained under GoalKDE is calibrated to the oracle test-time goal distribution; if not, Lemma 1 does not apply. The reader's REJECT verdict remains appropriate.","tokens_in":18603,"tokens_out":7926,"duration_ms":72166,"concrete_test":"Analytical test: In a simple two-goal MDP with known dynamics, compute the MaxEnt IRL objective under the oracle goal prior p(g) and the L_Info objective under μ(g)=GoalKDE's stationary proposal. Show that the minimizers differ when μ≠p(g), e.g., by evaluating the extra term E_{p_E(τ|g)}[log(p(g)/μ(g))]. If the two objectives have different optima, Lemma 1 cannot hold under GoalKDE. Computational complement: using the released code, train CIRL with GoalKDE in Reacher, then evaluate log qξ(g|τ) on oracle-sampled expert demonstrations (g∼p(g)); compare to a control trained with oracle goal sampling. A systematic deficit or miscalibration under GoalKDE would confirm the distribution-shift concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Appendix A.1's cancellation of prior terms: 'When the same prior p(g) is used,' the conditional KL equals the joint KL, and L_Info is derived from the joint p*(g,τ) = p(g)p*(τ|g). CIRL's actual pretraining uses GoalKDE (Algorithm 1, line 5: g = argmin_g KDE(D)), a deterministic, policy-dependent proposal distribution μ(g), not a sample from p(g). Therefore the variational posterior learned by L_Info approximates p_μ(g|τ) ∝ μ(g)p*(τ|g), not the expert posterior p(g|τ) ∝ p(g)p_E(τ|g). Recomputing the KL decomposition with μ in place of p(g) leaves an uncancelled term E_{p_E(τ|g)}[log(p(g)/μ(g))] in the surrogate, so minimizing L_Info is not equivalent to MaxEnt IRL under the test-time goal prior. This is exactly the distribution shift the authors acknowledge in Appendix D: 'most of the performance gap is likely due to distribution shift between goals explored by GoalKDE and those commanded at test time.' Since the theorem's key assumption is violated by the algorithm that was actually run, the 'consistent' claim is not proved for CIRL as implemented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CIRL, a self-supervised pretraining method for zero-shot imitation. CIRL trains a goal-conditioned maximum-entropy policy with contrastive RL, proposes training goals via a KDE-based low-density sampler (GoalKDE), and learns an amortized variational posterior q_xi(g|tau) over goals from trajectories. At test time, given a single demonstration, it infers a goal and commands the learned goal-conditioned policy. The paper claims a consistency guarantee (Lemma 1) that MaxEnt IRL over goal-conditioned rewards is equivalent to maximizing L_Info = E_{g~p(g), tau~p^*(tau|g)} log q_xi(g|tau), and reports experiments on JaxGCRL and URLB showing improved imitation scores over FB and 1-NN baselines.","tokens_in":18921,"tokens_out":11698,"duration_ms":99071,"significance":"The conceptual goal—reducing multi-task IRL to goal inference while accounting for the partition function—is appealing, and the paper ships reproducible code and conducts informative ablations. However, the central theoretical contribution is not established as stated. The proof of Lemma 1 effectively assumes the learned policy trajectory distribution equals the expert's, and the implemented GoalKDE goal sampler violates the common-prior assumption of the theorem. The paper therefore does not currently deliver a consistency guarantee for the algorithm that is actually evaluated. The empirical results may still be of interest, but the main novelty is the claimed consistency, and that claim is unsupported.","major_comments":[{"comment":"The proof never bridges the IRL objective and the L_Info objective. After writing min_theta E_p(g) D_KL(p_E(tau|g) || p^*(tau|g)), the proof switches to FA VI over p^*(tau,g) and shows that max_xi E_{p^*(g,tau)} log q_xi(g|tau) makes q_xi match p^*(g|tau). But matching the posterior of the learned policy is not the same as matching the expert posterior p_E(g|tau). The two coincide only if p^*(tau|g) = p_E(tau|g), which is assumed nowhere in the lemma and is not guaranteed by optimality of the learned policy unless the expert is itself the MaxEnt-optimal policy for the same entropy coefficient. Under that assumption the result is almost tautological: q is trained on samples from p^*(g,tau) and hence recovers p^*(g|tau) by definition. This does not constitute a consistency proof for IRL.","section":"Section 4.3 / Lemma 1 / Appendix A.1"},{"comment":"The theorem requires goals to be drawn from p(g), the same prior used at test time ('When the same prior p(g) is used'). GoalKDE instead selects g = argmin_g KDE(D), a deterministic, policy-dependent low-density proposal mu(g). Consequently the variational objective trains q on p_mu(g|tau) proportional to mu(g) p^*(tau|g), not the expert posterior under p(g). Recomputing the KL decomposition with mu leaves an uncancelled E[log(p(g)/mu(g))] term, so maximizing L_Info is not equivalent to MaxEnt IRL under the test-time prior. The authors' own Appendix D statement—'most of the performance gap is likely due to distribution shift between goals explored by GoalKDE and those commanded at test time'—concedes this distribution shift. Thus Lemma 1 does not apply to CIRL as implemented.","section":"Algorithm 1 line 5 / Section 4.4 / Appendix D"},{"comment":"The claim that the mean-field family 'without loss of generality' contains the true posterior is asserted without proof. The true posterior p^*(g|tau) is proportional to p(g) exp(sum_t r_g(s_t,a_t) - log Z_g); the goal-dependent factors p(g) and Z_g are global and do not factor over time. No argument shows they can be absorbed into per-timestep Gaussian conditionals q_xi(g|s_t,a_t), and the Gaussian MLP class is not closed under arbitrary such reparameterizations. This is not merely technical, because the corollary is used to justify the goal-inference architecture. The empirical superiority of mean field over full-trajectory inference in Figure 5 can stand on its own, but the corollary should be proved with appropriate capacity assumptions or removed.","section":"Section 4.2 / Corollary 1"}],"minor_comments":[{"comment":"The symbol '=⇒' is ambiguous. The paper should state whether Lemma 1 claims an equivalence, an implication, or a reduction, and in which direction. There is also a missing closing parenthesis in the KL expression.","section":"Equation (4)"},{"comment":"The sentence 'By additionally noting that the g we infer should have high mutual information with tau, we simplify...' is misleading. The simplification from D_KL(p^*(g|tau) || q_xi(g|tau)) to E log q_xi(g|tau) is just the forward KL objective with respect to xi; the mutual-information remark is unnecessary.","section":"Section 4.2"},{"comment":"In the chain-rule step, the notation 'E_{p(g)}[D_KL(p_E(tau,g) || p^*(tau,g))]' is not correct; the joint KL is D_KL(p_E(tau,g) || p^*(tau,g)) after defining the joints with the same prior. Also, the roles of theta and g are not consistently distinguished: the IRL objective is written as min_theta, but the goal-conditioned family is indexed by g.","section":"Appendix A.1"},{"comment":"The expert is trained with CRL under oracle goal sampling, while CIRL is trained with GoalKDE. The paper should clarify whether the expert satisfies the MaxEnt optimality assumption of Lemma 1; the reported entropy coefficient alpha = 1e-5 is very small, so the expert is close to deterministic and may not be the soft-Boltzmann policy used in the theory.","section":"Section 5 / Appendix C"},{"comment":"The admission that 'most of the performance gap is likely due to distribution shift between goals explored by GoalKDE and those commanded at test time' directly conflicts with the consistency claim in the main text. This is currently relegated to the appendix; it should be discussed in the main text where Lemma 1 is presented.","section":"Appendix D / Figure 8"},{"comment":"The FB inconsistency counterexample assumes the backward representation can be the one-hot occupancy vector. For learned FB representations this equality is not guaranteed, so the lemma is illustrative rather than a rigorous refutation of FB as a family. It is not central to CIRL, but the paper should temper the wording.","section":"Appendix A.2 / Lemma 2"}],"recommendation":"reject","confidential_remarks":"The paper has a useful empirical study and clear writing, but the central theoretical claim is not supported. The proof of Lemma 1 is circular without an explicit assumption that the expert is the MaxEnt-optimal policy, and the implemented GoalKDE sampler violates the same-prior condition of that theorem. These are load-bearing issues for the paper's main contribution, not presentation problems. I would encourage the authors to resubmit a version that either proves consistency for the actual algorithm (e.g., with a goal proposal correction term) or substantially qualifies the title and abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nWhat you should know up front: the headline theorem doesn't prove what the algorithm does. The method itself is a sensible integration of existing pieces—contrastive RL, KDE goal sampling, amortized inference—and the counterexample against Forward-Backward representations is a legitimate new observation. But the central consistency claim is not established for the system that produced the experiments.\n\nWhat's genuinely good: the FB counterexample (Lemma 2) is clean and to the point: occupancy-based reward inference can confuse an infrequently visited goal with a frequent one. That's a useful warning for anyone using successor representations for IRL. The CIRL pipeline is also a reasonable instantiation of the \"goals as summaries\" idea, and the experiments, while limited, are at least on standard benchmarks and the code is released.\n\nThe main problem is in the proof of Lemma 1. The equivalence between MaxEnt IRL and the variational goal-inference objective requires sampling goals from the same prior p(g) used in the expert's distribution and evaluating the learned policy's trajectory distribution p*(τ|g) for those goals. The algorithm instead proposes goals via GoalKDE, which is a deterministic, policy-dependent selection rule, not a sample from any fixed prior. The paper's own Appendix D admits that \"most of the performance gap is likely due to distribution shift between goals explored by GoalKDE and those commanded at test time.\" That admission is effectively a concession that the theorem's key assumption is violated by the implemented system. So the \"consistent\" in the title is not supported.\n\nThere are two smaller issues worth noting. The mean-field \"without loss of generality\" claim is too quick: the true posterior does not obviously factor as a product of per-state distributions, since it contains a single p(g) and a partition function Z_g. The argument needs to be made rigorous or downgraded to a heuristic. And the empirical section reports single runs without error bars or seeds, which is thin support for the sweeping superiority claims.\n\nVerdict: this is not a paper where the main theoretical result holds. But it's not a toy either—the method works in some environments, and the FB counterexample is worth publishing. I'd send it to a serious referee with a request to either fix the consistency proof for GoalKDE or change the claim, and to add proper statistics. The idea is good enough that it deserves another iteration.\n\nRecommendation: accept for peer review, expect major revision.","headline":"CIRL's consistency theorem doesn't apply to the GoalKDE-based algorithm that was actually run, though the FB counterexample is a genuine contribution.","tokens_in":19410,"tokens_out":3635,"would_cite":true,"duration_ms":30475,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot imitation reduces to goal inference: a self-supervised agent infers the demonstrator's goal from one trajectory and commands a contrastive goal-conditioned policy, and this procedure is proven consistent.","keywords":["zero-shot imitation","goal-conditioned reinforcement learning","inverse reinforcement learning","contrastive RL","maximum entropy","variational inference","goal inference","self-supervised pretraining"],"falsifier":"On a locomotion task, train CIRL with GoalKDE exploration but evaluate on expert trajectories whose true goals lie in a region GoalKDE rarely visited during training (e.g., a far corner of the arena). Plot the MAP-inferred goal against the true goal across many demonstrations: if the consistency claim holds for the algorithm as run, the inferred goals should be unbiased; any systematic bias toward the visited low-density region — and a corresponding drop in imitation score relative to oracle goal sampling — would refute the claim as applied.","tokens_in":18454,"feed_emoji":"🎯","tokens_out":7525,"duration_ms":64173,"temperature":0.7,"pith_summary":"The paper introduces CIRL, a self-supervised pretraining method for zero-shot imitation. The central claim is that when tasks can be summarized by a goal state, the multi-task inverse reinforcement learning problem reduces to goal inference: instead of recovering a reward function from a demonstration, the agent infers which goal the demonstrator was pursuing, then commands a goal-conditioned policy trained entirely through autonomous exploration. The paper proves that this reduction is consistent (Lemma 1) — that is, it recovers the correct distribution over expert goals by accounting for the relative difficulty of reaching different states, something occupancy-based methods like Forward-Backward fail to do (Lemma 2). The practical payoff is that an agent can be pretrained with no rewards, no demonstrations, and no preferences, and then imitate a single expert trajectory at test time. Experiments on goal-conditioned and reward-maximizing benchmarks show CIRL outperforms prior zero-shot imitation baselines.","feed_headline":"Goal inference makes zero-shot imitation provably consistent","feed_subtitle":"The paper proves goal inference is consistent and beats prior imitation baselines.","key_machinery":"The key machinery is Lemma 1, an identity that converts MaxEnt IRL into a variational goal-inference objective. The forward KL between expert and model trajectory distributions decomposes via the KL chain rule; because the same goal prior appears in both joints, the objective becomes a variational posterior-matching problem. The crucial trick is that trajectories are sampled from the MaxEnt policy (g∼p(g), τ∼p*(τ|g)), so the intractable partition function Z_g is accounted for implicitly in the sample distribution. This is coupled with two supporting components: a mean-field variational family (Corollary 1) that contains the true posterior for finite trajectories, and GoalKDE, an automatic go","core_discovery":"The paper's central claim is an equivalence (Lemma 1): under MaxEnt IRL with goal-conditioned rewards, minimizing the forward KL between expert and model trajectory distributions (with a shared goal prior) is equivalent to maximizing a variational goal-inference objective — min_θ E_{p(g)} D_KL(p_E(τ|g)∥p*(τ|g)) ⟹ max_ξ E_{g∼p(g),τ∼p*(τ|g)} log q_ξ(g|τ). The partition function encoding the relative difficulty of reaching goals is never computed; it is implicitly captured by sampling trajectories from the MaxEnt policy. The paper further proves (Corollary 1) that a mean-field variational posterior is without loss of generality for finite trajectories, and gives a counterexample (Lemma 2) showi","pith_inferences":["The consistency theorem requires the goal prior used in pretraining to equal the expert's goal prior. The deployed algorithm uses GoalKDE, which samples low-density states rather than drawing from p(g); the authors acknowledge in Appendix D that most of the measured performance gap stems from this distribution shift. The theoretical guarantee therefore does not directly cover the algorithm as run.","If GoalKDE were replaced by a method that samples goals from the true test-time prior, the imitation score should approach the oracle baseline whenever the variational goal-inference model is well calibrated; the gap observed between GoalKDE and oracle sampling is thus a measurable violation of the theorem's assumptions.","The variational sampling trick in Lemma 1 may extend beyond goal-reaching rewards to any reward family parameterized by a low-dimensional latent variable, though the mean-field corollary would not survive without the goal structure.","Lemma 2's counterexample generalizes to a broader lesson for IRL: any method that treats state visitation frequency as a proxy for reward will misread policies that spend time in easy-to-reach rather than rewarding states."],"forward_implications":["Agents can be pretrained for imitation entirely through self-supervised interaction — no rewards, no demonstrations, no human preference data — and then imitate a single expert trajectory at test time.","Because goal inference accounts for the relative difficulty of reaching states, an infrequently visited but intended state is correctly identified as the goal, something occupancy-based IRL methods provably fail to do.","By enlarging the goal space (e.g., adding velocity to position), the same framework can imitate reward-maximizing policies that are not explicitly goal-reaching, as demonstrated on standard benchmark tasks.","Mean-field goal inference, which predicts the goal independently from each state-action pair, is both computationally cheaper and, when the true posterior is mean-field, exactly as expressive as full-trajectory encoders.","The proven failure of Forward-Backward representations (Lemma 2) is a caution: using occupancy measures directly as rewards is fundamentally inadequate for zero-shot imitation."],"fun_headline_variants":["Goal inference proves zero-shot imitation consistency","CIRL: Provable zero-shot imitation via goal inference","Self-supervised goal inference yields consistent imitation","Imitation without rewards: goal inference guarantees intent","Zero-shot imitation: proof of consistent goal recovery"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The consistency proof assumes the goal distribution used during pretraining is the same as the expert's goal prior and that the learned MaxEnt policy is optimal; in practice GoalKDE explores low-density states rather than sampling from the test-time goal prior, so the theorem does not apply to the algorithm as run — a gap the authors attribute for most of the performance loss.","fun_headline_variants_meta":{"raw":{"variants":["Goal inference proves zero-shot imitation consistency","CIRL: Provable zero-shot imitation via goal inference","Self-supervised goal inference yields consistent imitation","Imitation without rewards: goal inference guarantees intent","Zero-shot imitation: proof of consistent goal recovery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1527,"prompt_tokens":778,"completion_tokens":749,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":678}},"tokens_in":522,"tokens_out":749,"duration_ms":6678,"temperature":1.0,"reasoning_tokens":678,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T09:07:14.427681+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a locomotion task, train CIRL with GoalKDE exploration but evaluate on expert trajectories whose true goals lie in a region GoalKDE rarely visited during training (e.g., a far corner of the arena). Plot the MAP-inferred goal against the true goal across many demonstrations: if the consistency claim holds for the algorithm as run, the inferred goals should be unbiased; any systematic bias toward the visited low-density region — and a corresponding drop in imitation score relative to oracle goal sampling — would refute the claim as applied.","supporting_citations":[],"review_version":1}