{"id":"59c484f1-8dff-417d-91ef-df704b969641","arxiv_id":"2506.01623","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MAGIK reuses a source RL policy for new analogous tasks by using a semi-supervised VAE to imagine target observations in source form, achieving zero-shot transfer in MiniGrid and Reacher.","lead":"MAGIK lets a reinforcement learning agent solve a new, similar task without practicing it, by using a variational autoencoder to re-imagine what it sees in terms of the old task. The authors show this works for simple pick-and-reach tasks, but the analogy mapping is specified by hand.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MAGIK's reported zero-shot transfer depends on a per-target, hand-crafted class-to-source mapping supplied by the experimenter (Sec. 6.1.1), not learned by the agent; without this mapping the method reduces to manual observation relabeling.","rationale":"The paper's central claim, as stated in the abstract and implemented in Algorithm 1, is that encoding a target observation, replacing its class latent with the source class, and executing the source policy on the imagined observation yields effective zero-shot transfer using only a small number of human-labelled examples. This would be a substantial result if the class-to-source correspondence were learned or emergent. The experiments show otherwise: every target task is accompanied by an experimenter-chosen mapping, and Target 3 uses a chained mapping that effectively encodes the target reward function. Nothing in the VAE objective or in Algorithm 1 discovers which source class corresponds to which target class; the architecture only performs the substitution once csource is supplied. The assumption in Section 4 that the target observation space is a subset of the source observation space makes the VAE's class swap possible, but it does not select the analogy. Therefore the strong empirical results in Tables 2 and 3 rest on additional human-provided task knowledge that is not counted among the labelled examples. The missing direct-relabeling baseline is important because if recoloring raw observations and running the same SAC policy reproduces MAGIK's scores, the imagination network is not the mechanism of transfer; the transfer is fully attributable to the user-specified mapping. This concern does not allege any experimental fraud or internal inconsistency; the reported numbers may be accurate. Rather, it questions whether the method as described supports the claimed novelty of an agent that maps to analogous goals. The reader's conditional verdict already identifies the same weakest assumption, and the suggested revisions (direct relabeling baseline, explicit statement that the analogy is user-provided) are appropriate. I therefore agree with the reader and recommend no change to the conditional verdict.","tokens_in":19772,"tokens_out":5733,"duration_ms":68850,"concrete_test":"Add a baseline that performs direct observation relabeling without the VAE: for MiniGrid, recolor red balls to green in each raw observation; for Reacher, replace the target-colour feature with the blue feature in the observation vector. Execute the same pretrained SAC policy with no imagination network, over the same 5 seeds and episode counts as Tables 2 and 3. Also run MAGIK on Target 3 with a single generic mapping (all observed classes mapped to green) to confirm that the chained per-target mapping is necessary for the reported red-ball score. If the direct relabeling baseline matches MAGIK's performance, or if the generic mapping fails Target 3, the reported transfer is attributable to the user-specified class mapping rather than to the imagination mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MAGIK enables zero-shot transfer by mapping target observations to analogous source states, requiring only a small number of human-labelled examples. The experiments, however, rely on a target-specific analogy mapping chosen by the experimenter for each task. In Section 6.1.1, MAGIK is 'instructed by tweaking the c latent': for Targets 1 and 2, red-ball observations are reclassified as green; for Target 3, Class 2 is mapped to Class 4 (no ball) and Class 3 is first mapped to Class 1 and then to Class 2. Algorithm 1 only implements a fixed csource substitution once this mapping has been selected. The Target 3 chain is particularly telling: it encodes the target reward structure (ignore green, pick red) by suppressing the green ball and converting red to green. This is additional target-task information that is not included in the reported '600 human-labelled examples' count, and it is not discovered by the agent. If the mapping is withheld or chosen incorrectly, MAGIK has no mechanism to infer the analogy; it simply executes the source policy on the relabeled observation. Thus the empirical support in Tables 2 and 3 validates a manually specified observation-relabeling procedure, not an agent that maps to analogous goals on its own. The paper should state clearly that the analogy is provided by the user, and it should compare against direct observation relabeling to separate the contribution of the VAE from the contribution of the hand-specified mapping.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MAGIK, a framework for zero-shot reinforcement-learning transfer across analogous tasks. A semi-supervised VAE trained on source-task observations is used to disentangle a class-agnostic continuous latent z from a class-specific discrete latent c; at deployment time, Algorithm 1 encodes a target observation, replaces c with a source-class value, decodes an imagined source-aligned observation, and executes the source SAC policy. Experiments in a custom MiniGrid environment (pick green vs. pick red) and a modified MuJoCo Reacher environment (reach different coloured targets) report performance close to fine-tuned SAC, with 600 (MiniGrid) or 250 (Reacher) labels, and compare against successor-feature variants and a GAN-based transfer method.","tokens_in":20076,"tokens_out":5371,"duration_ms":59226,"significance":"If the claims are appropriately scoped, the paper is a useful empirical demonstration of policy reuse through latent substitution: it is transparent, the code is promised to be available, and it includes comparisons to several successor-feature baselines and to fine-tuning. The approach is not circular in the fitting sense, since target-task achievements are not optimized against target rewards. The main weakness is that the analogy itself is not learned or discovered by the agent: Section 6.1.1 specifies a hand-authored class-to-source mapping for every target task, so the reported results validate a user-supplied relabeling strategy more than an autonomous analogy-mapping mechanism. This overstatement affects the central claim and must be addressed before the paper can be recommended.","major_comments":[{"comment":"Algorithm 1 takes csource as an input, and Section 6.1.1 supplies that input by hand for every target: Targets 1 and 2 map Class 1 (red ball) to Class 2 (green ball), and Target 3 uses a chained mapping in which Class 3 is first imagined as Class 1 and then as Class 2, while Class 2 is imagined as Class 4. The Target 3 chain encodes the target reward structure (ignore the green ball, convert the red ball to green) and is additional target-task information not included in the reported 600 labelled examples. As written, the paper demonstrates zero-shot execution of a user-specified observation relabeling, not an agent that 'maps entities in the target task to their analogues' on its own. Please either explicitly state that the analogy mapping is provided by the user and revise the abstract and introduction accordingly, or add a mechanism by which the mapping is inferred; in either case, add an experiment in which the mapping is withheld or chosen incorrectly.","section":"Section 6.1.1 and Algorithm 1"},{"comment":"The empirical section compares MAGIK to fine-tuned SAC and to successor-feature baselines, but it never compares against direct observation relabeling, such as recoloring the red ball pixels to green in MiniGrid or changing the target-colour feature to blue in Reacher and then executing the source policy. Such a baseline is necessary to separate the contribution of the VAE's disentanglement from the contribution of the hand-specified class-to-source mapping. If direct relabeling performs as well as MAGIK, the proposed imagination mechanism adds no measurable benefit; if it does not, the difference should be quantified and discussed.","section":"Tables 2 and 3 (missing baseline)"},{"comment":"The data-efficiency claim (H2) is supported only by point estimates averaged over five seeds with no significance tests or confidence intervals. Several comparisons have substantially overlapping error bars, for example Target 1 in MiniGrid: MAGIK 8.20 +/- 0.20 vs. SAC-Fine tuned 8.60 +/- 0.74, and Target 3: MAGIK 9.00 +/- 0.54 vs. SAC-Fine tuned 9.00 +/- 0.54. The statement that MAGIK 'achieves comparable performance' and the 16x reduction claim need statistical support, at minimum paired per-seed results, a paired test, or bootstrapped confidence intervals.","section":"Section 6.1.2, Tables 2 and 3"}],"minor_comments":[{"comment":"The first paragraph states that class labelling was simulated and that 'no human were involved in labeling', while the abstract says 'a small number of human-labelled examples'; please reconcile this by saying the labels are of the kind that would require human annotation and report them as an annotation budget rather than actual human labour.","section":"Section 6.1"},{"comment":"The 'Blue' row in Table 3 is the source task itself, not a transfer target. Please label it explicitly as a control or move it to a separate part of the table.","section":"Table 3"},{"comment":"The algorithm should state explicitly that csource is provided by the user or by a separate mapping module; the current pseudocode leaves the origin of csource implicit.","section":"Algorithm 1"},{"comment":"The sentence 'The SF assume the linearity of the reward' contains a grammar error and should be reworded, for example to 'Successor features assume that the reward is linear in the feature representation.'","section":"Section 2.1"},{"comment":"The supplementary table has duplicated rows for the Blue and Yellow conditions under 30K and 50K random samples; please correct the table so each condition appears once.","section":"Supplementary Table 6"},{"comment":"The code link appears as 'available here' with no visible URL in the manuscript text; please provide a working link.","section":"Footnote 2"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the central issue is the gap between the 'analogy mapping' language in the title and abstract and the hand-specified csource in Section 6.1.1. This is a load-bearing overclaim rather than a cosmetic flaw. I do not think rejection is necessary, because the empirical result is still a valid demonstration of policy reuse under a user-provided analogy, and the paper is honest about several limitations in Section 7. A revised version that reframes the contribution, adds a direct relabeling baseline, and provides statistical support for H2 could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: MAGIK is a clean latent-swapping approach for zero-shot RL transfer, and the experiments on MiniGrid and Reacher are solid as far as they go. The catch—and it's a real one—is that the analogy is not learned. The experimenter tells MAGIK which class to swap for each target (Section 6.1.1), and for Target 3 the prescribed chain essentially encodes the target reward function. So the paper demonstrates that a VAE can be used to relabel observations when a human supplies the mapping; it doesn't demonstrate an agent that discovers the analogy on its own.\n\nWhat's genuinely new: a semi-supervised VAE with HSIC to disentangle class identity (c) from structure (z), then swapping c to generate source-aligned observations for policy reuse. The results are strong: MAGIK reaches near fine-tuned performance on color-swapped MiniGrid and Reacher tasks, and it beats the Successor Feature baselines they tried. I believe the empirical tables support the narrow claim that latent swapping works for these two benchmarks.\n\nThe main soft spot is the missing baseline: direct observation relabeling. In Reacher, the state includes target color; simply replacing the target color feature with blue would likely match MAGIK. Without that baseline, the contribution of the VAE is unclear. The authors should also be upfront that the per-target class mapping is human-authored and not part of the '600 labeled examples' count. Minor issues: no significance tests, standard deviations are sometimes large, and the code link isn't a working URL.\n\nStill, the paper is honest about some limitations (Section 7) and includes useful ablations in the supplement. The idea is incremental relative to [9] and [17], but it's a reasonable and clearly written contribution. I'd send it to review. The right outcome is probably a conditional accept, requiring a relabeling baseline and a rewritten claim that the analogy is user-provided.","headline":"Clean latent-swapping trick for zero-shot RL transfer, but the analogy is hand-specified per target, so the 'zero-shot' claim oversells.","tokens_in":20630,"tokens_out":2329,"would_cite":false,"duration_ms":24559,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MAGIK claims RL agents can solve analogous tasks zero-shot by encoding a target observation, swapping its class latent for the source class, and decoding an imagined source-aligned observation for a pretrained policy.","keywords":["MAGIK","zero-shot transfer","reinforcement learning","analogical reasoning","semi-supervised variational autoencoder","latent disentanglement","policy reuse"],"falsifier":"A decisive test: withhold the hand-crafted class mapping for Target 3 and instead let MAGIK choose the substitution automatically from its classification head. If red-ball picking drops from 9.00 to the untransferred SAC level near 1.75 out of 10, the reported result relies on the human-specified chain, not on the imagination mechanism alone.","tokens_in":19548,"feed_emoji":"🎯","tokens_out":7124,"duration_ms":73680,"temperature":0.7,"pith_summary":"An RL agent that has learned one task can often be applied to a structurally similar task if it first 'imagines' the new situation as the old one. MAGIK claims to mechanize this: a semi-supervised variational autoencoder splits each observation into a class-agnostic part (layout, positions) and a class label (the rewarded object), and the agent swaps in the source class and runs its original policy on the decoded image. If true, skills would transfer without any interaction with the target environment, using only a small number of human-labelled examples instead of thousands of fine-tuning steps. The paper reports that this recipe matches fine-tuned SAC on colour-swapped MiniGrid and MuJoCo Reacher tasks.","feed_headline":"Imagining old tasks solves new RL goals with zero retraining","feed_subtitle":"A latent-class swap lets a frozen policy match fine-tuned scores on swapped-object tasks.","key_machinery":"The imagination network is a semi-supervised VAE with two latent variables: a continuous z for task-agnostic structure and a discrete c for task/class identity, sampled with Gumbel-Softmax. The two codes are pushed to be independent by an HSIC regulariser, and the decoder conditions c on z through FiLM layers. Algorithm 1 carries the argument: encode the target observation to z, decode with c set to the source class, then feed the imagined observation to the frozen source policy.","core_discovery":"The central claim is that zero-shot transfer between analogous RL tasks can be reduced to a latent-class substitution. On a target observation, the imagination network estimates the task-agnostic code z and the task-specific class c; replacing c with the source class and decoding produces an observation the source policy already knows how to handle. The paper argues that a semi-supervised objective plus an HSIC independence penalty is enough to keep z and c disentangled, and that this makes the substitution valid. Empirically, MAGIK reaches 8.20-9.00 out of 10 good picks in MiniGrid targets and 19.20-20.00 out of 20 reaches in Reacher, close to or matching fine-tuned SAC, with zero additional environment interaction and only 600/250 labelled examples.","pith_inferences":["A direct extension the paper leaves unstated: any analogy expressible as a one-step class substitution (colour, shape, label) should port to MAGIK unchanged, whereas analogies that require changing positions, counts, or relations will not, because z is held fixed.","The Target 3 protocol chains two class substitutions; this opens the door to composing imagination steps, so an agent could handle a target that requires removing one object and then relabelling another, provided the chain is specified.","The paper notes but does not test an active-learning variant of the label budget; a testable extension would query the classification head's uncertainty and should cut the 250-600 labelled examples further.","Because the source policy is frozen and the imagination network is trained offline, the same mechanism could be applied to any frozen policy, not only SAC, so long as the policy consumes the decoded imagined observations."],"forward_implications":["On MiniGrid Target 3, MAGIK picks the red ball 9.00 of 10 episodes, matching SAC fine-tuned (9.00), while untransferred SAC gets 1.75.","On Reacher, MAGIK reaches 19.20 of 20 for red and green and 19.60 for yellow, close to fine-tuned SAC (19.40-19.80), with zero target interaction.","Data cost drops: 600 labelled MiniGrid examples and 250 Reacher examples (under 1% of source data) replace roughly 10K fine-tuning interactions, giving up to a 16x reduction.","Transfer works when the reward changes and observation changes together, a setting where successor-feature baselines that assume linear rewards or shared dynamics struggle."],"supporting_citations":[{"why":"supplies the variational autoencoder objective that the imagination network extends to two latent variables.","marker":"[19]"},{"why":"provides the semi-supervised ELBO formulation used to train on labelled and unlabelled observations.","marker":"[20]"},{"why":"supplies the Hilbert-Schmidt Independence Criterion used as a regulariser to disentangle z and c.","marker":"[10]"},{"why":"supplies the Gumbel-Softmax trick used to sample the discrete class latent c.","marker":"[18]"},{"why":"supplies the FiLM conditioning layers that let the decoder combine the class-agnostic and class-specific latents.","marker":"[25]"},{"why":"is the Soft Actor-Critic algorithm whose pretrained policy is reused by MAGIK.","marker":"[12]"},{"why":"is the image-to-image transfer baseline that fails to transform red objects to green and is compared against.","marker":"[9]"},{"why":"is a successor-feature baseline that MAGIK outperforms in both environments.","marker":"[6]"},{"why":"is the prior transfer work that motivates the assumption that the source task contains all relevant transferable elements.","marker":"[17]"},{"why":"supplies the MiniGrid environment used for the image-based experiments.","marker":"[5]"}],"fun_headline_variants":["Imagination swaps task labels for zero-shot RL transfer","Frozen policy nails new tasks by imagining old analogs","Zero-shot RL: imagine old task to solve new one","Map target to source in latent space then zero-shot act"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes a human has already worked out the correct class-to-source mapping for each target task and tells the agent which class to imagine; without that hand-crafted instruction, MAGIK has no way to discover the analogy.","fun_headline_variants_meta":{"raw":{"variants":["Imagination swaps task labels for zero-shot RL transfer","Frozen policy nails new tasks by imagining old analogs","Zero-shot RL: imagine old task to solve new one","Map target to source in latent space then zero-shot act"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000848,"raw_usage":{"total_tokens":3636,"prompt_tokens":836,"completion_tokens":2800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":2735}},"tokens_in":452,"tokens_out":2800,"duration_ms":19347,"temperature":1.0,"reasoning_tokens":2735,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:36:47.549524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test: withhold the hand-crafted class mapping for Target 3 and instead let MAGIK choose the substitution automatically from its classification head. If red-ball picking drops from 9.00 to the untransferred SAC level near 1.75 out of 10, the reported result relies on the human-specified chain, not on the imagination mechanism alone.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the semi-supervised ELBO formulation used to train on labelled and unlabelled observations."},{"cited_title":"Gretton, R","cited_arxiv_id":null,"evidence_quote":"supplies the Hilbert-Schmidt Independence Criterion used as a regulariser to disentangle z and c."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the Gumbel-Softmax trick used to sample the discrete class latent c."},{"cited_title":"Haarnoja, A","cited_arxiv_id":null,"evidence_quote":"is the Soft Actor-Critic algorithm whose pretrained policy is reused by MAGIK."},{"cited_title":"Gamrian and Y","cited_arxiv_id":null,"evidence_quote":"is the image-to-image transfer baseline that fails to transform red objects to green and is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is a successor-feature baseline that MAGIK outperforms in both environments."},{"cited_title":"Higgins, A","cited_arxiv_id":null,"evidence_quote":"is the prior transfer work that motivates the assumption that the source task contains all relevant transferable elements."},{"cited_title":"Chevalier-Boisvert, B","cited_arxiv_id":null,"evidence_quote":"supplies the MiniGrid environment used for the image-based experiments."}],"review_version":1}