{"id":"1cdce403-d943-4831-b63a-f319109e4570","arxiv_id":"2505.08073","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A reverse world model trained on reversed replay data generates counterfactual prior states that significantly help non-experts identify why a model-based RL agent failed.","lead":"Researchers trained a reverse world model that predicts what an environment must have looked like for a reinforcement learning agent to choose a user-preferred action. In a 70-person study, people shown these counterfactual images were significantly better at identifying why the agent failed and reported higher trust and lower mental effort.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RWM's counterfactual fidelity is never directly tested, and the user study lacks an active-control arm, so the causal role of the explanation content remains unestablished.","rationale":"The reader's conditional verdict is appropriate. My stress-test sharpens the same weakest assumption: the RWM's counterfactual states are never checked against the policy's actual action preferences, and the human-study design confounds explanation content with the presence of extra images. A simple computational action-match test on the frozen policy would establish whether the images satisfy the paper's own definition of a counterfactual. The paper does report a statistically significant user-study effect (Table 1, H1), which is real evidence of improved performance; the issue is attribution, not fabrication. The absence of code and data further limits reproducibility, but that alone would not change the conditional verdict. If the action-match test passes and an active-control arm were added, the central claim would be substantially stronger; if it fails, the conditional recommendation should move toward rejection. Because the recommended verdict remains CONDITIONAL, no change from the reader's verdict is needed.","tokens_in":11192,"tokens_out":9050,"duration_ms":93909,"concrete_test":"For all four deviation points, take the RWM's generated latent (h^rev_t, z^rev_t), feed it to the frozen actor, and record whether argmax_a pi(a | h^rev_t, z^rev_t) equals the user-desired action a^d_t. Compare this action-match rate with (i) the rate obtained from the actual pre-perturbation state encodings and (ii) a shuffled baseline using RWM outputs from other scenarios. If the RWM states do not elicit the desired action at a rate close to the pre-perturbation baseline and substantially above the shuffled baseline, the generated images are not faithful counterfactuals by the paper's own definition, and the user-study accuracy gain cannot be attributed to their semantic content.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method's defining premise is that an RWM-generated state s^d_t is one in which the agent's policy would choose the user-desired action a^d_t (Section 3, Figure 2). Section 5 provides no evidence for that premise: the RWM is trained only with reconstruction error on time-reversed replay data, and no reconstruction numbers, generated-image examples, human ratings, or comparisons with ground-truth altered states are reported. All four test scenarios are out-of-distribution perturbations (objects removed/moved/obstructed) that the agent never experienced, and Section 8 concedes that RWM outputs are tied to states visited during training. If the generated images are noisy or reflect forward dynamics rather than policy preference, treatment-arm users could still improve on the cause-identification task simply because they are given four extra images (the treatment arm receives video plus eight images, the control only video plus four). Thus the central attribution—that the semantic content of RWM counterfactuals, not just additional visual information, increases understanding—is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an Explainable RL method based on a Reverse World Model (RWM) trained by reversing the temporal order of DreamerV3's replay-buffer transitions. The RWM is intended to generate counterfactual images showing what the environment should have looked like for the agent's policy to prefer a user-supplied desired action. The authors evaluate this method with a Prolific-hosted human study (N=70) in a modified Crafter environment with four failure scenarios. The treatment arm receives the execution video, four true snapshots, and four RWM-generated 'expected' images; the control arm receives only the video and the four true snapshots. The paper reports significantly higher cause-identification accuracy (64.86% vs. 26.52%, Fisher's exact p<0.00001), higher satisfaction, higher trust, and lower cognitive load in the treatment arm, and concludes that showing users what the world should have been like significantly improves their understanding of the agent's policy.","tokens_in":11426,"tokens_out":4354,"duration_ms":44518,"significance":"If the central attribution holds, the paper makes a useful contribution to explainable reinforcement learning for non-expert users: it offers a lightweight way to generate counterfactual explanations from the agent's own world model without a separate post-hoc training phase, and it provides evidence that such explanations can improve users' causal diagnosis of agent failures. The randomized between-subjects design, the use of a fictional domain to control for commonsense assumptions, and the inclusion of a free-response text box are strengths. However, the current evidence does not yet establish the mechanistic claim. The treatment arm differs from control not only in the semantic content of the explanations but also in the sheer number of images shown, and the RWM's outputs are never directly validated for fidelity or policy-alignment. These gaps are load-bearing for the paper's central conclusion.","major_comments":[{"comment":"The central premise—that an RWM-generated image depicts a state in which the agent's policy would choose the desired action—is never directly tested. Section 5 reports only that the RWM is trained with reconstruction error on time-reversed replay data, but the paper gives no reconstruction numbers, no example generated images, no human ratings of image content, and no comparison with ground-truth altered states. Section 8 concedes that RWM outputs are tied to states the agent visited during training, while all four test scenarios involve objects removed, moved, or obstructed—states the agent never experienced. Without direct evidence that the generated images are faithful counterfactuals (rather than blurry or arbitrary reconstructions), the observed treatment-arm improvements could be caused by any extra visual information. Please report reconstruction error, show qualitative examples, and verify behaviorally that the policy, when rolled out in the generated state, selects the desired action with high probability.","section":"§5, §6.1, §8"},{"comment":"The study confounds explanation content with information quantity. Treatment participants receive video plus eight images (four true snapshots and four RWM images), while control participants receive video plus four images. Consequently, the significant differences in cause identification (Table 1) and in satisfaction, trust, and cognitive load (Figure 5) could be driven by having more visual evidence rather than by the semantic content of the RWM counterfactuals. An active control arm—for example, showing four extra images that are not semantically related to the scenario, or RWM images from a different scenario, or randomly selected snapshots—is necessary to isolate the causal role of the explanation. Without such a baseline, hypotheses H1–H4 are not attributable to the proposed explanation mechanism.","section":"§6.1, Table 1, Figure 5"},{"comment":"The conceptual link from time-reversed replay data to policy-preferred counterfactual states is assumed rather than derived. Training a world model on reversed transitions approximates inverse dynamics p(z_{t-1}|z_t, a_{t-1}), but the explanation requires a state s^d_t such that π(s^d_t)=a^d_t. These are different objects: a state that is a plausible predecessor under the learned dynamics need not be one in which the policy prefers the counterfactual action. The authors should either provide a formal argument for when the RWM objective implies the required policy precondition, or empirically verify the property by evaluating the policy on generated states for each scenario. This verification is essential for the method to be usable in practice.","section":"§3, §5"}],"minor_comments":[{"comment":"Section 7.2 and Section 7.3 both state 'The satisfaction survey results are shown in Figure 5b.' The satisfaction subsection should reference Figure 5a, and the trust subsection should reference Figure 5b.","section":"§7.2, §7.3"},{"comment":"The abstract and conclusions state as a hypothesis that explanations can help users control the agent through environment manipulation, but no experiment tests this. Please label this explicitly as untested future work rather than implying it is supported by the reported study.","section":"Abstract, §9"},{"comment":"The sentence defining the RWM output, 'predicts the embedded state pφ(ẑ_{t-1}|h_{t-1}), where h_{t-1} is a function of h_t, z_t and a_{t-1}', is unclear and appears to contain a typo. Please state precisely the conditioning variables used at training and at inference, and define h_{t-1} in the same notation as Equation (1).","section":"§5"},{"comment":"The statement that there is a 1:16 chance of randomly selecting the correct combination assumes independent uniform selection from four objects and four verb phrases. Please clarify whether all 16 combinations were actually plausible and whether any combination could be ruled out by the snapshots alone.","section":"§6.1"},{"comment":"The per-scenario t-tests appear to use one-sided p-values; please specify the test direction and, if multiple comparisons were made, report whether any correction was applied.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the human study has good randomization and significant results, but the missing active control and the lack of any direct validation of RWM output fidelity leave the central causal claim unsupported. I believe this is fixable with a follow-up experiment and additional technical evaluations, so I recommend major revision rather than rejection. I see no concerns about novelty or citation practice; the related work is appropriately placed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the RWM is a neat idea—train a backwards transition model on the same replay data as DreamerV3's forward model, and use it to show users what the world should have looked like for the agent to prefer a user-desired action. That's a real contribution, and prior counterfactual XRL work (Olson, Huber, RACCER) hasn't done it this way.\n\nWhat the paper does well: it positions itself well against the literature, the method is simple and clean, and the human study is generally competently run—randomized, clear hypotheses, Fisher's exact tests with solid p-values. The effect sizes are large: 65% vs 27% correct cause identification.\n\nWhere it's soft: the central claim is that the RWM images causally improve understanding. The study doesn't support that as strongly as it claims. The treatment group receives four additional images on top of video and snapshots. Without an active-control condition that adds unrelated images, you can't attribute the improvement to the semantic content of the RWM outputs. This is not a minor footnote; it's the load-bearing inference of the paper.\n\nSecond, the RWM's counterfactual fidelity is never directly assessed. No reconstruction error, no human rating of image quality, no comparison against ground-truth altered states. The paper concedes in Section 8 that outputs are tied to states visited during training, and all four scenarios are out-of-distribution (objects moved/removed/obstructed). If the images are noisy or reflect forward dynamics, the user-study benefits could come from extra visual cues rather than from 'what should have been.'\n\nThe absence of code and data also limits reproducibility, though that's a common issue at this stage.\n\nIs the central idea salvageable? Yes. Adding a proper active-control baseline and a small fidelity evaluation (even a handful of generated images rated by humans) would make the paper much stronger. The RWM itself is worth publishing even without the human study, but as it stands, the strongest conclusion you can draw is 'RWM images, when shown alongside the execution video, help users identify causes'—not 'the counterfactual content is what matters.'\n\nWho should read this: anyone working on explainable RL or human-agent interaction. It deserves serious peer review, but the reviewer should push for a revised version that closes the confound and gives us some handle on the RWM's faithfulness.\n\nI'd engage with it—but I'd treat the user-study numbers as upper-bound estimates until the active control appears.","headline":"Novel Reverse World Model idea with a promising human study, but unfaithful counterfactuals and a missing active-control arm leave the main attribution unsupported.","tokens_in":11930,"tokens_out":2241,"would_cite":true,"duration_ms":21958,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Showing users what the world should have been like for an agent to take a preferred action significantly improves their ability to identify why the agent failed.","keywords":["Explainable Reinforcement Learning","Counterfactual Explanations","World Models","Reverse World Model","DreamerV3","User Study","Actionable Explanations","Policy Transparency"],"falsifier":"Run the four coffee-task scenarios with a third condition that shows users random or deliberately distorted extra images alongside the real snapshots; if that condition performs as well as the RWM condition, the benefit is not specific to the counterfactual content. Alternatively, compare each RWM-generated image to the ground-truth state after the corresponding environmental change and check whether the changed object (moved, obstructed, or removed) is actually visible in the predicted image.","tokens_in":11009,"feed_emoji":"🤖","tokens_out":7853,"duration_ms":67575,"temperature":0.7,"pith_summary":"The paper introduces a Reverse World Model, a companion to the agent's learned forward model of environment dynamics, that generates images of what the state of the world should have been for a reinforcement learning agent to choose an action the user prefers. The paper claims that such 'world-should-have-been' explanations give non-expert users an actionable understanding of an otherwise opaque policy: users cannot retrain the agent, but they can change its environment once they know what the agent is responding to. In a randomized human study with a kitchen-task agent, participants who received these counterfactual images identified the cause of agent failure more than twice as often as participants who saw only the agent's actual behavior, and also reported higher satisfaction, higher trust, and lower cognitive load. The paper positions this as an approach that works without retraining, without access to the live environment, and without a separate post-hoc explanation model.","feed_headline":"Seeing an agent's expected world state helps users spot its failures","feed_subtitle":"Reverse-world-model images raised failure-cause identification from 26.5% to 64.9%.","key_machinery":"The load-bearing object is the Reverse World Model (RWM), a second transition model trained alongside the forward DreamerV3 world model on the agent's replay buffer with the order of each sampled transition chunk reversed and actions shifted so that the model predicts the prior embedded state from the later observation and the action taken. The forward world model gives counterfactual futures, but the RWM gives the counterfactual past: the state in which the policy would have preferred the user's action over the action actually taken. Decoding the RWM's latent predictions into images produces the 'what the world should have been like' snapshots shown to users. The design deliberately reuses the agent's own training data—no second training phase, no environment access—so the explanation is internal to the agent's model of the world.","core_discovery":"On the paper's own terms, the central discovery is that explaining a sequential decision by inverting it—showing the state that would have made the agent take the user's desired action—significantly improves how well non-experts understand an RL agent's policy. The claim is supported by four hypotheses about accuracy, satisfaction, trust, and cognitive load, and the experiments find support for all four, with the accuracy improvement being the headline result: 64.86% correct cause identification in the treatment group versus 26.52% in the control, a difference the paper reports as significant at p < 0.00001 cumulatively. The mechanism is a Reverse World Model trained on the same DreamerV3 replay data with the temporal order of transitions reversed, which predicts the latent encoding of the prior state given a future state and an action; decoded images of those predicted prior states are what participants see. The paper frames this as an actionable explanation for non-AI experts, because it reveals the environmental feature the agent was responding to, which is something a user could in principle alter.","pith_inferences":["A direct test of the RWM's fidelity—comparing its predicted prior states against ground-truth altered states, or measuring pixel-level reconstruction error—would separate the claim that the content of the counterfactual images is what helps users from the alternative that any extra images would help.","The human result probably depends on the counterfactual being visually legible in a grid environment with discrete objects; the method's benefit in continuous, photorealistic environments remains an open question.","One could imagine a closed loop: if the RWM exposes the feature the agent expects, the same model could rank candidate world alterations by how much they shift the policy's preferred action, turning explanation into a control interface."],"forward_implications":["Users of an RL system could diagnose why an agent deviates from their expectations by looking at a difference image: the real state versus the state the agent's reverse model says it needed.","Because the explanation names the environmental feature driving the policy, it suggests a concrete intervention—move the object, remove the obstruction—that a non-expert can perform without touching the policy.","The same RWM approach could be applied to any model-based RL agent whose world model is trained on collected transitions, not just DreamerV3, as long as the transition data are available in a replay buffer.","Explanation quality becomes tied to world-model quality: the better the agent's learned dynamics, the more faithful the counterfactual states it can show a user."],"supporting_citations":[{"why":"Supplies the DreamerV3 model-based RL framework whose world model the Reverse World Model extends.","marker":"[Hafner et al., 2025]"},{"why":"Introduces world models as learned latent dynamics of an environment, the conceptual basis for counterfactual state prediction.","marker":"[Ha and Schmidhuber, 2018]"},{"why":"Provides the Crafter environment used to build the fictional coffee-making scenarios for the user study.","marker":"[Hafner, 2022]"},{"why":"Supplies the satisfaction and trust survey instruments used for hypotheses H2 and H3.","marker":"[Hoffman et al., 2023]"},{"why":"Supplies the NASA-TLX cognitive load measure used for hypothesis H4.","marker":"[Hart and Staveland, 1988]"},{"why":"Establishes counterfactual 'why P rather than Q' explanations as the natural form of human explanation, motivating the design.","marker":"[Miller, 2019]"},{"why":"The RACCER baseline that generates counterfactual states by searching reachable states, against which the paper contrasts its no-environment-access approach.","marker":"[Gajcin and Dusparic, 2024a]"},{"why":"Prior work generating counterfactual states via generative models; the paper's approach differs by avoiding a separate post-hoc training phase.","marker":"[Olson et al., 2021]"}],"fun_headline_variants":["Reverse world models reveal why RL agents pick actions","Show what world state would change an agent's mind","Inverted world models raise RL explanation accuracy to 65%","Counterfactual world states help users understand agent policies","See the state that would make an agent choose differently"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Reverse World Model's images accurately and intelligibly depict states in which the agent's policy would have chosen the user's desired action—the paper never directly evaluates the faithfulness of those images, so if they are noisy or misleading, the observed user-study gains might come from the extra visual material rather than from the counterfactual content.","fun_headline_variants_meta":{"raw":{"variants":["Reverse world models reveal why RL agents pick actions","Show what world state would change an agent's mind","Inverted world models raise RL explanation accuracy to 65%","Counterfactual world states help users understand agent policies","See the state that would make an agent choose differently"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000287,"raw_usage":{"total_tokens":1679,"prompt_tokens":933,"completion_tokens":746,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":669}},"tokens_in":549,"tokens_out":746,"duration_ms":7878,"temperature":1.0,"reasoning_tokens":669,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:03:44.463493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the four coffee-task scenarios with a third condition that shows users random or deliberately distorted extra images alongside the real snapshots; if that condition performs as well as the RWM condition, the benefit is not specific to the counterfactual content. Alternatively, compare each RWM-generated image to the ground-truth state after the corresponding environmental change and check whether the changed object (moved, obstructed, or removed) is actually visible in the predicted image.","supporting_citations":[{"cited_title":"Mastering diverse control tasks through world models","cited_arxiv_id":null,"evidence_quote":"Supplies the DreamerV3 model-based RL framework whose world model the Reverse World Model extends."},{"cited_title":"World models","cited_arxiv_id":null,"evidence_quote":"Introduces world models as learned latent dynamics of an environment, the conceptual basis for counterfactual state prediction."},{"cited_title":"Benchmarking the spectrum of agent capabilities,","cited_arxiv_id":null,"evidence_quote":"Provides the Crafter environment used to build the fictional coffee-making scenarios for the user study."},{"cited_title":"Hoffman, Shane T","cited_arxiv_id":null,"evidence_quote":"Supplies the satisfaction and trust survey instruments used for hypotheses H2 and H3."},{"cited_title":"Development of nasa-tlx (task load index): Re- sults of empirical and theoretical research","cited_arxiv_id":null,"evidence_quote":"Supplies the NASA-TLX cognitive load measure used for hypothesis H4."},{"cited_title":"Explanation in artificial intelli- gence: Insights from the social sciences","cited_arxiv_id":null,"evidence_quote":"Establishes counterfactual 'why P rather than Q' explanations as the natural form of human explanation, motivating the design."},{"cited_title":"Olson, Roli Khanna, Lawrence Neal, Fuxin Li, and Weng-Keen Wong","cited_arxiv_id":null,"evidence_quote":"Prior work generating counterfactual states via generative models; the paper's approach differs by avoiding a separate post-hoc training phase."}],"review_version":1}