{"id":"953ad7bd-a0a8-49f1-ae86-66db6bdd7eb0","arxiv_id":"2606.16933","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Distributional shift in RL is classified by which POMDP generative component changes (internal agent vs external environment) and by whether the time boundary is explicit, implicit, or hybrid.","lead":"This paper builds a taxonomy that locates distributional shift in RL inside the agent-environment generative process (state, observation, policy, transition, reward) and the train/test or online time boundary. It gives researchers a shared language for diagnosing why policies fail under ID/OOD or non-stationarity and for measuring recovery.","discovery_kind":"unification","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged empirical thinness.","rationale":"The paper’s strongest claim is definitional and structural, not empirical. Definitions 1–3 and the internal/external + boundary taxonomy follow directly from the chain-rule factorization of a POMDP interaction step and correctly recover classical dataset-shift categories when the RL process is projected onto (X,Y). No equation introduces an unstated boundedness, independence, or stationarity assumption that would break under ordinary RL regimes. The multi-agent extension (Def. 4) is likewise a straightforward marginalization. The reader’s weakest-assumption note already isolates the only real limitation—single-run gridworld illustrations that cannot rigorously demonstrate diagnostic distinctness. Because that limitation is already reflected in the CONDITIONAL verdict and does not undermine the formal unification claim, no further adjustment is warranted. Public multi-seed statistics and code would convert the verdict to ACCEPT; their absence does not convert it to REJECT.","tokens_in":29314,"tokens_out":553,"duration_ms":5461,"concrete_test":"Re-run the five explicit-boundary and five implicit-boundary conditions of §5 with ≥5 independent seeds, report mean±std of the five adaptation metrics of §6 (Δ_shift, Δ_drop, T_rec, Γ_post, A_adapt), and check whether the rank-order of degradation/recovery profiles across the five causal-origin factors remains stable; if the ordering collapses under multi-seed variance, the “diagnosable patterns” claim weakens while the taxonomy definitions themselves remain intact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Defs. 1–3 + §4 taxonomy) is a definitional re-factorization of the standard POMDP generative process (Eqs. 5–10) that correctly unifies ID/OOD and non-stationarity under explicit/implicit/hybrid boundaries. It is internally consistent with classical MDP/POMDP theory and does not rest on a hidden mathematical assumption that fails in a realistic regime. The only soft spot is the one the reader already named: the claim that the five-factor taxonomy yields “distinct and diagnosable degradation patterns” is supported only by single-run DQN curves on a hand-designed 30×30 gridworld (§5.5, “each experimental result is computed from a single independent run”). That is weak evidence of diagnostic power, not a flaw in the factorization itself. No stronger load-bearing concern (e.g., incompleteness of the five factors, inconsistency of the internal/external split, or circularity in the boundary definitions) is present.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes a causal-origin taxonomy of distributional shift in RL by transferring the classical supervised dataset-shift factorization to the POMDP generative process. It decomposes a single interaction step as s ~ p(s), o ~ p(o|s), a ~ π(a|o), s' ~ p(s'|s,a), r ~ p(r|s,a,s') (Eqs. 5–10), then defines distributional shift as any pre/post mismatch in the joint generative model that originates in at least one of these five factors (Definitions 1–3). Shifts are further classified as internal (agent-driven: p(o|s), π(a|o)) or external (environment-driven: p(s), p(s'|s,a), p(r|s,a,s')), and as explicit, implicit, or hybrid according to the shifted-time boundary. The taxonomy is extended to decentralized MARL (Definition 4), illustrated with single-run DQN experiments on a 30×30 gridworld under five hand-designed perturbations (§5), and accompanied by five adaptation metrics (Δ_shift, Δ_drop, T_rec, Γ_post, A_adapt) plus a post-hoc mapping of existing methods (Table 3).","tokens_in":29573,"tokens_out":746,"duration_ms":7672,"significance":"If accepted as a conceptual contribution, the paper supplies a clean, shared vocabulary that unifies ID/OOD generalization and non-stationarity under a single generative factorization. The internal/external split and the explicit/implicit/hybrid boundary distinction are useful diagnostic tools for method design and for interpreting why robustness is shift-type dependent (as suggested by Robust-Gymnasium and RL-ViGen). The evaluation metrics in §6 are a concrete, reusable addition that go beyond aggregate return. The formal core (Eqs. 5–10, Definitions 1–4) is definitional and consistent with standard POMDP theory; the main value is organizational rather than predictive. The empirical section is intentionally minimal and does not claim algorithmic novelty.","major_comments":[{"comment":"§5.5 and the claim that the taxonomy yields “distinct and diagnosable degradation patterns” rest on single independent runs with no error bars, no statistical tests, and a deliberately minimal 30×30 gridworld. The authors themselves note that each result is from one run. This is too thin to support a diagnostic claim; either expand to multi-seed statistics (or a second environment) or rephrase the empirical section as a purely illustrative demonstration of the definitions rather than evidence of diagnostic power.","section":null},{"comment":"The sufficiency claim for the five-factor factorization plus a single shifted-time boundary (Definitions 1–3, §4) is asserted rather than stress-tested. Mixed or higher-order cases (e.g., a visual change that simultaneously alters p(o|s) and p(r|s,a,s'), or continuous multi-agent co-adaptation that blurs the internal/external boundary) are only briefly acknowledged in the Discussion. A short subsection clarifying the taxonomy’s scope and known incompletenesses would strengthen the central claim without requiring new theory.","section":null}],"minor_comments":[],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a solid conceptual paper. The real contribution is a clean re-factorization of the standard POMDP generative process into internal (agent) vs external (environment) shifts, plus explicit/implicit/hybrid time boundaries, that finally puts ID/OOD generalization and non-stationarity under one roof. Definitions 1–3 and the five-factor split (p(s), p(o|s), π(a|o), p(s′|s,a), p(r|s,a,s′)) are definitional and consistent with classical MDP theory; they do not invent new math, but they organize the literature better than the trajectory-ratio or visitation-mismatch definitions that currently float around.\n\nWhat works: the mapping of prior methods (domain randomization, meta-RL, offline-to-online, non-stationary RL) onto those generative components is useful diagnostic language. The five recovery metrics (immediate impact, worst-case drop, recovery time, quality, efficiency) are practical and fill a real gap left by plain return/success curves. The MARL extension treating other agents’ policy updates as external is clean. Citation pattern is thorough and fair; no circularity.\n\nSoft spots are exactly where the reader and stress-test say: the empirical section is deliberately minimal—single-run DQN on a 30×30 gridworld with hand-designed perturbations, no error bars, no multi-seed stats, code not yet public. That is weak support for the claim of “distinct and diagnosable degradation patterns,” not a flaw in the factorization itself. Free parameters (window fractions, η, quantization bits, FOV roll) are user-chosen, as expected for a framework paper. Nothing load-bearing is broken.\n\nThis is for people who design robustness benchmarks, diagnose failure modes, or write surveys. It will not change algorithms tomorrow, but it gives the subfield a shared vocabulary. I would bring it to reading group, cite the taxonomy when I need to locate a shift, and send it to peer review. Accept with the usual request for multi-seed results and public code; the conceptual core already stands.","headline":"Clean conceptual taxonomy that unifies ID/OOD and non-stationarity via POMDP factors; useful framing, thin single-run evidence.","tokens_in":30211,"tokens_out":530,"would_cite":true,"duration_ms":5223,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Distributional shift in RL is a change in one of five generative factors of the agent-environment loop, not a vague train-test mismatch.","keywords":["distributional shift","reinforcement learning","POMDP","causal-origin taxonomy","ID/OOD generalization","non-stationarity","adaptation metrics"],"falsifier":"Find a realistic RL failure mode whose performance collapse cannot be attributed to any change among p(s), p(o|s), π(a|o), p(s′|s,a), or p(r|s,a,s′), or show that two different single-factor interventions produce statistically indistinguishable degradation and recovery profiles under the paper’s own metrics.","tokens_in":30181,"feed_emoji":"🧠","tokens_out":793,"duration_ms":7142,"temperature":0.7,"pith_summary":"Reinforcement learning agents fail when the world or their own behavior stops matching the conditions under which they were trained. The paper argues that those failures are not one generic problem but mismatches that can be localized to specific pieces of the interaction: how states are distributed, how observations are generated, how the policy chooses actions, how the environment transitions, or how rewards are assigned. By rewriting classical supervised dataset shift in terms of this five-factor generative process, the authors show that the familiar labels “in-distribution / out-of-distribution” and “stationary / non-stationary” are simply different ways of describing when and where that process changes. They further distinguish agent-driven (internal) from environment-driven (external) shifts and explicit, implicit, or hybrid time boundaries. The payoff is a shared vocabulary for diagnosing robustness failures and for measuring not only whether performance drops but how hard the drop is and how quickly an agent recovers.","feed_headline":"RL failure is a change in one of five interaction factors","feed_subtitle":"A taxonomy localizes train-test and non-stationary collapse to state, observation, policy, dynamics, or reward","key_machinery":"The five-factor POMDP generative factorization p(s,o,a,s′,r)=p(s)p(o|s)π(a|o)p(s′|s,a)p(r|s,a,s′), split into external (environment) and internal (agent) factors and crossed with a shifted-time boundary that may be explicit, implicit, or hybrid.","core_discovery":"Distributional shift in reinforcement learning is fully characterized as a mismatch between the pre-boundary and post-boundary joint generative models p(o,s,a,s′,r). That mismatch must arise from a change in at least one of five causal-origin factors—p(s), p(o|s), π(a|o), p(s′|s,a), or p(r|s,a,s′)—and the same factorization unifies classical ID/OOD generalization with non-stationarity under explicit, implicit, or hybrid shifted-time boundaries.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["RL shift is a change in one of five causal process factors","Taxonomy maps RL failure to state, obs, policy, dynamics or reward","Five generative factors unify ID/OOD and non-stationary RL shifts","Distributional collapse traces to p(s), p(o|s), policy, transition or reward","Shifted-time boundary localizes RL mismatch to five interaction origins"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That every practically important distributional shift can be cleanly located in one of those five factors plus a single time boundary, and that single-run gridworld experiments are enough to show the resulting degradation patterns are distinct and diagnosable.","fun_headline_variants_meta":{"raw":{"variants":["RL shift is a change in one of five causal process factors","Taxonomy maps RL failure to state, obs, policy, dynamics or reward","Five generative factors unify ID/OOD and non-stationary RL shifts","Distributional collapse traces to p(s), p(o|s), policy, transition or reward","Shifted-time boundary localizes RL mismatch to five interaction origins"]},"model":"grok-4.5","effort":"low","cost_usd":0.005398,"raw_usage":{"total_tokens":1551,"prompt_tokens":881,"num_sources_used":0,"completion_tokens":102,"cost_in_usd_ticks":53980000,"prompt_tokens_details":{"text_tokens":881,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":568,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":881,"tokens_out":102,"duration_ms":4654,"temperature":1.0,"reasoning_tokens":568,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T13:41:45.593238+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Find a realistic RL failure mode whose performance collapse cannot be attributed to any change among p(s), p(o|s), π(a|o), p(s′|s,a), or p(r|s,a,s′), or show that two different single-factor interventions produce statistically indistinguishable degradation and recovery profiles under the paper’s own metrics.","supporting_citations":[],"review_version":1}