{"id":"8164d2a0-7f86-4643-9b2e-13ce23c4c548","arxiv_id":"2509.10423","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"On a single panda-gym Reach run, state-action mutual information increases from 0.84 to 2.83 bits during training, while the paper claims that observation noise collapses information channels and action noise only mildly reduces them, enabling fault localization.","lead":"Using a discretized robotic reaching task, the paper reports that mutual information between states and actions grows during PPO training, and claims distinct mutual-information signatures can distinguish sensor faults from actuator faults. A generalist reader might care because such architecture-free signals could become deployment-time health monitors for reinforcement learning agents.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The observation-noise fingerprint is not established: Section V.B reports only a 0.058-bit MI(S;A) drop while the Abstract promises a deep collapse, and the 2,000-sample plug-in estimator over a 343,000-cell joint space has unquantified finite-sample bias.","rationale":"The strongest claim is the diagnostic fingerprint. For it to hold, the differences between the observation-noise and action-noise MI traces must reflect true changes in the state-action-observation statistics, not finite-sample estimation error. The paper's estimator is uncalibrated in exactly the regime where it matters: n=2,000 over a nominal 343,000-cell joint support. Because plug-in MI is biased upward, and because the perturbation changes the support size (unique states and actions), the estimated differences can be driven by bias differentials rather than by true MI changes. The internal contradiction between the Abstract's 'pronounced drops in state-action coupling' and Section V.B's 0.058-bit MI(S;A) decrease is corroborating evidence that the quoted numbers are not stable enough to build on. I do not claim the results are false; I claim the paper, as written, does not supply the statistical controls needed to distinguish the proposed fingerprints from artifacts. The reader's weakest-assumption analysis identified the same measurement-validity problem, and I agree with it. A revision with bias correction, multi-seed confidence intervals, and a no-perturbation null baseline could settle the question, but as it stands the central diagnostic claim is under-supported and the reader's REJECT verdict remains appropriate.","tokens_in":12325,"tokens_out":4650,"duration_ms":40368,"concrete_test":"Re-run the frozen-policy perturbation experiments for all four seeds, and for every 2,000-step window compute bias-corrected MI (e.g., Miller-Madow or NSB) plus 95% bootstrap confidence intervals, and also record the same signatures on an unperturbed healthy run using arbitrary 2,000-step blocks as a null baseline. If the corrected observation-noise MI(S;A) drop is not statistically distinguishable from the healthy baseline drift, or if corrected MI(A;S') drops under action noise overlap the observation-noise drops, the differential-diagnosis claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central diagnostic claim (Abstract; §IV.E) is that a deployer can distinguish sensor faults from actuator faults by watching MI(S;A), MI(A;S'), and MI(S;S'). Every value behind that claim is a plug-in estimate computed from 2,000-step sliding windows (Section III.D) over 1,000 discretized states × 343 actions = 343,000 possible (s,a) cells. With n=2,000 samples, the plug-in mutual-information estimator has a large upward bias (order (|supp|-1)/(2n ln 2) bits) with no Miller-Madow or NSB correction and no bootstrap/CI. The support itself changes across conditions: reported unique states rise from ~27 to ~38 under observation noise while unique actions collapse, so the bias differs between the very fingerprints being compared. Baseline variability is never shown, and only one representative seed is reported (Section III.C, F). The reported numbers are also internally inconsistent: Section V.B lists MI(S;A) dropping only 0.058 bits under observation noise, while the Abstract and Figure 3 describe a 'deep collapse' of state-action coupling. That 0.058-bit change is smaller than the action-noise-induced 0.177-bit drop in MI(A;S') used as the contrasting signature, so the claimed separation is not established by the data as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an information-theoretic monitoring framework for reinforcement learning agents, tracking plug-in estimates of H(S), H(A), MI(S;A), MI(A;S'), MI(S;S'), and MI(S,A;S') over a discretized panda-gym Reach task. During training, it reports that MI(S;A) grows from 0.84 to 2.83 bits while MI(S,A;S') follows an inverted U-curve. In deployment, it claims that observation noise (sensor faults) causes broad collapses across all information channels, whereas action noise (actuator faults) selectively reduces action-outcome predictability while preserving state-action coupling, thereby enabling differential fault localization. The paper also argues that these information signatures are architecture-agnostic quality indicators and proposes future adaptive RL systems based on them.","tokens_in":12592,"tokens_out":4843,"duration_ms":41520,"significance":"If the empirical claims held, the proposed framework would be a practically valuable, architecture-agnostic diagnostic tool for deployed RL systems, and the MI(S;S') noise-amplitude scaling would be a falsifiable, deployable prediction. The experimental design is controlled and simple, and the metrics are computed directly from observable counts rather than from fitted model parameters. However, the quantitative basis for the central diagnostic claim is not currently established: the reported numbers contradict one another, the estimator is used without bias correction or uncertainty quantification, and the paper itself concedes that formal statistical analysis across runs remains future work.","major_comments":[{"comment":"The headline claim of a 'deep collapse' of state-action coupling under observation noise is contradicted by the numbers in Section V.B. There, observation noise is reported to decrease MI(S;A) by only 0.058 bits, which is smaller than the 0.177-bit drop in MI(A;S') under action noise, the very contrasting signature the paper relies on. Fig. 3 describes a 'sharper MI collapse (below 1 bit)' for sensor noise, but the reported 0.058-bit change is not a deep collapse. As presented, the data do not establish the claimed separation between sensor-fault and actuator-fault fingerprints.","section":"Abstract, V.B, Fig. 3"},{"comment":"The ratio of MI(S;A) to state entropy is reported inconsistently. Section I.B gives growth from 29.8% to 37.3%, while Section V.D gives growth from 24.7% to 63.8%. Table 1's values imply 0.836/3.385 = 24.7% and 2.83/4.444 = 63.7%, so the Section I.B numbers are inconsistent with the paper's own table. This is not a presentation nitpick: the ratio trajectory is used to argue that the agent develops selective attention despite growing state entropy.","section":"I.B, V.D, Table 1"},{"comment":"The reported peak location of MI(S,A;S') is inconsistent across sections. Section IV.A states the metric 'peaks at 3.46 bits around step 80 k'; Section IV.B says the early phase (0-25,000 steps) 'coincides with MI(S,A;S') peaking at 3.464 bits'; Section V.A says it peaks 'around 20 k steps'. These three placements imply different learning-phase interpretations, so the inverted-U narrative is not anchored to a stable empirical result.","section":"IV.A, IV.B, V.A"},{"comment":"The diagnostic conclusions rest on plug-in mutual information estimates from 2,000-step sliding windows over a joint state-action space of 1,000 x 343 = 343,000 cells. With 2,000 samples, the plug-in estimator has a substantial upward bias; no Miller-Madow or NSB bias correction, bootstrap confidence intervals, or sample-size analysis is provided. Moreover, the support size itself changes across conditions (unique actions collapse from ~150 to ~61 while unique states rise from ~27 to ~38 under observation noise), so the bias differs between the very fingerprints being compared. Section V.F concedes that 'formal statistical analysis across runs remains for future work,' but such analysis is precisely what is needed to support the differential fault-localization claim. Without it, differences of 0.058 to 0.177 bits cannot be distinguished from estimation noise.","section":"III.D, III.C, V.F"}],"minor_comments":[{"comment":"The citation to Sedlmeier et al. appears as [30] in Section II.C, but reference [30] is the PPO paper by Schulman et al.; the Sedlmeier paper is [31]. Several other citations in the reference list are also malformed or duplicated, e.g., reference [13] begins 'Genewein, T., Genewein, Tim, ...'.","section":"II.C, References"},{"comment":"The environment description is inconsistent. Section III.B describes the state as a continuous 3D distance-to-target vector and actions as Cartesian end-effector displacement commands, while Section IV.A describes the agent navigating between adjacent cells in a discrete grid with 'movement directions between adjacent cells.' These are different tasks, and the manuscript should describe one consistent experimental setup.","section":"III.B, IV.A"},{"comment":"There are broken cross-references: the Table 1 caption refers to 'Figure X,' and Section IV.E refers to 'Table 9' and 'Figure 8' when no such numbered objects exist. The H(A) relative change in Table 1 is also arithmetically wrong: 0.03 / 8.366 is approximately 0.36%, not 0.40%.","section":"Table 1, IV.E"}],"recommendation":"reject","confidential_remarks":"The manuscript's central empirical claim is not supported by its own reported numbers, and the measurement methodology lacks the uncertainty quantification needed to compare small MI differences. These are load-bearing problems rather than local presentation issues, so I recommend rejection even though the underlying idea is potentially useful. If the authors can provide corrected numbers, proper bias-corrected estimators with confidence intervals, and multi-seed statistical analysis, a substantially revised submission could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The differential-diagnosis idea is genuinely new, and it's the reason to read the paper. The claim that observation noise and action noise leave different fingerprints across MI(S;A), MI(A;S'), and MI(S;S') is a legitimate extension of prior MI-in-RL work, and I don't know of another paper that tries to localize sensor vs actuator faults this way. The learning-curve observations are not new, but the authors cite the relevant MI literature and frame the deployment problem well.\n\nThe trouble is the numbers do not support the abstract. The abstract promises 'pronounced drops in state-action coupling' under observation noise; Section V.B reports MI(S;A) dropping 0.058 bits, which is a tenth of the MI(A;S') drop and barely larger than the action-noise case's +0.003. That is not a deep collapse. The MI(S,A;S') peak is placed at 20k, 25k, and 80k steps in different sections, and the MI(S;A)/H(S) ratio is given as 29.8–37.3% in the intro but 24.7–63.8% in Table 1 and Section V.D. These are not cosmetic typos; the diagnostic fingerprint rests on exactly these values.\n\nThe estimation concern is real too. Plug-in MI from 2,000-step windows over a 343,000-cell joint space is strongly biased upward, the support changes across conditions, and there is no bias correction, no confidence interval, and only one seed reported. A baseline like CUSUM or a simple KL test is never run, so we don't know whether a cheap alternative already gives the same diagnosis.\n\nOn the plus side, the metrics are computed directly from empirical counts, so there's no circularity. The limitations section is honest about the single-run issue, even if the 'we verified 4 runs' line isn't backed with data. The paper is clearly written and the question matters.\n\nWho is this for? Someone deciding whether to write a monitoring tool for deployed RL, or a researcher working on drift detection. It deserves a serious referee because the claim is important and the flaws, while load-bearing, are identifiable and fixable. But as written, the central conclusion is not established. I'd send it out, then tell the authors to fix the contradictions, compute bias-corrected estimates with multi-seed intervals, and compare against a baseline. The idea survives; this version doesn't.","headline":"The differential-diagnosis idea is genuinely new, but the paper's own numbers contradict its central claim, so as written it cannot support the conclusion.","tokens_in":13163,"tokens_out":3374,"would_cite":false,"duration_ms":29027,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mutual-information channel traces distinguish sensor faults from actuator faults in a trained RL agent.","keywords":["mutual information","reinforcement learning","fault diagnosis","policy analysis","distribution shift","information signature","robotic control","policy coherence"],"falsifier":"Recompute the same set of metrics on the same frozen policy using either held-out transitions or a bias-corrected estimator; if the observation-noise run no longer shows a deeper $MI(S;S')$ collapse than the action-noise run, or if the sign of the 0.058-bit $MI(S;A)$ change reverses, the claimed sensor-versus-actuator fingerprint is an estimator artifact rather than a property of the policy.","tokens_in":12071,"feed_emoji":"🤖","tokens_out":11883,"duration_ms":87321,"temperature":0.7,"pith_summary":"This paper claims that a small set of Shannon mutual-information quantities, computed only from observed states, actions, and next states, reveals both how a reinforcement-learning (RL) agent learns and what kind of fault it is suffering once deployed. In a simulated robotic-arm reaching task, the authors find that successful training is marked by a 238% rise in $MI(S;A)$ even as state entropy grows, together with an inverted-U trajectory in $MI(S,A;S')$ that peaks during early exploration and falls during exploitation. When noise is injected after training, action-space noise produces a modest parallel drop in all MI channels while state-action coupling stays intact, whereas observation-space noise produces a deep collapse across all channels, led by a 1.83-bit drop in $MI(S;S')$. If these fingerprints hold, a deployer could watch the MI traces and localize a failure to sensors or actuators without any architectural change or reward-based signal. The evidence base is one discretized task, four training seeds, and controlled frozen-policy perturbation runs.","feed_headline":"Mutual-information traces separate sensor from actuator faults","feed_subtitle":"A trained arm's MI channels collapse differently for sensor vs actuator faults, enabling fault localization without extra hardware","key_machinery":"The load-bearing machinery is a bank of Shannon information estimates computed by counting observed transitions (plug-in estimates): state entropy $H(S)$, action entropy $H(A)$, next-state entropy $H(S')$, mutual information $MI(S;A)$, $MI(A;S')$, $MI(S,A;S')$, and $MI(S;S')$, all on a discretized 10-by-10-by-10 state grid and 7-by-7-by-7 action grid. Training curves accumulate all data in 5,000-step windows to display maturation; deployment monitoring uses 2,000-step sliding windows to make abrupt shifts visible. The diagnostic argument works by comparing which channels drop when a single controlled noise source is added to the frozen policy.","core_discovery":"On the paper's own terms, the discovery is that policy coherence has a measurable information signature. Across 200,000 training steps in a discretized reaching task, $MI(S;A)$ climbs from 0.84 to 2.83 bits while state entropy $H(S)$ rises from 3.39 to 4.44 bits, and $MI(S,A;S')$ forms an inverted U that peaks near 3.46 bits early in training and settles near 1.93 bits, matching the shift from exploration to exploitation. After training, with the policy frozen, injecting Gaussian noise into action outputs leaves state-action coupling nearly unchanged ($MI(S;A)$ +0.003 bits) while reducing action-to-next-state predictability ($MI(A;S')$ -0.177 bits), and injecting noise into observations collapses the whole information profile, with $MI(S;S')$ falling by 1.83 bits and $MI(A;S')$ by 0.668 bits. The paper takes these distinct channel-wise drops to be differential fingerprints that localize a fault to sensor or actuator, with the $MI(S;S')$ nadir depth scaling with noise variance for severity estimation.","pith_inferences":["Editorial inference: the channel decomposition suggests a natural test the paper does not run: during a genuine environmental shift, only the state-related channels should move while $MI(A;S')$ stays intact.","Editorial inference: the 0.058-bit drop in $MI(S;A)$ under observation noise is likely near the noise floor of a plug-in estimator on 2,000 samples, so the robust diagnostic signal is the much larger $MI(S;S')$ and $MI(A;S')$ collapses.","Editorial inference: if the inverted-U in $MI(S,A;S')$ transfers to other tasks, it could serve as a training-progress monitor that flags when an agent stops building a world model and starts exploiting, enabling automated curriculum decisions."],"forward_implications":["A monitoring system that already sees the agent's state-action-next-state log can detect the onset of sensor or actuator degradation without adding hardware or modifying the policy.","The depth of the $MI(S;S')$ drop gives a graded severity estimate for actuator noise, since the paper reports a larger nadir at noise variance 1.0 than at 0.1.","Because the metrics are defined on observable distributions rather than on network internals, the same signatures could be applied to any policy implementation, not just the PPO agent studied here.","The training-phase rise in $MI(S;A)$ relative to state entropy offers a candidate early-warning indicator of policy misalignment that could precede visible reward decline."],"supporting_citations":[{"why":"Supplies the simulated robotic-arm Reach environment on which all training and perturbation data are collected.","marker":"[12]"},{"why":"Supplies the PPO algorithm whose trained policy is analyzed and later subjected to noise injection.","marker":"[30]"},{"why":"Provides the Shannon entropy and mutual-information definitions the paper computes from empirical counts.","marker":"[7]"},{"why":"Serves as a standard drift-detection baseline (CUSUM) that the proposed diagnostic is compared against.","marker":"[28]"},{"why":"Serves as an adaptive-windowing drift-detection baseline that only signals change without localizing its source.","marker":"[6]"},{"why":"Serves as a kernel two-sample test baseline for distribution shift, contrasted with the channel-wise MI diagnosis.","marker":"[15]"},{"why":"Provides the information-bottleneck framing used to interpret rising MI(S;A) as selective compression.","marker":"[37]"}],"fun_headline_variants":["MI fingerprints isolate sensor from actuator faults","Learning's MI curve separates sensor from actuator decay","Mutual information channels tag sensor and actuator faults","Fault localization via mutual-information signatures","Info-theoretic fingerprints reveal which RL part breaks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The diagnostic fingerprints rest on the assumption that mutual-information values computed from 2,000 transitions are accurate enough to compare channels in a space with 343,000 possible state-action pairs; the paper provides no bias correction, confidence intervals, or sample-size analysis.","fun_headline_variants_meta":{"raw":{"variants":["MI fingerprints isolate sensor from actuator faults","Learning's MI curve separates sensor from actuator decay","Mutual information channels tag sensor and actuator faults","Fault localization via mutual-information signatures","Info-theoretic fingerprints reveal which RL part breaks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000543,"raw_usage":{"total_tokens":2654,"prompt_tokens":1055,"completion_tokens":1599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":1531}},"tokens_in":671,"tokens_out":1599,"duration_ms":9570,"temperature":1.0,"reasoning_tokens":1531,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:54:37.559646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the same set of metrics on the same frozen policy using either held-out transitions or a bias-corrected estimator; if the observation-noise run no longer shows a deeper $MI(S;S')$ collapse than the action-noise run, or if the sign of the 0.058-bit $MI(S;A)$ change reverses, the claimed sensor-versus-actuator fingerprint is an estimator artifact rather than a property of the policy.","supporting_citations":[],"review_version":2}