{"id":"8744154d-af86-4cb1-b208-aded670320cb","arxiv_id":"2504.14645","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"REACT evolves initial states under a joint diversity-and-certainty fitness to produce demonstrations that score higher on a fidelity proxy in gridworlds and early continuous control, but fidelity-based optimization wins for mature policies.","lead":"REACT is an evolutionary tool that varies the starting position of an RL agent and keeps the most diverse, uncertain, and behaviorally different rollouts to explain what the policy does. In gridworlds and an early-stage robot arm task it beats random starting points on the paper's fidelity metric, but for well-trained continuous-control policies it does not.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on an unvalidated return-dispersion metric, and by that metric REACT fails to beat Random in two of three FetchReach settings, so 'improves interpretability in continuous environments' is unsupported.","rationale":"The Reader correctly identified the ad hoc fidelity metric as the weakest link, and I agree that Eq. 7 is not validated as an interpretability measure. I also agree that the paper has genuine empirical content: the code is shared, the ablations and hyperparameter study are useful, and the gridworld Random/REACT comparisons are internally consistent enough to be reproducible. However, the concern is more specific and slightly stronger than the Reader's statement. The paper's own Table 1 shows REACT below Random in two of three FetchReach stages, so the continuous part of the headline claim is not merely under-validated; it is contradicted by the primary reported metric. The fallback to qualitative trajectory spread in Section 6.5 is reasonable as an exploratory observation but cannot carry the quantitative claim of 'significantly improves interpretability.' The discrete results survive this critique better, but they still depend on S being a meaningful proxy, which is untested. A CONDITIONAL verdict remains appropriate: the method is plausible and the discrete experimental pipeline is useful, but the paper should either validate S against human judgment or a recognized explanation-quality measure, report statistical tests, and explicitly restrict the continuous claim to early-stage policies.","tokens_in":949,"tokens_out":1209,"duration_ms":65088,"concrete_test":"Run a human-participant or externally validated interpretation study on the existing demonstration sets. For each environment and training stage, present the REACT, Random, and Fidelity demonstration sets and ask evaluators fixed factual questions about the policy, such as 'From which starting regions does the policy fail?', 'Which states make the agent hesitate?', or 'What strategy does it use near holes?' Measure answer accuracy, coverage of true edge cases, or time-to-answer. Additionally, compute S for every set against a fixed reference distribution of returns rather than the set's own mean. If REACT shows no significant advantage over Random in the human study, or if the fixed-reference S no longer ranks REACT first, then the claim that S tracks interpretability is unsupported and the abstract's continuous claim must be narrowed or removed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The quantitative backbone is the fidelity statistic S in Eq. 7, but S is never validated as a measure of interpretability. As defined, S is a length-weighted dispersion of cumulative rewards around the mean absolute reward; it reports how spread out the returns are, not whether the demonstrations explain the policy's decisions, cover the state-action space, or improve human understanding. Section 6.1 asserts that higher S means a more representative summary, but no evidence links S to human explanation quality. This matters because Table 1 undermines the continuous half of the headline claim. In FetchReach, REACT's fidelity IQM exceeds Random only for SAC-50k (0.842 vs 0.106). At SAC-100k, REACT scores 1.373 vs Random 1.681; at SAC-150k, REACT scores 0.137 vs Random 0.173. By the paper's primary metric, REACT is worse than random initialization in two of three continuous settings, and Section 6.5 falls back on qualitative trajectory spread to argue interpretability. The discrete results are also fragile: the fidelity-optimized baseline scores below Random in both gridworlds (0.255 vs 0.355; 0.291 vs 1.486), showing that S can be driven down by concentrating on one reward mode, which a pure dispersion statistic would do. Thus the headline claim rests on an unvalidated proxy that, by the paper's own numbers, fails to support the continuous part of the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces REACT, an evolutionary framework that perturbs a policy's initial states and selects a small set of demonstration trajectories intended to expose diverse, edge-case, or uncertain behaviors of a fixed reinforcement learning policy. Trajectories are scored by a joint surrogate fitness F (Eq. 5) combining global diversity, local diversity, and policy certainty, and the resulting demonstration set is evaluated by a reward-spread fidelity statistic S (Eq. 7). Experiments compare REACT against random initial states, the training initial state, a fidelity-optimized baseline, and ablated fitness components in FlatGrid11, HoleyGrid11, and FetchReach at three SAC training stages. The authors report higher fidelity IQMs for REACT over random and fidelity baselines in the two gridworlds and in early-stage FetchReach, and they argue for qualitative interpretability gains in continuous control through trajectory spread.","tokens_in":19452,"tokens_out":4056,"duration_ms":40436,"significance":"If the central claim held, REACT would be a useful model-agnostic tool for post-hoc policy analysis: it requires no gradient access, produces a compact set of user-reviewable demonstrations, and the authors release code and videos. A genuine strength is that the optimized joint fitness F (Eq. 5) is not identical to the evaluation metric S (Eq. 7), so the discrete results are not forced by construction, and the fitness function contains no fitted constants. The hyperparameter study and ablations also provide useful practical guidance. However, the paper's significance is currently capped by three issues: the evaluation metric S is asserted, not validated, as a measure of interpretability; the continuous results in Table 1 contradict the abstract's claim of improvement in both discrete and continuous environments; and the word 'significantly' is used without any significance tests. These problems are fixable by reframing claims and adding validation, but as written the headline contribution is stronger than the evidence.","major_comments":[{"comment":"The fidelity metric S is the quantitative backbone for the interpretability claim, but the paper never validates S as a measure of interpretability. Section 6.1 states that 'a higher fidelity score indicates that the selected demonstrations provide a more representative summary' and calls this 'intuitive,' yet no evidence links S to human understanding, explanation quality, or coverage of decision-relevant state-action space. Since the abstract's headline claim is that REACT 'significantly improves interpretability,' this unvalidated proxy is load-bearing and needs either external validation (e.g., a human study or correlation with an established interpretability measure) or a substantial weakening of the claim to 'diversity of generated trajectories.'","section":"§6.1, Eq. (7)"},{"comment":"The abstract states that optimizing trajectory selection 'significantly improves interpretability of RL policies in both discrete and continuous environments,' but Table 1 shows that in FetchReach at SAC-100k and SAC-150k, REACT's final fidelity IQM (1.373 and 0.137) is below the Random baseline (1.681 and 0.173). Section 6.5 acknowledges this and falls back to qualitative trajectory spread, but no quantitative measure of that spread is reported, so the continuous half of the headline claim is unsupported by the paper's primary metric. The abstract and the 'Discussion and Summary' section should be revised to state that REACT helps in early-stage continuous policies and that fidelity-based optimization is more effective for mature policies.","section":"Table 1 and Abstract"},{"comment":"The fidelity-optimized baseline performs worse than Random under the fidelity metric it directly optimizes: in FlatGrid11 it achieves 0.255 versus Random's 0.355, and in HoleyGrid11 0.291 versus 1.486. If the reported S is the same objective being optimized, evolutionary search should not monotonically decrease it below a random initialization unless the optimization is stuck, the baseline is mis-specified, or the reported fidelity is computed differently from the optimized fidelity. The paper's explanation that 'directly maximizing fidelity does not necessarily yield the most informative demonstrations' addresses interpretability, not why the numerical metric drops; this discrepancy needs to be resolved before S can be trusted as the paper's main evaluation statistic.","section":"§6.3, Table 1"},{"comment":"The paper repeatedly uses the word 'significantly' (abstract, Sections 6.3 and 6.4) but reports no significance tests, confidence intervals, or paired comparisons. Section 6.1 says results are 'averaged over ten random seeds to ensure statistical significance,' but averaging alone does not establish significance, and with 10 seeds the reported standard deviations in Table 1 overlap for several comparisons (e.g., FlatGrid11 REACT versus Random, and FetchReach SAC-150k REACT versus Random). The authors should report the rliable IQM confidence intervals or equivalent pairwise tests for the headline comparisons, and should remove the word 'significantly' where no test supports it.","section":"§6.1, Sections 6.3-6.5"}],"minor_comments":[{"comment":"The heading 'Continous Robotic Control' contains a typo; it should read 'Continuous Robotic Control.'","section":"§6.5 heading"},{"comment":"The notation |T| is overloaded: it denotes the size of the demonstration set in the weighting factor and the total trajectory length in the denominator. Using a distinct symbol, such as L_T, for the summed trajectory length would avoid ambiguity.","section":"Eq. (7)"},{"comment":"The normalization factor ⌈P⌉ is not defined precisely. The text says it is the 'maximum state distance' max_{s,s'} ||s - s'||_2, but it is unclear why the ceiling notation is used or how this quantity is computed in continuous state spaces; please define it explicitly.","section":"Eq. (2)"},{"comment":"The hyperparameter choices (encoding length, population size, crossover and mutation probabilities) are selected based on experiments in the same environments that are later used for the main evaluation, with no held-out validation or correction for selection bias. This should be stated as a limitation, or the final comparisons should use a separate validation split.","section":"§6.2"},{"comment":"The notation 'children_{pc,F(P)}' in Algorithm 1 is unclear; it should state explicitly how crossover uses the fitness values and what the offspring size is relative to the population size.","section":"Alg. 1, line 8"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and the code release is a genuine asset, but the paper currently overclaims in the abstract: the continuous results do not support the stated improvement, and the fidelity metric is not validated as an interpretability measure. If the authors reframe the contribution as a tool for generating diverse demonstrations for policy visualization, add significance testing or confidence intervals, resolve the fidelity-optimized baseline anomaly, and clearly separate discrete from continuous claims, the paper would be a solid contribution. I see no novelty or citation concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a modest but real contribution to a real problem. REACT treats trajectory-set selection for policy inspection as an evolutionary search over initial states, with a fitness balancing local state coverage, global set diversity, and policy certainty. That joint fitness is not in the prior work I know, and the code is public. The gridworld results look solid: REACT beats random and the ablations by a wide margin on the paper's own fidelity metric. The component ablations tell a sensible story. Credit where due: for an early-stage SAC policy, REACT also lifts fidelity well above random (0.842 vs 0.106). The hyperparameter study is real work, not filler.\n\nNow the soft spots. The continuous half of the headline claim does not hold up. In SAC-100k and SAC-150k, Random gets higher fidelity than REACT (1.681 vs 1.373; 0.173 vs 0.137). The paper's own text concedes that fidelity-based optimization outperforms REACT on mature policies, so the abstract's claim of improved interpretability in both discrete and continuous environments is an overclaim. The qualitative trajectory-spread argument is fine, but it should be framed as a qualitative observation, not a quantitative result.\n\nThe bigger problem is the fidelity metric S itself: a length-weighted dispersion of cumulative rewards around the mean absolute reward. The paper asserts higher S means better explanatory coverage, but nothing ties S to any human judgment of interpretability. That makes the main evaluation circular: the surrogate fitness is evaluated with an unvalidated proxy from the same paper. The stress-test note is right about this. The fidelity-optimized baseline also behaves strangely, scoring below Random in both gridworlds. The authors explain that as over-concentrating on one reward mode, which a pure dispersion metric would tend to do, but it remains a warning that S can be gamed trivially.\n\nThe paper also says \"significantly\" throughout without reporting significance tests; a paired test over the ten seeds would be cheap. Hyperparameters were tuned on the same environments used for evaluation, which inflates the numbers. Minor for an exploratory paper, but they should say so explicitly.\n\nBottom line: this is not a breakthrough and not a waste of time. The method is clearly described, the package seems reusable, and the discrete results are genuinely encouraging. The evaluation needs sharpening before it can support the continuous claim. A serious referee would send it back for revision, but I would not desk-reject it. Bring it to reading group if anyone works on RL interpretability or policy testing; I would not cite it myself until the fidelity metric is validated.","headline":"REACT is a plausibly useful tool for diverse trajectory selection, but the continuous-control claim is overreach and the evaluation rests on an unvalidated fidelity proxy.","tokens_in":20078,"tokens_out":2140,"would_cite":false,"duration_ms":19229,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that evolving starting states against a diversity-and-certainty fitness yields demonstrations that expose a reinforcement-learning policy's edge-case behavior better than random or fidelity-optimized baselines.","keywords":["reinforcement learning","interpretability","explainable reinforcement learning","evolutionary optimization","trajectory selection","surrogate fitness","demonstration diversity","fidelity"],"falsifier":"Ask human participants to answer behavioral questions about a policy after viewing either REACT-selected or random trajectories; if the sets with higher fidelity IQM are not the ones that enable more accurate answers, the surrogate fidelity measure fails as a stand-in for interpretability.","tokens_in":18931,"feed_emoji":"🧭","tokens_out":8275,"duration_ms":66464,"temperature":0.7,"pith_summary":"This paper claims that a fixed reinforcement-learning policy can be understood through a small, carefully chosen set of demonstration trajectories, and that the right way to choose them is to evolve starting states against a surrogate fitness function rather than to maximize reward or even the fidelity score directly. The REACT framework combines global diversity among trajectories, local state coverage within a trajectory, and the policy's action certainty into a joint score, then runs a genetic algorithm over encoded initial states. In two gridworld environments, the selected sets reach fidelity IQMs of 0.744 and 2.130, compared with 0.355 and 1.486 for random starting states and 0.255 and 0.291 for fidelity-optimized baselines. In a continuous robotic reaching task, the same approach gives the largest fidelity gain for an early-stage policy, while for a converged policy direct fidelity optimization wins on the score even though REACT's trajectories cover more of the workspace. The result, on the paper's own terms, is a model-agnostic inspection tool that surfaces edge cases and uncertainty in any simulatable policy.","feed_headline":"Searching initial states reveals a policy's hidden behaviors","feed_subtitle":"REACT evolves starting positions to build a demonstration set whose fidelity score beats random and fidelity-optimized baselines.","key_machinery":"Three quantities define the objective. Local diversity $D_l(\\tau)=|\\{s\\in\\tau\\}|/|P|$ is the fraction of the discretized position space the trajectory visits. Certainty $C(\\tau)=\\frac{1}{|\\tau|}\\sum_{s,a\\in\\tau}\\pi(a|s)$ is the policy's average confidence in its own actions. Global diversity $D_g(\\tau,T)$ is the minimum one-way distance from the candidate to the already selected trajectories, normalized by the maximum state distance. The joint fitness $F(\\tau,T)=D_g(\\tau,T)+F_l$ combines these through the minimum-distance ratio term $F_l=\\min_{t\\in T}\\left\\|\\left(\\frac{D_l}{C}\\right)(\\tau)-\\left(\\frac{D_l}{C}\\right)(t)\\right\\|^2$, which pushes a candidate to be far in the diversity-versus-uncertainty plane from its nearest neighbor in the set. REACT encodes initial states as bit strings, evolves them by tournament selection, single-point crossover, and bit-flip mutation, and prunes the population to keep the highest-fitness individuals, returning the surviving trajectories as the demonstration pool.","core_discovery":"The paper's central claim is that optimizing trajectory selection with the joint surrogate fitness $F(\\tau,T)=D_g(\\tau,T)+F_l$, where $D_g$ is the normalized one-way distance from the candidate trajectory to the demonstration set and $F_l=\\min_{t\\in T}\\left\\|\\left(\\frac{D_l}{C}\\right)(\\tau)-\\left(\\frac{D_l}{C}\\right)(t)\\right\\|^2$ is the minimum squared gap in the local-diversity/certainty ratio, produces demonstration sets that expose the policy's diverse and edge-case behavior better than random initial states or direct optimization of the fidelity score. The evidence is the final fidelity IQMs in Table 1: $0.744$ in FlatGrid11 and $2.130$ in HoleyGrid11, versus $0.355$ and $1.486$ for random states and $0.255$ and $0.291$ for the fidelity-optimized baseline. The authors interpret high fidelity as high explanatory coverage of the policy's decision space. They also report that in continuous control the joint fitness is most useful for early-stage policies, while for a mature policy direct fidelity optimization yields higher scores even though REACT's trajectories show wider spatial coverage.","pith_inferences":["A direct test of the paper's interpretability claim would replace the fidelity score with human ratings of explanation quality and check whether the two rankings agree across REACT, random, and fidelity-optimized sets.","The joint fitness could be turned into an adversarial curriculum by feeding REACT's diverse trajectories back into training, hardening the policy against out-of-distribution starting states; the paper mentions this direction but does not test it.","If local diversity were generalized beyond positions to object layouts or task parameters, the same evolutionary loop could interrogate policies over a much larger space of novel situations."],"forward_implications":["For partially trained or intermediate policies, REACT-selected demonstrations achieve higher fidelity IQMs than random starting states and than optimizing fidelity directly, so the method can reveal what an agent has learned mid-training.","In the two gridworld environments, the full joint fitness beats each of its components (certainty, local diversity, global diversity) and a simple sum of them, showing the minimum-distance ratio formulation is doing real work.","In continuous control, the benefit of diversity-based selection shrinks as the policy converges; for mature policies, direct fidelity optimization is the better surrogate, and REACT's advantage becomes qualitative (wider spatial coverage) rather than quantitative.","Because the method only needs to sample trajectories from a fixed policy, it is model-agnostic and can be applied to any RL agent whose behavior can be simulated."],"supporting_citations":[{"why":"supplies the fidelity metric (Eq. 7) that REACT is evaluated against and that the fidelity-optimized baseline optimizes","marker":"[15]"},{"why":"provides the interquartile-mean and optimality-gap statistics used for all fidelity comparisons","marker":"[1]"},{"why":"provides the FlatGrid11, HoleyGrid11, and FetchReach environments used in the experiments","marker":"[3]"},{"why":"supplies the PPO algorithm used to train the gridworld policies","marker":"[40]"},{"why":"supplies the SAC algorithm used to train the continuous-control policies at three training stages","marker":"[16]"},{"why":"defines the one-way distance between trajectories used in the global diversity term (Eq. 3)","marker":"[26]"},{"why":"the earlier conference version introducing the REACT algorithm that this article extends with ablations and hyperparameter analysis","marker":"[5]"}],"fun_headline_variants":["Fitness-based trajectory search exposes RL policy behaviors","Surrogate fitness boosts explainable RL demonstrations","Optimizing initial states reveals diverse RL policies","Evolutionary search finds edge-case policy behaviors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fidelity score used to quantify demonstration quality actually measures how well humans can understand the policy; the paper does not validate this link, so if score and understanding come apart, the main claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Fitness-based trajectory search exposes RL policy behaviors","Surrogate fitness boosts explainable RL demonstrations","Optimizing initial states reveals diverse RL policies","Evolutionary search finds edge-case policy behaviors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000442,"raw_usage":{"total_tokens":2255,"prompt_tokens":975,"completion_tokens":1280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1224}},"tokens_in":591,"tokens_out":1280,"duration_ms":9384,"temperature":1.0,"reasoning_tokens":1224,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:44:29.876787+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask human participants to answer behavioral questions about a policy after viewing either REACT-selected or random trajectories; if the sets with higher fidelity IQM are not the ones that enable more accurate answers, the surrogate fidelity measure fails as a stand-in for interpretability.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems 34:12222–12236","cited_arxiv_id":null,"evidence_quote":"supplies the fidelity metric (Eq. 7) that REACT is evaluated against and that the fidelity-optimized baseline optimizes"},{"cited_title":"Advances in neural information processing systems 34:29304–29320","cited_arxiv_id":null,"evidence_quote":"provides the interquartile-mean and optimality-gap statistics used for all fidelity comparisons"},{"cited_title":"https://github.com/philippaltmann/hyphi-gym/","cited_arxiv_id":null,"evidence_quote":"provides the FlatGrid11, HoleyGrid11, and FetchReach environments used in the experiments"},{"cited_title":"GeoInformatica 12:117–142","cited_arxiv_id":null,"evidence_quote":"defines the one-way distance between trajectories used in the global diversity term (Eq. 3)"},{"cited_title":"In: Proceedings of the 16th International Joint Conference on Computational Intelligence","cited_arxiv_id":null,"evidence_quote":"the earlier conference version introducing the REACT algorithm that this article extends with ablations and hyperparameter analysis"}],"review_version":1}