{"id":"ba2c7f7b-1670-4d66-9d05-1106b8abdc9c","arxiv_id":"2607.03847","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A neuro-symbolic pipeline for open-ended critical-scenario attribution improves intervention-based CTR and RIR by about 18% and 14% over LLM baselines across four decision-making environments.","lead":"NeSy-CSA turns “why is this scenario critical?” into a constrained neuro-symbolic pipeline: filter factors, build a dependency graph of subtasks, then mix symbolic checks with evidence-bound LLM steps. If it holds up, safety teams can turn rare failure cases into reusable, intervenable explanations instead of opaque LLM text.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Result-level CTR/RIR gains may credit easily editable correlates rather than true causal drivers of criticality.","rationale":"The reader correctly isolates the load-bearing premise: bounded interventions on Extract(R★) parameters are treated as evidence that the named factors drove criticality. The paper’s strongest claim is framed as beating LLM baselines under this protocol (Abstract; Table II), which the tables support, so REJECT is unwarranted. The concern is real and medium-severity: without a sham/random-parameter control or classical causal baselines, CTR/RIR can reward methods that surface high-leverage editable correlates rather than true causes. Process-level F1 and ablations strengthen the “structured reasoning” story but do not validate the causal interpretation of the result-level metrics. Verdict remains CONDITIONAL; the concrete sham-Extract test would settle whether the effectiveness numbers are overstated. Agreement with the reader is full on the weakest assumption.","tokens_in":22164,"tokens_out":687,"duration_ms":5812,"concrete_test":"On the same 200 critical samples per environment, recompute CTR/RIR for (i) NeSy-CSA, (ii) LLM-only, and (iii) a sham Extract that returns a random non-empty subset of scenario parameters of matching cardinality, all under identical ε=0.3 and direction rules. If sham CTR/RIR approaches or exceeds NeSy-CSA (or the NeSy–LLM gap shrinks by >50%), the intervention metric is not a faithful test of attribution quality and the 18%/14% claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that NeSy-CSA yields more effective attribution (CTR +18.32%, RIR +13.67% over LLM baselines; Abstract, Table II) rests on the intervention protocol in §III.E.b (Eqs. 34–36, Fig. 4). Extract(R★_i) yields parameter indices Θ★_i; only those are perturbed by ±ρ ε (u_d−l_d) with ε calibrated to ~0.3 (Fig. 6). CTR/RIR then count how often criticality flips or reward rises. This is a valid test of “editable knobs named by the text” only if the extracted parameters are the true drivers rather than correlated, high-leverage variables that any method can name. Nothing in the protocol distinguishes the two: no sham-factor or random-parameter control, no comparison against classical causal/fault baselines, and Extract itself is an unvalidated LLM/text step. Ablations (Table V) and process F1 (Table III) show internal structure helps, but they do not close the gap between “intervention works under this budget” and “attribution identified the cause.” If the gains largely come from naming easy-to-edit correlates, the headline effectiveness claim overstates causal attribution quality.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes NeSy-CSA, a neuro-symbolic framework for open-ended critical scenario attribution in decision-making agents. Given a critical sample, historical data, and domain priors, it (i) refines LLM-generated candidate factors via semantic deduplication and data-driven filtering (Welch t-test, BH correction, Hedges' g; Eqs. 7–14), (ii) builds a reusable predecessor-dependent subtask graph with deterministic Cont/Pred/IO validation (Eqs. 16–19), and (iii) executes formalizable subtasks via dynamically composed atomic symbolic procedures while routing the rest to evidence-constrained LLM inference (Eqs. 20–27). Evaluation combines process-level Precision/Recall/F1 against expert reference subtasks (Table III) with result-level CTR and RIR under bounded interventions on Extract(R*) parameters (Eqs. 34–36, Fig. 4). Across ACAS Xu, CoopNavi, BipedalWalker, and CARLA, NeSy-CSA reports average CTR/RIR gains of 18.32% and 13.67% over three LLM baselines (Table II), with ablations (Tables V–VI) and a CoopNavi case study (Fig. 7).","tokens_in":22535,"tokens_out":1341,"duration_ms":10383,"significance":"If the intervention protocol is accepted as a proxy for attribution quality, the work is a useful contribution to scenario-based testing: it turns discovered critical cases into structured, reusable explanations rather than isolated failures, and it does so without requiring a fixed causal graph or complete symbolic knowledge base. Strengths include multi-environment evaluation with three-run means and stds, process-level expert matching, component ablations, token-cost analysis of one-time vs per-sample graph construction, and an explicit hybrid routing design that is more inspectable than pure LLM attribution. The CTR/RIR metrics are falsifiable behavioral tests and a practical step beyond reference-free textual explanations. The main significance is methodological—constraining open-ended LLM attribution with factor filtering, validated subtask structure, and selective symbolic execution—rather than a new causal identification theorem.","major_comments":[{"comment":"§III.E.b, Eqs. (34)–(36), Fig. 4, and Table II: The central effectiveness claim (CTR +18.32%, RIR +13.67%) treats successful bounded interventions on Extract(R*_i) as evidence of correct causal attribution. This only shows that the named parameters are high-leverage editable knobs under budget ε (calibrated ~0.3 in Fig. 6). Without sham-factor / random-parameter controls, or comparison to classical causal/fault-diagnosis baselines discussed in §II.B, the protocol cannot distinguish true drivers from correlated easy-to-edit variables. Extract itself is an unvalidated text step. Please add at least one negative control (e.g., intervene on non-attributed or randomly selected parameters of matched cardinality) and report whether NeSy-CSA still outperforms baselines under that control; otherwise the causal reading of CTR/RIR should be substantially softened.","section":null},{"comment":"§IV.A.4 and Table II: Baselines are restricted to LLM-only, LLM+Tool, and LLM+CoT. The related-work section motivates traditional fault diagnosis and causal/counterfactual methods, yet none appear as experimental baselines. For environments where state variables and intervention spaces are structured (especially ACAS Xu), a simple causal or sensitivity baseline would test whether the neuro-symbolic machinery is necessary for the reported gains. Absence of such baselines weakens the claim that NeSy-CSA specifically advances open-ended attribution beyond existing structured methods.","section":null},{"comment":"§III.B–D and Algorithm 1: Factor filtering uses the same critical/non-critical partition of D that later defines Dc for CTR/RIR. While re-simulation is partly independent, there is mild circularity: factors are retained precisely because they differ between groups (Eq. 14), then interventions on those factors are scored by flipping criticality. Please clarify train/eval separation (e.g., filter factors on a held-out subset of D, evaluate CTR/RIR only on unseen critical samples) or quantify sensitivity of Table II to this reuse.","section":null}],"minor_comments":[{"comment":"Fig. 1 and Abstract: The +18.32% / +13.67% figures are averages over four environments relative to the mean of three LLM baselines; state this aggregation explicitly in the figure caption and abstract to avoid reading them as per-environment or vs. the best baseline.","section":null},{"comment":"§IV.A.3: Free parameters α, δ, η_d, η_m, ε are fixed after limited calibration (Fig. 6 only for ACAS Xu). A short sensitivity table for δ and η across environments would strengthen robustness claims.","section":null},{"comment":"Eq. (20) and §III.D.a: Formalizable(ST_t) ∧ Verifiable(ST_t) is central to routing but only described qualitatively. A brief operational definition or example of the LLM’s routing decision would improve reproducibility.","section":null},{"comment":"Table III: Process-level F1 uses the best of three expert reference sets (Eq. 32). Report also mean F1 over the three experts to show sensitivity to reference choice.","section":null},{"comment":"Typographical / presentation: arXiv id and some figure labels use mixed underscore styles (ACAS_Xu vs ACAS Xu); unify. Ensure Algorithm 1 line numbers match the narrative references in §III.C–D.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid systems paper for scenario-based testing / safe decision-making venues, but the causal interpretation of CTR/RIR is currently overstated relative to the evidence. If the authors add negative intervention controls and either a classical baseline or a clearly limited claim, it would be a reasonable accept after revision. Scope fit is good for automation/robotics/safety-testing journals; less so for pure causal-inference venues without the controls above."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is that NeSy-CSA is a concrete, multi-stage pipeline—LLM factor generation, Welch/BH/Hedges filtering, a validated predecessor subtask graph, and on-demand symbolic procedures from a small atomic library plus evidence-constrained neural inference—that turns critical-scenario explanation into something inspectable and intervenable. Across ACAS Xu, CoopNavi, BipedalWalker, and CARLA it lifts CTR and RIR by roughly 18% and 14% over LLM-only/Tool/CoT (Table II), with ablations (Tables V–VI), process F1 against expert references (Table III), and a clear CoopNavi case study that shows the evidence chain.\n\nWhat is actually new is not any single piece (factor screening, CoT-style graphs, tool use, and neuro-symbolic routing all exist) but the packaging for open-ended attribution without a fixed rule base, plus the two-level evaluation that treats process structure and behavioral effect separately. The math is standard and carefully applied; the citation pattern is fair on scenario testing, classical diagnosis, and recent neuro-symbolic work. One-time graph construction is a practical win on tokens without hurting quality.\n\nThe soft spot the stress-test flags is real but not fatal: CTR/RIR only show that parameters named by the attribution text are high-leverage under a calibrated budget ε≈0.3. Without sham-factor or random-parameter controls, and without classical causal/fault baselines, you cannot fully separate true drivers from easy-to-edit correlates. Extract itself is an unvalidated text step, routing is soft, and code/data are not released. Those are the right things to tighten, not reasons to dismiss the tables.\n\nThis is for people who care about scenario-based testing and safety analysis of decision agents and who want reusable explanations rather than more discovery fuzzing. It deserves a serious referee. I would bring it to reading group, cite the evaluation protocol and the hybrid executor design if I work in this area, and recommend peer review rather than desk reject.","headline":"Solid engineering pipeline for open-ended critical-scenario attribution that beats LLM baselines on intervention metrics; the causal claim is softer than the abstract implies but the work is still worth engaging.","tokens_in":23132,"tokens_out":525,"would_cite":true,"duration_ms":5055,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"NeSy-CSA turns open-ended critical-scenario attribution into structured, traceable neuro-symbolic reasoning that improves intervention effectiveness over LLM baselines.","keywords":["neuro-symbolic reasoning","critical scenario attribution","scenario-based testing","traceable reasoning","intervention evaluation","decision-making agents","LLM attribution"],"falsifier":"On held-out critical scenarios from the same four environments, extract parameters from NeSy-CSA conclusions, apply the paper’s bounded interventions, and check whether criticality-transition and reward-improvement rates remain near the reported levels; if they fall to baseline while process-level graph scores stay high, the result-level claim fails.","tokens_in":23036,"feed_emoji":"🧭","tokens_out":698,"duration_ms":5609,"temperature":0.7,"pith_summary":"Critical scenarios found by scenario-based testing of decision-making agents are only useful if we can explain why they are critical. That attribution problem is open-ended: causes can involve different state variables, interaction patterns, and failure mechanisms across tasks, so fixed diagnostic templates are too narrow and unconstrained LLM explanations are hard to inspect or trust. This paper claims that the gap can be closed by NeSy-CSA, a neuro-symbolic pipeline that first selects a compact set of data-supported factors, then organizes reasoning as a reusable predecessor-dependent subtask graph, and finally executes formalizable steps with dynamically composed symbolic procedures while routing semantic steps to evidence-constrained neural inference. Because open-ended attribution has no gold answers, the authors evaluate both the structural validity of the generated reasoning graph against expert references and the behavioral effect of intervening on parameters named by the attribution under a bounded budget. Across four environments the method raises two intervention metrics, criticality transition rate and reward improvement rate, by about 18% and 14% over LLM-only, tool-augmented, and chain-of-thought baselines, supporting the claim that discovered critical cases can become reusable knowledge for later testing and safety analysis.","feed_headline":"Critical scenarios get traceable reasons that actually change outcomes","feed_subtitle":"Neuro-symbolic attribution lifts intervention success ~18% and ~14% over LLM baselines across four agents.","key_machinery":"NeSy-CSA: key-factor refinement (knowledge-guided generation, semantic deduplication, statistical and effect-size filtering), a validated predecessor-dependent subtask graph G*, and neuro-symbolic hybrid execution that routes formalizable/verifiable subtasks to symbolic procedures built from atomic functions and the rest to evidence-constrained neural inference.","core_discovery":"Open-ended critical scenario attribution can be made both flexible and inspectable by constraining it at three levels—factor selection, predecessor-dependent subtask structure, and hybrid symbolic/neural execution—so that conclusions remain grounded in intermediate evidence and can be checked by controlled interventions that change criticality or reward.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["NeSy-CSA makes open-ended scenario criticality attributions traceable and intervenable","Neuro-symbolic factors plus evidence graphs lift attribution effect 18% and 14%","Hybrid symbolic-neural steps turn critical scenarios into inspectable causes","Constrained factor selection and dependency graphs ground open-ended attribution","Intervention tests confirm NeSy-CSA reasons actually shift criticality outcomes"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The method treats a successful bounded local edit of parameters named in the attribution text—flipping a critical outcome or raising reward—as evidence that those factors were the true drivers of criticality rather than merely correlated or easy-to-game knobs.","fun_headline_variants_meta":{"raw":{"variants":["NeSy-CSA makes open-ended scenario criticality attributions traceable and intervenable","Neuro-symbolic factors plus evidence graphs lift attribution effect 18% and 14%","Hybrid symbolic-neural steps turn critical scenarios into inspectable causes","Constrained factor selection and dependency graphs ground open-ended attribution","Intervention tests confirm NeSy-CSA reasons actually shift criticality outcomes"]},"model":"grok-4.5","effort":"low","cost_usd":0.003764,"raw_usage":{"total_tokens":1224,"prompt_tokens":799,"num_sources_used":0,"completion_tokens":81,"cost_in_usd_ticks":37640000,"prompt_tokens_details":{"text_tokens":799,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":344,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":799,"tokens_out":81,"duration_ms":3187,"temperature":1.0,"reasoning_tokens":344,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T23:32:15.803124+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On held-out critical scenarios from the same four environments, extract parameters from NeSy-CSA conclusions, apply the paper’s bounded interventions, and check whether criticality-transition and reward-improvement rates remain near the reported levels; if they fall to baseline while process-level graph scores stay high, the result-level claim fails.","supporting_citations":[],"review_version":1}