REVIEW 3 major objections 7 minor 42 references
A cognitive-EEG agent can answer natural-language analysis requests while a fail-closed harness, not the model, decides what evidence may be released.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-31 02:36 UTC pith:RLANLBN6
load-bearing objection Solid architecture paper: process-level fail-closed control is real and carefully evaluated; intent alignment is still mostly curator- and catalogue-bound, which the authors mostly admit. the 3 major comments →
CogEEGAgent: Toward Autonomous Cognitive EEG Analysis with Grounded Execution and Selection-Aware Verification
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
CogEEGAgent establishes that cognitive-EEG workflows can be automated under bounded autonomy by separating semantic authority from scientific authority. The language model only chooses a registered route; a Scientific Control Plane materializes typed contracts, commits one discovery selection before opening hidden confirmation once, and releases only evidence-bound reports. Sealed studies show higher exact routing than a matched deterministic router, release of supported analyses with blocked capability and reuse hazards, and held-out confirmation restoring false-positive control under adaptive search.
What carries the argument
The EEG-specific scientific harness—especially the Scientific Control Plane with trace–partition–evidence binding—commits one discovery candidate before single-use participant-disjoint confirmation and authorizes release only from typed, replayed evidence. Paradigm-Conditioned Verification audits completed workflows post hoc but does not replace that prospective boundary.
Load-bearing premise
The guarantee holds only if the frozen registry, deterministic executor, commitment ledger, confirmation opener, and scorer behave as specified, while the curator’s catalogue and the model’s semantic fit remain trusted outside automatic enforcement.
What would settle it
Run the sealed composed campaign with underspecified or wrong-template requests and check whether any report still releases without a matching committed, participant-disjoint confirmation return, or whether held-out confirmation fails to keep adaptive-search false positives near the nominal rate on the same candidate families.
If this is right
- Natural-language EEG questions can be answered with receipts that link contract, commitment, confirmation, and report rather than model prose alone.
- Scientific agents can be scored on semantic routing, commitment order, evidence lineage, and release—not only task completion.
- When full adaptive-search history cannot be trusted, held-out confirmation is a practical fail-closed substitute for uncorrected best-p selection.
- The same commit-then-confirm interfaces can wrap other registered scientific executors that expose a finite catalogue and a reserved confirmation resource.
- Model revisions can be swapped against fixed contracts without changing release semantics.
Where Pith is reading between the lines
- The remaining failure mode is semantic abstention under underspecification: process safety can block release while still selecting the wrong reserved contract, so intent validation is the next bottleneck.
- Domains without a separable confirmation split can reuse the post-hoc audit path but cannot claim the same selection-aware guarantee.
- Catalogue quality becomes a first-class scientific object: incomplete or biased registered families would make process-compliant releases still scientifically narrow.
- Completion metrics that ignore successful statistical returns will systematically overrate models that narrate effects without calling tools.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents CogEEGAgent, an LLM agent for cognitive-EEG analysis built on MNE-Python, in which the LLM performs bounded semantic routing (mapping natural language to a registered analysis template) while deterministic components own contract materialization, participant-disjoint discovery/confirmation, commitment, falsification, and evidence-bound release. Three prospective studies support the claims: (i) a sealed 60-request routing benchmark in which Qwen2.5-14B achieves 39/40 exact routes versus 33/40 for a matched field-weighted BM25 router, with shared preflight forcing abstention on all 20 required-abstention requests; (ii) a 21-case externally model-authored, outcome-blind campaign in which three supported ERP analyses (ERN, N400, MMN) are released with participant-disjoint confirmation and all hazard/reuse requests are blocked, though the three underspecified requests are misrouted to already-reserved contracts and contained only by lifecycle ordering; (iii) a 100,000-repetition policy stress test showing held-out confirmation reduces adaptive-search FPR from 16.5% to 4.9%. Deterministic enforcement is exercised by 20/20 and 188/188 fault-injection fixtures.
Significance. If the results hold, this is a useful and unusually disciplined contribution to the scientific-agents literature: the separation of semantic from scientific authority, the executable commit-before-confirmation state machine, and the participant-disjoint release path operationalize established sample-splitting policy in an agent setting. Strengths that materially raise confidence: sealed protocols with hash-bound ledgers and scorer-only gold opened after model calls stop; deterministic fault-injection matrices with exact match counts; externally authored outcome-blind requests; explicit preservation of failed predecessors (the No-Go P3/N170 pilot, §F.2); participant-disjoint confirmation with sign-flip and leave-one-out falsification; and public code. The paper is also commendably transparent about its own boundary (Table 1, §4.8, Suppl. A). The main residual risk is that the headline "fail-closed" framing exceeds the demonstrated enforcement scope, which is process-level rather than intent-level.
major comments (3)
- [Abstract/§5 vs §B.1 Table 5 and §4.1 Table 9] The abstract's closing claim of "fail-closed control over inference and release" is not matched by the paper's own semantic-misfit evidence. In the composed campaign, all three policy-designated underspecified requests were routed to already-reserved supported contracts (0/3 correct abstentions, Table 5/§B.1) and release was prevented only because the supported requests happened to run first; the authors state containment "is not order-invariant." Table 9 further shows full preflight releases 40/40 internally consistent wrong-template substitutions. In the counterfactual ordering, the system would release a statistically valid, fully audited, participant-disjoint analysis of a question the user did not ask. Two requests: (a) report the counterfactual explicitly — the runtime is deterministic, so either re-run the campaign with underspecified requests ordered first or state precisely what
- [§4.1 and Abstract, routing claim] The abstract states the agent "maps language to registered analyses more accurately than a matched deterministic router" without the qualification the body itself supplies: the exact paired test gives p=.070, the template-block sensitivity interval (−2.5 to 35.0 points) crosses zero, the corpus is designer-visible and synthetic, and implementation was finalized after the wording was visible (§4.8). The body correctly calls the advantage "descriptive and corpus-specific"; the abstract and contribution 1 should carry the same hedge, since routing superiority is one of the three legs of the evidence chain and will be the most-cited number (39/40 vs 33/40).
- [§4.2/§4.6, breadth of the positive release evidence] The positive end-to-end evidence is three supported releases (8/8, 10/10, 19/19 participants), one LLM, one frozen request ordering, and familiar pre-loaded data. This is appropriate as a demonstration, but the abstract's "these studies establish bounded autonomy" is stronger than n=3 semantic successes plus a 3/3 semantic-abstention failure supports. Please temper "establish" (e.g., "demonstrate ... under the stated trusted-boundary and catalogue assumptions") or add evidence (additional phrasings, a second model on the composed path, or the order-permutation sensitivity requested above).
minor comments (7)
- [Table 4] Row label renders as "Best-p16.5" — the policy name and the FPR entry appear to have lost a column separator.
- [§4.4 / Table 11] PCVR's injected-error detection F1 is modest overall (.442; .400–.471 per paradigm). The main text notes PCVR is "nearly inert in this conservative-model regime" but does not report the F1; adding it would prevent over-reading of the post-hoc audit path's detection power.
- [Suppl. §B.1] The external authoring session showed a fallback from Claude Fable 5 to Opus 4.8 and account-level memory status was unverifiable. A sentence on why this poses negligible contamination risk (e.g., no registry identifiers or outcomes were provided) would strengthen the provenance account, which is otherwise admirably detailed.
- [§3.3 / Suppl. §H] The effect-evidence score f and its cluster-count fallback min(1, 0.5·n_clusters) are ad hoc; since τ=0.3 halts on cumulative effect change, a brief sensitivity note on these choices would help.
- [Table 8] The Native arm's "Rel." column (e.g., 12/12 released for unsupported-template requests) counts pre-guard model routes; a one-line caption clarification would avoid confusion with the post-preflight release counts elsewhere.
- [§N] ds003061 (Delorme 2021) appears only in the auxiliary scope section; a pointer in the main text or a datasets summary would help readers track which data enter which evaluation.
- [§4.1 / Suppl. §C] Please state the Qwen2.5-14B serving/quantization details (or a precise pointer to the runtime manifest) in the main text, since the single-GPU feasibility claim in §4.5 depends on it.
Circularity Check
No significant circularity: empirical systems claims evaluated against sealed external benchmarks, not self-defined predictions.
full rationale
CogEEGAgent is an engineering/systems paper whose load-bearing claims are empirical (routing accuracy vs FW-BM25, composed campaign terminal actions, held-out FPR reduction) and architectural (trace–partition–evidence binding under stated trusted-component assumptions). The release-safety argument in Supplementary A is a conditional process argument from the state machine and trusted boundary, not a derivation of a quantity defined as its own target. Routing, campaign, and policy studies use sealed gold, outcome-blind external request authoring, participant-disjoint splits, and independent scorers opened only after model calls stop—so reported matches are not forced by fitting the evaluation target into the system definition. Self-citations (Hou et al. 2025/2026; Zeng et al. 2026) supply EEG domain context and motivation; they are not invoked as uniqueness theorems or as the sole warrant for the control-plane guarantees. Operationalizing known sample-splitting and multiple-testing policies (Dwork et al.; Bonferroni; max-T) inside an agent harness is application, not renaming a known result as a first-principles discovery. Scope limits (semantic fit outside deterministic enforcement; order-dependent containment of underspecified requests) are correctness/coverage issues, not circular reductions of outputs to inputs. No self-definitional loop, fitted-input-as-prediction, or load-bearing self-citation chain was found.
Axiom & Free-Parameter Ledger
free parameters (4)
- PCVR max repair iterations K and effect-change halt τ=0.3 =
τ=0.3; K described as max repair iterations (loop up to 3 in narrative)
- Frozen ROI–window candidate family sizes =
9 P3 / 12 N170 in policy test; 9 per component in group campaigns
- Significance level α and trial-count hard/soft thresholds =
conventional α implied (0.05); trial thresholds 5 and 5–9; rejection soft warning >50%
- Held-out split fractions and eligibility minima =
within-condition 50/50 in v3 predecessor; campaign D/C counts 8/8, 10/10, 19/19
axioms (5)
- ad hoc to paper Model routes and prose are untrusted; frozen registry, deterministic executor/Reporter, atomic commitment ledger, exclusive confirmation opener, and source-frozen scorer behave as specified.
- domain assumption A finite public catalogue of admissible registered analyses plus typed deterministic tool returns is available, and a confirmation resource can be reserved and opened once.
- domain assumption Sample-splitting / holdout reuse control limits false positives from adaptive analysis (Dwork et al. 2015 style validity concern).
- domain assumption MNE-Python operations and standard ERP windows/contrasts (ERP CORE / Luck practice) are acceptable scientific primitives for recorded workflows.
- ad hoc to paper Semantic fit between natural language and registered template can remain a model/curator responsibility outside deterministic enforcement.
invented entities (4)
-
Scientific Control Plane with trace–partition–evidence binding
independent evidence
-
Scientific Contract (15-field typed contract)
independent evidence
-
Paradigm-Conditioned Verification (PCVR)
independent evidence
-
CogEEGBench / composed sealed campaigns
no independent evidence
read the original abstract
Electroencephalography (EEG) analysis in cognitive studies requires specialized expertise and involves many defensible choices over contrasts, channels, time windows, and statistical tests. LLM agents can translate varied natural-language questions into analysis choices, offering a flexible interface for automation. Yet fluent reports alone cannot establish that an agent selected the requested analysis or evaluated a confirmatory claim independently of adaptive search. We present CogEEGAgent, a cognitive-EEG analysis agent grounded in MNE-Python. Its EEG-specific scientific harness separates semantic from scientific authority. The LLM interprets intent and proposes registered analyses, while deterministic components validate typed contracts, control confirmation access, and authorize evidence-bound release. On a prespecified routing benchmark, CogEEGAgent maps language to registered analyses more accurately than a matched deterministic router, while matched preflight makes both systems abstain whenever required. In an externally model-authored, outcome-blind campaign, the complete system releases supported analyses with participant-disjoint confirmation and blocks prespecified capability hazards and lifecycle-reuse requests. Policy stress testing shows that held-out confirmation curbs false positives from uncorrected adaptive search. Together, these studies establish bounded autonomy and an auditable automation framework for cognitive-EEG workflows. More broadly, they show how scientific agents can combine flexible language understanding with fail-closed control over inference and release.
Figures
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2511.09947 , year=
EEGAgent: A Unified Framework for Automated EEG Analysis Using Large Language Models , author=. arXiv preprint arXiv:2511.09947 , year=
-
[2]
arXiv preprint arXiv:2605.06584 , year=
NeuroAgent: LLM Agents for Multimodal Neuroimaging Analysis and Research , author=. arXiv preprint arXiv:2605.06584 , year=
-
[3]
arXiv preprint arXiv:2602.13473 , year=
NeuroWeaver: An Autonomous Evolutionary Agent for Exploring the Programmatic Space of EEG Analysis Pipelines , author=. arXiv preprint arXiv:2602.13473 , year=
-
[4]
El Agente Gr\'
Bai, Jiaru and Aldossary, Abdulrahman and Swanick, Thomas and others , journal=. El Agente Gr\'
-
[5]
arXiv preprint arXiv:2603.20356 , year=
Agentproof: Static Verification of Agent Workflow Graphs , author=. arXiv preprint arXiv:2603.20356 , year=
-
[6]
arXiv preprint arXiv:2604.08401 , year=
Verify Before You Commit: Towards Faithful Reasoning in LLM Agents via Self-Auditing , author=. arXiv preprint arXiv:2604.08401 , year=
-
[7]
arXiv preprint arXiv:2604.02485 , year=
Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models , author=. arXiv preprint arXiv:2604.02485 , year=
-
[8]
arXiv preprint arXiv:2509.05363 , year=
SasAgent: Multi-Agent AI System for Small-Angle Scattering Data Analysis , author=. arXiv preprint arXiv:2509.05363 , year=
-
[9]
arXiv preprint arXiv:2605.01423 , year=
HepScript: A Dual-Use DSL for Human-AI Collaborative Data Analysis Workflows in High-Energy Physics , author=. arXiv preprint arXiv:2605.01423 , year=
-
[10]
arXiv preprint arXiv:2408.06292 , year=
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery , author=. arXiv preprint arXiv:2408.06292 , year=
-
[11]
Frontiers in Neuroscience , volume=
MEG and EEG Data Analysis with MNE-Python , author=. Frontiers in Neuroscience , volume=
-
[12]
NeuroImage , volume=
ERP CORE: An Open Resource for Human Event-Related Potential Research , author=. NeuroImage , volume=
-
[13]
2014 , publisher=
An Introduction to the Event-Related Potential Technique , author=. 2014 , publisher=
2014
-
[14]
arXiv preprint arXiv:2308.03188 , year=
Automatically Correcting Large Language Models: Surveying the Landscape of Diverse Self-Correction Strategies , author=. arXiv preprint arXiv:2308.03188 , year=
-
[15]
arXiv preprint arXiv:2512.23518 , year=
Single LLM Debate, MoLaCE: Mixture of Latent Concept Experts Against Confirmation Bias , author=. arXiv preprint arXiv:2512.23518 , year=
-
[16]
arXiv preprint arXiv:2604.02617 , year=
AutoVerifier: An Agentic Automated Verification Framework Using Large Language Models , author=. arXiv preprint arXiv:2604.02617 , year=
-
[17]
arXiv preprint arXiv:2604.03406 , year=
SASAV: Self-Directed Agent for Scientific Analysis and Visualization , author=. arXiv preprint arXiv:2604.03406 , year=
-
[18]
IEEE Transactions on Biomedical Engineering , volume=
BCI2000: A General-Purpose Brain-Computer Interface (BCI) System , author=. IEEE Transactions on Biomedical Engineering , volume=
-
[19]
2024 , eprint=
EEG-GPT: Exploring Capabilities of Large Language Models for EEG Classification and Interpretation , author=. 2024 , eprint=
2024
-
[20]
Nature , volume=
Variability in the analysis of a single neuroimaging dataset by many teams , author=. Nature , volume=. 2020 , doi=
2020
-
[21]
2024 , eprint=
ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery , author=. 2024 , eprint=
2024
-
[22]
2025 , eprint=
BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology , author=. 2025 , eprint=
2025
-
[23]
Journal of Neuroscience Methods , year=
Maris, Eric and Oostenveld, Robert , title=. Journal of Neuroscience Methods , year=
-
[24]
and Bekhti, Yousra and Raimondo, Federico and Gramfort, Alexandre , journal=
Jas, Mainak and Engemann, Denis A. and Bekhti, Yousra and Raimondo, Federico and Gramfort, Alexandre , journal=. Autoreject: Automated artifact rejection for. 2017 , publisher=
2017
-
[25]
Scientific Reports , volume=
EEG is better left alone , author=. Scientific Reports , volume=. 2023 , doi=
2023
-
[26]
2019 , publisher=
Pion-Tonachini, Luca and Kreutz-Delgado, Ken and Makeig, Scott , journal=. 2019 , publisher=
2019
-
[27]
Nature Neuroscience , volume=
Parameterizing neural power spectra into periodic and aperiodic components , author=. Nature Neuroscience , volume=. 2020 , doi=
2020
-
[28]
Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses , journal=
Lakens, Dani. Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses , journal=. 2017 , volume=
2017
-
[29]
and Appelhoff, Stefan and Gorgolewski, Krzysztof J
Pernet, Cyril R. and Appelhoff, Stefan and Gorgolewski, Krzysztof J. and Flandin, Guillaume and Phillips, Christophe and Delorme, Arnaud and Oostenveld, Robert , journal=. 2019 , doi=
2019
-
[30]
and Gramfort, Alexandre and Maurits, Natasha and Michel, Christoph M
Pernet, Cyril and Garrido, Marta I. and Gramfort, Alexandre and Maurits, Natasha and Michel, Christoph M. and Pang, Elizabeth and Salmelin, Riitta and Schoffelen, Jan Mathijs and Valdes-Sosa, Pedro A. and Puce, Aina , journal=. Issues and recommendations from the. 2020 , doi=
2020
-
[31]
Proceedings of Machine Learning and Systems (MLSys) , volume=
Accounting for Variance in Machine Learning Benchmarks , author=. Proceedings of Machine Learning and Systems (MLSys) , volume=. 2021 , url=
2021
-
[32]
Statistical Comparisons of Classifiers over Multiple Data Sets , journal=
Dem. Statistical Comparisons of Classifiers over Multiple Data Sets , journal=. 2006 , url=
2006
-
[33]
Jo, Hyejeong and Yang, Yiqian and Han, Juhyeok and Duan, Yiqun and Xiong, Hui and Lee, Won Hee , year=. Are. 2405.06459 , archivePrefix=
-
[34]
Hou, D. and Wu, Z. and Jiang, L. and Li, Z. and Lin, F. and Yamada, K. D. , year=. Same Brain, Different Prediction: How Preprocessing Choices Undermine. 2605.07212 , archivePrefix=
-
[35]
Frontiers in Human Neuroscience , volume=
Task-constrained self-initiated attention shifts are indexed by frontal-midline theta ramping , author=. Frontiers in Human Neuroscience , volume=. 2025 , doi=
2025
-
[36]
Subject-Specific Analysis of Self-Initiated Attention Shifts from
Zeng, Yuwen and Hou, Dengzhe and Zhang, Zhang and Sun, Sai and Huang, Yongsong and Tseng, Chia-huei and Shioiri, Satoshi , year=. Subject-Specific Analysis of Self-Initiated Attention Shifts from. 2605.18251 , archivePrefix=
-
[37]
arXiv preprint arXiv:2606.25400 , year=
BrainAgent: A Large Language Model-Driven Multi-Agent Framework for Autonomous Brain Signal Understanding , author=. arXiv preprint arXiv:2606.25400 , year=
-
[38]
Wu, Zihao and Xu, Steven and Chen, Bowen and Wan, Shaowen and Li, Yiwei and Ruan, Wei and Lyu, Yanjun and Li, Siyuan and Zhu, Dajiang and Liu, Tianming and Zhao, Lin , year=. 2604.23674 , archivePrefix=
-
[39]
PLOS ONE , volume=
A Comparison Study of Canonical Correlation Analysis Based Methods for Detecting Steady-State Visual Evoked Potentials , author=. PLOS ONE , volume=. 2015 , doi=
2015
-
[40]
Delorme, Arnaud , year=
-
[41]
Science , volume=
The Reusable Holdout: Preserving Validity in Adaptive Data Analysis , author=. Science , volume=. 2015 , doi=
2015
-
[42]
and Zaragoza, Hugo and Taylor, Michael J
Robertson, Stephen E. and Zaragoza, Hugo and Taylor, Michael J. , booktitle=. Simple. 2004 , publisher=. doi:10.1145/1031171.1031181 , url=
arXiv 2004
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.