REVIEW 4 major objections 5 minor 83 references
BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that comprehensive EEG understanding can be benchmarked as instruction-conditioned analytical execution, and that structured agent workflows outperform autonomous code execution for most of 13 evaluated LLMs.
desk verdict A solid benchmark whose headline paradigm comparison is confounded by task-specific tools, and which reports only two of four subsets so far. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a three-part evaluation pipeline. First, the benchmark is organized as a hierarchy of subsets, reusable tasks, and data-bound instances, where each instance's ground truth comes from a deterministic analysis script that fixes specific analytical choices such as Welch window length, the 8–13 Hz alpha band, and a 1.5×IQR outlier rule. Second, a black-box protocol gives the target system only the instruction and input files, keeping the reference package and validation configuration hidden, and records execution actions separately for audit. Third, multi-unit validation uses an LLM-based Parser Agent to map free-form reports into a structured JSON schema, then scores the extracted fields, the original report, and generated artifacts through numerical, categorical, set, sequence, semantic, and artifact validation. The same instances and scoring are run under two execution paradigms—BrainAgent's structured, tool-mediated agent workflow and CodeAct's autonomous code execution—so the effect of the execution paradigm can be isolated while holding instructions, data, and evaluation criteria constant.
What would settle it
Ask a panel of clinical EEG experts to independently score a sample of free-form reports from the completed Foundational and Sleep runs against the benchmark's own instructions, without seeing the reference values, and compare their scores with the Semantic Judge and overall scores. If expert agreement with the benchmark is materially lower than the 95.54% LLM-auditor agreement reported for one Sleep run, or if experts rate a substantial fraction of the reference scripts' parameter choices as indefensible, the benchmark would be measuring conformity to one analytical convention rather than scientific EEG understanding.
Extended reading notes
Core claim
BrainBench's central claim is that comprehensive EEG understanding can be operationalized as instruction-conditioned analytical execution: a target system receives an instruction and input files, performs the analysis, and produces a free-form scientific report plus artifacts when required. Each instance has a hidden deterministic reference script that computes the ground-truth numbers, labels, sets, sequences, or artifacts from the bound data, and an LLM-based Parser Agent extracts explicitly reported fields from the free-form report without judging correctness. The completed evaluation on Foundational Analysis and Sleep Assessment shows substantial but uneven performance: the best overall score stays below 80, BrainAgent raises the performance floor and compresses differences among models, while CodeAct keeps a higher ceiling for its strongest configuration. The paper states this shows that comprehensive EEG understanding is not determined by the underlying LLM alone; it emerges from the interaction between model capability, analytical task, and execution paradigm.
Load-bearing premise
The whole score depends on the assumption that the benchmark's fixed reference scripts—specific frequency bands, window lengths, and outlier thresholds—and its LLM-based semantic judge define correct EEG analysis, rather than one defensible analytical convention among several.
Editorial extensions
If this is right
- If the central claim is right, future EEG LLM systems can be compared on identical instructions, data, and scoring, replacing system-specific demonstrations with a common leaderboard.
- Structured agentic execution should be expected to lift quantitative accuracy and artifact reliability for most models, while categorical, set, sequence, and semantic scores will vary by model and subset.
- Scores should fall as task difficulty rises, and the benefit of workflow structure should shrink on the hardest tasks, marking long-horizon evidence integration as the current capability boundary.
- Because the benchmark's Semantic Judge disagreed with an independent LLM auditor on about 4.5% of judged semantic decisions in one audited run, model gaps smaller than that uncertainty should not be over-read as real differences.
Reading between the lines
- We infer that extending the benchmark to the unpublished Neurocognitive and Physiological Integration subsets would likely show the largest BrainAgent-over-CodeAct gap on medium-complexity cross-modal tasks, where many tool calls must be coordinated, and a smaller gap on open-ended reasoning tasks, mirroring the completed subsets.
- We infer a testable robustness check: re-scoring a sample of instances under alternative defensible parameter choices, such as a different alpha-band edge or a different outlier threshold, and measuring whether model rankings survive would separate scientific understanding from adherence to the benchmark's fixed analytical conventions.
- We infer that a human-expert audit of the Semantic Judge across all four subsets, not just one model run, would convert the benchmark's internal consistency check into evidence about whether the scores measure genuine EEG scientific validity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BrainBench, a benchmark for instruction-conditioned EEG understanding, comprising four subsets (Foundational Analysis, Sleep Assessment, Neurocognitive Assessment, Physiological Integration), 172 tasks, over 4K real-data instances, and 17 datasets. The evaluation protocol takes a natural-language instruction and real EEG/physiological recordings as input and requires the target system to produce a free-form report and optional artifacts, which are scored through six validation units (numerical, categorical, set, sequence, semantic, artifact). The authors evaluate 13 LLMs under two execution paradigms: autonomous code execution with CodeAct and a structured multi-agent framework with BrainAgent. Results are reported for Foundational Analysis and Sleep Assessment, with BrainAgent outperforming CodeAct for most models on these subsets; the best overall score remains below 80. The paper concludes that comprehensive EEG understanding is not determined by the underlying LLM alone but emerges from the interaction between model capability, analytical task, and execution paradigm.
Significance. If the benchmark is valid, it addresses a real gap: most EEG benchmarks focus on fixed decoding tasks, whereas BrainBench attempts to evaluate workflow-level, instruction-conditioned analysis. The construction is detailed and largely reproducible in design: deterministic reference scripts, a three-expert cross-review of tasks, six validation units, a black-box execution protocol, and an audit of the parser (99.8% agreement) and semantic judge (95.5% agreement) on one run. The scale of evaluation (100K+ executions) is a strength. However, the present manuscript does not support the headline claims: results for two of the four subsets are missing, and the BrainAgent advantage is plausibly confounded by task-specific tools in its toolset. The benchmark itself may be valuable after the evaluation is completed and the paradigm comparison is cleansed of leakage, but the current findings are premature.
major comments (4)
- [D.2.1 and D.2.2, Table 8] The assertion in Section D.2.1 that no tool encodes the solution, task-specific parameters, or decision rule of any task is contradicted by the tool descriptions in Section D.2.2. For example, OximetryAnalyzer computes ODI3 and ODI4 (matching SA-40), RespiratoryEventAnalyzer detects apnea, hypopnea, and subtypes (matching SA-30/32/35), SleepArousalAnalyzer detects arousals (matching SA-29/31), WorkloadRuleClassifier applies a spectral decision rule for workload classification (matching NA-21/24/26), and ValenceRBTransformer/ArousalRBTransformer carry pretrained DEAP-based emotion checkpoints (matching NA-01/02). These tools directly implement the workflows and decision rules that CodeAct models must reconstruct from general-purpose libraries. Therefore, the BrainAgent advantage reported in Section 4.1 cannot be cleanly attributed to the execution paradigm; it may reflect task-specific tool leakage. This issue is load-bearing for the paper's central claim about the interaction between model capability and execution paradigm.
- [4.1, Table 1] The paper claims in the abstract, introduction, and conclusion that 13 LLMs are evaluated across all four subsets and that comprehensive EEG understanding is jointly determined by model, task, and paradigm. However, Table 1 reports results only for Foundational Analysis and Sleep Assessment; no results are presented for Neurocognitive Assessment or Physiological Integration. Section 4.1 explicitly frames these as 'currently completed results,' but the accompanying conclusions generalize beyond the reported subsets. The missing subsets are necessary to support the paper's claims about breadth and about paradigm effects across the full benchmark. The authors should either provide the missing results or substantially qualify their conclusions.
- [E.5 and 3.2.3] The Semantic Judge is a central component of the evaluation, used to score 'scientifically grounded conclusions' in many tasks (e.g., FA-30, NA-06). Its audit in Appendix E.5 compares the judge only against another LLM (Qwen3.7 Max) on a single model-subset run, and the paper itself states that this is 'not a human-annotated estimate of absolute evaluator accuracy.' Without human-expert validation of the judge rubrics and scoring, the benchmark's claim to measure scientific EEG understanding is not established. This is not merely a presentation issue: the semantic validation scores feed directly into the headline results and the conclusion that BrainAgent improves evidence-grounded interpretation.
- [C.1, Listing 1] The deterministic ground-truth scripts fix analysis parameters that are not fully specified in the instructions and that competent analysts could legitimately choose differently. For instance, Listing 1 fixes n_fft equal to 4 times the sampling rate, defines the alpha band as 8–13 Hz, and the text elsewhere fixes an IQR multiplier of 1.5. A model that uses another reasonable spectral estimation configuration may produce numerically different results and be penalized. Since the ground truth defines 'scientifically correct' for each task, the benchmark's absolute scores conflate scientific understanding with adherence to the authors' parameter choices. The paper should justify these parameter choices or provide a sensitivity analysis showing that scores are robust to reasonable alternatives.
minor comments (5)
- [Figure 1 caption] The caption states '173 tasks' while the abstract, Section 3, and the task inventories consistently report 172 tasks; correct the discrepancy.
- [Table 1] Several entries have missing spaces, making the numeric table difficult to read (e.g., 'Gemini 3.6 Flash 77.38 54.8572.9559.5175.1757.18' and 'Claude Opus 581.47 84.9766.4173.5373.9479.25'). Reformate the table.
- [Section 3.3] The subset is called 'Physiological Integration' in the abstract and elsewhere, but Section 3.3 refers to 'Physiological Signal Integration'; use one consistent name.
- [Appendix E.3] The text writes '( ho=0.868)' and similar expressions where the Greek letter rho is intended; typeset the symbol consistently.
- [Abstract and Section 5] The paper states that code and benchmark 'will be released soon' without a data/code availability statement or repository link; please include concrete availability details to support reproducibility.
Circularity Check
BrainAgent's advantage over CodeAct is partly by construction: BrainBench defines the task space and BrainAgent's toolset is built to cover it, including analyzers that directly compute benchmark ground-truth metrics and pretrained emotion checkpoints.
-
self definitional
[Appendix D.2.1 (BrainAgent architecture and benchmark adaptation)]
"To cover the evaluation capability boundary defined by BrainBench, we equip BrainAgent with a capability-oriented toolset comprising reusable EEG analysis operations rather than task- or instance-specific solutions. No tool encodes the solution, task-specific parameters, reference answer, or decision rule of any particular task or instance."
The benchmark's evaluation capability boundary is defined by its 172 tasks and their validation configurations; equipping BrainAgent's toolset to cover that boundary means the toolset is constructed from the same task definitions that will later measure it. The headline finding that BrainAgent outperforms CodeAct (68.78 vs 62.83) is then partly a property of the toolset's match to the evaluation, not an independent measurement of the execution paradigm. Appendix D.2.2's task-specific analyzers confirm this reduction.
-
self definitional
[Appendix D.2.2, Table 8 (Sleep Assessment tools)]
"OximetryAnalyzer Computes ODI3, ODI4, sleep/wake mean and minimum SpO2, T90, and T80. RespiratoryEventAnalyzer Detects apnea, hypopnea, apnea subtype, and RERA-like events from respiratory, oximetry, and arousal evidence. SleepArousalAnalyzer Detects rule-based EEG arousal-like events using robust high-frequency envelope thresholds."
BrainBench's SA-40 target is exactly 'sleep-period ODI3 and ODI4'; SA-30/32/35 ask for apnea/hypopnea/subtype classification and indices; SA-29/31 ask for arousal detection and arousal index. Bundling analyzers that compute these quantities means BrainAgent executes the task by calling a prebuilt function, while CodeAct must reconstruct the logic from general-purpose libraries. The reported BrainAgent advantage on those numerical and categorical units is therefore the analyzer's output, not the LLM's instruction-to-conclusion transformation, making the measured paradigm difference partly by construction.
1 more flagged steps
-
fitted input called prediction
[Appendix D.2.2, Table 8 (Neurocognitive Assessment tools)]
"ValenceRBTransformer Converts EEG into DEAP-style differential-entropy tokens and performs binary valence inference with a local RBTransformer checkpoint. ArousalRBTransformer Uses the same DEAP-style differential-entropy preprocessing and RBTransformer pipeline to perform binary arousal classification."
Tasks NA-01 and NA-02 require classifying an EEG segment's emotional polarity and arousal/valence level. A pretrained RBTransformer checkpoint is a fitted emotion classifier; routing BrainAgent through it converts the LLM's task into a checkpoint call. The resulting emotion labels are predictions of the fitted model, not of the LLM's emergent EEG understanding, so crediting them to BrainAgent's structured agentic execution is equivalent to renaming a fitted model's output as the system's understanding.
full rationale
Most of BrainBench itself is not circular: ground truth is generated by deterministic scripts from public datasets, the parser and semantic pipeline is held fixed across both paradigms, and CodeAct results (e.g., Claude Opus 5's 79.25 overall score) show that the comparison is not entirely forced. However, the central paradigm comparison is partially circular because BrainAgent's toolset was deliberately built to cover the benchmark's task space and includes analyzers that directly implement the exact quantities the benchmark scores (ODI3/ODI4, apnea/hypopnea, arousal, workload rules, and emotion classification with pretrained DEAP-style checkpoints). This contradicts Appendix D.2.1's claim that no tool encodes a decision rule, and it means the measured BrainAgent advantage on numerical and categorical tasks is in part the output of prebuilt or fitted tools rather than the LLM's transformation of instruction to conclusion. No load-bearing self-citation chain was found for the benchmark's absolute validity; the Semantic Judge audit against another LLM is a validity limitation, not a circularity. The score of 6 reflects partial circularity: some headline predictions reduce by construction, while semantic, sequence, and hard reasoning tasks retain independent content.
Assumptions & free parameters
free parameters (5)
- Numerical validation tolerance tau =
e.g., 5% of GT in FA-30-1; 10% of GT in SA-36-6
- Metric weights w_im =
e.g., 50/50 in SA-36; 40/30/30 in NA-06
- Difficulty coefficients d_i =
1.0 (Easy), 1.5 (Medium), 2.0 (Hard)
- Ground-truth script parameters =
n_fft=4*sfreq, bandpass 0.5-40 Hz, alpha 8-13 Hz, IQR multiplier 1.5
- CodeAct execution budgets =
20 interaction rounds, 150 s per execution, 720 s per instance, temperature 0.7
assumptions (4)
- ad hoc to paper The deterministic reference scripts define the scientifically correct analysis for each instruction.
- domain assumption The Parser Agent and Semantic Judge, both LLMs, accurately score free-form outputs.
- domain assumption Standard EEG conventions are treated as fixed ground truth.
- ad hoc to paper The first five subjects of each dataset are representative of the population.
Cite this review
Pith. "Pith review of BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding." pith.science (2026). https://pith.science/paper/UVXRMU75
@misc{pith2026260804156,
author = {Pith},
title = {Pith review of: BrainBench: Benchmarking Large Language Models for Comprehensive EEG Understanding},
year = {2026},
howpublished = {\url{https://pith.science/paper/UVXRMU75}},
note = {Machine review of arXiv:2608.04156}
}
read the original abstract
Electroencephalography (EEG) analysis extends beyond assigning predefined labels to recordings; it requires workflows connecting natural-language instructions, signal processing, quantitative evidence, and scientific interpretation. We term this capability \emph{comprehensive EEG understanding}. Existing evaluations, however, primarily target isolated decoding tasks or system-specific demonstrations, leaving the competence of large language models (LLMs) insufficiently quantified. We introduce \benchmarkname{}, a unified benchmark for comprehensive, instruction-conditioned EEG understanding. It comprises four subsets---Foundational Analysis, Sleep Assessment, Neurocognitive Assessment, and Physiological Integration---covering 17 datasets, \numcases{} tasks, and over \numinstances{} real-data instances. Given an instruction and EEG recordings with optional physiological signals, a system must perform the analysis and produce a scientifically grounded report and, when required, artifacts. Outputs are assessed through numerical, categorical, set, sequence, semantic, and artifact validation. We evaluate \nummodels{} representative LLMs across more than 100K executions under two paradigms: autonomous code execution with CodeAct and structured agentic analysis with BrainAgent. Results vary substantially across models, subsets, difficulty levels, and execution paradigms, showing that EEG competence depends on the model and its operationalization. \benchmarkname{} provides a reproducible testbed for advancing LLM-based EEG understanding. The code and benchmark will be released soon, with evaluation results continuously updated.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
All-night dynamics of the human sleep eeg
Daniel Aeschbach and Alexander A Borbely. All-night dynamics of the human sleep eeg. Journal of sleep research, 2(2):70–81, 1993
1993
-
[2]
Bci2000: a general-purpose brain-computer interface (bci) system.IEEE Transactions on biomedical engineering, 51(6):1034–1043, 2004
Gerwin Schalk, Dennis J McFarland, Thilo Hinterberger, Niels Birbaumer, and Jonathan R Wolpaw. Bci2000: a general-purpose brain-computer interface (bci) system.IEEE Transactions on biomedical engineering, 51(6):1034–1043, 2004
2004
-
[3]
Eeg dynamics in patients with alzheimer’s disease.Clinical neurophysiology, 115(7):1490–1505, 2004
Jaeseung Jeong. Eeg dynamics in patients with alzheimer’s disease.Clinical neurophysiology, 115(7):1490–1505, 2004
2004
-
[4]
Eeg and meg: relevance to neuroscience.Neuron, 80(5):1112–1128, 2013
Fernando Lopes da Silva. Eeg and meg: relevance to neuroscience.Neuron, 80(5):1112–1128, 2013
2013
-
[5]
Cbramod: A criss-cross brain foundation model for eeg decoding
Jiquan Wang, Sha Zhao, Zhiling Luo, Yangxuan Zhou, Haiteng Jiang, Shijian Li, Tao Li, and Gang Pan. Cbramod: A criss-cross brain foundation model for eeg decoding. InInternational conference on learning representations, volume 2025, pages 75310–75346, 2025
work page 2025
-
[6]
Personalized sleep staging leveraging source-free unsupervised domain adaptation
Yangxuan Zhou, Sha Zhao, Jiquan Wang, Haiteng Jiang, Shijian Li, Benyan Luo, Tao Li, and Gang Pan. Personalized sleep staging leveraging source-free unsupervised domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 14529–14537, 2025
work page 2025
-
[7]
Yuchen Zhou, Jiamin Wu, Zichen Ren, Zhouheng Yao, Weiheng Lu, Kunyu Peng, Qihao Zheng, Chunfeng Song, Wanli Ouyang, and Chao Gou. Csbrain: A cross-scale spatiotemporal brain foundation model for eeg decoding.Advances in Neural Information Processing Systems, 38: 87150–87195, 2026
work page 2026
-
[8]
Yassine El Ouahidi, Jonathan Lys, Philipp Thölke, Nicolas Farrugia, Bastien Pasdeloup, Vincent Gripon, Karim Jerbi, and Giulia Lioi. Reve: A foundation model for eeg-adapting to any setup with large-scale pretraining on 25,000 subjects.Advances in Neural Information Processing Systems, 38:22541–22577, 2026
work page 2026
Show all 83 references
-
[9]
Committee report: publication guidelines and recommendations for studies using electroencephalography and magnetoen- cephalography.Psychophysiology, 51(1):1–21, 2014
Andreas Keil, Stefan Debener, Gabriele Gratton, Markus Junghöfer, Emily S Kappenman, Steven J Luck, Phan Luu, Gregory A Miller, and Cindy M Yee. Committee report: publication guidelines and recommendations for studies using electroencephalography and magnetoen- cephalography.P...
2014
-
[10]
A revised glossary of terms most commonly used by clinical electroencephalographers and updated proposal for the report format of the eeg findings
Nick Kane, Jayant Acharya, Sandor Beniczky, Luis Caboclo, Simon Finnigan, Peter W Kaplan, Hiroshi Shibasaki, Ronit Pressler, and Michel JAM Van Putten. A revised glossary of terms most commonly used by clinical electroencephalographers and updated proposal for the report forma...
2017
-
[11]
Issues and recommendations from the ohbm cobidas meeg committee for reproducible eeg and meg research.Nature neuroscience, 23(12):1473–1483, 2020
Cyril Pernet, Marta I Garrido, Alexandre Gramfort, Natasha Maurits, Christoph M Michel, Elizabeth Pang, Riitta Salmelin, Jan Mathijs Schoffelen, Pedro A Valdes-Sosa, and Aina Puce. Issues and recommendations from the ohbm cobidas meeg committee for reproducible eeg and meg res...
2020
-
[12]
Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020. 10
1901
-
[13]
React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022
2022 arXiv
-
[14]
Eeg agent: A unified framework for automated eeg analysis using large language models
Sha Zhao, Mingyi Peng, Haiteng Jiang, Tao Li, and Shijian Li. Eeg agent: A unified framework for automated eeg analysis using large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 18063–18071, 2026
2026
-
[15]
Eeg-ai: An agentic system for ai-assisted semi-automated eeg preprocessing and artifact removal.Journal of Neuroscience Methods, 432:110759, 2026
Abdelrahman Abdou, Martin Ivanov, Sarmed Shaya, Alice Rueda, Fatemeh Gholamali Nezhad, Ilya Demchenko, Mohammad Amin Kamaleddin, Paul A Frewen, Benjamin T Dunkley, Brendan Brady, et al. Eeg-ai: An agentic system for ai-assisted semi-automated eeg preprocessing and artifact rem...
2026
-
[16]
Neuroskill (tm): Proactive real-time agentic system capable of modeling human state of mind.arXiv preprint arXiv:2603.03212, 2026
Nataliya Kosmyna and Eugene Hauptmann. Neuroskill (tm): Proactive real-time agentic system capable of modeling human state of mind.arXiv preprint arXiv:2603.03212, 2026
2026
-
[17]
Embracing trustworthy brain-agent collaboration as paradigm extension for intelligent assistive technologies.Advances in Neural Information Processing Systems, 38, 2026
Yankai Chen, Xinni Zhang, Yifei Zhang, Yangning Li, Henry Zou, Chunyu Miao, Weizhi Zhang, Steve Xue Liu, and Philip S Yu. Embracing trustworthy brain-agent collaboration as paradigm extension for intelligent assistive technologies.Advances in Neural Information Processing Syst...
2026
-
[18]
Sleeplm: Natural-language intelligence for human sleep
Zongzhe Xu, Zitao Shuai, Eideen Mozaffari, Ravi Shankar Aysola, Rajesh Kumar, and Yuzhe Yang. Sleeplm: Natural-language intelligence for human sleep. InForty-third International Conference on Machine Learning, 2026
2026
-
[19]
Neuroweaver: An autonomous evolutionary agent for exploring the programmatic space of eeg analysis pipelines.arXiv preprint arXiv:2602.13473, 2026
Guoan Wang, Shihao Yang, Jun-En Ding, and Feng Liu. Neuroweaver: An autonomous evolutionary agent for exploring the programmatic space of eeg analysis pipelines.arXiv preprint arXiv:2602.13473, 2026
2026 arXiv
-
[20]
Brainagent: A large language model-driven multi-agent framework for autonomous brain signal understanding
Yangxuan Zhou, Sha Zhao, Jiquan Wang, Shijian Li, and Gang Pan. Brainagent: A large language model-driven multi-agent framework for autonomous brain signal understanding. arXiv preprint arXiv:2606.25400, 2026
2026 arXiv
-
[21]
Brainpilot: Automating brain discovery with agentic research.arXiv preprint arXiv:2607.15079, 2026
Haoxuan Li, Tianci Gao, Jianhe Li, Yang Fan, Runze Shi, Weiran Wang, Tianxiang Zhao, Zezhao Wu, Xiaoyang Jiang, Qihui Zhang, et al. Brainpilot: Automating brain discovery with agentic research.arXiv preprint arXiv:2607.15079, 2026
2026 arXiv
-
[22]
Neural signals generate clinical notes in the wild.arXiv preprint arXiv:2601.22197, 2026
Jathurshan Pradeepkumar, Zheng Chen, and Jimeng Sun. Neural signals generate clinical notes in the wild.arXiv preprint arXiv:2601.22197, 2026
2026 arXiv
-
[23]
Cerebragloss: Instruction-tuning a large vision-language model for fine-grained clinical eeg interpretation
Wei Gu, Luo Tianming, Qiran Zhang, Mohan Ye, Xiao Shen, Wenxin Chen, Yunhuan Li, Yichen Zhang, Jing Hong, Bao-liang Lu, et al. Cerebragloss: Instruction-tuning a large vision-language model for fine-grained clinical eeg interpretation. InThe Fourteenth International Conference...
-
[24]
Adabrain-bench: Benchmarking brain foundation models for brain-computer interface applications.arXiv preprint arXiv:2507.09882, 2025
Jiamin Wu, Zichen Ren, Junyu Wang, Pengyu Zhu, Yonghao Song, Mianxin Liu, Qihao Zheng, Lei Bai, Wanli Ouyang, and Chunfeng Song. Adabrain-bench: Benchmarking brain foundation models for brain-computer interface applications.arXiv preprint arXiv:2507.09882, 2025
2025 arXiv
-
[25]
Eeg-fm-bench: A comprehensive benchmark for the systematic evaluation of eeg foundation models.arXiv preprint arXiv:2508.17742, 2025
Wei Xiong, Jiangtong Li, Jie Li, and Kun Zhu. Eeg-fm-bench: A comprehensive benchmark for the systematic evaluation of eeg foundation models.arXiv preprint arXiv:2508.17742, 2025
2025
-
[26]
Brain4fms: A benchmark of foundation models for electrical brain signal.arXiv preprint arXiv:2602.11558, 2026
Fanqi Shen, Enhong Yang, Jiahe Li, Junru Hong, Xiaoran Pan, Zhizhang Yuan, Meng Li, and Yang Yang. Brain4fms: A benchmark of foundation models for electrical brain signal.arXiv preprint arXiv:2602.11558, 2026
2026
-
[27]
Omnieeg-bench: A standardized evaluation benchmark for eeg foundation models.arXiv preprint arXiv:2606.00815, 2026
Ziling Lu, Zongsheng Li, Xinke Shen, Kexin Lou, Yingyue Xin, Xiaoqi Chen, Shinan Wang, Xiang Chen, Jiahao Fan, Chenyu Huang, et al. Omnieeg-bench: A standardized evaluation benchmark for eeg foundation models.arXiv preprint arXiv:2606.00815, 2026
2026 arXiv
-
[28]
Neuroatlas: Benchmarking foundation models for clinical eeg and brain-computer interfaces.arXiv preprint arXiv:2605.14698, 2026
Konstantinos Kontras, Trui Osselaer, Stylianos G Mouslech, Angeliki-Ilektra Karaiskou, Guido Gagliardi, Thomas Strypsteen, Mohammad Hossein Badiei, Anku Rani, Maarten Vanmarcke, Miguel Bhagubai, et al. Neuroatlas: Benchmarking foundation models for clinical eeg and brain-compu...
2026 arXiv
-
[29]
Hearts: Benchmarking llm reasoning on health time series.arXiv preprint arXiv:2603.06638, 2026
Sirui Li, Shuhan Xiao, Mihir Joshi, Ahmed Metwally, Daniel McDuff, Wei Wang, and Yuzhe Yang. Hearts: Benchmarking llm reasoning on health time series.arXiv preprint arXiv:2603.06638, 2026
2026 arXiv
-
[30]
Executable code actions elicit better llm agents
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. InForty-first International Conference on Machine Learning, 2024
2024
-
[31]
Eegnet: a compact convolutional network for eeg-based brain-computer interfaces.arXiv preprint arXiv:1611.08024, 2016
Vernon J Lawhern, Amelia J Solon, Nicholas R Waytowich, Stephen M Gordon, Chou P Hung, and Brent J Lance. Eegnet: a compact convolutional network for eeg-based brain-computer interfaces.arXiv preprint arXiv:1611.08024, 2016
2016 arXiv
-
[32]
Large brain model for learning generic representations with tremendous eeg data in bci
Wei-Bang Jiang, Liming Zhao, and Bao-Liang Lu. Large brain model for learning generic representations with tremendous eeg data in bci. InInternational Conference on Learning Representations, volume 2024, pages 16405–16426, 2024
2024
-
[33]
Brainomni: A brain foundation model for unified eeg and meg signals.Advances in Neural Information Processing Systems, 38:41179–41212, 2026
Qinfan Xiao, Ziyun Cui, Chi Zhang, Siqi Chen, Wen Wu, Andrew Thwaites, Alexandra Woolgar, Bowen Zhou, and Chao Zhang. Brainomni: A brain foundation model for unified eeg and meg signals.Advances in Neural Information Processing Systems, 38:41179–41212, 2026
2026
-
[34]
Eegmamba: An eeg foundation model with mamba.Neural Networks, page 107816, 2025
Jiquan Wang, Sha Zhao, Zhiling Luo, Yangxuan Zhou, Shijian Li, and Gang Pan. Eegmamba: An eeg foundation model with mamba.Neural Networks, page 107816, 2025
2025
-
[35]
Deeper- brain: A neuro-grounded eeg foundation model towards universal bci.arXiv preprint arXiv:2601.06134, 2026
Jiquan Wang, Sha Zhao, Yangxuan Zhou, Yiming Kang, Shijian Li, and Gang Pan. Deeper- brain: A neuro-grounded eeg foundation model towards universal bci.arXiv preprint arXiv:2601.06134, 2026
2026
-
[36]
Eeg-fm-audit: A systematic evaluation and analysis pipeline for eeg foundation models.arXiv preprint arXiv:2605.26910, 2026
Xianheng Wang, Yige Yang, and Damien Coyle. Eeg-fm-audit: A systematic evaluation and analysis pipeline for eeg foundation models.arXiv preprint arXiv:2605.26910, 2026
2026 arXiv
-
[37]
Eeg-bench: A benchmark for eeg foundation models in clinical applications.arXiv preprint arXiv:2512.08959, 2025
Ard Kastrati, Josua Bürki, Jonas Lauer, Cheng Xuan, Raffaele Iaquinto, and Roger Wattenhofer. Eeg-bench: A benchmark for eeg foundation models in clinical applications.arXiv preprint arXiv:2512.08959, 2025
2025
-
[38]
Neural- bench: A unifying framework to benchmark neuroai models.arXiv preprint arXiv:2605.08495, 2026
Hubert Banville, Stéphane d’Ascoli, Simon Dahan, Jérémy Rapin, Marlène Careil, Yohann Benchetrit, Jarod Lévy, Saarang Panchavati, Antoine Ratouchniak, Elisa Cascardi, et al. Neural- bench: A unifying framework to benchmark neuroai models.arXiv preprint arXiv:2605.08495, 2026
2026 arXiv
-
[39]
Clef: Eeg foundation model for learning clinical semantics.arXiv preprint arXiv:2605.10817, 2026
Peng Cao, Ali Mirzazadeh, Jong Woo Lee, Aleksandar Videnovic, and Dina Katabi. Clef: Eeg foundation model for learning clinical semantics.arXiv preprint arXiv:2605.10817, 2026
2026 arXiv
-
[40]
Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023
2023
-
[41]
Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36: 68539–68551, 2023
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36: 68539–68551, 2023
2023
-
[42]
Neurochat: A neuroadaptive ai chatbot for customizing learning experiences
Dünya Baradari, Nataliya Kosmyna, Oscar Petrov, Rebecah Kaplun, and Pattie Maes. Neurochat: A neuroadaptive ai chatbot for customizing learning experiences. InProceedings of the 7th ACM Conference on Conversational User Interfaces, pages 1–21, 2025
2025
-
[43]
Chatbci: A p300 speller bci leveraging large language models for improved sentence composition in realistic scenarios.arXiv preprint arXiv:2411.15395, 2024
Jiazhen Hong, Weinan Wang, and Laleh Najafizadeh. Chatbci: A p300 speller bci leveraging large language models for improved sentence composition in realistic scenarios.arXiv preprint arXiv:2411.15395, 2024
2024
-
[44]
Isruc-sleep: A comprehensive public dataset for sleep researchers.Computer methods and programs in biomedicine, 124:180–192, 2016
Sirvan Khalighi, Teresa Sousa, José Moutinho Santos, and Urbano Nunes. Isruc-sleep: A comprehensive public dataset for sleep researchers.Computer methods and programs in biomedicine, 124:180–192, 2016. 12
2016
-
[45]
2020 international brain–computer interface competition: A review.Frontiers in human neuroscience, 16:898300, 2022
Ji-Hoon Jeong, Jeong-Hyun Cho, Young-Eun Lee, Seo-Hyun Lee, Gi-Hwan Shin, Young-Seok Kweon, José del R Millán, Klaus-Robert Müller, and Seong-Whan Lee. 2020 international brain–computer interface competition: A review.Frontiers in human neuroscience, 16:898300, 2022
2020
-
[46]
Wei Liu, Jie-Lin Qiu, Wei-Long Zheng, and Bao-Liang Lu. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition.IEEE Transactions on Cognitive and Developmental Systems, 14(2):715–729, 2021
2021
-
[47]
Electroencephalogram (eeg)-based computer-aided technique to diagnose major depressive disorder (mdd).Biomedical Signal Processing and Control, 31: 108–115, 2017
Wajid Mumtaz, Likun Xia, Syed Saad Azhar Ali, Mohd Azhar Mohd Yasin, Muhammad Hussain, and Aamir Saeed Malik. Electroencephalogram (eeg)-based computer-aided technique to diagnose major depressive disorder (mdd).Biomedical Signal Processing and Control, 31: 108–115, 2017
2017
-
[48]
Electroencephalograms during mental arithmetic task performance.Data, 4 (1):14, 2019
Igor Zyma, Sergii Tukaev, Ivan Seleznov, Ken Kiyono, Anton Popov, Mariia Chernykh, and Oleksii Shpenkov. Electroencephalograms during mental arithmetic task performance.Data, 4 (1):14, 2019
2019
-
[49]
Inter-database validation of a deep learning approach for automatic sleep scoring.arXiv preprint arXiv:2009.10365, 2020
Diego Alvarez-Estevez and Roselyne M Rijsman. Inter-database validation of a deep learning approach for automatic sleep scoring.arXiv preprint arXiv:2009.10365, 2020
2009 arXiv
-
[50]
Montreal archive of sleep studies: an open-access resource for instrument benchmarking and exploratory research.Journal of sleep research, 23(6):628–635, 2014
Christian O’reilly, Nadia Gosselin, Julie Carrier, and Tore Nielsen. Montreal archive of sleep studies: an open-access resource for instrument benchmarking and exploratory research.Journal of sleep research, 23(6):628–635, 2014
2014
-
[51]
You snooze, you win: the physionet/computing in cardiology challenge 2018
Mohammad M Ghassemi, Benjamin E Moody, Li-Wei H Lehman, Christopher Song, Qiao Li, Haoqi Sun, Roger G Mark, M Brandon Westover, and Gari D Clifford. You snooze, you win: the physionet/computing in cardiology challenge 2018. In2018 Computing in Cardiology Conference (CinC), vol...
2018
-
[52]
The sleep heart health study: design, rationale, and methods.Sleep, 20(12):1077–1085, 1997
Stuart F Quan, Barbara V Howard, Conrad Iber, James P Kiley, F Javier Nieto, George T O’Connor, David M Rapoport, Susan Redline, John Robbins, Jonathan M Samet, et al. The sleep heart health study: design, rationale, and methods.Sleep, 20(12):1077–1085, 1997
1997
-
[53]
A large finer-grained affective computing eeg dataset.Scientific Data, 10(1):740, 2023
Jingjing Chen, Xiaobin Wang, Chen Huang, Xin Hu, Xinke Shen, and Dan Zhang. A large finer-grained affective computing eeg dataset.Scientific Data, 10(1):740, 2023
2023
-
[54]
Refed: a subject real-time dynamic labeled eeg-fnirs synchronized recorded emotion dataset.Advances in Neural Information Processing Systems, 38, 2026
Xiaojun Ning, Jing Wang, Zhiyang Feng, Tianzuo Xin, Shuo Zhang, Shaoqi Zhang, Zheng Lian, Yi Ding, Youfang Lin, and Ziyu Jia. Refed: a subject real-time dynamic labeled eeg-fnirs synchronized recorded emotion dataset.Advances in Neural Information Processing Systems, 38, 2026
2026
-
[55]
Open multi-session and multi-task eeg cognitive dataset for passive brain- computer interface applications.Scientific Data, 10(1):85, 2023
Marcel F Hinss, Emilie S Jahanpour, Bertille Somon, Lou Pluchon, Frédéric Dehais, and Raphaëlle N Roy. Open multi-session and multi-task eeg cognitive dataset for passive brain- computer interface applications.Scientific Data, 10(1):85, 2023
2023
-
[56]
Multimodal phenotyping dataset of driving fatigue.Scientific Data, 2026
Jiayi Li, Cong Fu, Jinbu Tang, Ligang Zhou, Hao Chen, Wei Zhou, Chen Chen, and Jingchun Luo. Multimodal phenotyping dataset of driving fatigue.Scientific Data, 2026
2026
-
[57]
Seed-vii: A multimodal dataset of six basic emotions with continuous labels for emotion recognition.IEEE Transactions on Affective Computing, 16(2):969–985, 2024
Wei-Bang Jiang, Xuan-Hao Liu, Wei-Long Zheng, and Bao-Liang Lu. Seed-vii: A multimodal dataset of six basic emotions with continuous labels for emotion recognition.IEEE Transactions on Affective Computing, 16(2):969–985, 2024
2024
-
[58]
Deap: A database for emotion analysis; using physiological signals.IEEE transactions on affective computing, 3(1): 18–31, 2011
Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. Deap: A database for emotion analysis; using physiological signals.IEEE transactions on affective computing, 3(1): 18–31, 2011
2011
-
[59]
Simultaneous acquisition of eeg and nirs during cognitive tasks for an open access dataset.Scientific data, 5(1):180003, 2018
Jaeyoung Shin, Alexander V on Lühmann, Do-Won Kim, Jan Mehnert, Han-Jeong Hwang, and Klaus-Robert Müller. Simultaneous acquisition of eeg and nirs during cognitive tasks for an open access dataset.Scientific data, 5(1):180003, 2018
2018
-
[60]
A multimodal approach to estimating vigilance using eeg and forehead eog.arXiv preprint arXiv:1611.08492, 2016
Wei-Long Zheng and Bao-Liang Lu. A multimodal approach to estimating vigilance using eeg and forehead eog.arXiv preprint arXiv:1611.08492, 2016. 13
2016 arXiv
-
[61]
Qwen3.5: Towards native multimodal agents
Qwen Team. Qwen3.5: Towards native multimodal agents. Technical report, February 2026. URLhttps://qwen.ai/blog?id=qwen3.5
2026
-
[62]
Qwen3.7: The agent frontier
Qwen Team. Qwen3.7: The agent frontier. Technical report, May 2026. URL https: //qwen.ai/blog?id=qwen3.7
2026
-
[63]
Qwen3.7-Plus: Multimodal agent intelligence
Qwen Team. Qwen3.7-Plus: Multimodal agent intelligence. Technical report, May 2026. URL https://qwen.ai/blog?id=qwen3.7-plus
2026
-
[64]
Glm-5: from vibe coding to agentic engineering
Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026
2026 arXiv
-
[65]
Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026
Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026
2026
-
[66]
Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026
2026 arXiv
-
[67]
The minimax-m2 series: Mini activations unleashing max real-world intelligence.arXiv preprint arXiv:2605.26494, 2026
Aili Chen, Aonian Li, Baichuan Zhou, Bangwei Gong, Binyang Jiang, Boji Dan, Changqing Yu, Chao Wang, Cheng Ma, Cheng Zhong, et al. The minimax-m2 series: Mini activations unleashing max real-world intelligence.arXiv preprint arXiv:2605.26494, 2026
2026 arXiv
-
[68]
GPT-5.6 system card
OpenAI. GPT-5.6 system card. System card, July 2026. URL https:// deploymentsafety.openai.com/gpt-5-6/gpt-5-6.pdf
2026
-
[69]
Gemini 3.6 Flash model card
Google DeepMind. Gemini 3.6 Flash model card. Model card, July 2026. URL https: //deepmind.google/models/model-cards/gemini-3-6-flash/
2026
-
[70]
meta_info
Anthropic. Claude Opus 5 system card. System card, July 2026. URL https://www. anthropic.com/system-cards. 14 A DATASETSTATISTICS BrainBench incorporates 17 unique datasets spanning research-oriented EEG recordings, clinical polysomnography, neurocognitive experiments, and mul...
2026
-
[71]
delta global mean PLV
-
[72]
theta global mean PLV
-
[73]
alpha global mean PLV
-
[74]
beta global mean PLV
-
[75]
dominant synchronization band ### STRICT CONSTRAINTS (MANDATORY)
-
[76]
Output ONLY a valid JSON object
-
[77]
DO NOT include Markdown code blocks
-
[78]
DO NOT include any conversational text or explanations
-
[79]
delta_global_mean_plv
Keys must be EXACTLY "delta_global_mean_plv", "theta_global_mean_plv", " alpha_global_mean_plv", "beta_global_mean_plv", and "dominant_synchronization_band"
-
[80]
The four PLV values must be float or null
-
[81]
dominant_synchronization_band
"dominant_synchronization_band" must be one of "delta", "theta", "alpha", "beta", or null. ### OUTPUT TEMPLATE {"delta_global_mean_plv": <float|null>, "theta_global_mean_plv": <float|null>, " alpha_global_mean_plv": <float|null>, "beta_global_mean_plv": <float|null>, " dominan...
-
[82]
The code will run in a persistent Python kernel and the output will be returned as an Observation
Execute Python code by enclosing it in <execute>...</execute>. The code will run in a persistent Python kernel and the output will be returned as an Observation. Top-level variables from earlier snippets remain available
-
[83]
input/HR.npy
When the task is complete, provide the requested final report enclosed in <solution>...</solution>. The text inside <solution> is returned to the user, so it must follow the task’s requested output format and contain all results. Use code to inspect the provided files instead ...
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.