REVIEW 4 major objections 5 minor 34 references
A Modular Workflow for Multimodal Reading Experiments
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper presents a modular workflow that synchronizes eye tracking, EEG, and interaction data to link gaze to words, sentences, and areas of interest in freely chosen online text.
desk verdict A genuinely integrative pipeline for naturalistic reading studies, but the 'general basis' claim outruns the evidence; conditional acceptance with code, data, and tempered language would make it solid. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the linking of the three data types in both time and space. A common-clock synchronization protocol records eye tracker, EEG, and input-device streams into one file; a browser plug-in maps raw gaze coordinates to the displayed text and emits word-fixation onsets; the EEG stream is segmented on those onsets and cleaned with artifact rejection and time-frequency analysis; and a linguistic feature extractor annotates the read text. The workflow's reusability rests on the modularity of these components, each of which can be swapped or extended.
What would settle it
Run the complete pipeline on a set of known web pages read by participants whose fixations are manually annotated word-by-word, compare the pipeline's fixation-to-word assignments with the manual annotations, and compute the fraction of fixation onsets assigned to a neighboring word or sentence; if that fraction is non-trivial or grows with smaller fonts or dynamic layouts, the fixation-locked EEG results cannot be trusted.
Extended reading notes
Core claim
The central claim is that a modular pipeline can integrate real-time gaze-to-text mapping with post-hoc fixation-triggered EEG segmentation, and that this integration makes multimodal experiments on self-selected online reading practical rather than confined to tightly controlled trial-based paradigms. In the reported use case, participants search and read news on two topics; sentences are selected for follow-up ratings based on the longest fixation durations or the strongest theta-band power increases after first-fixation onset. The paper presents the setup as both a concrete study environment and a general template: the sensor streams, analysis scripts, and selection criteria are replaceable, so the same structure can be adapted to emotion, confusion, comprehension, or language-learning questions.
Load-bearing premise
The load-bearing premise is that the eye tracker's gaze-to-word assignment remains accurate enough in real web pages that every word-fixation onset marks the correct text; if a substantial share of fixations land on the wrong word or sentence, the EEG epochs, the selected sentences, and the neural conclusions would all be distorted.
Editorial extensions
If this is right
- Gaze metrics can be computed live during a reading session and immediately used to select sentences for post-task ratings, closing the loop between measurement and follow-up questions in one visit.
- EEG can be segmented on the first fixation of each word in free reading, so neural responses such as theta power changes or fixation-related potentials can be tied to specific linguistic content without predefined trials.
- Because the data are synchronized to a common clock, mouse and keyboard events can be aligned with gaze and brain signals, enabling analyses of search behavior alongside reading.
- With the sensors and analysis modules replaceable, the same workflow can be adapted to study topics beyond selective exposure, such as emotion during reading, comprehension difficulty, or foreign-language processing.
- Linguistic annotation of the read text gives researchers partial control over uncontrolled naturalistic stimuli, allowing them to account for text properties in later analysis.
Reading between the lines
- One open risk is that the gaze-to-word assignment used as the backbone has been validated on its own but not as part of the full pipeline; if fixation onsets are assigned to the wrong word at sentence boundaries, the EEG epochs and the sentences selected for rating would shift.
- We would add that a stability analysis of the sentence-selection criteria across thresholds would show whether the same passages are consistently flagged, since different plausible criteria might select different sentence subsets.
- The architecture could support fully adaptive experiments in which EEG features computed between sessions change the stimuli shown in later tasks, because the components already compute metrics in near-real-time.
- For low-density mobile EEG settings, the paper itself notes that artifact correction is less reliable, so an extension using the available eye-openness signal for regression-based ocular correction may be needed before the workflow transfers to mobile setups.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a modular, web-based workflow for conducting naturalistic online-reading experiments with synchronized eye tracking, EEG, and interaction data. The workflow uses Lab Streaming Layer (LSL) for temporal alignment, the EyeLiveMetrics browser plugin for gaze-to-word mapping, and MNE-Python for EEG preprocessing and fixation-triggered epoching. A use case on selective exposure in online news reading is described, in which sentences are selected for post-task self-report based on fixation duration and theta-band power changes. The manuscript claims that the workflow 'provides a general basis for multimodal research on reading' and related cognitive processes.
Significance. If the workflow were demonstrated to be reliable end-to-end, it would be a valuable contribution to the growing literature on co-registered eye-tracking and EEG in naturalistic reading, particularly because it targets real-time or near-real-time integration and within-session stimulus selection. The use of existing, open components (LSL, MNE-Python, EyeLiveMetrics) and the modular design are sensible and potentially reusable. However, the paper's central claim is currently supported only by component-level validation of the gaze-to-word mapping from prior work; there is no empirical evidence that the integrated pipeline actually runs as described, and the Discussion itself concedes that accurate fixation detection is essential but does not quantify its accuracy in this setup. The significance therefore depends on the outcome of additional validation that is not present in the manuscript.
major comments (4)
- [§5 and §6] The central claim that the workflow 'provides a general basis for multimodal research on reading' (Abstract, also echoed in §6 and §7) is not supported by any end-to-end validation. Section 5, which is presented as a use case, describes hardware, procedures, and selection criteria but reports no number of participants, no indication of whether the pipeline ran to completion for any session, no fixation-to-word agreement rates, no EEG epoch yields, and no output metrics of any kind. The Discussion in §6 explicitly states that 'Accurate fixation detection is essential whenever downstream analyses depend on fixation-locked windows,' yet no accuracy evidence for the integrated pipeline is provided. As written, the paper demonstrates an architecture, not a working workflow.
- [§3.4, §3.5, and §6] The load-bearing premise for fixation-triggered EEG segmentation is that the EyeLiveMetrics gaze-to-word mapping is correct at the level of individual fixations. The cited validation in [9] reports Pearson correlations 'of ≤0.96 across all fixation metrics' against Tobii Pro Lab, but aggregate fixation-metric correlations do not establish per-fixation word-assignment agreement. Systematic misassignments, such as a consistent one-word offset around short or high-frequency words, can leave aggregate correlations high while substantially biasing the word-level epochs described in §3.5 and the sentence-selection criteria in §5 (longest fixation duration; theta power in the 500–1000 ms and 1000–1500 ms windows). The manuscript needs to report a per-fixation or per-word assignment accuracy measure, or at minimum justify why the aggregate validation suffices.
- [§3.3, §3.5] The temporal alignment between EyeLiveMetrics's gaze output and the LSL-recorded EEG stream is not specified. Section 3.5 asserts that 'as gaze and EEG data are aligned with LSL, we extract the fixation onset for each fixated word,' but it is unclear whether EyeLiveMetrics fixations are timestamped in the LSL clock, whether they are later re-aligned, and whether any clock-offset correction is applied. Without this information, the synchronization claim is an assumption rather than a demonstrated property of the system. The paper should describe the exact timestamping path from the raw gaze samples to the word-level fixation onsets used for epoching.
- [§2.4] The novelty claim that 'no existing approach integrates the co-registration of EEG, gaze, and other sensor data with a real-time analysis pipeline for naturalistic online reading' is stated without a systematic comparison to existing co-registration toolkits and pipelines (e.g., Dimigen et al. [4] and subsequent work on fixation-related potentials in natural reading). If the claim is intended to cover only the specific combination of browser-DOM gaze mapping plus real-time feedback selection, it should be scoped more precisely; otherwise, the manuscript overstates its novelty relative to the cited literature.
minor comments (5)
- [§3.2, Figure 2 caption] In the Figure 2 caption, 'extend of user reported feedback' should be 'extent of user-reported feedback'.
- [§1, last paragraph of Intro] The sentence beginning 'By combining multimodal co-registration...' is long and lists several benefits; it would be clearer split into two sentences.
- [§5, Pre-experiment] The phrase 'we collected participants’ general attitude' should be 'we measured participants’ general attitudes' for clarity, and the scale reference [24] appears to be a book on political attitudes; please clarify whether this is the actual instrument used.
- [§5, Post-Experiment] The sentence 'Using these criteria, a maximum of 60 sentences were rated per participant' is ambiguous about whether the maximum was reached in practice; report the actual distribution or state that no data are reported.
- [§2.1, footnote 1] The footnote says 'Only two out of seven datasets (SEED [33, 5], and SEED-IV) include gaze information,' but the main text says 'only two datasets include eye-tracking data (SEED-IV [32] and MAHNOB-HCI [29])'. These two counts are inconsistent and should be reconciled.
Circularity Check
No circular derivation; reliance on the authors' prior component validation is not load-bearing in a circular sense.
full rationale
The paper is a systems/workflow description rather than a derivation with fitted parameters or predictions. The central claim is that the pipeline integrates eye tracking, EEG, and interaction streams via Lab Streaming Layer and links gaze to browser-based text at word, sentence, and AOI levels. Each integration step is described from external building blocks (LSL, MNE-Python, ICLabel, elfen) and the authors' own EyeLiveMetrics plugin. The validation of EyeLiveMetrics is cited from prior peer-reviewed work [9]; although the author sets overlap, that citation is component-level external evidence and not an input to which this paper's central claim reduces. The theta-band selection windows are motivated by literature and by the authors' earlier CHI abstract [21], but they are presented as one configurable example criterion rather than as a predicted result derived in this paper. No equation or metric is defined in terms of the outcome it is claimed to predict; the use case reports no fitted parameters and no quantitative claim whose value is forced by construction. The acknowledged limitations, such as gaze-to-word assignment noise propagating into EEG epoching, concern validation completeness rather than circularity. Consequently, no specific circular step can be quoted, and the appropriate score is at the low end of the non-circular range.
Assumptions & free parameters
assumptions (4)
- domain assumption Lab Streaming Layer provides accurate temporal synchronization of eye tracking, EEG, and interaction streams.
- domain assumption EyeLiveMetrics fixation detection and word-level gaze mapping are accurate enough for fixation-locked EEG epoching.
- domain assumption ICA with ICLabel can remove ocular and muscle artifacts from low-density EEG sufficiently to preserve fixation-related potentials.
- domain assumption Theta band power increases in the 500-1500 ms window after fixation onset reflect memory and evaluative processing.
Cite this review
Pith. "Pith review of A Modular Workflow for Multimodal Reading Experiments." pith.science (2026). https://pith.science/paper/PGTV4ASR
@misc{pith2026260805966,
author = {Pith},
title = {Pith review of: A Modular Workflow for Multimodal Reading Experiments},
year = {2026},
howpublished = {\url{https://pith.science/paper/PGTV4ASR}},
note = {Machine review of arXiv:2608.05966}
}
read the original abstract
We introduce a web-based modular workflow for real-time multimodal experiments in naturalistic online reading. The workflow integrates eye tracking, EEG, and interaction data from mouse and keyboard, synchronizes them via Lab Streaming Layer, and links gaze to browser-based text at the word, sentence, and AOI levels. It is designed as a reusable experimental procedure that can be adapted to different sensors, tasks, and analysis goals. As a use case, we apply the workflow to a study of selective exposure in online news search and reading. During the experiment, gaze-derived measures are computed online, while EEG and other synchronized streams are processed immediately after task sessions based on fixation-triggered segmentation. The resulting behavioral, neural, and linguistic metrics support selecting text passages for targeted post-task rating or labelling within the same lab session. The workflow thus provides a general basis for multimodal research on reading and related cognitive processes, and supports the empirical validation, in ecological contexts, of constructs that are typically operationalised through self-report measures.
Figures
Reference graph
Works this paper leans on
-
[9]
Hienert, D., Schmidt, H., Kr ¨amer, T., Kern, D., 2024. Eyelivemetrics: Real-time analysis of online reading with eye tracking, in: Proceedings of the 2024 Symposium on Eye Tracking Research and Applications, Association for Computing Machinery, New York, NY , USA. pp. 1–7. URL:https://doi.org/10.1145/3649902.3656495, doi:10.1145/3649902.3656495
-
[4]
Coregistration of eye movements and eeg in natural reading: analyses and review
Dimigen, O., Sommer, W., Hohlfeld, A., Jacobs, A.M., Kliegl, R., 2011. Coregistration of eye movements and eeg in natural reading: analyses and review. Journal of experimental psychology: General 140, 552
work page 2011
-
[1]
Ablin, P., Cardoso, J.F., Gramfort, A., 2018. Faster ica under orthogonal constraint, in: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE Press, Calgary, AB, Canada. pp. 4464–4468
work page 2018
-
[2]
Boonprakong, N., Tag, B., Goncalves, J., Dingler, T., 2025. How do hci researchers study cognitive biases? a scoping review, in: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY , USA. pp. 1–20. URL:https://doi.org/10.1145/3706598.3713450, doi:10.1145/3706598.3713450
arXiv 2025
-
[3]
Optimizing the ica-based removal of ocular eeg artifacts from free viewing experiments
Dimigen, O., 2020. Optimizing the ica-based removal of ocular eeg artifacts from free viewing experiments. NeuroImage 207, 116117. URL:https://www.sciencedirect.com/science/article/pii/S1053811919307086, doi:https://doi.org/10. 1016/j.neuroimage.2019.116117
-
[5]
Duan, R.N., Zhu, J.Y ., Lu, B.L., 2013. Differential entropy feature for EEG-based emotion classification, in: 6th International IEEE/EMBS Conference on Neural Engineering (NER), IEEE. pp. 81–84
work page 2013
-
[6]
Ehinger, B.V ., Dimigen, O., 2019. Unfold: an integrated toolbox for overlap correction, non-linear modeling, and regression-based eeg analysis. PeerJ 7, e7838. URL:https://doi.org/10.7717/peerj.7838, doi:10.7717/peerj.7838
-
[7]
Fischer, P., Reinweber, M., V ogrincic, C., Sch ¨afer, A., Schienle, A., V olberg, G., 2013. Neural mechanisms of selective exposure: An eeg study on the processing of decision-consistent and inconsistent information. International Journal of Psychophysiology 87, 13–18. doi:https://doi.org/10.1016/j.ijpsycho.2012.10.011
Show all 34 references
-
[8]
MEG and EEG data analysis with MNE-Python
Gramfort, A., Luessi, M., Larson, E., Engemann, D.A., Strohmeier, D., Brodbeck, C., Goj, R., Jas, M., Brooks, T., Parkkonen, L., H¨am¨al¨ainen, M.S., 2013. MEG and EEG data analysis with MNE-Python. Frontiers in Neuroscience 7, 1–13. doi:10.3389/fnins.2013.00267
2013
-
[10]
Emotion recog- nition in eeg signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., Garc ´ıa, D.L., Gorriz, J.M., Acharya, U.R., 2023. Emotion recog- nition in eeg signals using deep learning methods: A review. Computers in Biology and Medicine 165, 107450. URL:https:// www.sciencedirect.co...
2023 doi
-
[11]
Autoreject: Automated artifact rejection for meg and eeg data
Jas, M., Engemann, D.A., Bekhti, Y ., Raimondo, F., Gramfort, A., 2017. Autoreject: Automated artifact rejection for meg and eeg data. NeuroImage 159, 417–429. URL:https://www.sciencedirect.com/science/article/pii/S1053811917305013, doi:https:// doi.org/10.1016/j.neuroimage.20...
2017 doi
-
[12]
A theory of reading: from eye fixations to comprehension
Just, M.A., Carpenter, P.A., 1980. A theory of reading: from eye fixations to comprehension. Psychological review 87, 329
1980
-
[13]
Eeg alpha and theta oscillations reflect cognitive and memory performance: a review and analysis
Klimesch, W., 1999. Eeg alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain research reviews 29, 169–195
1999
-
[14]
Theta synchronization and alpha desynchronization in a memory task
Klimesch, W., Doppelmayr, M., Schimke, H., Ripper, B., 1997. Theta synchronization and alpha desynchronization in a memory task. Psychophysiology 34, 169–176
1997
-
[15]
Theta oscillations and the erp old/new effect: inde- pendent phenomena? Clinical Neurophysiology 111, 781–793
Klimesch, W., Doppelmayr, M., Schwaiger, J., Winkler, T., Gruber, W., 2000. Theta oscillations and the erp old/new effect: inde- pendent phenomena? Clinical Neurophysiology 111, 781–793. URL:https://www.sciencedirect.com/science/article/pii/ S1388245700002546, doi:https://doi....
2000 doi
-
[16]
Episodic and semantic memory: an analysis in the eeg theta and alpha band
Klimesch, W., Schimke, H., Schwaiger, J., 1994. Episodic and semantic memory: an analysis in the eeg theta and alpha band. Electroen- cephalography and clinical Neurophysiology 91, 428–441
1994
-
[17]
Choice and preference in media use: Advances in selective exposure theory and research
Knobloch-Westerwick, S., 2014. Choice and preference in media use: Advances in selective exposure theory and research. Routledge, London. 9
2014
-
[18]
Investigating the influence of rsvp display parameters on working memory load using electroen- cephalography
Kosch, T., Schmidt, A., Chuang, L., 2019. Investigating the influence of rsvp display parameters on working memory load using electroen- cephalography. URL:https://neuroadaptive.org/wp-content/uploads/2022/07/NAT19_Programme.pdf#page=24
2019
-
[19]
Kosch, T., Schmidt, A., Thanheiser, S., Chuang, L.L., 2020. One does not simply rsvp: Mental workload to select speed reading parameters using electroencephalography, in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Association for Computing Ma...
2020
-
[20]
The lab streaming layer for synchronized multimodal recording
Kothe, C., Shirazi, S.Y ., Stenner, T., Medine, D., Boulay, C., Grivich, M.I., Mullen, T., Delorme, A., Makeig, S., 2024. The lab streaming layer for synchronized multimodal recording. doi:10.1101/2024.02.13.580071
2024 doi
-
[21]
Kr ¨amer, T., Hienert, D., Chiossi, F., Kosch, T., Kern, D., 2025. Escaping the filter bubble: Evaluating electroencephalographic theta band synchronization as indicator for selective exposure in online news reading, in: Proceedings of the Extended Abstracts of the CHI Confere...
2025
-
[22]
Kr ¨amer, T., Kern, D., Kosch, T., Hienert, D., 2026. Topic matters: How linguistic properties can shape reading behaviour in selective exposure studies, in: Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, Association for Comp...
2026
-
[23]
Mne-icalabel: Automatically annotating ica components with iclabel in python
Li, A., Feitelberg, J., Saini, A.P., H ¨ochenberger, R., Scheltienne, M., 2022. Mne-icalabel: Automatically annotating ica components with iclabel in python. Journal of Open Source Software 7, 4484. doi:10.21105/joss.04484
2022 doi
-
[24]
Triggerpunkte
Mau, S., Lux, T., Westheuser, L., 2023. Triggerpunkte. Suhrkamp Verlag, Berlin, Germany
2023
-
[25]
elfen: A python package for efficient linguistic feature extraction for natural language datasets, in: Croce, D., Leidner, J., Moosavi, N.S
Maurer, M., 2026. elfen: A python package for efficient linguistic feature extraction for natural language datasets, in: Croce, D., Leidner, J., Moosavi, N.S. (Eds.), Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (V...
2026 doi
-
[26]
A review on eeg-based multimodal learning for emotion recognition
Pillalamarri, R., Shanmugam, U., 2025. A review on eeg-based multimodal learning for emotion recognition. Artificial Intelligence Review 58, 131. URL:https://doi.org/10.1007/s10462-025-11126-9, doi:10.1007/s10462-025-11126-9
2025 doi
-
[27]
Iclabel: An automated electroencephalographic independent component classifier, dataset, and website
Pion-Tonachini, L., Kreutz-Delgado, K., Makeig, S., 2019. Iclabel: An automated electroencephalographic independent component classifier, dataset, and website. NeuroImage 198, 181–197. doi:10.1016/j.neuroimage.2019.05.026
2019 doi
-
[28]
Big Five Inventory (BFI-10)
Rammstedt, B., Kemper, C.J., Klein, M.C., Beierlein, C., Kovaleva, A., 2014. Big Five Inventory (BFI-10). Technical Report. GESIS Leibniz Institute for the Social Sciences. URL:https://zis.gesis.org/DoiId/zis76, doi:10.6102/zis76
2014 doi
-
[29]
A multimodal database for affect recognition and implicit tagging
Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M., 2011. A multimodal database for affect recognition and implicit tagging. IEEE transac- tions on affective computing 3, 42–55
2011
-
[30]
Adolescents environmental emotion perception by integrating eeg and eye movements
Su, Y ., Li, W., Bi, N., Lv, Z., 2019. Adolescents environmental emotion perception by integrating eeg and eye movements. Frontiers in Neurorobotics 13. doi:10.3389/fnbot.2019.00046
2019
-
[31]
Zhao, L.M., Li, R., Zheng, W.L., Lu, B.L., 2019. Classification of five emotions from eeg and eye movement signals: Complementary representation properties, in: 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 611–614. doi:10.1109/ NER.2019.8717055
2019
-
[32]
Emotionmeter: A multimodal framework for recognizing human emotions
Zheng, W., Liu, W., Lu, Y ., Lu, B., Cichocki, A., 2018. Emotionmeter: A multimodal framework for recognizing human emotions. IEEE Transactions on Cybernetics , 1–13doi:10.1109/TCYB.2018.2797176
2018
-
[33]
Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks
Zheng, W.L., Lu, B.L., 2015. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development 7, 162–175. doi:10.1109/TAMD.2015.2431497
2015
-
[34]
Detecting reading-induced confusion using eeg and eye tracking
Zhuang, H., Baradari, D., Kosmyna, N., Balyan, A., Albrecht, C., Chen, S., Maes, P., 2025. Detecting reading-induced confusion using eeg and eye tracking. URL:https://arxiv.org/abs/2508.14442,arXiv:2508.14442. 10
2025 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.