REVIEW 3 major objections 5 minor 17 references
LabelECG: A Web-based Tool for Distributed Electrocardiogram Annotation
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read LabelECG is a browser-based tool that let four physicians annotate about 15,000 ECG records in three months, with the labels supporting a clinical AI competition.
desk verdict A useful tool paper that currently can't be checked: the authors need to ship the artifact before the central claim that LabelECG works as described should be taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the three-system web architecture. A user-management system stores accounts and controls registration through administrator-issued verification codes; an annotation-management system stores ECG files, parameters, automatic diagnoses and user labels; and a visualization service renders multi-lead ECG traces in the browser. The annotation workflow is built around the confirm/unsure buttons, which push a decision to a diagnosis database and advance to the next record, plus review rights for expert accounts. This division lets dozens of remote annotators work on one dataset simultaneously and lets experts police label quality without any annotator touching raw files.
What would settle it
Release the tool, invite several users to annotate the same ECG cohort concurrently in a browser session, and check that every confirm/unsure decision is stored and visible to the expert reviewer. If annotations are lost, users cannot access the same dataset simultaneously, or the tool cannot be deployed on a local network as promised, the central claim of a functioning distributed annotation system is refuted.
Extended reading notes
Core claim
LabelECG is a client-server system with three cooperating parts: user management, annotation management, and ECG visualization. Multiple users register, choose a dataset, view ECG traces in the browser, and record diagnoses, marking each record as "confirmed" or "unsure"; expert accounts can review and revise all annotations. The paper reports that four physicians used the system to annotate about 15,000 records in about three months and that this effort "guaranteed the success" of the competition it was built for.
Load-bearing premise
The paper's claim depends on the reported implementation actually providing the described multi-user annotation, data-management, and review functions without critical defects, since no runnable code or independent test is supplied.
Editorial extensions
If this is right
- If LabelECG works as reported, research groups can assemble large labeled ECG datasets by recruiting clinicians to annotate from any device, without building custom data-management infrastructure.
- The reported throughput implies a small team can label on the order of 1,250 records per physician per month for 12-lead resting ECGs, making large-cohort annotation projects feasible.
- The tool's design, with local deployment on a private network plus browser access, means hospitals concerned about data sharing can still run distributed annotation.
- Because the current version focuses on diagnosis-level labels, the promised additions of beat and fiducial-point annotation would extend the same infrastructure to more granular ECG labeling tasks.
Reading between the lines
- The authors compare LabelECG to an image-annotation tool that enabled a famous large image dataset; by analogy, if the promised public release is reliable, LabelECG could play a similar role for ECG deep learning, but the paper itself does not claim to have created such a dataset.
- The paper does not report inter-annotator agreement or label-quality metrics; a natural extension would be to measure agreement between annotators and against the expert review to see whether distributed browser annotation preserves clinical-grade accuracy.
- The causal statement that the annotations "guaranteed the success" of the competition is asserted rather than demonstrated; an independent replication of the workflow on a smaller cohort would test whether the tool alone, rather than the doctors' expertise, drives throughput.
- Since the tool is said to be freely accessible, a testable consequence is that a third party could run the released version and reproduce a small-scale multi-user annotation session; if the release does not happen or the session fails, the paper's central claim rests on an unrepeatable deployment.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents LabelECG, a web-based tool for distributed annotation of electrocardiograms (ECGs). It describes the tool's functions, including user registration, dataset selection, ECG visualization, diagnosis annotation, and personal/peer review, and it details a three-system architecture (user management, annotation management, and a Lightwave-based visualization system). The authors report that LabelECG was deployed with doctors from four hospitals to annotate about 15,000 12-lead resting ECG records in about three months, and that the resulting annotations supported the First China ECG Intelligent Competition. The paper states that LabelECG will be made freely accessible on the Internet, but it provides no URL, code, demo, or independent evaluation.
Significance. If the described functionality and deployment are real, LabelECG would address a real need: building large, carefully annotated ECG datasets for deep learning applications. The paper's strengths include a plausible architecture built around the established Lightwave viewer, docker-based deployment that supports local-area-network use for privacy-sensitive data, and a concrete real-world deployment context (the competition). However, the paper provides no runnable artifact, no demo URL, no quantitative evaluation, no inter-annotator agreement statistics, and no system logs; the central claims therefore rest entirely on the authors' narrative, architecture diagrams, and screenshots. As a systems/tool paper, this lack of verifiability is a significant weakness that must be addressed before the claims can be accepted.
major comments (3)
- [Section 3.2, Annotation Management System] The concurrency control in the distributed annotation workflow is underspecified. The text says that after the first-time annotation, "the system automatically shows the record next to the ones they previously annotated" (Section 3.2), implying a per-user private sequence, but no locking, record-assignment, or conflict-resolution protocol is described. Multiple annotators could therefore be assigned the same record independently, and the paper does not explain how duplicate or conflicting annotations are prevented or reconciled. Because "distributed" and "simultaneously" annotation is a central claim of the paper, please specify the actual concurrency mechanism (e.g., per-record queue, lease/lock, optimistic concurrency, or a later review pass) and provide evidence from the implementation that it works.
- [Section 4, Competition deployment] The main quantitative claim that about 15,000 records were annotated in about three months and that this "guaranteed the success of the competition" is supported only by self-report. No system logs, timestamps, per-user annotation counts, reviewer statistics, or quality metrics (e.g., inter-annotator agreement against a gold standard) are provided. Readers cannot check whether the annotation throughput was plausible or whether the resulting labels were accurate. At minimum, the paper should report the number of annotators, the per-record or per-week annotation rate, the distribution of "confirmed" vs. "unsure" labels, and any validation of the expert-review process.
- [Abstract and Section 1] The paper promises that LabelECG "will be freely accessible on the Internet" (Abstract) and "will be accessible online" (Section 1), but no URL, repository, or demo is given anywhere in the text or references. For a systems paper, the software itself is the primary contribution; without an accessible artifact, the claimed functionality is unverifiable. Please provide a link to the source code and/or an online demo, or if access is intentionally restricted, give a detailed, reproducible description of the deployment environment and a precise specification of the architecture so that others could reimplement the system.
minor comments (5)
- [Abstract] The abstract contains the misspelling "La-belECG"; the tool name should be "LabelECG" throughout.
- [Section 4] "With the assistances of LabelECG" is grammatically incorrect; it should be "With the assistance of LabelECG."
- [Table 1] The Functions column for EcgEditor and BSS_ECG says "diagnose annotating", which is inconsistent and likely a typo for "diagnosis annotating" or "diagnostic annotation"; please unify the terminology across all rows.
- [Section 3.2] "the first-time users need to choose the first data to annotate" should be rephrased to "the first record to annotate" to maintain consistent terminology and improve readability.
- [References] The reference list has inconsistent capitalization (e.g., "nature" for the journal Nature) and several incomplete entries; please align all references with a consistent style (e.g., ensuring journal names are capitalized and page numbers are present).
Circularity Check
No circularity: LabelECG is a systems/tool paper with no derivation whose claims reduce to their own inputs.
full rationale
This is a systems paper describing the design, implementation, and deployment of a web-based ECG annotation tool. There is no mathematical derivation, no fitted parameter, and no predictive model whose output is constructed from its input. The central claim is that LabelECG is a functioning distributed annotation tool that supported roughly 15,000 records being annotated by doctors from four hospitals in about three months. That claim is supported by the architecture description in Section 3, the function descriptions in Section 3.1, and the deployment narrative in Section 4. No equation is derived from an assumption, and no quantity is renamed or re-imported as a prediction. The paper cites PhysioNet's Lightwave system as an incorporated visualization component, but that is a stated external dependency, not a load-bearing self-citation. The authors' conclusions about the competition's success are self-reported deployment outcomes rather than derived results, so there is no derivation chain in which an output is equivalent to an input by construction. Accordingly, the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The LabelECG architecture (frontend, node.js/Express backend, three databases, Lightwave CGI) is sufficient to support concurrent use by dozens of users without data corruption or performance degradation.
- domain assumption The annotation workflow (confirm/unsure buttons and expert review) produces clinically valid ground truth for training deep learning models.
- domain assumption PhysioNet's Lightwave system correctly renders ECG data on desktop and mobile browsers as claimed.
Cite this review
Pith. "Pith review of LabelECG: A Web-based Tool for Distributed Electrocardiogram Annotation." pith.science (2026). https://pith.science/paper/755UYCZQ
@misc{pith2026190806553,
author = {Pith},
title = {Pith review of: LabelECG: A Web-based Tool for Distributed Electrocardiogram Annotation},
year = {2026},
howpublished = {\url{https://pith.science/paper/755UYCZQ}},
note = {Machine review of arXiv:1908.06553}
}
read the original abstract
Electrocardiography plays an essential role in diagnosing and screening cardiovascular diseases in daily healthcare. Deep neural networks have shown the potentials to improve the accuracies of arrhythmia detection based on electrocardiograms (ECGs). However, more ECG records with ground truth are needed to promote the development and progression of deep learning techniques in automatic ECG analysis. Here we propose a web-based tool for ECG viewing and annotating, LabelECG. With the facilitation of unified data management, LabelECG is able to distribute large cohorts of ECGs to dozens of technicians and physicians, who can simultaneously make annotations through web-browsers on PCs, tablets and cell phones. Along with the doctors from four hospitals in China, we applied LabelECG to support the annotations of about 15,000 12-lead resting ECG records in three months. These annotated ECGs have successfully supported the First China ECG intelligent Competition. La-belECG will be freely accessible on the Internet to support similar researches, and will also be upgraded through future works.
Reference graph
Works this paper leans on
-
[1]
Clifford, G.D., Azuaje, F. and McSharry, P. : Advanced methods and tools for ECG data analysis. Artech house, Boston (2006)
work page 2006
-
[2]
Journal of the American College of Cardiology, 70(9), 1183-1192
Schlä pfer J, Wellens, HJ : Computer-interpreted electrocardiograms: benefits and limit a- tions. Journal of the American College of Cardiology, 70(9), 1183-1192. (2017)
work page 2017
-
[3]
Marquette 12SL ECG Analysis Program , Statement of Validation and Accuracy , http://gehealthcare. com, last accessed 2019/7/13
work page 2019
-
[4]
In Computers in Cardiology, 2005, (pp
Macfarlane PW, Devine B, Clark E : The university of Glasgow (Uni -G) ECG analysis program. In Computers in Cardiology, 2005, (pp. 451-454). IEEE, (2016)
work page 2016
-
[5]
LeCun Y, Bengio Y, Hinton G: Deep learning. nature, 521(7553), 436 (2015)
work page 2015
-
[6]
Journal of the American College of Cardiology
Krittanawong C, Zh ang H, Wang Z, Aydar M, Kitai T : Artificial intelligence in precision cardiovascular medicine. Journal of the American College of Cardiology. 22; 69(21):2657- 64 (2017)
work page 2017
-
[7]
Hannun AY, Rajpurkar P, Haghpanahi M, Tison GH, Bourn C, Turakhia MP, Ng AY : Cardiologist-level arrhythmia detection an d classification in ambulatory electrocardi o- grams using a deep neural network. Nature medicine. 25(1):65 (2019)
work page 2019
-
[8]
Nature medicine 25(1):70 (2019)
Attia ZI, Kapa S, Lopez -Jimenez F, McKie PM, Ladewig DJ, Satam G, Pellikka PA, E n- riquez-Sarano M, Nosewort hy PA, Munger TM, Asirvatham SJ : Screening for cardiac contractile dysfunction using an artificial intelligence –enabled electrocardiogram. Nature medicine 25(1):70 (2019)
work page 2019
Show all 17 references
-
[9]
Internatio nal journal of computer vision , 1;77(1-3):157-73 (2008)
Russell BC, To rralba A, Murphy KP, Freeman WT : LabelMe: a database and web -based tool for image annotation. Internatio nal journal of computer vision , 1;77(1-3):157-73 (2008)
2008
-
[10]
Imagenet: A large -scale hierarchical image database
Deng J, Dong W, Socher R, Li LJ, Li K, Fei -Fei L. Imagenet: A large -scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition , pp. 248-255 (2009)
2009
-
[11]
The First Ch ina ECG Intelligent Competition, http://mdi.ids.tsinghua.edu.cn/#/, last a c- cessed 2019/7/13
2019
-
[12]
SigViewer: Visualizing Multimodal Signals Stored in XDF (Extensible Data Format) Files
Lin Y, Brunner C, Sajda P, Faller J. SigViewer: Visualizing Multimodal Signals Stored in XDF (Extensible Data Format) Files. arXiv preprint arXiv:1708.06333. 2017 Aug 12
2017 arXiv
-
[13]
EcgEditor, https://github.com/Unisens/EcgEditor, last accessed 2019/7/13
2019
-
[14]
ECG Viewer, https://github.com/jramshur/ECG_Viewer, last accessed 2019/7/13
2019
-
[15]
BSS_ECG, https://github.com/AdnanHidic/bss_ecg, last accessed 2019/7/13
2019
-
[16]
WaveformECG: A Platform for Visualizing, Annotat- ing, and Analyzing ECG Data
Winslow RL, Granite S, Jurado C. WaveformECG: A Platform for Visualizing, Annotat- ing, and Analyzing ECG Data. Computing in science & engineering, 26;18(5):36 (2016)
2016
-
[17]
Circulation , 101(23):e215-20 (2000)
Goldberger AL, Amaral LA, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE , Moody GB, Peng CK, Stanley HE : PhysioBank, PhysioToolkit, and PhysioNet : comp o- nents of a new research resource for complex ph ysiologic signals. Circulation , 101(23):e215-20 (2000)
2000
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.