{"id":"7e75f292-4967-41cc-b4e1-68ed93c42e5f","arxiv_id":"1908.06553","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A web-based tool for distributed ECG annotation, reported to have enabled 15,000 records to be labeled by four doctors in three months.","lead":"LabelECG is a web-based tool for distributed ECG annotation, letting multiple doctors view and label heart recordings from any browser. The authors report using it to annotate about 15,000 resting ECGs in three months, supporting a Chinese ECG competition.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No public artifact or independent evidence exists that LabelECG actually implements the described distributed annotation workflow; the central claim is currently unverifiable.","rationale":"The reader identified the absence of a runnable artifact as the weakest assumption, and my reading agrees. The central claim that LabelECG is a working distributed annotation tool, and the quantitative deployment claim of 15,000 records in three months, are supported only by the paper's own descriptions and stated outcomes. There is no independent verification mechanism. I considered whether the underspecified concurrency model in Section 3.2 might be a separate technical defect, but without the artifact it remains an untested design detail rather than a demonstrated flaw. The overclaim in Section 4 that LabelECG 'guaranteed the success of the competition' is rhetorical but minor. Since the reader's conditional verdict already hinges on artifact availability, my concern does not change the recommended action: accept only if the authors supply a working public deployment or a reproducible repository, plus a minimal verification of the annotation workflow.","tokens_in":5358,"tokens_out":3102,"duration_ms":35663,"concrete_test":"Obtain the promised public link or Docker image, deploy LabelECG locally, create two user accounts, upload a small shared dataset (e.g., 100 records), and have both users annotate simultaneously; then inspect the backend database to verify that each record is assigned to exactly one user at a time, all annotations are stored without loss, and the expert review workflow is functional. If the supplied artifact cannot be deployed or concurrent annotation corrupts or duplicates data, the central functional claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LabelECG is a functioning web-based distributed annotation tool that enabled roughly 15,000 ECG records to be annotated in three months (Sections 1 and 4). The only supporting evidence is the architecture diagram (Fig. 3), screenshots (Fig. 2), and the authors' narrative. No source code, demo URL, or public deployment is provided; the paper says LabelECG 'will be freely accessible on the Internet' but gives no address. The deployment statistics are self-reported: there are no system logs, per-user counts, timestamps, or inter-annotator agreement figures to verify the three-month/15,000-record claim. Additionally, Section 3.2 leaves the concurrency model underspecified: the interface shows each user the next unannotated record, but no locking or record-assignment protocol is described, leaving open the possibility that simultaneous annotators duplicate or overwrite work. None of this is a disagreement with scientific consensus; it is simply that the paper's core assertion, that LabelECG works as described, cannot currently be checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LabelECG, a web-based tool for distributed annotation of electrocardiograms (ECGs). It describes the tool's functions, including user registration, dataset selection, ECG visualization, diagnosis annotation, and personal/peer review, and it details a three-system architecture (user management, annotation management, and a Lightwave-based visualization system). The authors report that LabelECG was deployed with doctors from four hospitals to annotate about 15,000 12-lead resting ECG records in about three months, and that the resulting annotations supported the First China ECG Intelligent Competition. The paper states that LabelECG will be made freely accessible on the Internet, but it provides no URL, code, demo, or independent evaluation.","tokens_in":5501,"tokens_out":3913,"duration_ms":39114,"significance":"If the described functionality and deployment are real, LabelECG would address a real need: building large, carefully annotated ECG datasets for deep learning applications. The paper's strengths include a plausible architecture built around the established Lightwave viewer, docker-based deployment that supports local-area-network use for privacy-sensitive data, and a concrete real-world deployment context (the competition). However, the paper provides no runnable artifact, no demo URL, no quantitative evaluation, no inter-annotator agreement statistics, and no system logs; the central claims therefore rest entirely on the authors' narrative, architecture diagrams, and screenshots. As a systems/tool paper, this lack of verifiability is a significant weakness that must be addressed before the claims can be accepted.","major_comments":[{"comment":"The concurrency control in the distributed annotation workflow is underspecified. The text says that after the first-time annotation, \"the system automatically shows the record next to the ones they previously annotated\" (Section 3.2), implying a per-user private sequence, but no locking, record-assignment, or conflict-resolution protocol is described. Multiple annotators could therefore be assigned the same record independently, and the paper does not explain how duplicate or conflicting annotations are prevented or reconciled. Because \"distributed\" and \"simultaneously\" annotation is a central claim of the paper, please specify the actual concurrency mechanism (e.g., per-record queue, lease/lock, optimistic concurrency, or a later review pass) and provide evidence from the implementation that it works.","section":"Section 3.2, Annotation Management System"},{"comment":"The main quantitative claim that about 15,000 records were annotated in about three months and that this \"guaranteed the success of the competition\" is supported only by self-report. No system logs, timestamps, per-user annotation counts, reviewer statistics, or quality metrics (e.g., inter-annotator agreement against a gold standard) are provided. Readers cannot check whether the annotation throughput was plausible or whether the resulting labels were accurate. At minimum, the paper should report the number of annotators, the per-record or per-week annotation rate, the distribution of \"confirmed\" vs. \"unsure\" labels, and any validation of the expert-review process.","section":"Section 4, Competition deployment"},{"comment":"The paper promises that LabelECG \"will be freely accessible on the Internet\" (Abstract) and \"will be accessible online\" (Section 1), but no URL, repository, or demo is given anywhere in the text or references. For a systems paper, the software itself is the primary contribution; without an accessible artifact, the claimed functionality is unverifiable. Please provide a link to the source code and/or an online demo, or if access is intentionally restricted, give a detailed, reproducible description of the deployment environment and a precise specification of the architecture so that others could reimplement the system.","section":"Abstract and Section 1"}],"minor_comments":[{"comment":"The abstract contains the misspelling \"La-belECG\"; the tool name should be \"LabelECG\" throughout.","section":"Abstract"},{"comment":"\"With the assistances of LabelECG\" is grammatically incorrect; it should be \"With the assistance of LabelECG.\"","section":"Section 4"},{"comment":"The Functions column for EcgEditor and BSS_ECG says \"diagnose annotating\", which is inconsistent and likely a typo for \"diagnosis annotating\" or \"diagnostic annotation\"; please unify the terminology across all rows.","section":"Table 1"},{"comment":"\"the first-time users need to choose the first data to annotate\" should be rephrased to \"the first record to annotate\" to maintain consistent terminology and improve readability.","section":"Section 3.2"},{"comment":"The reference list has inconsistent capitalization (e.g., \"nature\" for the journal Nature) and several incomplete entries; please align all references with a consistent style (e.g., ensuring journal names are capitalized and page numbers are present).","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a tool paper whose central claim is the existence and utility of a deployed system, yet the manuscript currently provides no artifact, no URL, and no independent evaluation. The concurrency model is also under-specified, and the throughput/quality claims are self-reported. These issues are significant but addressable: the authors could add a repository link, a demo, or at least detailed deployment logs and a precise concurrency protocol, and they could strengthen Section 4 with quantitative evidence. I therefore support a major revision rather than outright rejection. I would also suggest the editor ask for an explicit statement of what will be made publicly available and under what license."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the quick take: LabelECG is a sensible, needed tool—a web-based distributed ECG annotation system—and the deployment story (15,000 records, four hospitals, three months, supporting a competition) is a useful data point. But the paper gives no way to verify the central claim that the tool works as described. No code, no demo, no deployment logs. The promise that it \"will be freely accessible on the Internet\" is made without a URL.\n\nWhat's new: the paper identifies a real bottleneck—deep learning for ECGs needs large annotated datasets, but existing annotation tools are offline or inaccessible. LabelECG's design choices (docker-based deployment, multi-user collaboration, unified data management, expert review, integration with Lightwave for visualization and Physionet datasets) are all reasonable and practical. The two-team review workflow is a nice touch. The authors are honest about the current limits—no beat-level annotation, Chinese-only interface—which is refreshing.\n\nSoft spots: the biggest is verifiability. The architecture is described at a high level, but the concurrency model is underspecified. The UI shows the next unannotated record, but there is no description of locking, record assignment, or how simultaneous annotators avoid duplicating work. The 15,000-record/three-month number is self-reported, with no per-user counts or inter-annotator agreement. And \"guaranteed the success of the competition\" is an overclaim. Nothing here suggests dishonesty; it's just not yet checkable.\n\nFor a systems paper, that's a load-bearing gap. The right fix is to make the artifact available—a public repo, a demo instance, or at least a detailed deployment walkthrough with test data. The authors promise accessibility, so it's fair to ask for it before the main claim is taken at face value.\n\nThis deserves serious peer review: the tool addresses a real need, and the reported scale is nontrivial. I'd condition acceptance on releasing the artifact and adding a short section on concurrency and review semantics.\n\nRecommendation: send it to review, but insist on the artifact or a demo before accepting.","headline":"A useful tool paper that currently can't be checked: the authors need to ship the artifact before the central claim that LabelECG works as described should be taken at face value.","tokens_in":6084,"tokens_out":2055,"would_cite":false,"duration_ms":20465,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LabelECG is a browser-based tool that let four physicians annotate about 15,000 ECG records in three months, with the labels supporting a clinical AI competition.","keywords":["electrocardiogram","ECG annotation","distributed annotation","web-based tool","clinical data labeling","deep learning dataset","arrhythmia detection","data management"],"falsifier":"Release the tool, invite several users to annotate the same ECG cohort concurrently in a browser session, and check that every confirm/unsure decision is stored and visible to the expert reviewer. If annotations are lost, users cannot access the same dataset simultaneously, or the tool cannot be deployed on a local network as promised, the central claim of a functioning distributed annotation system is refuted.","tokens_in":5149,"feed_emoji":"🫀","tokens_out":5543,"duration_ms":49079,"temperature":0.7,"pith_summary":"This paper presents LabelECG, a web-based tool for distributed annotation of electrocardiograms (ECGs). The central claim is that the tool, by managing data centrally and letting many physicians annotate through ordinary web browsers, enabled doctors from four hospitals to annotate about 15,000 12-lead resting ECG records in roughly three months. The resulting labels supported the First China ECG Intelligent Competition. The authors argue this matters because deep-learning ECG analysis needs large, carefully labeled datasets, and LabelECG is meant to make building such datasets practical.","feed_headline":"Web tool lets doctors label 15,000 ECGs in 3 months","feed_subtitle":"Browser-based LabelECG distributes ECG labeling across clinicians, building annotated datasets deep learning needs.","key_machinery":"The load-bearing mechanism is the three-system web architecture. A user-management system stores accounts and controls registration through administrator-issued verification codes; an annotation-management system stores ECG files, parameters, automatic diagnoses and user labels; and a visualization service renders multi-lead ECG traces in the browser. The annotation workflow is built around the confirm/unsure buttons, which push a decision to a diagnosis database and advance to the next record, plus review rights for expert accounts. This division lets dozens of remote annotators work on one dataset simultaneously and lets experts police label quality without any annotator touching raw files.","core_discovery":"LabelECG is a client-server system with three cooperating parts: user management, annotation management, and ECG visualization. Multiple users register, choose a dataset, view ECG traces in the browser, and record diagnoses, marking each record as \"confirmed\" or \"unsure\"; expert accounts can review and revise all annotations. The paper reports that four physicians used the system to annotate about 15,000 records in about three months and that this effort \"guaranteed the success\" of the competition it was built for.","pith_inferences":["The authors compare LabelECG to an image-annotation tool that enabled a famous large image dataset; by analogy, if the promised public release is reliable, LabelECG could play a similar role for ECG deep learning, but the paper itself does not claim to have created such a dataset.","The paper does not report inter-annotator agreement or label-quality metrics; a natural extension would be to measure agreement between annotators and against the expert review to see whether distributed browser annotation preserves clinical-grade accuracy.","The causal statement that the annotations \"guaranteed the success\" of the competition is asserted rather than demonstrated; an independent replication of the workflow on a smaller cohort would test whether the tool alone, rather than the doctors' expertise, drives throughput.","Since the tool is said to be freely accessible, a testable consequence is that a third party could run the released version and reproduce a small-scale multi-user annotation session; if the release does not happen or the session fails, the paper's central claim rests on an unrepeatable deployment."],"forward_implications":["If LabelECG works as reported, research groups can assemble large labeled ECG datasets by recruiting clinicians to annotate from any device, without building custom data-management infrastructure.","The reported throughput implies a small team can label on the order of 1,250 records per physician per month for 12-lead resting ECGs, making large-cohort annotation projects feasible.","The tool's design, with local deployment on a private network plus browser access, means hospitals concerned about data sharing can still run distributed annotation.","Because the current version focuses on diagnosis-level labels, the promised additions of beat and fiducial-point annotation would extend the same infrastructure to more granular ECG labeling tasks."],"supporting_citations":[{"why":"provides the ECG visualization service that LabelECG embeds as a CGI application.","marker":"[17]"},{"why":"is the image-annotation tool the authors cite as the model for how a labeling tool can enable a large dataset.","marker":"[9]"},{"why":"is the competition that the 15,000 annotated records were created to support.","marker":"[11]"},{"why":"demonstrates a deep network trained on a large ECG corpus, motivating the need for more labeled records.","marker":"[7]"},{"why":"similarly shows a neural network trained on tens of thousands of ECG pairs, motivating large labeled datasets.","marker":"[8]"},{"why":"is the only previous web-based ECG annotation tool, used as the baseline contrast for LabelECG's online distributed access.","marker":"[16]"}],"fun_headline_variants":["Distributed web tool labels 15k ECGs in 3 months","Browser-based ECG annotation scales to 15k records","LabelECG: Web platform for collaborative ECG annotation","Web tool powers 15k ECG annotations for AI training","Multi-device web app accelerates ECG labeling for AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's claim depends on the reported implementation actually providing the described multi-user annotation, data-management, and review functions without critical defects, since no runnable code or independent test is supplied.","fun_headline_variants_meta":{"raw":{"variants":["Distributed web tool labels 15k ECGs in 3 months","Browser-based ECG annotation scales to 15k records","LabelECG: Web platform for collaborative ECG annotation","Web tool powers 15k ECG annotations for AI training","Multi-device web app accelerates ECG labeling for AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00053,"raw_usage":{"total_tokens":2493,"prompt_tokens":822,"completion_tokens":1671,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":1590}},"tokens_in":438,"tokens_out":1671,"duration_ms":13886,"temperature":1.0,"reasoning_tokens":1590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:39:56.974790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Release the tool, invite several users to annotate the same ECG cohort concurrently in a browser session, and check that every confirm/unsure decision is stored and visible to the expert reviewer. If annotations are lost, users cannot access the same dataset simultaneously, or the tool cannot be deployed on a local network as promised, the central claim of a functioning distributed annotation system is refuted.","supporting_citations":[{"cited_title":"Circulation , 101(23):e215-20 (2000)","cited_arxiv_id":null,"evidence_quote":"provides the ECG visualization service that LabelECG embeds as a CGI application."},{"cited_title":"Internatio nal journal of computer vision , 1;77(1-3):157-73 (2008)","cited_arxiv_id":null,"evidence_quote":"is the image-annotation tool the authors cite as the model for how a labeling tool can enable a large dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is the competition that the 15,000 annotated records were created to support."},{"cited_title":"Nature medicine","cited_arxiv_id":null,"evidence_quote":"demonstrates a deep network trained on a large ECG corpus, motivating the need for more labeled records."},{"cited_title":"Nature medicine 25(1):70 (2019)","cited_arxiv_id":null,"evidence_quote":"similarly shows a neural network trained on tens of thousands of ECG pairs, motivating large labeled datasets."},{"cited_title":"WaveformECG: A Platform for Visualizing, Annotat- ing, and Analyzing ECG Data","cited_arxiv_id":null,"evidence_quote":"is the only previous web-based ECG annotation tool, used as the baseline contrast for LabelECG's online distributed access."}],"review_version":1}