{"id":"49b700f8-d734-455e-be5f-76614a267c3a","arxiv_id":"2501.03053","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A telehealth tongue dataset with 5,109 images and a multi-branch network that detects eight tongue attributes, slightly improving F1 over backbones and approaching practitioner agreement.","lead":"This paper introduces a publicly promised dataset of 5,109 smartphone tongue images and a multi-branch network that detects eight tongue attributes for remote diagnosis. It reports modest F1 gains over standard backbones, with performance near that of human practitioners on several attributes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No inter-annotator agreement is reported, and the claimed F1 gain over ResNet50 is concentrated in Pale and Ecchymosis, the two attributes most sensitive to uncontrolled lighting; label noise is therefore an uneliminated rival explanation for the headline improvement.","rationale":"Reader's weakest assumption matches the most load-bearing point: the label layer is the foundation for every number in Table 3. I considered the alternative that the model-vs-model comparison is confounded because SignNet includes the ATFE preprocessing that the ResNet50 baseline does not, but the ablation in Table 4 (ResNet50+upright vs SignNet) does isolate the architecture, so that concern is smaller. I also considered the cross-validation and hold-out mixing, but the 1% and 4% improvement claim is computed on the same five-fold split for all models, so the protocol issue mainly affects the human comparison rather than the headline model comparison. The label-reliability issue, by contrast, affects the ground truth itself: if the two attributes that drive the average-F1 gain are labeled inconsistently under uncontrolled lighting, the central claim could be an artifact of annotation noise. The proposed kappa check is decisive and inexpensive; until it is run, a conditional verdict is appropriate.","tokens_in":18119,"tokens_out":8318,"duration_ms":77707,"concrete_test":"Recruit two or more TCM practitioners who did not participate in the original labeling, blind them to the original annotations and to each other, and have them independently re-annotate a random sample of at least 200 images from the held-out test set using the same eight binary attributes. Compute per-attribute Fleiss' kappa, or Cohen's kappa for each pair. If kappa for Pale or Ecchymosis is below about 0.4, the labels are too unreliable to support the reported 4-point F1 gain, and the comparison should be rerun on adjudicated or high-agreement labels. If all attributes show kappa at or above 0.6, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Quantitative Results) is that SignNet raises average F1 by roughly 4 points and accuracy by 1 point over ResNet50. That claim presupposes that the three professional practitioners' binary labels are correct and consistent for single static smartphone images, but the paper reports no inter-annotator agreement. The Dataset section states that final annotations were completed by three practitioners, and the Quantitative Results section notes that annotations were made through direct observation in a clinical setting, whereas the test images were captured at home with uncontrolled lighting; a label can therefore encode what an annotator saw live even when the image does not visually support it. The concern is mechanically load-bearing because the average-F1 advantage of SignNet over ResNet50 in Tables 3-4 is driven almost entirely by Pale (F1 32.68 to 46.00) and Ecchymosis (F1 24.72 to 38.02), the two attributes the authors themselves identify as vulnerable to overly bright or dark lighting conditions. With only 624 Pale and 471 Ecchymosis positive images in the whole dataset and no kappa statistic, random or systematic label noise in these two classes could produce the reported gain without any true improvement. The numbers are not internally inconsistent, but the ground truth is unvalidated, so both the model-vs-model comparison and the claim of approaching practitioner-level performance rest on an unverified foundation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TongueDx, a telehealth-oriented tongue image dataset with 5109 images from 4650 subjects and eight binary attribute labels, and proposes a Sign-Oriented multi-label attribute detection framework. The framework's Adaptive Tongue Feature Extraction (ATFE) module detects the tongue with YOLOv5-MobileNetV3 (trained on GroundingDINO boxes), segments it with MobileSAM, orients it upright, and separates it into body and edge regions. The SignNet classifier then uses whole-tongue, body, and edge branches plus explicit color and fur branches with self-attention fusion to predict the eight attributes. In experiments, SignNet is reported to improve average accuracy by about 1% and average F1 by about 4% over ResNet50, with ablations attributing gains to the ATFE module and the color/fur branches; the authors also compare the model against registered practitioners and laypersons.","tokens_in":18408,"tokens_out":4937,"duration_ms":46527,"significance":"If the label foundation holds, TongueDx would be a genuinely useful public resource for remote tongue diagnosis, and the paper's explicit modeling of attribute-sign-region relationships is a sensible architectural idea that could transfer to other structured medical image tasks. The ablation study is well designed at the component level: it separately evaluates detection, segmentation, upright orientation, and the color and fur branches, and the reported numbers are internally consistent with the proposed mechanism. The manuscript also includes a candid supplementary discussion of stability and lighting limitations. However, the headline comparisons against ResNet50 and against practitioners rest on unvalidated ground-truth labels and on an incompletely specified evaluation protocol, so the strength of the contribution is not yet established.","major_comments":[{"comment":"The manuscript does not report any inter-annotator agreement for the eight binary attribute labels, and it states that final annotations were completed by three professional practitioners through direct observation in a clinical setting, while test images were captured at home under uncontrolled lighting. This creates an unquantified train/test label mismatch: a label may encode what was observed live even when the image alone does not support it. The concern is load-bearing because the average-F1 advantage of SignNet over ResNet50 in Table 3 is concentrated in Pale (32.68 to 46.00) and Ecchymosis (24.72 to 38.02), the two attributes the authors themselves identify in the Supplementary Material as most sensitive to lighting. Please report per-attribute inter-annotator agreement (for example Cohen's or Fleiss' kappa) on a representative sample, describe the adjudication process, and clarify whether the three practitioners used in the human evaluation are the same individuals who created the training labels. Without this evidence, the model-model and model-human comparisons rest on unverified ground truth.","section":"The TongueDx Dataset; Quantitative Results"},{"comment":"Table 3 is captioned 'five-fold cross-validation,' but the Supplementary describes a five-fold cross-validation combined with a separate hold-out test set of 895 subjects, with the hold-out test set reserved for human comparison. These are different protocols, and the paper does not state clearly whether the numbers in Tables 3 and 4 come from the five training folds, from the hold-out test set, or from a combination of both. Please specify which split produced every reported metric, confirm whether the hold-out test set is excluded from the five folds, and describe exactly how model selection was performed (the Supplementary mentions selecting on validation F1). This is essential for interpreting the headline 1%/4% improvements and for reproducing the human-evaluation results.","section":"Experiments; Supplementary: Five-fold Cross-validation Setting"},{"comment":"The localization evaluation is circular: GroundingDINO's outputs are used as the ground-truth bounding boxes for training YOLOv5-MobileNetV3, and the reported 'over 99% mAP' measures agreement with that same teacher model, not with true tongue location. This does not validate the ATFE claim of 'high-quality annotations' or robustness. Please evaluate the detection network on a human-verified subset of bounding boxes, or at least report the agreement between GroundingDINO and human-annotated boxes, so that the localization step is independently validated.","section":"Tongue Detection and Segmentation"},{"comment":"The claim that SignNet 'approaches the performance of practitioners' is based on single averaged accuracy/F1 rows for practitioners, with no variance and no statistical test, and no significance tests are reported for any model-model difference either. Given the large standard deviations in Pale and Ecchymosis in Table 3, please provide paired or matched comparisons (for example per-fold paired tests or bootstrap confidence intervals) for SignNet versus ResNet50 and for SignNet versus practitioner performance on the same test images. Without such tests, the strength of the 4% F1 claim and the practitioner-level claim cannot be assessed.","section":"Quantitative Results"}],"minor_comments":[{"comment":"Equation (8) is malformed as printed: it reads alpha_j = median / F_j, but the surrounding text says m is the median frequency; it should read alpha_j = m / F_j, with m defined in the equation rather than only in prose.","section":"Loss Function"},{"comment":"Please standardize attribute names across tables, figures, and text: for example, 'Furrthick' in Supplementary Table 5 and 'Tipsidered' in Table 5 should be consistent with 'FurThick' and 'TipSideRed' used elsewhere.","section":"Supplementary Table 5; main text"},{"comment":"Figure 5 is hard to read because several labels are garbled or truncated (for example the TIUO/TEBRS inset and the mathematical symbols in the branch labels); please provide a higher-resolution version with clean and complete notation.","section":"Figure 5"},{"comment":"The contour filtering condition in Algorithm 1 is underspecified: step 3 says 'filter points where angle between points > alpha' without defining how that angle is computed or how the threshold interacts with the later smoothing steps; please provide a precise geometric definition.","section":"Algorithm 1"},{"comment":"The ROC discussion states that SignNet and ResNet50+ATFE are 'fairly comparable to the baseline ResNet,' which is in tension with the claimed 4% average F1 improvement in Table 3; please clarify whether the ROC/AUC comparison uses a different protocol or subset, or reconcile the two statements.","section":"Supplementary: ROC Curve Analysis"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for a medical-imaging or applied-AI venue, and the promised public dataset addresses a real gap in telehealth tongue diagnosis. The architecture is sensible and the ablation is informative, but the revision must supply the missing label-reliability evidence and a precise evaluation protocol before the headline claims can be accepted. I would not recommend rejection at this stage; the required evidence appears obtainable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: TongueDx is the real contribution; SignNet is a sensible architecture and the ablations support it, but the headline F1 gain over ResNet50 is not established, because the ground-truth labels were assigned in person rather than from the images, and the reported comparison to practitioners mixes cross-validation and hold-out numbers.\n\nWhat's genuinely new: TongueDx is the first tongue dataset collected under telehealth-like conditions—smartphone/laptop cameras, uncontrolled lighting, some beauty filters—with eight binary attribute labels, and the authors promise to release it. Prior public datasets are device-controlled or have disappeared. The ATFE preprocessing (detection, segmentation, upright orientation) is a reasonable standardization and the Table 4 ablation shows each component adds a little. SignNet's whole-tongue/body/edge branches with attention fusion is well motivated by the attribute-sign table, and the color/fur branches appear to earn their keep.\n\nThe soft spots are load-bearing. The paper states that annotations were made through direct observation in a clinical setting, while the test images were captured at home. No inter-annotator agreement is reported. When the label reflects what the practitioner saw live rather than what the single image actually shows, the training target is partly independent of the input. The F1 gain over ResNet50 is concentrated in Pale and Ecchymosis (32.68→46.00 and 24.72→38.02), exactly the two attributes the authors flag as most lighting-sensitive. Label noise in those rare classes could plausibly produce the reported gain without a real algorithmic improvement.\n\nSecond, the human comparison isn't apples-to-apples. Table 3 reports five-fold cross-validation for the models, but the layperson/practitioner rows look like a single evaluation on a fixed hold-out test. Without model numbers on that same test set, 'approaches practitioner performance' is not supported as reported. Third, there are no significance tests and the Pale/Ecchymosis variances are large; the average improvement may not be robust. The >99% mAP for YOLOv5 is just agreement with the GroundingDINO teacher, so it's a speedup claim, not an accuracy claim—fine if stated, but easy to misread.\n\nIf you work in telehealth or TCM imaging, this dataset is worth watching. The paper deserves a serious referee: ask for inter-annotator agreement (kappa), model evaluation on the same hold-out set as the humans, and release of data and code. I'd bring it to reading group mainly to discuss the label-leakage issue; the architecture itself is not a methodological landmark.","headline":"The TongueDx dataset is a genuinely useful contribution, but the headline F1 gain rests on unvalidated labels and a mixed evaluation protocol, so treat the quantitative claim as not established.","tokens_in":18940,"tokens_out":4429,"would_cite":false,"duration_ms":39171,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a sign-oriented multi-branch network, combined with adaptive tongue detection, segmentation, and upright orientation, recognizes eight tongue attributes from uncontrolled telehealth images with near-practitioner…","keywords":["tongue diagnosis","multi-label classification","telemedicine","deep learning","tongue image dataset","Traditional Chinese Medicine","sign-oriented network","image segmentation"],"falsifier":"Have an independent panel of practitioners re-annotate a random sample of the test-set images and measure inter-annotator agreement against the original labels; if agreement on Pale or Ecchymosis is at or near chance (for example, kappa below 0.4), the claimed improvements over baseline and the comparison with practitioners would be an artifact of label noise rather than a real signal.","tokens_in":17906,"feed_emoji":"👅","tokens_out":5150,"duration_ms":67964,"temperature":0.7,"pith_summary":"The paper claims that tongue diagnosis, traditionally done by looking at the tongue surface, can be automated for telehealth images that are low-resolution and unevenly lit. It proposes a pipeline that first locates, segments, and rotates each tongue into a standard upright view, then runs a Sign-oriented Network (SignNet) that separates the whole tongue, tongue body, and tongue edge into branches aligned with how practitioners read color, shape, region, and fur signs. The authors report that SignNet raises average accuracy by about 1% and average F1-score by about 4% over ResNet50, and that it outperforms laypersons and approaches registered practitioners on eight binary tongue attributes. A new public dataset of 5109 smartphone and laptop tongue images from 4650 subjects, labeled for eight attributes plus demographics, is introduced to support remote diagnosis research. If right, the framework would give telehealth systems a way to extract clinically meaningful tongue signs from ordinary consumer-camera images.","feed_headline":"Sign-aware network reads eight tongue signs from telehealth photos","feed_subtitle":"A multi-branch network plus adaptive image extraction beats generic ResNet50 and nears expert accuracy.","key_machinery":"The central machinery is SignNet, a three-branch network that mirrors the paper's attribute-sign relationship table: the whole-tongue branch predicts color category and fur presence, the body branch handles body-region attributes such as crack, and the edge branch handles edge attributes such as tooth marks, with self-attention fusing color and fur features into each regional branch. Before this, the Adaptive Tongue Feature Extraction (ATFE) module standardizes input by detecting the tongue with a lightweight YOLOv5-MobileNetV3 network, segmenting it with MobileSAM, and rotating it upright with a contour-based tongue-tip and top alignment algorithm. A tongue body and edge separation algorithm with an adaptive edge width generates the regional inputs. The loss combines weighted binary cross-entropy to counter class imbalance.","core_discovery":"The paper's central claim is that a sign-oriented, multi-branch network, fed by an adaptive tongue extraction module, can recognize eight tongue attributes (Pale, TipSideRed, RedSpot, Ecchymosis, Crack, ToothMark, FurThick, FurYellow) in uncontrolled telehealth images better than generic deep backbones. The load-bearing numbers are that ResNet50+ATFE gains about 2% average F1 over plain ResNet50, and SignNet adds a further improvement of about 1% average accuracy and about 4% average F1 over ResNet50. The design explicitly encodes the practitioner's sign relationships: color cues are reorganized into four simplified color categories, fur is reduced to presence or absence, and edge-specific versus body-specific attributes are processed in separate branches fused by self-attention. The authors also position the TongueDx dataset as a uniquely telehealth-oriented public resource, since existing tongue datasets were captured with standardized devices or are no longer available.","pith_inferences":["Because no inter-annotator agreement is reported, the reported margins over practitioners could be as much a measure of label noise as of model skill; an independent re-annotation study would settle this.","The color-branch simplification into four color categories could be seen as a learned soft color-calibration, and a testable extension is to replace it with explicit color constancy preprocessing on paired controlled and uncontrolled images.","The upright-orientation algorithm explicitly assumes the tongue appears in a normal, non-upside-down orientation, so extending to arbitrary pose would require a more general keypoint or rotation regressor.","If the public release includes demographic labels, the dataset could support age- and gender-stratified analysis of tongue signs, which the current experiments do not touch."],"forward_implications":["Remote tongue screening becomes feasible with ordinary smartphone or laptop cameras, not just standardized tongue-imaging devices.","The TongueDx dataset, if released as promised, gives the community a common benchmark for multi-label tongue attribute recognition under telehealth conditions.","The ATFE standardization (detect-segment-orient) transfers a benefit of about a 2% average F1 gain to a generic backbone, suggesting preprocessing is as important as the classifier.","SignNet's color, fur, body, and edge decomposition provides a template for embedding clinical sign knowledge into multi-label medical image classifiers.","Performance approaching registered practitioners suggests automated pre-screening could support, not replace, remote tongue assessment."],"supporting_citations":[{"why":"GroundingDINO generates the high-quality bounding-box pseudo-labels used to train the lightweight detector.","marker":"Liu et al. 2023"},{"why":"MobileSAM segments the tongue inside the predicted box in the final pipeline.","marker":"Zhang et al. 2023"},{"why":"YOLOv5, combined with MobileNetV3, forms the lightweight real-time detection backbone.","marker":"Jocher et al. 2022"},{"why":"MobileNetV3 supplies the efficient backbone for the tongue localization network.","marker":"Howard et al. 2019"},{"why":"TransFG is the architecture behind the reproduced strongest baseline from Yuan et al.","marker":"He et al. 2022"},{"why":"This gastric-cancer tongue diagnosis method is the baseline that SignNet is directly compared against.","marker":"Yuan et al. 2023"},{"why":"Prior multi-label tongue analysis on controlled TFDA-1 images represents the gap SignNet addresses for telehealth settings.","marker":"Jiang et al. 2022"},{"why":"The upright-orientation prior inspired the tongue rotation algorithm used in the ATFE module.","marker":"Fu et al. 2008"}],"fun_headline_variants":["Sign-oriented AI reads eight tongue signs from telehealth photos","Multi-label tongue diagnosis for remote health checks","Adaptive tongue features boost telehealth diagnosis accuracy","New framework spotlights eight tongue attributes for telemedicine","Remote tongue diagnosis gets sign-aware deep learning boost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the three practitioners' labels on single static smartphone images are accurate and consistent enough across all eight attributes, particularly Pale and Ecchymosis, for the reported accuracy and F1 gaps to be meaningful.","fun_headline_variants_meta":{"raw":{"variants":["Sign-oriented AI reads eight tongue signs from telehealth photos","Multi-label tongue diagnosis for remote health checks","Adaptive tongue features boost telehealth diagnosis accuracy","New framework spotlights eight tongue attributes for telemedicine","Remote tongue diagnosis gets sign-aware deep learning boost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000538,"raw_usage":{"total_tokens":2565,"prompt_tokens":913,"completion_tokens":1652,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":1581}},"tokens_in":529,"tokens_out":1652,"duration_ms":11337,"temperature":1.0,"reasoning_tokens":1581,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:57:13.468502+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent panel of practitioners re-annotate a random sample of the test-set images and measure inter-annotator agreement against the original labels; if agreement on Pale or Ecchymosis is at or near chance (for example, kappa below 0.4), the claimed improvements over baseline and the comparison with practitioners would be an artifact of label noise rather than a real signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"YOLOv5, combined with MobileNetV3, forms the lightweight real-time detection backbone."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MobileNetV3 supplies the efficient backbone for the tongue localization network."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This gastric-cancer tongue diagnosis method is the baseline that SignNet is directly compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior multi-label tongue analysis on controlled TFDA-1 images represents the gap SignNet addresses for telehealth settings."}],"review_version":1}