Pith. sign in

REVIEW 3 major objections 5 minor 30 references

CaresAI at SMM4H-HeaRD 2026: Predicting TNM Staging

T0 review · 3 major / 5 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LightGBM with TF-IDF is the strongest baseline for independent T/N/M prediction from pathology reports, but still fails to generalise reliably.

desk verdict Shared-task system note: LightGBM+TF-IDF beats their BERT/WRN setups on TCGA TNM, with honest caveats and a clear drop on the second test set; useful as a workshop baseline, not a research advance. read the letter →

arxiv 2607.03466 v1 pith:BPVRRNS2 submitted 2026-07-03 cs.CL cs.AIcs.LG

Joseph Itopa Abubakar , Jorge Jarme , Favour Igwezeke , Mary Adewunmi This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords TNMstagingpathologyreportsTF-IDFLightGBMbiomedicalembeddingsmulti-labelclassificationclassimbalanceclinicalNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cancer staging depends on three labels—tumor size and local invasion (T), lymph-node involvement (N), and distant metastasis (M)—that are usually read by hand from free-text pathology reports. This paper treats each label as a separate multi-label classification problem on TCGA reports and systematically compares classical features (TF-IDF) and biomedical embeddings (ClinicalBERT, BioBERT, PubMedBERT) with logistic regression, LightGBM, feed-forward nets and wide residual networks. On the training distribution LightGBM with TF-IDF reaches the highest AUROCs (0.937 T, 0.952 N, 0.831 M). The same pipeline scores Macro-F1 0.938 on the first held-out test set, yet falls to 0.858 on a second test set, revealing that class imbalance, long-document truncation and distribution shift still dominate. The authors therefore present a fast, reproducible baseline that is not yet safe for clinical deployment.

What carries the argument

Independent multi-label classification of T, N and M, each filtered and trained separately, using either sparse TF-IDF vectors or averaged ClinicalBERT/BioBERT/PubMedBERT embeddings fed to LightGBM (or WRN).

What would settle it

A third external test set drawn from a different hospital system or cancer-type mix that shows Macro-F1 remaining above 0.90 for all three stages (or a controlled ablation that restores performance once class weights, long-context models and oversampling are applied).

Watch

Extended reading notes

Core claim

Among the systems examined, LightGBM trained on TF-IDF features is the best-performing approach for independent T, N and M prediction from TCGA pathology reports, delivering training-phase AUROCs of 0.9368 (T), 0.9524 (N) and 0.8311 (M) and Codabench Macro-F1 scores of 0.978/0.957/0.879 on test set 1 (dropping to 0.807/0.767/1.0 on test set 2). Combining the three biomedical embeddings improves deep models, yet none of them surpass the classical pipeline on the official metrics; the resulting system is therefore an efficient reproducible baseline that still requires further optimisation before real-world use.

Load-bearing premise

Models fitted on the moderately imbalanced TCGA training distribution (M1 only about 7 percent of M cases; rare advanced T/N stages) will transfer to unseen test distributions when the three labels are filtered and trained independently.

Editorial extensions

If this is right

  • An efficient TF-IDF + LightGBM pipeline can serve as the default baseline for future shared tasks on TNM extraction.
  • Independent modelling of T, N and M allows partial labels to be used without discarding entire reports.
  • Performance collapse under mild distribution shift shows that any production system must include explicit imbalance handling and long-document encoding.
  • The same feature–model stack can be reused for pre-screening or registry triage once generalisation is improved.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because M1 is rare and often only implied rather than stated, keyword or short-context models will systematically under-call metastasis unless long-range clinical reasoning is added.
  • The 512-token limit of standard BERT models is a hard ceiling for TCGA reports; any competitive system will need hierarchical or long-context encoders.
  • A stacked ensemble that keeps LightGBM on TF-IDF as the primary learner and uses deep embeddings only for residual correction may close the generalisation gap with little extra cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a shared-task system for independent T, N, and M staging from TCGA pathology reports (SMM4H-HeaRD 2026 Task 6). The authors cast the problem as three separate classification tasks, compare TF-IDF and mean-pooled ClinicalBERT/BioBERT/PubMedBERT embeddings with LR, LightGBM, stacked ensembles, FFNN, and Wide Residual Networks, and conclude that LightGBM+TF-IDF is their best setup (training-phase AUROC 0.9368/0.9524/0.8311 for T/N/M; Codabench Macro-F1 0.978/0.957/0.879 on test set 1, lower on test set 2). They provide a reproducible pipeline and correctly caution that the system is not yet suitable for real-world clinical use because of class imbalance, long-document truncation, and limited generalization.

Significance. As a shared-task system description the work is useful: it supplies a simple, computationally efficient LightGBM+TF-IDF baseline, releases code, and documents the practical failure modes (M1 rarity ≈7%, 512-token truncation, Macro-F1 drop from 0.938 to 0.858 across official test sets). Those contributions are appropriate for a workshop proceedings track. The scientific advance relative to the cited Kefeli et al. TCGA staging work is modest; the main value is the transparent empirical comparison and the explicit clinical-use caveat rather than a new modeling principle.

major comments (3)
  1. Results and Discussion / Table 1: The central claim that LightGBM+TF-IDF is the best-performing approach rests on Table 1 numbers obtained from an internal 80/20 split of the training set, yet the same section also reports a validation micro-F1 of 0.94 and an AUROC of 0.96 that are never broken down by model or category and are never aligned with the official Codabench protocol. Without a single, consistent evaluation table that maps each submitted system to the shared-task metrics, the superiority claim is not cleanly established.
  2. Results and Discussion (baseline comparison): The paper asserts that an AUROC of 0.96 on the validation set exceeds the organizer/Kefeli range of 0.815–0.942, but does not state which model produced the 0.96 figure, whether the label sets and metric definitions match, or whether the comparison uses the same external validation corpus. This comparison is load-bearing for the claim of an improved baseline and must be made precise or removed.
  3. Abstract, §2.1 and throughout: T, N and M are repeatedly called “multi-label” classification tasks, yet the label definitions (T remapped to 0–3, N 0–3, M binary) and the independent filtering of each category describe multi-class (or binary) problems. The terminology error propagates into metric interpretation (micro-F1 vs Macro-F1) and should be corrected so that the experimental design is unambiguous.
minor comments (5)
  1. Table 1 keys: WRN is glossed as “wide recurrent network” while the text and references describe Wide Residual Networks (Zagoruyko & Komodakis). Correct the key and keep residual/recurrent terminology consistent.
  2. Abstract vs Table 2: Macro-F1 aggregates are reported as 0.938 (test 1) and 0.858 (test 2); verify that these are simple averages of the per-category scores shown in Table 2 and state the aggregation explicitly.
  3. §2.1: Clarify how missing T/N/M labels are handled after independent filtering (sample sizes per task after filtering are not given).
  4. Code Availability: The GitHub link is present but the hyperparameter list is only titled “Hyperparameters” with no values; include the concrete LightGBM/TF-IDF settings used for the submitted runs.
  5. Minor typos: “Code A vailability”, “Wide Residual Neural Networks” vs residual/recurrent, and occasional grammar slips in the Abstract and Limitations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: standard empirical multi-class classification with external labels, held-out metrics, and non-author baselines.

full rationale

This is a shared-task empirical NLP/ML paper. T/N/M labels are taken directly from TCGA clinical metadata (not engineered from the model outputs); features are TF-IDF or frozen pretrained embeddings; classifiers (LR, LightGBM, FFNN, WRN) are trained and scored on an 80/20 training split plus Codabench test sets. Reported AUROC/F1/Macro-F1 numbers are ordinary evaluation metrics, not quantities that reduce by construction to fitted constants or definitional identities. Citations to Kefeli et al. (dataset/baseline), Huang/Lee/Gu (BERT variants), Ke (LightGBM), and Zagoruyko (WRN) are external method/data references with no author overlap and no uniqueness theorems or ansatzes imported as load-bearing premises. There is no self-definitional loop, no fitted-parameter-renamed-as-prediction, and no renaming of a known result as a derivation. Metric inconsistencies noted by the skeptic (training-phase vs validation, micro vs macro, residual vs recurrent) are correctness/reporting issues, not circularity. The derivation chain is simply 'train classifier on features + labels → score on held-out data,' which is self-contained against external benchmarks.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The work rests on standard ML practice plus domain assumptions about TCGA labels and independent T/N/M modeling. There are no invented physical entities. Free parameters are the usual model and featurization choices; none are claimed as universal constants. The central performance claim depends on those choices and on the assumption that the shared-task label distribution is a fair proxy for the evaluation the authors care about.

free parameters (5)
  • LightGBM hyperparameters (depth, learning rate, leaves, etc.)
    Best reported system depends on LightGBM configuration; text points to a hyperparameters section/GitHub but does not fix values in the manuscript body.
  • TF-IDF vocabulary / n-gram / max-features settings
    Winning representation is TF-IDF; exact vectorizer settings are not specified in the paper text and affect sparsity and performance.
  • BERT embedding aggregation (mean pooling of ClinicalBERT/BioBERT/PubMedBERT)
    Dense features are produced by averaging pretrained embeddings; pooling and combination choices are free design decisions that change the representation.
  • Internal 80/20 train-test split used for Table 1 experiments
    Reported training-phase AUROC/F1 depend on this split choice, kept fixed across experiments.
  • T label remapping from {1–4} to {0–3}
    Modeling choice that defines the classification target for T; stated in Dataset/Methodology sections.
assumptions (4)
  • domain assumption TCGA structured clinical metadata T/N/M labels are correct ground truth for the free-text pathology reports.
    All supervised metrics treat these metadata labels as gold; invoked throughout Dataset and Methodology.
  • domain assumption Predicting T, N, and M independently (with per-category filtering of missing labels) is a valid problem formulation for staging support.
    Stated as the task framing; ignores joint stage constraints that clinicians use.
  • ad hoc to paper Mean-pooled 512-token biomedical BERT embeddings and/or TF-IDF capture enough signal from long pathology reports for staging.
    Limitations section admits truncation of reports longer than 512 tokens and long-document challenges; still used as primary deep representations.
  • domain assumption Standard classification metrics (AUROC, F1, Macro-F1) on the shared-task splits are adequate evidence of utility as a clinical baseline.
    Conclusion and abstract evaluate success primarily via these scores while also saying the system is not clinically ready.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CaresAI at SMM4H-HeaRD 2026: Predicting TNM Staging." pith.science (2026). https://pith.science/paper/BPVRRNS2

@misc{pith2026260703466,
  author       = {Pith},
  title        = {Pith review of: CaresAI at SMM4H-HeaRD 2026: Predicting TNM Staging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BPVRRNS2}},
  note         = {Machine review of arXiv:2607.03466}
}
read the original abstract

This study aims to predict Tumor, Node, and Metastasis (TNM) stage labels independently, with the Cancer Genome Atlas (TCGA) pathology report as the sixth shared task of SMM4H-HeaRD 2026. The problem is framed as three multi-label classification tasks. We explore both classical and deep learning approaches using Term Frequency-Inverse Document Frequency (TF-IDF) features and embeddings from ClinicalBERT, BioBERT, and PubMedBERT. These representations are used with Logistic Regression (LR), Light Gradient Boosting Machine (LightGBM), Feed-Forward Neural Networks (FFNN), and Wide Residual Networks (WRN). Our results show that individual embeddings perform similarly to the TNM label classification, while their combination improves its predictive ability. WRN achieves AUROC scores of 0.839 (T), 0.8502 (N), and 0.803 (M) with F1-scores of 0.622, 0.702, and 0.9337, respectively, for the training phase. LightGBM with TF-IDF performs best with AUROC scores of 0.9368 (T), 0.9524 (N), and 0.8311 (M) and F1-scores of 0.7559 (T), 0.7384 (N), and 0.7017 (M) during the training phase. Furthermore, the result of the Codabench for the test sets indicates a Macro-F1 score of 0.978, 0.957, and 0.879 for the T, N, and M categories respectively for test set 1; while test set 2 records a Macro-F1 score for T, N, and M is 0.807, 0.767, 1.0 respectively. However, performance declined during the evaluation phase of the test sets, a drop from 0.938 to 0.858 of test set 1 to 2, for the Macro-F1 score across all stages; suggesting limitations in model generalizability, sensitivity to class imbalance, and challenges in processing lengthy clinical documents. Although this study provides an efficient baseline model and a reproducible pipeline, further optimization and validation are required before it can be considered suitable for use in a real-world clinical setting.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 1 canonical work pages

  1. [1]

    2017 , publisher =

    TNM Classification of Malignant Tumours , edition =. 2017 , publisher =

    2017

  2. [2]

    Information Processing & Management , volume =

    Salton, Gerard and Buckley, Christopher , title =. Information Processing & Management , volume =

  3. [3]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Ke, Guolin and Meng, Qi and Finley, Thomas and Wang, Taifeng and Chen, Wei and Ma, Weidong and Ye, Qiwei and Liu, Tie-Yan , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  4. [4]

    JAMA Oncology , year =

    Pan, Jian-Ji and Mai, Hai-Qiang and Ng, Wai Tong and others , title =. JAMA Oncology , year =

  5. [5]

    Expert Review of Anticancer Therapy , year =

    Keung, Emily Z and Gershenwald, Jeffrey E , title =. Expert Review of Anticancer Therapy , year =

  6. [6]

    Annals of Oncology , year =

    Park, Y H and Lee, S J and Cho, E Y and others , title =. Annals of Oncology , year =

  7. [7]

    Chapter 2 - The basics of deep learning , booktitle =

    Xichuan Zhou and Haijun Liu and Cong Shi and Ji Liu , keywords =. Chapter 2 - The basics of deep learning , booktitle =. 2022 , isbn =. doi:https://doi.org/10.1016/B978-0-32-385783-3.00009-0 , url =

    work page doi:10.1016/b978-0-32-385783-3.00009-0 2022

  8. [8]

    Contemporary Oncology , year =

    Tomczak, Katarzyna and Czerwinska, Patrycja and Wiznerowicz, Maciej , title =. Contemporary Oncology , year =

Show all 30 references
  1. [9]

    Patterns , year =

    Kefeli, Jenna and Tatonetti, Nicholas , title =. Patterns , year =

  2. [10]

    Artificial Intelligence Review , year =

    Hands, Isaac and Kavuluru, Ramakanth , title =. Artificial Intelligence Review , year =

  3. [11]

    and Tsang, Kevin K

    Kefeli, Jenna and Berkowitz, Jacob and Acitores Cortina, Jose M. and Tsang, Kevin K. and Tatonetti, Nicholas P. , title =. Nature Communications , year =

  4. [12]

    Proceedings of the ACM Conference on Health, Inference, and Learning , year =

    Huang, Kexin and Altosaar, Jaan and Ranganath, Rajesh , title =. Proceedings of the ACM Conference on Health, Inference, and Learning , year =

  5. [13]

    Bioinformatics , year =

    Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo , title =. Bioinformatics , year =

  6. [14]

    ACM Transactions on Computing for Healthcare , year =

    Gu, Yu and Tinn, Robert and Cheng, Hao and Lucas, Michael and Usuyama, Naoto and Liu, Xiaodong and Naumann, Tristan and Gao, Jianfeng and Poon, Hoifung , title =. ACM Transactions on Computing for Healthcare , year =

  7. [15]

    BMC Medical Informatics and Decision Making , year =

    Jo, Eunbeen and Cho, Eunbi and Lee, Yebin and Song, Sanghoun and Joo, Hyung Joon , title =. BMC Medical Informatics and Decision Making , year =

  8. [16]

    Scientific Reports , year =

    Veeranki, Sai Pavan Kumar and Abdulnazar, Akhila and Kramer, Diether and Kreuzthaler, Markus and Lumenta, David Benjamin , title =. Scientific Reports , year =

  9. [17]

    Proceedings of the British Machine Vision Conference , year =

    Zagoruyko, Sergey and Komodakis, Nikos , title =. Proceedings of the British Machine Vision Conference , year =

  10. [18]

    arXiv preprint arXiv:2412.21022 , year =

    Petridis, Christos , title =. arXiv preprint arXiv:2412.21022 , year =

    arXiv

  11. [19]

    De Angeli, Kevin and Gao, Shang and Danciu, Ioana and Durbin, Eric B. and Wu, Xiao-Cheng and Stroup, Antoinette and Doherty, Jennifer and Schwartz, Stephen and Wiggins, Charles and Damesyn, Mark and Coyle, Linda and Penberthy, Lynne and Tourassi, Georgia D. and Yoon, Hong-Jun ...

  12. [20]

    and Doherty, Jennifer and Stroup, Antoinette and Coyle, Linda and Penberthy, Lynne and Tourassi, Georgia , title =

    De Angeli, Kevin and Gao, Shang and Alawad, Mohammed and Yoon, Hong-Jun and Schaefferkoetter, Noah and Wu, Xiao-Cheng and Durbin, Eric B. and Doherty, Jennifer and Stroup, Antoinette and Coyle, Linda and Penberthy, Lynne and Tourassi, Georgia , title =. BMC Bioinformatics , year =

  13. [21]

    and Doherty, Jennifer and Stroup, Antoinette and Coyle, Linda and Tourassi, Georgia , title =

    Gao, Shang and Alawad, Mohammed and Young, M Todd and Gounley, John and Schaefferkoetter, Noah and Yoon, Hong Jun and Wu, Xiao-Cheng and Durbin, Eric B. and Doherty, Jennifer and Stroup, Antoinette and Coyle, Linda and Tourassi, Georgia , title =. IEEE Journal of Biomedical an...

  14. [22]

    Scientific Reports , year =

    Jamil, Arslan and Hanif, Muhammad Kashif and Sarwar, Muhammad Umer and Khan, Muhammad Irfan , title =. Scientific Reports , year =

  15. [23]

    IEEE Access , year =

    Le, Thanh-Dung and Jouvet, Philippe and Noumeir, Rita , title =. IEEE Access , year =

  16. [24]

    Entropy , year =

    Feng, Xiaoyue and Liang, Yanchun and Shi, Xiaohu and Xu, Dong and Wang, Xu and Guan, Renchu , title =. Entropy , year =

  17. [25]

    and Lu-Yao, Grace and Yang, Christopher C

    Chang, Chia-Hsuan and Lucas, Mary M. and Lu-Yao, Grace and Yang, Christopher C. , title =. IEEE International Conference on Healthcare Informatics (ICHI) , year =

  18. [26]

    and Liechty, Benjamin L

    Saluja, Rachit and Rosenthal, Jacob and Windon, Annika and Artzi, Yoav and Pisapia, David J. and Liechty, Benjamin L. and Sabuncu, Mert R. , title =. Scientific Reports , year =

  19. [27]

    Machine Learning , volume =

    Leo Breiman , title =. Machine Learning , volume =. 2001 , doi =

    2001

  20. [28]

    Weinberger , title =

    Chuan Guo and Geoff Pleiss and Yu Sun and Kilian Q. Weinberger , title =. Proceedings of the 34th International Conference on Machine Learning , pages =

  21. [29]

    ARPN Journal of Engineering and Applied Sciences , volume=

    A statistical comparison of logistic regression and different Bayes classification methods for machine learning , author=. ARPN Journal of Engineering and Applied Sciences , volume=

  22. [30]

    Global cancer burden growing, amidst mounting need for services , year =

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.