REVIEW 3 major objections 5 minor 31 references
LLM clinical knowledge can classify keloids versus hypertrophic scars by writing local feature programs, not by looking at patient photos.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-31 04:16 UTC pith:SEFH4HQT
load-bearing objection Clean methods paper: image-blind LLM feature programs beat foundation probes on a real three-hospital scar task, with honest nesting and real soft spots on site count and search overfitting. the 3 major comments →
ScaFE: Data-Efficient Scar Classification with LLM-Generated Clinical Feature Programs
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Under patient-level leave-one-site-out evaluation on 600 photographs from three hospitals, ScaFE reaches 81.0% site-macro balanced accuracy—10 points above the strongest baseline—by transferring LLM clinical knowledge into deterministic local feature programs rather than direct vision-language diagnosis, and it retains a 11.8-point lead at 10% of the development data.
What carries the argument
ScaFE (Scar Feature Engineering): a validation-guided loop in which a web-enabled LLM generates evidence-linked Python feature programs that run in a restricted local sandbox; only aggregate errors, confusion counts, balanced accuracy, and SHAP summaries return for iterative repair, after which a fixed Random Forest learns the decision boundary on the structured features.
Load-bearing premise
Ordinary clinical photographs, measured by short deterministic programs under a restricted library, carry enough keloid-versus-hypertrophic-scar signal to generalize to an unseen hospital when the language model only sees aggregate validation feedback.
What would settle it
Rerun the same leave-one-site-out protocol on a new hospital or prospective cohort where the true distinction depends on palpation, three-dimensional growth, or history the sandbox cannot measure; if ScaFE then falls to baseline levels while image models hold up, the central claim fails.
If this is right
- Cross-site scar triage can keep photographs inside the hospital while still using hosted LLM knowledge at search time.
- Low-data medical image tasks that already have named clinical scales can be attacked as feature-program search rather than end-to-end fine-tuning.
- Every retained feature ships with a source passage and executable definition, so auditors can inspect why a prediction was made without opening the LLM.
- Refinement with only aggregate SHAP and confusion feedback can raise program executability and evidence coverage without exposing patient-level outputs.
Where Pith is reading between the lines
- The same bounded program-synthesis pattern could transfer to other photo-only dermatology distinctions that clinicians already describe with named visual attributes.
- If sandbox libraries cannot express the decisive visual cues, the method will plateau even with perfect literature retrieval—suggesting a natural stress test against tasks that need depth or tactile proxies.
- Governance teams may prefer this split (hosted LLM for code, local RF for decisions) as a template whenever data-residency rules block direct multimodal API calls.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. ScaFE proposes transferring clinical knowledge from a web-enabled LLM into deterministic, locally executed Python feature programs for keloid vs. hypertrophic scar classification, rather than sending images to a VLM or training end-to-end image models. Programs are generated from retrieved clinical evidence, refined over T rounds using only aggregate validation diagnostics (confusion counts, balanced accuracy, invalid/constant rates, global SHAP), then paired with a fixed Random Forest. On 600 unique-patient photographs from three hospitals under nested leave-one-site-out evaluation, ScaFE reports 81.0% site-macro balanced accuracy (+10.0 pp over BiomedCLIP) and retains a large lead at 10% development data (72.0%, +11.8 pp), with refinement raising executability from 66.7% to 95.0% and strong evidence-pass rates after filtering.
Significance. If the gains reflect stable, clinically grounded measurements rather than two-site photographic idiosyncrasies, the work is a meaningful contribution to data-efficient and governance-compatible medical image learning. The separation of roles—LLM as knowledge source, sandboxed code as measurer, lightweight learner as decision boundary—is cleanly formulated (§3–4) and addresses a real tension between hosted multimodal models and hospital data constraints. Strengths include nested patient-level leave-one-site-out, matched partitions, near-chance leakage/permutation controls, ablations that isolate literature grounding and semantically aligned feedback (including shuffled-feedback), and data-efficiency curves that re-run the full search inside each subsample rather than only refitting the classifier. The audit of executability, evidence gating, and SHAP faithfulness is also a useful template for program-synthesis pipelines in medicine.
major comments (3)
- [§5.1, Eqs. (3)–(5), Algorithm 1] §5.1 and Eqs. (3)–(5)/Alg. 1: The central cross-site claim rests on three leave-one-site-out folds in which program search repeatedly optimizes on D_val drawn from only the two development hospitals. The manuscript correctly notes that three sites cannot support population-of-hospitals inference, but this understates a load-bearing risk: with M×T=12 candidates and multi-round SHAP/confusion feedback, the loop can amplify any visually measurable cue that separates KD/HS inside the two-site pool (illumination, camera mix, skin-tone/body-site imbalance, labeling style) even when evidence records name clinical constructs. Handcrafted+RF at 61.3% shows the gain is search-driven; ablations remove literature/SHAP but do not break dependence on the two-site val loop. Please add analyses that stress this risk—e.g., reporting retained feature definitions and distributions by site, site-stratified
- [§5.6, §4.2–4.3] §5.6 (RQ4) and §4.2–4.3: The claim that LLM knowledge is transferred into clinically meaningful measurements is only weakly evidenced in the main text. Expert assessment of source–feature mappings and visual proxies is deferred to Supplementary A.6, while the main results emphasize BAcc and executability. For the knowledge-transfer story to carry the +10 pp claim, the main paper should characterize the final feature sets (names, operational definitions, top SHAP features, examples of what each measures on images) and show that high-importance dimensions correspond to intended scar attributes rather than acquisition correlates. Without that, “evidence-grounded” largely means bibliographic linkage plus local executability, not demonstrated construct validity.
- [§5.2, Table 1] §5.2 baselines: Local VLM-direct uses MedGemma-1.5-4B-IT with label-likelihood normalization and lands at 66.2%, below frozen BiomedCLIP probing. Given that a core motivation is superiority over direct VLM diagnosis (and citation of Shiraishi et al.), the comparison should include a stronger multimodal baseline under the same local/no-train constraint (larger medical VLM if deployable, or multiple prompting/decoding regimes with the same budget as M×T). Otherwise the “rather than direct VLM decisions” contrast risks overstating the margin against a weak instantiation of that paradigm, even though the foundation-model probe comparisons remain informative.
minor comments (5)
- [Abstract, §5.6] Abstract vs. body wording on evidence rates is inconsistent across versions (e.g., “verified evidence for 91.7% of the final features” vs. candidate evidence-pass 91.7% with final filtering to 100% coverage). Align terminology so “candidate pass” and “retained coverage” are not conflated.
- [Table 1, §5.3] Table 1 in the main text reports only BAcc; macro-F1, AUROC, ECE, and CIs are supplementary. For the primary comparison table, at least CIs on the site-macro average (or the paired improvement CI already mentioned in text) would help readers without opening the supplement.
- [Figure 2] Figure 2(b) mixes executability/contract/evidence rates with validation BAcc on one axis; consider dual axes or separate panels so scale differences are clearer.
- [§5.1] Implementation cites gpt-4.1-2025-04-14 and access dates in 2026; ensure model IDs, prompts, and archived run artifacts are pinned for reproducibility as claimed in §5.1/A.2.
- [§1–2] Minor prose duplicates and spacing artifacts appear (e.g., concatenated words in the introduction and related work). A copy-edit pass would improve readability.
Circularity Check
No significant circularity: held-out site performance is not defined by the validation objective, and claims rest on sealed leave-one-site-out comparisons rather than by-construction identities.
full rationale
ScaFE’s load-bearing claim is empirical cross-site balanced accuracy under patient-level leave-one-site-out (Table 1; §5.1–5.3), not a first-principles identity. The derivation chain is: LLM + web evidence → candidate feature programs g → local execution → fixed RF fitted on D_fit → selection by validation BAcc on D_val (Eqs. 3–5, 11, 14; Alg. 1) → freeze g* and h* → single evaluation on sealed D_test. Repeated use of D_val is explicitly labeled an optimization set (“only the sealed test set estimates generalization,” §4.3), which is standard nested model selection, not Eq-X-equals-Eq-Y circularity: the reported 81.0% site-macro BAcc and +10.0 pp vs BiomedCLIP are measured on hospitals never seen during search or fitting. SHAP/confusion feedback guide repair but never expose test labels or patient-level outputs to the LLM (§4.3). Baselines (handcrafted+RF, ResNet, EfficientNet, DINOv3, Derm Foundation, BiomedCLIP, local VLM) and negative controls (random encoder, label permutation) are external comparators on the same partitions. Related-work self-citation (Ye et al. 2024 on prototypes) is not load-bearing for the accuracy claim. No uniqueness theorem, fitted scalar renamed as prediction, or definitional reduction of the test metric to the search objective appears in the equations or evaluation protocol. Overfitting to two-site photographic cues is a generalization risk, not circularity under this pass’s criteria.
Axiom & Free-Parameter Ledger
free parameters (6)
- M candidates per round =
4
- T refinement rounds =
3
- LLM sampling temperature =
0.2
- RF hyperparameters (n_trees, max_depth, min_leaf, feature subsample) =
500 trees, depth 8, min leaf 3, sqrt(Kg) features
- Input resize resolution for ScaFE =
512×512
- fit/val split ratio inside development sites =
80% fit / 20% val
axioms (5)
- domain assumption Named visually assessable scar concepts from instruments such as VSS/POSAS can be operationalized as deterministic numeric measurements on 2D RGB photos sufficient for KD vs HS discrimination.
- ad hoc to paper Aggregate validation diagnostics (confusion counts, BAcc, invalid/constant rates, global SHAP) without patient-level outputs suffice for an LLM to repair and improve feature programs.
- ad hoc to paper A restricted local execution interface (approved libraries, no network/labels/metadata) yields faithful, non-leaky measurements of the intended clinical attributes.
- domain assumption Leave-one-site-out on three hospitals with patient-level splits is an adequate test of cross-site generalization for the claimed margins.
- standard math Standard supervised learning and bootstrap testing assumptions (i.i.d. patients within site after de-duplication; RF/variance estimates meaningful at N=200/site).
invented entities (2)
-
ScaFE validation-guided feature-program search loop
no independent evidence
-
Evidence-gated executable scar feature programs
no independent evidence
read the original abstract
Classifying pathological scars from clinical photographs requires distinguishing keloids from hypertrophic scars despite limited expert-labeled data and substantial acquisition variation across hospitals. End-to-end image models remain data-dependent, whereas sending photographs to a hosted vision-language model (VLM) may conflict with local data-governance requirements and yields decisions that are difficult to reproduce and audit. We introduce ScaFE (Scar Feature Engineering), which transfers clinical knowledge from a large language model (LLM) into deterministic, executable feature programs instead of asking the model to diagnose images. A web-enabled LLM retrieves clinical evidence and synthesizes programs that measure visually assessable scar attributes. Candidate programs execute in a restricted local environment, and only aggregate validation statistics and feature-level SHAP summaries are returned for iterative repair and refinement; raw images and patient-level outputs remain local. A lightweight Random Forest then operates on the resulting structured representation. On 600 photographs from three hospitals under leave-one-site-out evaluation, ScaFE achieves 81.0% site-macro balanced accuracy, exceeding the strongest baseline, BiomedCLIP, by 10.0 percentage points. With only 10% of the development data, ScaFE retains 72.0% balanced accuracy and an 11.8-point lead. Iterative refinement also raises the executable-program rate from 66.7% to 95.0%, with verified evidence for 91.7% of the final features. These results show that LLM knowledge can support data-efficient, cross-site medical image classification through local and auditable feature programs rather than direct VLM decisions.
Figures
Reference graph
Works this paper leans on
-
[1]
Angus and Ferguson, Mark W
Bayat, Ardeshir and McGrouther, D. Angus and Ferguson, Mark W. J. Skin scarring. BMJ. 2003
2003
-
[2]
Keloids and hypertrophic scars: pathophysiology, classification, and treatment
Berman, Brian and Maderal, Andrea and Raphael, Briana. Keloids and hypertrophic scars: pathophysiology, classification, and treatment. Dermatologic Surgery. 2017
2017
-
[3]
The Vancouver Scar Scale : an administration tool and its interrater reliability
Baryza, Mary Jo and Baryza, Gregory A. The Vancouver Scar Scale : an administration tool and its interrater reliability. Journal of Burn Care & Rehabilitation. 1995. doi:10.1097/00004630-199509000-00013
-
[4]
Draaijers, Lieneke J. and Tempelman, Fenike R. H. and Botman, Yvonne A. M. and Tuinebreijer, Wim E. and Middelkoop, Esther and Kreis, Robert W. and van Zuijlen, Paul P. M. The Patient and Observer Scar Assessment Scale : a reliable and feasible tool for scar evaluation. Plastic and Reconstructive Surgery. 2004. doi:10.1097/01.PRS.0000122207.28773.56
arXiv 2004
-
[5]
and Ko, Justin and Swetter, Susan M
Esteva, Andre and Kuprel, Brett and Novoa, Roberto A. and Ko, Justin and Swetter, Susan M. and Blau, Helen M. and Thrun, Sebastian. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017. doi:10.1038/nature21056
-
[6]
Deep residual learning for image recognition
He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016
2016
-
[7]
EfficientNet : Rethinking model scaling for convolutional neural networks
Tan, Mingxing and Le, Quoc V. EfficientNet : Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning. 2019
2019
-
[8]
a rli, Nathanael and Chowdhery, Aakanksha and Mansfield, Philip and Demner-Fushman, Dina and Ag \
Singhal, Karan and Azizi, Shekoofeh and Tu, Tao and Mahdavi, S. Sara and Wei, Jason and Chung, Hyung Won and Scales, Nathan and Tanwani, Ajay and Cole-Lewis, Heather and Pfohl, Stephen and Payne, Perry and Seneviratne, Martin and Gamble, Paul and Kelly, Chris and Babiker, Abubakr and Sch \"a rli, Nathanael and Chowdhery, Aakanksha and Mansfield, Philip an...
2023
-
[9]
Capabilities of GPT-4 on medical challenge problems
Nori, Harsha and King, Nicholas and McKinney, Scott Mayer and Carignan, Dean and Horvitz, Eric. Capabilities of GPT-4 on medical challenge problems. arXiv preprint arXiv:2303.13375. 2023
Pith/arXiv arXiv 2023
-
[10]
Liu, Yuan and Jain, Ayush and Eng, Clara and Way, David H. and Lee, Kang and Bui, Peggy and Kanada, Kimberly and de Oliveira Marinho, Guilherme and Gallegos, Jessica and Gabriele, Sara and Gupta, Vishakha and Singh, Nalini and Natarajan, Vivek and Hofmann-Wellenhof, Rainer and Corrado, Greg S. and Peng, Lily H. and Webster, Dale R. and Ai, Dennis and Huan...
2020
-
[11]
Pre-trained multimodal large language model enhances dermatological diagnosis using
Zhou, Juexiao and He, Xiaonan and Sun, Liyuan and Xu, Jiannan and Chen, Xiuying and Chu, Yuetan and Zhou, Longxi and Liao, Xingyu and Zhang, Bin and Afvari, Shawn and Gao, Xin , journal=. Pre-trained multimodal large language model enhances dermatological diagnosis using. 2024 , doi=
2024
-
[12]
Representation learning: A review and new perspectives
Bengio, Yoshua and Courville, Aaron and Vincent, Pascal. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2013
2013
-
[13]
A few useful things to know about machine learning
Domingos, Pedro. A few useful things to know about machine learning. Communications of the ACM. 2012
2012
-
[14]
Neurosymbolic AI : The 3rd wave
Garcez, Artur d'Avila and Lamb, Luis C. Neurosymbolic AI : The 3rd wave. Artificial Intelligence Review. 2023. doi:10.1007/s10462-023-10448-w
-
[15]
Concept bottleneck models
Koh, Pang Wei and Nguyen, Thao and Tang, Yew Siang and Mussmann, Stephen and Pierson, Emma and Kim, Been and Liang, Percy. Concept bottleneck models. International Conference on Machine Learning. 2020
2020
-
[16]
Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest CT
Xie, Yutong and Xia, Yong and Zhang, Jianpeng and Song, Yang and Feng, Dagan and Fulham, Michael and Cai, Weidong. Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest CT. IEEE Transactions on Medical Imaging. 2019
2019
-
[17]
2023 , doi=
Li, Chunyuan and Wong, Cliff and Zhang, Sheng and Usuyama, Naoto and Liu, Haotian and Yang, Jianwei and Naumann, Tristan and Poon, Hoifung and Gao, Jianfeng , journal=. 2023 , doi=
2023
-
[18]
Aesthetic Plastic Surgery , volume=
The potential of chat-based artificial intelligence models in differentiating between keloid and hypertrophic scars: a pilot study , author=. Aesthetic Plastic Surgery , volume=. 2024 , doi=
2024
-
[19]
Random forests
Breiman, Leo. Random forests. Machine learning. 2001
2001
-
[20]
Medical image analysis , volume=
A survey on deep learning in medical image analysis , author=. Medical image analysis , volume=. 2017 , doi=
2017
-
[21]
2024 , url=
Ye, Hangting and Fan, Wei and Song, Xiaozhuang and Zheng, Shun and Zhao, He and Guo, Dandan and Chang, Yi , booktitle=. 2024 , url=
2024
-
[22]
A Unified Approach to Interpreting Model Predictions , url =
Lundberg, Scott M and Lee, Su-In , booktitle =. A Unified Approach to Interpreting Model Predictions , url =
-
[23]
and Naumann, Tristan and Wang, Sheng and Poon, Hoifung , title =
Zhang, Sheng and Xu, Yanbo and Usuyama, Naoto and Xu, Hanwen and Bagga, Jaspreet and Tinn, Robert and Preston, Sam and Rao, Rajesh and Wei, Mu and Valluri, Naveen and Wong, Cliff and Tupini, Andrea and Wang, Yu and Mazzola, Matt and Shukla, Swadheen and Liden, Lars and Gao, Jianfeng and Crabtree, Angela and Piening, Brian and Bifulco, Carlo and Lungren, M...
2025
-
[24]
Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages =
Chen, Tianqi and Guestrin, Carlos , title =. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages =. 2016 , doi =
2016
-
[25]
arXiv preprint arXiv:2508.10104 , year =
Sim. arXiv preprint arXiv:2508.10104 , year =
-
[26]
Kiraly, Atilla P. and Baur, Sebastien and Philbrick, Kenneth and Mahvar, Fereshteh and Yatziv, Liron and Chen, Tiffany and Sterling, Bram and George, Nick and Jamil, Fayaz and Tang, Jing and Bailey, Kai and Ahmed, Faruk and Goel, Akshay and Ward, Abbi and Yang, Lin and Sellergren, Andrew and Matias, Yossi and Hassidim, Avinatan and Shetty, Shravya and Gol...
-
[27]
and Kirmizibayrak, Can and Golden, Daniel , title =
Sellergren, Andrew and Gao, Chufan and Mahvar, Fereshteh and Kohlberger, Timo and Jamil, Fayaz and Traverse, Madeleine and Tono, Alberto and Sadjad, Bashir and Yang, Lin and Lau, Charles and Yatziv, Liron and Chen, Tiffany and Sterling, Bram and Philbrick, Kenneth and Tiwari, Richa and Liu, Yun and Jajoo, Madhuram and Sankarapu, Chandrashekar and Vispute,...
-
[28]
Peter and Ge, Zongyuan , title =
Yan, Siyuan and Yu, Zhen and Primiero, Clare and Vico-Alonso, Cristina and Wang, Zhonghua and Yang, Litao and Tschandl, Philipp and Hu, Ming and Ju, Lie and Tan, Gin and Tang, Vincent and Ng, Aik Beng and Powell, David and Bonnington, Paul and See, Simon and Magnaterra, Elisabetta and Ferguson, Peter and Nguyen, Jennifer and Guitera, Pascale and Banuls, J...
2025
-
[29]
Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =
Yan, Siyuan and Hu, Ming and Jiang, Yiwen and Li, Xieji and Fei, Hao and Tschandl, Philipp and Kittler, Harald and Ge, Zongyuan , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =
-
[30]
and Hla, Michael and Moor, Michael and Adithan, Subathra and Kwak, Stephen and Topol, Eric J
Rao, Vishwanatha M. and Hla, Michael and Moor, Michael and Adithan, Subathra and Kwak, Stephen and Topol, Eric J. and Rajpurkar, Pranav , title =. Nature , volume =. 2025 , doi =
2025
-
[31]
and Diao, James A
Buckley, Thomas A. and Diao, James A. and Srivastava, Cam N. and Brodeur, Peter G. and Rajpurkar, Pranav and Rodman, Adam and Manrai, Arjun K. , title =. Nature Communications , volume =. 2026 , doi =
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.