{"id":"6553188e-b12f-4677-926d-615623daf785","arxiv_id":"2607.28538","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LLM-written, evidence-linked local feature programs plus a Random Forest reach 81% leave-one-hospital scar classification accuracy, 10 points above BiomedCLIP, without image upload.","lead":"ScaFE turns an LLM into a clinical feature engineer: it writes local, auditable code that measures scar traits from photos, then a small Random Forest classifies keloid vs hypertrophic scar. On 600 images from three hospitals it beats strong vision baselines by about 10 points and stays ahead with only 10% of the training data, without sending images to the model.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The load-bearing risk is that validation-guided program search on only two development sites can overfit site-shared photographic cues that do not transfer beyond the three-hospital cohort.","rationale":"The reader correctly flags that ordinary photos and sandboxable programs may lack full clinical signal, and that three-site external validity plus non-reproducible web-LLM search keep the paper short of full ACCEPT. That is real but slightly mis-centered: the manuscript already frames ScaFE as assistive visual support and shows large margins over handcrafted features and a local VLM, so “photos are incomplete” alone does not overturn the comparative claim inside this cohort. The more load-bearing internal risk is whether validation-guided synthesis on two development sites invents stable clinical measurements or merely two-site-discriminative visual programs that still succeed on a third similar hospital. Experimental hygiene (patient-level splits, re-search under subsampling, leakage controls) is strong, so I do not move to REJECT; I keep CONDITIONAL and only sharpen the condition—code/artifacts plus a true held-out-hospital or multi-cohort check—rather than treating palpation incompleteness as the decisive weak assumption. Agreement with the reader is partial: same verdict band and reproducibility/external-validity caution, different primary failure mode.","tokens_in":13799,"tokens_out":749,"duration_ms":14854,"concrete_test":"Hold out one fixed hospital as a never-touched meta-test. On the other two, run the full ScaFE search under 2-fold site-out (or pooled development with an inner site-blocked val) exactly as in §5.1, freeze g*/h*, then evaluate once on the meta-test hospital. If meta-test BAcc falls to within ~2–3 points of BiomedCLIP (or the +10 pp margin collapses), the original LOSO gains were partly two-site search overfitting rather than transferable feature programs.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (81.0% site-macro BAcc, +10.0 over BiomedCLIP; +11.8 at 10% data) rests on nested leave-one-site-out with T=3 refinement rounds that repeatedly query D_val from the two development hospitals (Eqs. 3–5, §4.3, Alg. 1). The paper correctly notes D_val is an optimization set, not a sealed test, and that three sites cannot support population-of-hospitals inference (§5.1). The weakest link is therefore not merely that photographs omit palpation (the reader’s point), but that the LLM+SHAP loop can discover and amplify any visually measurable cue that separates KD/HS inside the two-site development pool—illumination, camera mix, skin-tone/body-site imbalance, or labeling style—even when those cues are not the clinical constructs named in the evidence records. Handcrafted+RF (same RF, no LLM search) is only 61.3%, so the gain is search-driven; with M×T=12 candidates and multi-round feedback, that search has enough capacity to fit two-site idiosyncrasies that still look like “cross-site” success when the third hospital is drawn from a similar acquisition era. Ablations remove literature/SHAP but do not replace the two-site val loop with a search that never sees the held-out site’s domain. If the reported margin is partly two-site photographic collusion rather than stable clinical measurements, the headline generalization and data-efficiency claims weaken.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"ScaFE proposes transferring clinical knowledge from a web-enabled LLM into deterministic, locally executed Python feature programs for keloid vs. hypertrophic scar classification, rather than sending images to a VLM or training end-to-end image models. Programs are generated from retrieved clinical evidence, refined over T rounds using only aggregate validation diagnostics (confusion counts, balanced accuracy, invalid/constant rates, global SHAP), then paired with a fixed Random Forest. On 600 unique-patient photographs from three hospitals under nested leave-one-site-out evaluation, ScaFE reports 81.0% site-macro balanced accuracy (+10.0 pp over BiomedCLIP) and retains a large lead at 10% development data (72.0%, +11.8 pp), with refinement raising executability from 66.7% to 95.0% and strong evidence-pass rates after filtering.","tokens_in":14209,"tokens_out":1520,"duration_ms":42105,"significance":"If the gains reflect stable, clinically grounded measurements rather than two-site photographic idiosyncrasies, the work is a meaningful contribution to data-efficient and governance-compatible medical image learning. The separation of roles—LLM as knowledge source, sandboxed code as measurer, lightweight learner as decision boundary—is cleanly formulated (§3–4) and addresses a real tension between hosted multimodal models and hospital data constraints. Strengths include nested patient-level leave-one-site-out, matched partitions, near-chance leakage/permutation controls, ablations that isolate literature grounding and semantically aligned feedback (including shuffled-feedback), and data-efficiency curves that re-run the full search inside each subsample rather than only refitting the classifier. The audit of executability, evidence gating, and SHAP faithfulness is also a useful template for program-synthesis pipelines in medicine.","major_comments":[{"comment":"§5.1 and Eqs. (3)–(5)/Alg. 1: The central cross-site claim rests on three leave-one-site-out folds in which program search repeatedly optimizes on D_val drawn from only the two development hospitals. The manuscript correctly notes that three sites cannot support population-of-hospitals inference, but this understates a load-bearing risk: with M×T=12 candidates and multi-round SHAP/confusion feedback, the loop can amplify any visually measurable cue that separates KD/HS inside the two-site pool (illumination, camera mix, skin-tone/body-site imbalance, labeling style) even when evidence records name clinical constructs. Handcrafted+RF at 61.3% shows the gain is search-driven; ablations remove literature/SHAP but do not break dependence on the two-site val loop. Please add analyses that stress this risk—e.g., reporting retained feature definitions and distributions by site, site-stratified","section":"§5.1, Eqs. (3)–(5), Algorithm 1"},{"comment":"§5.6 (RQ4) and §4.2–4.3: The claim that LLM knowledge is transferred into clinically meaningful measurements is only weakly evidenced in the main text. Expert assessment of source–feature mappings and visual proxies is deferred to Supplementary A.6, while the main results emphasize BAcc and executability. For the knowledge-transfer story to carry the +10 pp claim, the main paper should characterize the final feature sets (names, operational definitions, top SHAP features, examples of what each measures on images) and show that high-importance dimensions correspond to intended scar attributes rather than acquisition correlates. Without that, “evidence-grounded” largely means bibliographic linkage plus local executability, not demonstrated construct validity.","section":"§5.6, §4.2–4.3"},{"comment":"§5.2 baselines: Local VLM-direct uses MedGemma-1.5-4B-IT with label-likelihood normalization and lands at 66.2%, below frozen BiomedCLIP probing. Given that a core motivation is superiority over direct VLM diagnosis (and citation of Shiraishi et al.), the comparison should include a stronger multimodal baseline under the same local/no-train constraint (larger medical VLM if deployable, or multiple prompting/decoding regimes with the same budget as M×T). Otherwise the “rather than direct VLM decisions” contrast risks overstating the margin against a weak instantiation of that paradigm, even though the foundation-model probe comparisons remain informative.","section":"§5.2, Table 1"}],"minor_comments":[{"comment":"Abstract vs. body wording on evidence rates is inconsistent across versions (e.g., “verified evidence for 91.7% of the final features” vs. candidate evidence-pass 91.7% with final filtering to 100% coverage). Align terminology so “candidate pass” and “retained coverage” are not conflated.","section":"Abstract, §5.6"},{"comment":"Table 1 in the main text reports only BAcc; macro-F1, AUROC, ECE, and CIs are supplementary. For the primary comparison table, at least CIs on the site-macro average (or the paired improvement CI already mentioned in text) would help readers without opening the supplement.","section":"Table 1, §5.3"},{"comment":"Figure 2(b) mixes executability/contract/evidence rates with validation BAcc on one axis; consider dual axes or separate panels so scale differences are clearer.","section":"Figure 2"},{"comment":"Implementation cites gpt-4.1-2025-04-14 and access dates in 2026; ensure model IDs, prompts, and archived run artifacts are pinned for reproducibility as claimed in §5.1/A.2.","section":"§5.1"},{"comment":"Minor prose duplicates and spacing artifacts appear (e.g., concatenated words in the introduction and related work). A copy-edit pass would improve readability.","section":"§1–2"}],"recommendation":"major_revision","confidential_remarks":"Solid, timely systems paper with a clear privacy/audit angle and better-than-average experimental hygiene for LLM-in-the-loop medical imaging. I do not see equation-level circularity; the skeptic concern about two-site photographic collusion is the real threat to the headline claim and is fixable with tighter claims plus main-text feature audits rather than new multi-hospital data. Fit is appropriate for a CV/medical imaging venue that values methods plus careful evaluation. Borderline major vs minor revision; I chose major because the generalization and construct-validity points are load-bearing for how the abstract is currently written."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a practical pipeline paper, not a theory one. They keep photos off the LLM, turn web-grounded clinical concepts into sandboxed Python features, refine with only aggregate confusion/SHAP stats, and freeze before opening the held-out hospital. On 600 unique patients, three leave-one-site-out folds, they report 81% site-macro BAcc versus BiomedCLIP at 71%, and the lead grows under 10% development data when they re-run the full search inside each subsample. That last detail matters; a lot of “low-data” curves cheat by freezing features discovered on the full set.\n\nWhat is actually new is the packaging, not any single ingredient. Concept bottlenecks, handcrafted scar scores, medical VLMs, and LLM codegen already exist. The useful move is the evidence contract plus local execution plus aggregate-only repair, aimed at governance and audit rather than “let the VLM diagnose.” Experimental hygiene is above average for this area: matched partitions, permutation/random-encoder controls near chance, ablations that hit literature, SHAP, and shuffled feedback, second-LLM check, calibration numbers. RF as fixed evaluator is a reasonable choice and they swap classifiers without the story collapsing.\n\nSoft spots, in proportion. Three hospitals is thin for any population-of-sites claim; they say so. The stress-test worry is fair: multi-round search on two-site validation can amplify shared photographic cues (camera mix, lighting, body-site imbalance) that still “transfer” to a third similar site. Handcrafted+RF at ~61% shows the gain is search-driven, which cuts both ways. Photographs also omit palpation and history, so the clinical ceiling is bounded by what short deterministic code can measure. Reproducibility is the other real gap—live web search and a dated hosted LLM are not bit-stable, and without released programs/artifacts a referee cannot fully audit the features. None of that looks like a load-bearing contradiction in the reported numbers; it is external validity and artifact risk.\n\nWho it is for: people doing clinical imaging under data-governance constraints, or anyone interested in LLM-as-feature-engineer rather than LLM-as-doctor. I would bring it to a methods reading group if the room cares about medical ML or neurosymbolic pipelines. It deserves serious peer review, not a desk reject—ask for code, pinned evidence logs, and a clearer limit statement on three-site generalization. I would engage.","headline":"Clean methods paper: image-blind LLM feature programs beat foundation probes on a real three-hospital scar task, with honest nesting and real soft spots on site count and search overfitting.","tokens_in":14833,"tokens_out":616,"would_cite":true,"duration_ms":18158,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"LLM clinical knowledge can classify keloids versus hypertrophic scars by writing local feature programs, not by looking at patient photos.","keywords":["scar classification","keloid","hypertrophic scar","LLM feature engineering","program synthesis","data-efficient medical imaging","cross-site generalization","auditable clinical AI"],"falsifier":"Rerun the same leave-one-site-out protocol on a new hospital or prospective cohort where the true distinction depends on palpation, three-dimensional growth, or history the sandbox cannot measure; if ScaFE then falls to baseline levels while image models hold up, the central claim fails.","tokens_in":14605,"feed_emoji":"🔬","tokens_out":882,"duration_ms":18575,"temperature":0.7,"pith_summary":"Distinguishing keloids from hypertrophic scars in ordinary clinical photos is hard when labeled cases are scarce and hospitals capture images differently. Sending photos to a hosted vision model raises privacy and audit problems, and end-to-end networks need a lot of specialist data. This paper argues that a language model can instead act as a feature engineer: it retrieves clinical sources, writes short deterministic programs that measure visually assessable scar attributes, and revises those programs using only aggregate validation statistics and feature-importance summaries while raw images stay inside the hospital. A simple Random Forest then classifies the resulting table of measurements. On 600 photos from three hospitals, the approach beats strong image baselines under leave-one-site-out testing and keeps a large lead when only a tenth of the development data is available. The point is that clinical knowledge can enter the pipeline as inspectable code rather than as an opaque image diagnosis.","feed_headline":"LLM writes scar features, beats image models by 10 points","feed_subtitle":"Local programs, not photo uploads: 81% cross-hospital accuracy with far less labeled data","key_machinery":"ScaFE (Scar Feature Engineering): a validation-guided loop in which a web-enabled LLM generates evidence-linked Python feature programs that run in a restricted local sandbox; only aggregate errors, confusion counts, balanced accuracy, and SHAP summaries return for iterative repair, after which a fixed Random Forest learns the decision boundary on the structured features.","core_discovery":"Under patient-level leave-one-site-out evaluation on 600 photographs from three hospitals, ScaFE reaches 81.0% site-macro balanced accuracy—10 points above the strongest baseline—by transferring LLM clinical knowledge into deterministic local feature programs rather than direct vision-language diagnosis, and it retains a 11.8-point lead at 10% of the development data.","pith_inferences":["The same bounded program-synthesis pattern could transfer to other photo-only dermatology distinctions that clinicians already describe with named visual attributes.","If sandbox libraries cannot express the decisive visual cues, the method will plateau even with perfect literature retrieval—suggesting a natural stress test against tasks that need depth or tactile proxies.","Governance teams may prefer this split (hosted LLM for code, local RF for decisions) as a template whenever data-residency rules block direct multimodal API calls."],"forward_implications":["Cross-site scar triage can keep photographs inside the hospital while still using hosted LLM knowledge at search time.","Low-data medical image tasks that already have named clinical scales can be attacked as feature-program search rather than end-to-end fine-tuning.","Every retained feature ships with a source passage and executable definition, so auditors can inspect why a prediction was made without opening the LLM.","Refinement with only aggregate SHAP and confusion feedback can raise program executability and evidence coverage without exposing patient-level outputs."],"fun_headline_variants":["ScaFE: LLM feature programs hit 81% scar accuracy, +10 pts","Local LLM scar features beat BiomedCLIP by 10 points","Data-efficient scar classification via LLM-written programs","Cross-hospital 81% scar accuracy without sending photos","LLM programs retain 11.8-pt lead at 10% labeled data"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Ordinary clinical photographs, measured by short deterministic programs under a restricted library, carry enough keloid-versus-hypertrophic-scar signal to generalize to an unseen hospital when the language model only sees aggregate validation feedback.","fun_headline_variants_meta":{"raw":{"variants":["ScaFE: LLM feature programs hit 81% scar accuracy, +10 pts","Local LLM scar features beat BiomedCLIP by 10 points","Data-efficient scar classification via LLM-written programs","Cross-hospital 81% scar accuracy without sending photos","LLM programs retain 11.8-pt lead at 10% labeled data"]},"model":"grok-4.5","effort":"low","cost_usd":0.002086,"raw_usage":{"total_tokens":956,"prompt_tokens":858,"num_sources_used":0,"completion_tokens":78,"cost_in_usd_ticks":20864000,"prompt_tokens_details":{"text_tokens":858,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":20,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":858,"tokens_out":78,"duration_ms":2266,"temperature":1.0,"reasoning_tokens":20,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T04:16:00.982790+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Rerun the same leave-one-site-out protocol on a new hospital or prospective cohort where the true distinction depends on palpation, three-dimensional growth, or history the sandbox cannot measure; if ScaFE then falls to baseline levels while image models hold up, the central claim fails.","supporting_citations":[],"review_version":1}