{"id":"f7420dac-1605-4ece-8785-fbc1188dac75","arxiv_id":"2607.14195","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A random forest on six curvature/torsion features classifies ICA-C1 tortuosity with 0.92/0.86 Macro-F1 in binary/ternary tasks.","lead":"This paper combines discrete-geometry tortuosity features, information-gain feature selection, and random forest classification to automatically grade internal carotid artery (ICA-C1) tortuosity from CT angiography centerlines. On 379 vessels it reports Macro-F1 scores of 0.92 (binary severe/non-severe) and 0.86 (three-class morphological grading).","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full-data information-gain feature selection before nested cross-validation leaks label information into the outer folds, so the reported Macro-F1 values may be optimistically biased.","rationale":"The reader's CONDITIONAL verdict is appropriate. The paper is transparent about many limitations and the geometric modeling is plausible, but the headline performance numbers are the central claim, and the evaluation protocol has a concrete leak: information-gain feature selection uses the full labeled dataset before nested cross-validation. This can inflate the reported Macro-F1 even though the RF hyperparameters are selected in an inner loop. The reader's stated weakest assumption—subjective physician labels—is a real and important limitation for clinical interpretation, but it is a premise about the target construct rather than a defect in the reported benchmark. The feature-selection leakage is more directly load-bearing for the numerical claim itself. I agree with the reader that external validation and inter-rater reliability are needed, but the immediate, checkable threat to the reported F1 scores is the selection-bias path. Since the reader already assigns CONDITIONAL, my concern does not change the verdict; it sharpens the reason why the conditional should remain in place.","tokens_in":18481,"tokens_out":3802,"duration_ms":41981,"concrete_test":"Re-run the full pipeline with feature selection moved inside the outer training folds: for each of the 5 outer folds, recompute information gain and Spearman correlation on the training fold only, select the six features, then train RF and evaluate on the held-out fold. Compare the resulting out-of-fold Macro-F1 with the reported 0.9206/0.8626. If the drop exceeds roughly 0.02, the full-data selection is materially optimistic and the reported numbers should be revised downward.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim (Macro-F1 0.9206 binary, 0.8626 ternary) rests on an evaluation protocol in which the six-feature subset is selected using the full labeled dataset before cross-validation. Section IV-A explicitly states that information gain was calculated from the physician's ternary morphological classification and that Spearman correlation pruning was then applied to reduce the initial 13 features to 6. Section III-D-2 describes a nested 5-fold cross-validation, but that nesting protects only hyperparameter selection; the outer fold labels have already influenced which features are retained. Because the feature-selection step is supervised and uses all 379 samples, the out-of-fold predictions are not strictly independent of the test labels. This is a classic selection-bias pathway and tends to inflate Macro-F1, especially with N=379 and highly correlated geometric features. Thus the headline numbers may overstate generalization even if the geometric derivations and the RF implementation are correct. This concern is more specific than the label-subjectivity issue: it threatens the descriptive benchmark itself, not only its clinical interpretation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a hybrid framework for classifying ICA-C1 vascular tortuosity using 379 clinical centerlines (193 subjects). Thirteen geometric features are computed from centerlines by discrete least-squares derivative estimation (curvature, torsion, integrals), and a cascade of information-gain ranking and Spearman correlation pruning reduces them to six features (TI, AC, TC, AC/AT, AT, TT). A random forest is compared with LR, SVM, and XGBoost under nested 5-fold cross-validation; the authors report Macro-F1 0.9206 for binary severe/non-severe screening and 0.8626 for ternary straight/low/high grading. A Morphological Risk Index (MRI) is defined from RF impurity importances with thresholds set at midpoints of group means. The paper also illustrates the insufficiency of TI with matched phantom curves.","tokens_in":18762,"tokens_out":9978,"duration_ms":99623,"significance":"The geometric feature set and the phantom demonstration in Section II-B are useful and clearly presented; the ablation and model comparisons are appropriate in spirit, and the authors explicitly label the MRI as exploratory and list limitations. If the evaluation protocol were corrected (feature selection nested inside CV, subject-level grouping, uncertainty quantification), the central claim of high Macro-F1 would be a meaningful, reproducible contribution to objective tortuosity grading. At present, however, the headline numbers are not trustworthy because of selection bias and unaccounted data dependence.","major_comments":[{"comment":"The IG ranking and Spearman pruning are computed once on the full labeled dataset before the nested cross-validation. Thus the outer-fold test labels participate in choosing the six-feature subset, so the 'out-of-fold' predictions are not independent of the test labels. With 379 samples and highly correlated features, this selection bias tends to inflate Macro-F1. Please move the entire feature-selection cascade inside each outer training fold and report selection stability, or justify why the bias is negligible.","section":"IV-A / III-D-2"},{"comment":"The data are 379 vessels from 193 subjects. Vessel-level k-fold splitting can place the two vessels of one subject in both training and test, inflating generalisation. Report subject-level grouped cross-validation or a mixed-effects/random-intercept analysis.","section":"III-A / III-D-2"},{"comment":"Only point estimates are given. The difference RF vs XGBoost in binary is 0.0009, and the ablation differences are less than 0.01; without foldwise standard deviations, confidence intervals, or paired tests these differences are uninterpretable. Add fold-level results and CIs.","section":"Tables IV, VI, VIII"},{"comment":"The ternary confusion matrices sum to 93, 112, and 174 for Straight, Low, and High rows, while Table III gives group sizes N=93, 174, 112. The row/sample-size labeling is therefore inconsistent. The text's comparison of high-tortuosity misclassifications (RF 15 vs LR/SVM 31) relies on these rows; correct the table and recompute the error counts.","section":"Table VII"},{"comment":"MRI thresholds are midpoints of full-sample group means and are not evaluated on held-out data. These are in-sample fitted decision rules; as published they are not validated reference ranges. Please report cross-validated or external performance of the MRI thresholds, or explicitly state that they are descriptive full-sample statistics with no decision-making claim.","section":"Eqs. (28)-(29)"},{"comment":"The binary label definition is not stated. The text says binary and ternary labels are not a one-to-one mapping, yet the Table V class sizes (267/112) coincide with merging 'straight'+'low' vs 'high'. Specify how 'severe' was derived, and report the annotation protocol and inter-rater reliability for the physician's ternary grades used as labels.","section":"III-A / IV-A"}],"minor_comments":[{"comment":"The text mentions an 'inner 3-fold loop' for hyperparameter search but later says '5-fold nested OOF'; clarify the number of inner folds and report grid ranges.","section":"III-D-2"},{"comment":"'Fig. V' should be 'Table V'.","section":"IV-C-1"},{"comment":"The number of quantile bins K for IG discretization is not given; without it the IG ranking is not reproducible.","section":"III-C-1"},{"comment":"The ablation tolerance ('10^-3 scale') is not defined; specify how the tolerance interval was set.","section":"III-D-3"},{"comment":"Consider reporting data/code availability and the exact RF hyperparameters (number of trees, max depth, class weights) to make the experiments reproducible.","section":"Methods"},{"comment":"The 'peaks' at 3 and 5 features are within 0.01 of the 6-feature baseline, but without variance estimates the statement that these are 'equivalent' is not supported.","section":"Table VIII"}],"recommendation":"major_revision","confidential_remarks":"The geometric contribution is solid and the paper is within the journal's scope. My main concern is methodological: the full-data feature selection and vessel-level CV make the headline F1 scores optimistic. These are fixable by re-running the analysis with selection nested in CV and subject-level grouping. The confusion-matrix count inconsistency should be checked before resubmission. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the six-feature subset (TI, AC, TC, AC/AT, AT, TT) for ICA-C1 grading and the reported Macro-F1s (0.92 binary, 0.86 ternary) are credible as a descriptive benchmark on these 379 centerlines, but not as evidence of generalizable performance. The main reason is methodological: feature selection by information gain and Spearman pruning is done once on the whole dataset, then the nested CV only protects hyperparameter choice. So the outer-fold test labels influenced which features were kept. That's a classic selection-bias leak, and with 13 candidates and N=379 it can inflate F1 by a few points. The paper even says the IG values were computed from the physician's labels. The stress-test note is right.\n\nWhat's genuinely useful: the paper gives a clean, well-specified geometric feature set for a clinically relevant vessel, with a nice worked example showing TI cannot distinguish a Salkowski curve from a helix with identical L/D. The discrete derivative formulas are clearly presented and reproducible. The ablation study is sensible, and the limitations section is honest — they explicitly say the MRI is exploratory and needs external validation.\n\nSoft spots beyond the selection leak: no confidence intervals or foldwise variance; no adjustment for two vessels per subject (193 patients, 379 vessels); the ground-truth labels are one physician's visual grading, so the model measures agreement with that grading, and inter-rater reliability is not addressed. The MRI thresholds in Eqs. (28)-(29) are fitted to the same in-sample group means, so any performance claim for them would be circular; the paper is careful not to claim performance, but it should say so explicitly. And there's a concrete inconsistency: the flowchart lists MRI_ternary weights as AC 0.3541, TI 0.2858, etc., while Eq. (26) gives 0.3273 TI, 0.2434 AC, etc. One of these is wrong.\n\nBottom line: this is a solid, honest engineering paper with a fixable evaluation-protocol flaw. The right referee response is major revision: move feature selection inside the CV loop (or at least report both variants), add confidence intervals and per-subject clustering, and fix the MRI formula. The phantom and the feature set are worth keeping. I'd be comfortable sending it out.","headline":"A competent feature-selection-and-classification study whose headline F1s are probably a bit inflated because the six features were chosen on the full dataset before cross-validation; fixable, and worth a referee.","tokens_in":19246,"tokens_out":2789,"would_cite":false,"duration_ms":27572,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Six curvature and torsion features, selected by information gain, let a random forest reproduce physician tortuosity grades (Macro-F1 0.92 binary, 0.86 ternary).","keywords":["blood vessel tortuosity","internal carotid artery","discrete geometry","curvature and torsion","feature selection","information gain","random forest","morphological risk index"],"falsifier":"Have independent neuroradiologists re-grade the same 379 centerlines and measure inter-rater agreement (e.g., Cohen's kappa); then test the trained six-feature model on a multi-center external dataset. If kappa is far below the model's Macro-F1, or if Macro-F1 drops substantially on the external data, the model is encoding a single rater's or single center's idiosyncrasies rather than a generalizable morphological rule.","tokens_in":18383,"feed_emoji":"🩻","tokens_out":11737,"duration_ms":93664,"temperature":0.7,"pith_summary":"Physicians grade ICA-C1 vessel tortuosity visually, which is experience-dependent, and the standard Tortuosity Index (L/D) cannot distinguish in-plane bending from out-of-plane twisting. This paper attempts to establish that a reproducible, geometry-only substitute exists: on 379 clinical centerlines, it computes pointwise curvature and torsion with a discrete derivative estimator, aggregates them into 13 features, prunes to six non-redundant ones via Information Gain and Spearman correlation, and trains a Random Forest. The reported Macro-F1 is 0.9206 for binary severe/non-severe screening and 0.8626 for ternary straight/low/high grading, with a Morphological Risk Index summarizing the model's feature importances. If the pattern holds, clinicians could obtain an objective numerical tortuosity readout that supports both coarse screening and finer morphological distinctions.","feed_headline":"Matches physician tortuosity grades with six geometry features","feed_subtitle":"On 379 carotid centerlines, it achieves F1 0.92 (binary) and 0.86 (three-way) without manual grading.","key_machinery":"The central mechanism is a discrete-geometry derivative estimator: at each centerline point, a constrained least-squares fit (via Lagrange multipliers) over a sliding window yields stable first, second, and third derivatives r', r'', r'''; these feed the standard Frenet formulas for curvature κ(t) and torsion τ(t). Aggregating these per-point values produces 13 global, averaged, and length-normalized features. A two-stage filter—Information Gain ranking followed by Spearman correlation pruning (|ρ|>0.95)—removes redundant/low-ranked features, yielding the six-feature set. The Random Forest classifier maps these six features to labels, with cost-sensitive class weights to handle imbalance and","core_discovery":"The authors aim to establish that a distance-based metric like the Tortuosity Index is geometrically ambiguous (a Salkowski curve and a helix can share identical L/D values) and that a compact set of curvature- and torsion-derived features can replace subjective visual grading. From 379 ICA-C1 centerlines, they estimate pointwise curvature and torsion via a sliding-window least-squares derivative method, build 13 aggregate features, prune to six (TI, AC, TC, AC/AT, AT, TT) by Information Gain and Spearman correlation, and train a Random Forest. Reported nested out-of-fold Macro-F1 is 0.9206 for binary severe/non-severe screening and 0.8626 for ternary straight/low/high grading. Ablation show","pith_inferences":["A natural extension is to apply the same discrete-geometry feature pipeline to other vessels (e.g., coronary or cerebral arteries) and retrain the classifier, since the mathematics does not depend on the carotid anatomy.","A multi-reader agreement study would clarify whether the model is learning a stable morphological signal or encoding one rater's idiosyncrasies; if model F1 exceeds human-human agreement, the model could serve as a reference standard.","The phantom-based argument (identical TI for Salkowski and helix) suggests any proposed replacement metric should be tested against a library of such geometric-ambiguity pairs before being adopted for clinical use.","The MRI thresholds are sample-based and require external validation; a concrete next test would be to correlate MRI with procedural outcomes such as catheter navigation time or first-pass recanalization."],"forward_implications":["If the reported performance generalizes, tortuosity assessment could become an automated, reproducible measurement rather than a subjective visual judgment.","The six-feature set is small enough to be checked by hand after centerline extraction, making it a practical candidate for integration into CTA reading workflows.","The ablation result implies that a fast binary screen only needs three curvature features (TI, AC, TC), while torsion features become necessary when grading into three morphological classes.","The Morphological Risk Index provides a single numeric summary that could be tracked over time or compared across patients, though the paper explicitly frames it as exploratory.","Because the labels come from a physician's prior visual classification, the framework should be read as quantifying that clinical consensus, not as discovering an independent ground truth."],"fun_headline_variants":["Six geometric features outmatch distance-based tortuosity metrics","F1 0.92: Six features classify carotid tortuosity like a pro","Geometry beats distance: six features replace manual tortuosity grading","Automated tortuosity grading: six geometry features, RF, F1 0.92","Curvature and torsion features predict vascular tortuosity at F1 0.92"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reported scores are computed against the physicians' previous visual morphological classification, so if those ratings are inconsistent or rater-dependent, the model is matching a subjective reference rather than an objective geometric ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Six geometric features outmatch distance-based tortuosity metrics","F1 0.92: Six features classify carotid tortuosity like a pro","Geometry beats distance: six features replace manual tortuosity grading","Automated tortuosity grading: six geometry features, RF, F1 0.92","Curvature and torsion features predict vascular tortuosity at F1 0.92"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000874,"raw_usage":{"total_tokens":3677,"prompt_tokens":863,"completion_tokens":2814,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":2711}},"tokens_in":607,"tokens_out":2814,"duration_ms":16795,"temperature":1.0,"reasoning_tokens":2711,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T03:03:03.991020+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent neuroradiologists re-grade the same 379 centerlines and measure inter-rater agreement (e.g., Cohen's kappa); then test the trained six-feature model on a multi-center external dataset. If kappa is far below the model's Macro-F1, or if Macro-F1 drops substantially on the external data, the model is encoding a single rater's or single center's idiosyncrasies rather than a generalizable morphological rule.","supporting_citations":[],"review_version":1}