{"id":"882a6ca8-039b-4694-ba53-8c3a23f03fff","arxiv_id":"2506.03202","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A support vector regressor predicts FEM-simulated skull shapes from 3D photos with R2=0.95, but validation on real patient outcomes is poor (R2=-3.58).","lead":"This paper trains a machine learning model to predict head shape changes after spring-assisted craniosynostosis surgery using only 3D photographs instead of CT scans. The model matches its own simulations well, but fails against real post-surgery scans, so it is not yet a usable planning tool.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's R2=0.95 is synthetic-only; the real-patient validation (R2=-3.58) directly refutes the clinical prediction claim, and the 2.70 mm error claim is not benchmarked against a null model.","rationale":"The reader's weakest_assumption correctly identifies the same load-bearing concern: the synthetic FEM-based training accuracy does not transfer to real surgical outcomes, and the validation R2=-3.58 is direct evidence. I agree with the REJECT verdict because the paper's own abstract and conclusion present the model as a tool for real clinical prediction, not merely as a surrogate for FEM simulations. The variance explained by the model on real patients is negative, meaning the mean follow-up shape is a better predictor than the ML model. This is not a borderline case of an unverified assumption; it is a central claim contradicted by the manuscript's own independent validation. The paper transparently lists limitations, which is good scientific practice, but transparency does not make the clinical claim defensible. The proposed null-model benchmark would settle whether the reported 2.70 mm error represents any genuine predictive skill, and if it does not, the only defensible framing is a feasibility study for a FEM surrogate, not a validated surgical planning tool. No additional concerns are needed; this single issue is sufficient to sustain the rejection.","tokens_in":11834,"tokens_out":2976,"duration_ms":27582,"concrete_test":"Compute the mean surface error and R2 of a null predictor that always outputs the mean postoperative shape for the 13 validation patients, using the same scaling and normalization as in Section 2.5.1. If the null model achieves R2 near 0 and mean error at or below 2.70 mm, then the claimed 2.70 mm real-patient accuracy is clinically meaningless and the abstract's central claim fails. Additionally, recompute the validation R2 after removing the growth-rescaling step to confirm that the negative R2 is robust to that preprocessing choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the SVM surrogate predicts real postoperative skull shapes from 3D photographs, with the abstract reporting R2=0.95 and MSE/MAE below 0.13. The R2=0.95 is computed on a held-out split of 2356 FEM simulations (Section 3.3), not on any real surgical outcome. The independent validation in Section 3.4 on 13 real SAC patients yields R2=-3.58, MSE=1.41, and MAE=0.88, which is worse than simply predicting the mean postoperative shape. The Discussion attempts to recover clinical relevance by citing an error rate of 2.70±0.58 mm and comparing it to Borghi et al. 2020, but no baseline error for the mean-shape predictor is reported. If a constant prediction equal to the average follow-up shape already achieves a mean surface error at or below 2.70 mm on these 13 patients, then the model adds no predictive value. The load-bearing assumption is therefore not FEM realism per se, but the equivalence between synthetic test accuracy and clinical utility; this equivalence is empirically rejected at the validation stage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for real-time prediction of postoperative skull shape after spring-assisted cranioplasty (SAC) for sagittal craniosynostosis. The method uses population-average skull and soft-tissue thickness to convert 3D photographs into synthetic skull models, simulates surgical outcomes with finite element modelling (FEM) across a design of experiments (2356 simulations), and trains a multi-output support vector regressor to predict FEM output shape modes from patient age, surgical parameters, spring properties, and preoperative shape modes. The abstract reports a test-set R² of 0.95 with MSE and MAE below 0.13. The paper also reports an independent validation on 13 real patients in Section 3.4, where the model achieves R² = -3.58, MSE = 1.41, and MAE = 0.88.","tokens_in":12084,"tokens_out":2967,"duration_ms":32121,"significance":"If the reported accuracy held on real patients, the tool would be a valuable non-ionising alternative to CT-based FEM planning for SAC. The pipeline is methodologically coherent, uses an independent external cohort for validation, and the authors are transparent about the poor validation performance in Section 3.4. However, the central claim of the paper—that the surrogate model predicts real surgical outcomes from 3D photographs—is contradicted by that validation. The negative R² means the model is worse than predicting the mean postoperative shape, so the proposed tool currently has no demonstrated clinical predictive value. The abstract and discussion substantially overstate the findings by foregrounding the synthetic test metrics and the mean surface error without a null-model baseline.","major_comments":[{"comment":"The abstract's central claim ('the resulting multi-output support vector regressor model achieves a R2 metric of 0.95 and MSE and MAE below 0.13') is based solely on the held-out split of FEM simulations in Section 3.3. The independent validation on 13 real patients in Section 3.4 reports R² = -3.58, MSE = 1.41, and MAE = 0.88. A negative R² means the model performs worse than a trivial predictor returning the mean postoperative shape. Presenting the synthetic R² without the validation result is misleading and does not support the stated goal of predicting surgical outcomes for new patients.","section":"Abstract and Section 3.3-3.4"},{"comment":"The Discussion claims clinical usefulness from a mean surface error of 2.70 ± 0.58 mm and compares this to Borghi et al. (2020). This comparison is not benchmarked against a null model. A constant prediction equal to the average follow-up shape could plausibly achieve a mean surface error at or below 2.70 mm on these 13 patients, in which case the model adds no predictive value. The paper does not report the baseline surface error for such a null predictor, so the error-rate claim does not establish that the model generalises to real outcomes.","section":"Section 4 (Discussion)"},{"comment":"The load-bearing assumption that a surrogate trained on FEM outputs can generalise to real surgical outcomes is empirically rejected by the validation in Section 3.4. The FEM models are built from population-averaged skull/skin thickness, uniform material properties, and artificial suture locations (Sections 2.2-2.3), and the trained model reproduces those simulations well. However, the validation shows no predictive signal on real postoperative 3D photographs (R² = -3.58). The Discussion lists possible causes such as growth and population-averaged parameters, but the magnitude of the negative R² indicates that the model is not merely noisy; it fails to capture the actual surgical response. This invalidates the clinical prediction claim unless the model is retrained or fundamentally revised.","section":"Sections 2.2-2.4 and 3.4"},{"comment":"The validation preprocessing is under-specified: the postoperative 3D photographs are 'scaled down to omit the growth factor' that was not in the FEM, but the scaling method (e.g., global uniform scaling, landmark-based alignment, or volume normalisation) is not described. Because the validation metric is a surface-distance error, the scaling choice directly affects the reported 2.70 mm error and the R² value. Without details and a sensitivity analysis, the validation result cannot be fully interpreted or reproduced.","section":"Section 2.5.1"}],"minor_comments":[{"comment":"Equations (2) and (3) use inconsistent notation: the preoperative vector is indexed to N_preop while the postoperative vector is indexed to N_postop, but the text later states the mean shapes are both in ℝ^{N×3}; the relationship between N_preop and N_postop and the template's node count should be clarified.","section":"Section 2.4.1"},{"comment":"The text states the CT population has 30 patients and the validation population has 13 patients, but the age ranges are given as '5.8 ± 1.15 months' and '5.1 ± 1.0' respectively; the second entry is missing the unit 'months'.","section":"Section 2.1"},{"comment":"The SSM retains 11 modes for both input and output shapes, but the output modes account for only 90% of variance while the input modes account for 94%. The potential impact of discarding 10% of output variance on the prediction metrics is not discussed.","section":"Section 3.2"},{"comment":"The predicted-versus-actual plots for the first four modes would benefit from showing the R² and error metrics for each individual mode, since the aggregate R² of 0.95 may conceal modes with poor prediction that nonetheless contribute to the final shape error.","section":"Figure 12"},{"comment":"The paper states that bone and suture are assigned viscoelastic material properties via a Prony series, but the actual parameter values are only cited to earlier work; for reproducibility, the key material constants (or a reference table) should be included.","section":"Section 2.3"}],"recommendation":"reject","confidential_remarks":"The manuscript is transparent about the validation failure in Section 3.4, which is a credit to the authors. However, the abstract and discussion frame the study as a success, and the load-bearing claim of clinical prediction is contradicted by the data. A negative R² on the independent cohort is not a fixable presentation issue; it undermines the core contribution. If the authors wish to resubmit, they would need to either substantially revise the surrogate and validation methodology or reframe the paper as a negative-result study with a much more cautious abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this paper builds a real thing—an ML surrogate of the group's own FEM pipeline—and then honestly reports that it fails on the 13-patient external validation. The problem is the abstract doesn't. It sells the synthetic R2=0.95 as if it were clinical accuracy. The external validation gives R2=-3.58, which is worse than predicting the average follow-up head for everyone. That's not a small gap; it's the central claim collapsing.\n\nWhat's new: the trained multi-output SVR on 2356 simulated SAC outcomes is new relative to their prior FEM-only work, and the real-patient validation set is a legitimate independent test. Credit where due: they ran the validation, reported the numbers, and listed the simplifications (population mean thicknesses, uniform material properties, no growth). Many papers in this space stop at the synthetic test set.\n\nSoft spots, in proportion. The key one is presentational: the abstract reports R2=0.95 with no caveat, and the Discussion's 'error rate of 2.70±0.58 mm' has no null baseline. If a constant mean-shape prediction already lands at 2.70 mm average error on those 13 patients, the model adds nothing. The paper doesn't tell us. That's a fixable omission but it's load-bearing for any clinical claim. The FEM realism assumptions were always the weak link, and the negative validation confirms they don't transfer. The paper reads as a feasibility study whose conclusions overreach.\n\nWho it's for: people working on FEM-ML surrogates in craniofacial surgery, and anyone interested in honest negative external validation. The pipeline is coherent and the writing is clear. But as a clinical planning tool paper it isn't close to ready. A revised version that narrows the title and abstract to 'surrogate of FEM simulations' and adds the null baseline would be defensible.\n\nRecommendation: I'd send it to peer review, not desk reject. The negative external validation is exactly the kind of result reviewers need to see, and the methodological pieces are real. The referee's job would be to force the abstract in line with the evidence and to require the null baseline. If the authors refuse to reframe, reject; if they do, it becomes a modest but honest contribution. I'd cite it for the validation numbers, not for the clinical claim.","headline":"A genuinely tested surrogate model whose abstract oversells it: the synthetic R2=0.95 is real, the real-patient validation (R2=-3.58) is honestly reported but fatal to the clinical claim.","tokens_in":12664,"tokens_out":1964,"would_cite":false,"duration_ms":20797,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-output support vector regressor trained on finite-element simulations predicts post-operative skull shape from 3D photographs, with $R^2 = 0.95$ on synthetic test data.","keywords":["Spring assisted cranioplasty","Surrogate model","Statistical shape modelling","Sagittal craniosynostosis","Finite element modelling","Support vector regression","3D photography","Surgical planning"],"falsifier":"Take a prospective cohort of at least 30 sagittal craniosynostosis patients, feed each preoperative 3D photograph and the actual surgical parameters into the trained model, and compare predicted surfaces with 3-week follow-up scans using the same $R^2$ and per-vertex error metrics as the paper's validation; if $R^2$ stays near or below zero rather than approaching the synthetic 0.95, the synthetic FEM training data do not transfer to real surgery.","tokens_in":11614,"feed_emoji":"🦴","tokens_out":9062,"duration_ms":86022,"temperature":0.7,"pith_summary":"The authors aim to turn surgical planning for spring-assisted cranioplasty, the main treatment for sagittal craniosynostosis (a condition in which the skull's midline suture fuses early, producing a long narrow head), into a radiation-free, real-time prediction. They build synthetic skulls from 30 preoperative CT scans by applying population-average soft-tissue thickness, skull thickness, and suture locations, run 2,356 finite-element simulations spanning different osteotomy placements and spring properties, then train machine-learning regressors to reproduce the simulated post-operative skull shapes. Their multi-output support vector regressor reaches $R^2 = 0.95$ on held-out synthetic cases, with mean squared error 0.042 and mean absolute error 0.122. On 13 real patients with pre- and post-operative 3D photographs, the same model gives $R^2 = -3.58$ and an average surface error of $2.70 \\pm 0.58$ mm, which the authors attribute to growth and to the simplifying assumptions of the synthetic model. If that gap closes, clinicians could compare surgical scenarios in seconds from a 3D photograph alone, without CT scans.","feed_headline":"3D-photo model forecasts post-surgery skull shape, R2 = 0.95","feed_subtitle":"Swaps CT-based planning for a 3D photo, cutting FEM computation from hours to seconds.","key_machinery":"The engine is a statistical shape model (SSM) paired with a multi-output support vector regressor. SSM applies principal component analysis to skull meshes, representing each skull as the population mean plus a weighted combination of eigenmodes; the same compression is applied to the 2,356 finite-element output meshes. The regressor learns the mapping from input mode weights plus age, osteotomy parameters, and spring stiffness and free length to output mode weights, so a prediction is a full skull surface. This is what reduces hours of finite-element computation to a function evaluation.","core_discovery":"The paper's central claim is that post-operative skull shape after spring-assisted cranioplasty can be predicted as a function of patient age, surgical parameters (osteotomy position and width), spring characteristics, and a compressed description of the pre-operative head shape, using a machine-learning surrogate of finite-element simulations. The surrogate is a multi-output support vector regressor trained on 2,356 synthetic outcomes generated by finite-element models built from population-averaged skull and skin thickness and average suture positions; on the synthetic test set it achieves $R^2 = 0.95$, MSE = 0.042, and MAE = 0.122. The authors further claim this enables outcome prediction from 3D photographs in real time, eliminating CT and the engineering expertise otherwise needed for finite-element planning. They report an average surface error of $2.70 \\pm 0.58$ mm on 13 real patients, alongside a validation $R^2$ of -3.58, and identify growth and population-averaged modelling assumptions as the main sources of the discrepancy.","pith_inferences":["My reading: the negative real-patient $R^2$ is evidence that the bottleneck is the finite-element model's fidelity, not the regressor, since the regressor fits the synthetic dataset almost perfectly; improvements should target growth modelling and personalised thickness and suture inputs before model architecture.","A testable extension would be to retrain the surrogate on FEM cases that include per-patient skull thickness and suture geometry inferred from photographs, and check whether real-patient $R^2$ rises above the current -3.58; the paper cites skull-inference from photos as future work.","If validated, the same photo-to-surrogate pipeline could transfer to other cranial vault procedures or to longitudinal follow-up, where the surrogate is trained once on a population and evaluated on surface scans.","The clinically meaningful quantity is the $2.70 \\pm 0.58$ mm surface error on real patients, not the synthetic $R^2$; whether that error is acceptable depends on a clinical threshold the paper has not defined."],"forward_implications":["Preoperative planning can be done from a 3D photograph and planned surgical parameters, with no CT scan and no radiation exposure.","Surgeons could simulate many spring placements and choices in seconds, rather than committing to one configuration based on experience and age.","Because the surrogate inherits the finite-element model's behaviour, the machine-learning step introduces little additional error on synthetic data; the practical limit of the tool is how faithfully the FEM represents real surgery.","The same model could be inverted to search for surgical parameters that maximise the cranial index, a direction the paper explicitly identifies for future work."],"supporting_citations":[{"why":"Supplies the mechanical behaviour (stiffness and free length) of the three spring models used in the finite-element simulations and establishes the FEM approach for spring-assisted cranioplasty.","marker":"Borghi et al., 2017"},{"why":"Provides the patient-specific computational model and the osteotomy replication method on which the synthetic skull pipeline builds.","marker":"Borghi et al., 2018"},{"why":"Defines the population-specific material properties (elastic and viscoelastic) assigned to bone and sutures in the FEM.","marker":"Borghi et al., 2020"},{"why":"Gives the radiation-free framework for creating synthetic skulls from 3D photographs, including suture placement and the measurement protocol.","marker":"Garate Andikoetxea et al., 2023"},{"why":"Documents the surgical technique and quantitative outcomes across 100 spring-assisted cranioplasty cases, including spring insertion and osteotomy geometry.","marker":"Rodgers et al., 2017"},{"why":"Demonstrates the FEM-plus-machine-learning surrogate approach in medicine and motivates its use for real-time predictions.","marker":"Madani et al., 2019"},{"why":"Provides the statistical shape modelling formalism (mean shape plus eigenmodes) used to compress input and output meshes.","marker":"Pascoletti, 2008"},{"why":"Shows statistical shape modelling applied to spring-assisted cranioplasty, linking surgical parameters to head shape.","marker":"Rodriguez-Florez et al., 2017a"}],"fun_headline_variants":["3D photo + ML predicts skull shape after craniosynostosis surgery","AI plans skull surgery from photos instead of CT scans","Real-time skull-shape forecast for craniosynostosis surgery","ML surrogate simulates skull surgery outcomes from 3D photos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that finite-element simulations built from population-average skull and skin thickness, average suture positions, and uniform material properties faithfully reproduce what actually happens when a real baby's skull is operated on.","fun_headline_variants_meta":{"raw":{"variants":["3D photo + ML predicts skull shape after craniosynostosis surgery","AI plans skull surgery from photos instead of CT scans","Real-time skull-shape forecast for craniosynostosis surgery","ML surrogate simulates skull surgery outcomes from 3D photos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2993,"prompt_tokens":1090,"completion_tokens":1903,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":706,"completion_tokens_details":{"reasoning_tokens":1830}},"tokens_in":706,"tokens_out":1903,"duration_ms":13338,"temperature":1.0,"reasoning_tokens":1830,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:31:23.318030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a prospective cohort of at least 30 sagittal craniosynostosis patients, feed each preoperative 3D photograph and the actual surgical parameters into the trained model, and compare predicted surfaces with 3-week follow-up scans using the same $R^2$ and per-vertex error metrics as the paper's validation; if $R^2$ stays near or below zero rather than approaching the synthetic 0.95, the synthetic FEM training data do not transfer to real surgery.","supporting_citations":[{"cited_title":", author Schievano, S","cited_arxiv_id":null,"evidence_quote":"Supplies the mechanical behaviour (stiffness and free length) of the three spring models used in the finite-element simulations and establishes the FEM approach for spring-assisted cranioplasty."},{"cited_title":", author Rodriguez-Florez, N","cited_arxiv_id":null,"evidence_quote":"Provides the patient-specific computational model and the osteotomy replication method on which the synthetic skull pipeline builds."},{"cited_title":", author Rodriguez Florez, N","cited_arxiv_id":null,"evidence_quote":"Defines the population-specific material properties (elastic and viscoelastic) assigned to bone and sutures in the FEM."},{"cited_title":", author Ajami, S","cited_arxiv_id":null,"evidence_quote":"Gives the radiation-free framework for creating synthetic skulls from 3D photographs, including suture placement and the measurement protocol."},{"cited_title":", author Glass, G.E","cited_arxiv_id":null,"evidence_quote":"Documents the surgical technique and quantitative outcomes across 100 spring-assisted cranioplasty cases, including spring insertion and osteotomy geometry."},{"cited_title":", year 2008","cited_arxiv_id":null,"evidence_quote":"Provides the statistical shape modelling formalism (mean shape plus eigenmodes) used to compress input and output meshes."}],"review_version":1}