{"id":"1e038079-3600-46c2-a427-57f1c9ed52c3","arxiv_id":"2608.11037","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Inference-time aggregation of patient images, disorder centroids, and hybrid nearest-neighbor/centroid scores improves mean per-disorder top-1 retrieval accuracy on GMDB by up to 14.8 percentage points without retraining the GM-Arc encoder.","lead":"This paper shows that combining multiple photos of the same patient and summarizing reference cases per disorder at retrieval time improves AI rankings of rare genetic disorders from facial images. The gains are achieved without retraining the underlying face encoder, so the approach could be dropped into existing GestaltMatcher-style systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rare-disorder results use overlapping 70/30 repeats, not true 10-fold cross-validation, so the GMDB-Rare and GMDB-Multi-Rare gains may not be as reliable as reported.","rationale":"The paper is a careful empirical study with thorough ablations, honest limitations, paired cluster-bootstrap testing, and split-gallery sensitivity checks, so I would not reject it. The reader's weakest assumption concerns whether arithmetic means of GM-Arc embeddings are meaningful in cosine-distance space; that is a legitimate generalization concern, but the paper's own distance-level fusion results and proportional nearest-neighbor analyses partially bound it, and the within-GMDB results are empirical. The cross-validation discrepancy is more load-bearing because it is internal, concrete, and directly affects the headline GMDB-Rare numbers, which are the clinically most important part of the claim. If the overlapping-split protocol is confirmed, the reported p-values and accuracy gains for unseen rare disorders need to be re-established on a patient-disjoint partition before the 'unseen disorder' claim can be taken at face value. The recommended verdict remains conditional on the authors clarifying or correcting the evaluation protocol and confirming the results under a true patient-disjoint cross-validation.","tokens_in":33407,"tokens_out":12706,"duration_ms":121389,"concrete_test":"Run a true 10-fold patient-disjoint cross-validation on GMDB-Rare: assign each of the 1,240 rare patients to exactly one test fold; for each fold, build the rare gallery from the other nine-tenths of patients, combine it with the full GMDB-Freq gallery, and evaluate the single-image baseline, centroid-only scoring, and the full λ=0.75 framework with the same code. Recompute the mean per-disorder top-1 and top-5 accuracy and the cluster-bootstrap p-values on GMDB-Rare and GMDB-Multi-Rare. If the reported gains (e.g., +4.41 pp on GMDB-Rare and +8.17 pp on GMDB-Multi-Rare at top-1) shrink materially or lose significance, the central rare-disorder claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states that GMDB-Rare (1,554 images, 1,240 patients) was evaluated with 10-fold cross-validation, but the per-fold numbers are 455.7 evaluation images and 1,098.3 gallery images. A true 10-fold split would put roughly 155 images in each test fold; 455.7 is about 29.3% of the total, and 10 × 455.7 ≈ 4,557 ≈ 3 × 1,554, meaning each patient appears in the test portion of about three of the ten 'folds'. Section 3.2 implicitly confirms recurrence by weighting bootstrap resamples for patients appearing across splits. The reported GMDB-Rare and GMDB-Multi-Rare mean per-disorder accuracies and p-values are therefore averages over non-independent, overlapping random 70/30 splits rather than a patient-disjoint cross-validation. The central claim for unseen rare disorders (top-1 from 19.38% to 23.79% on GMDB-Rare, and 18.54% to 26.71% on GMDB-Multi-Rare) rests on this protocol. The point estimates may still be unbiased, but the protocol is misreported, and the variance and p-value estimates need re-derivation under a proper patient-disjoint partition.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an inference-time multi-level evidence aggregation framework for rare-genetic-disorder facial phenotype retrieval. It keeps the GM-Arc encoder fixed and combines: (i) embedding-level fusion of multiple test images per patient (Eq. 11), (ii) patient-weighted disorder centroids summarizing gallery patients (Eq. 7), and (iii) hybrid individual-centroid scoring with a single tuned weight lambda (Eq. 9). In a unified gallery containing both GMDB-Freq and GMDB-Rare disorders, the full framework reports consistent mean per-disorder top-N accuracy gains over the single-image nearest-neighbor baseline, e.g., top-1 from 38.52% to 48.82% on GMDB-Freq and from 19.38% to 23.79% on GMDB-Rare. The evaluation includes component ablations, lambda sensitivity analyses, stratified analyses, and a paired two-stage cluster bootstrap.","tokens_in":33661,"tokens_out":10278,"duration_ms":92160,"significance":"If the results hold, the method is practically valuable because it improves retrieval without retraining the encoder, and the component design is transparent and simple. The manuscript has several strengths: the evaluation uses held-out test sets with identity-leakage prevention, the hybrid weight lambda was selected on a held-out validation set and fixed, a post hoc sensitivity analysis documents robustness, component-wise ablations are reported for both unified and split galleries, and the authors commit to releasing code. However, the rare-disorder evaluation protocol is misreported as 10-fold cross-validation when the numbers show overlapping 70/30 splits, which bears directly on the rare-disorder claims and their p-values.","major_comments":[{"comment":"Section 3.1 and Table 1: the procedure labeled '10-fold cross-validation' is not a patient-disjoint 10-fold partition. GMDB-Rare has 1,554 images; the per-fold evaluation set contains 455.7 ± 7.7 images, so 10 × 455.7 ≈ 4,557 ≈ 3 × 1,554, meaning each patient's images are evaluated in about three of the ten 'folds'. The per-fold evaluation also contains all 361 disorders (one patient per disorder on average), whereas a true 10-fold split of 1,240 patients would put only about 124 patients per fold. The same arithmetic holds for GMDB-Multi-Rare (159.9 × 10 ≈ 3 × 528). Thus the reported GMDB-Rare and GMDB-Multi-Rare results are averages over repeated random 70/30 splits, not over cross-validation folds. The headline rare-disorder improvements (top-1 from 19.38% to 23.79% and from 18.54% to 26.71%) rest on this protocol and need to be re-derived with patient-disjoint partitions or explicitly reported as repeated subsampling.","section":"Section 3.1, Table 1"},{"comment":"Section 3.2: the statistical inference for the rare subsets inherits the same problem. The two-stage cluster bootstrap uses weights 'for patients recurring across the ten cross-validation splits,' which confirms that the same patients contribute to multiple folds. Paired bootstrap p-values computed over these overlapping fold averages do not provide a valid test for a patient-disjoint evaluation: the effective sample size is inflated approximately threefold, so p < 0.001 for GMDB-Rare and GMDB-Multi-Rare may substantially overstate confidence. The authors should either run a genuinely patient-disjoint 10-fold or k-fold evaluation, or switch to a repeated random-subsampling design with inference that treats the patient as the primary resampling unit across all appearances, and report confidence intervals for the fold-level mean accuracy under that design.","section":"Section 3.2"}],"minor_comments":[{"comment":"Section 2.2 describes the aggregation operators as 'parameter-free,' but Eq. (9) introduces the tuned weight lambda. Consider saying 'not requiring additional encoder training' or 'having only one tuned hyperparameter' to avoid a contradiction.","section":"Section 2.2"},{"comment":"Figure 3 is dense; the four curves per panel are distinguished only by line style, and the caption's 'thin lines' may be hard to read in print. Direct labeling or a table of the same numbers would improve clarity.","section":"Figure 3"},{"comment":"Section 4.4 reports p = 0.003 for GMDB-Multi-Freq but p < 0.001 for the other subsets; please state explicitly whether the same paired bootstrap scheme was used for all four subsets and whether any correction for testing four evaluation subsets was considered.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is well written and the experimental design is mostly careful, but the discrepancy between '10-fold cross-validation' and the reported per-fold sizes is a substantive issue that should be fixed before acceptance. The authors already weight bootstrap resamples for patients appearing across splits, which indicates awareness of the overlap; the right fix is to either run true patient-disjoint folds or to relabel the design as repeated random subsampling and adjust the inference accordingly. I would not reject on this basis, but I would not accept before the rare-disorder numbers are recomputed. In addition, the controlled-access data and code-only-upon-acceptance policy makes independent verification difficult; please consider asking for a reproducibility appendix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this paper shows a genuinely useful inference-time trick—aggregate multiple test images, disorder centroids, and a hybrid of both—and it works in their experiments. The gains are large and fairly consistent across evaluation subsets. But the rare-disorder evaluation is not actually 10-fold cross-validation; it's three overlapping 70/30 repeats. That matters because the headline rare-disorder gains rest on that protocol.\n\nWhat's new: the individual components are all known (template pooling, class centroids, hybrid scoring), but this is the first careful study of combining them at inference time for facial phenotype retrieval without retraining the encoder. That's a practical contribution—existing GestaltMatcher deployments could get a lift just by changing how they score. The experiments are mostly well done: identity leakage is handled, the bootstrap is paired and clustered, and the lambda sensitivity analysis is honest and post hoc.\n\nThe biggest problem is the CV description. Section 3.1 says GMDB-Rare was evaluated with 10-fold CV, but the numbers don't fit: 1,554 images total, 455.7 in the evaluation fold means 29.3% test, not 10%. Ten times 455.7 is roughly three times the dataset, so each patient lands in the test portion about three times across the 'folds.' This is repeated 70/30 splits, not patient-disjoint CV. The paper even acknowledges recurring patients in the bootstrap weighting. That doesn't necessarily make the point estimates wrong, but it means the p-values and confidence intervals are computed from non-independent samples, and the protocol is misreported. This needs a correction or a re-run with proper patient-disjoint folds before the rare-disorder claims can be taken at face value.\n\nOther soft spots are minor: code is promised but not yet released; the evaluation is on a single controlled-access database and one reproduced encoder; and the geometric assumption that averaging embeddings in cosine space is meaningful is acknowledged but not validated. These are addressable, not fatal.\n\nWho this is for: anyone building or using GestaltMatcher-style retrieval, and the clinical genetics community that deploys it. It's an empirical paper with a practical payoff, not a methodological breakthrough.\n\nRecommendation: yes, send it to peer review, but require the authors to fix the CV protocol or redo the rare-disorder evaluation with true patient-disjoint splits. The rest of the paper is careful enough that this should be revisable.","headline":"Useful inference-time aggregation for GestaltMatcher, but the rare-disorder evaluation is three overlapping 70/30 repeats mislabeled as 10-fold CV, so those gains need re-verification.","tokens_in":34208,"tokens_out":3290,"would_cite":true,"duration_ms":28350,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that pooling facial evidence across a patient's multiple images, a disorder's gallery patients, and local versus global disorder scores at inference time improves rare-disorder retrieval accuracy on every GMDB evaluation…","keywords":["rare genetic disorders","facial phenotyping","medical image retrieval","evidence aggregation","GestaltMatcher","disorder prioritization","inference-time fusion","Clinical Face Phenotype Space"],"falsifier":"Run the same aggregation pipeline with a different facial-phenotype encoder or on a future GMDB release and compare against the single-image baseline; if patient-weighted centroids or embedding-level fusion reduce top-1 accuracy for rare disorders, the geometric premise fails. A more direct check is to test a cohort of disorders with age-dependent facial presentations and inspect whether the disorder centroid falls in a low-density region of the embedding space, in which case nearest-neighbor distances should beat centroid distances and the mean-embedding representative is the wrong summary.","tokens_in":33241,"feed_emoji":"🧬","tokens_out":5450,"duration_ms":47026,"temperature":0.7,"pith_summary":"The paper asks whether rare genetic disorder prioritization by facial phenotype can be made more reliable by pooling evidence that is already available in a retrieval database, rather than by training a better encoder. It claims that three simple inference-time operations — averaging a patient's face images into one embedding, representing each disorder by a patient-weighted centroid of its gallery embeddings, and blending that centroid distance with the nearest-neighbor distance — improve disorder ranking accuracy on the GestaltMatcher Database (GMDB). Top-1 accuracy rises from 38.52% to 48.82% on frequent disorders and from 19.38% to 23.79% on rare disorders, with larger gains for patients who have several images. Because the encoder is untouched, the method can be added to existing GestaltMatcher-style retrieval pipelines without retraining.","feed_headline":"Averaging face evidence lifts rare-disorder retrieval by up to 15 points","feed_subtitle":"Patient-weighted centroids plus hybrid scoring beat single-image matching on every GMDB evaluation subset.","key_machinery":"The central object is the Clinical Face Phenotype Space (CFPS), the fixed embedding space produced by the GM-Arc encoder ensemble, in which cosine distance measures facial phenotypic similarity. Three parameter-free operators act on this space: patient-level embedding fusion, which averages the test patient's image embeddings; patient-weighted disorder centroids, which average per-patient mean gallery embeddings so that patients with many images do not dominate; and hybrid individual-centroid scoring, $D_{\\text{hybrid}} = \\lambda D_{\\text{centroid}} + (1-\\lambda) D_{\\text{NN}}$ with $\\lambda=0.75$, which combines global disorder-level evidence with local nearest-neighbor evidence. These operators carry the argument by changing only how existing embeddings and distances are combined, leaving the encoder fixed.","core_discovery":"The central claim is that a patient or a disorder is better modelled as a set of observations in the Clinical Face Phenotype Space (CFPS) than as an isolated image or point, and that aggregating these observations at inference time produces more robust disorder rankings. The full framework uses patient-weighted disorder centroids, a hybrid distance with weight $\\lambda=0.75$ between the centroid distance and the nearest-neighbor distance, and embedding-level averaging of a patient's test images before scoring. The paper reports consistent improvements in mean per-disorder top-$N$ accuracy across GMDB-Freq, GMDB-Rare, and multi-image subsets, including a +14.82 percentage-point top-1 gain on GMDB-Multi-Freq, and shows that the true-disorder rank improves for far more patients than it worsens.","pith_inferences":["A natural extension, not tested in the paper, is to make the hybrid weight $\\lambda$ adaptive to gallery size or disorder representation; the paper's own post hoc sensitivity analysis suggests rare disorders would benefit from more centroid-dominant weighting.","The same aggregation principle could transfer to other set-based medical image retrieval tasks, such as multiple views or time points of the same lesion, although this is speculative beyond the facial phenotype setting.","The paper's database-growth analysis implies that patient-weighted centroids will become more consequential as reference databases accumulate more images per patient over time, a consequence the authors note but do not quantify in the current gallery."],"forward_implications":["Existing GestaltMatcher-style retrieval systems can adopt the method by changing only inference-time scoring, with no retraining or architecture modification.","Retrieval for ultra-rare disorders represented by very few gallery patients improves, because disorder centroids stabilize sparse gallery evidence.","Patients with multiple clinical photographs benefit most; top-1 accuracy on GMDB-Multi-Freq rises from 46.12% to 60.94%.","The unified-gallery results indicate the method remains effective when well-represented and rarely represented disorders are ranked together, which mirrors real diagnostic retrieval.","The hybrid weighting parameter lets a practitioner balance local image-level evidence against disorder-level summaries, with centroid-dominant weighting preferable for sparsely represented disorders."],"supporting_citations":[{"why":"Supplies the fixed GestaltMatcher-Arc encoder ensemble and the single-image nearest-neighbor retrieval baseline that all aggregation components build on.","marker":"Hustinx et al. (2023)"},{"why":"Establishes GestaltMatcher retrieval in the Clinical Face Phenotype Space and introduces the GMDB evaluation setup for rare-disorder prioritization.","marker":"Hsieh et al. (2022)"},{"why":"Documents GMDB v1.1.4, the controlled-access database used for all galleries, evaluation subsets, and the multi-image growth statistics.","marker":"Lesmann et al. (2024)"},{"why":"Provides the founding evidence that ordinary facial photographs contain diagnostically relevant gestalt information and can be embedded in a clinical face phenotype space.","marker":"Ferry et al. (2014)"},{"why":"Supplies the ArcFace angular-margin face representation that GM-Arc fine-tunes, defining the embedding geometry on which cosine-distance aggregation operates.","marker":"Deng et al. (2019)"}],"fun_headline_variants":["Averaging face evidence lifts rare-disorder retrieval by up to 15 points","Patient-weighted centroids boost rare-disorder facial retrieval by up to 15 points","Multi-level evidence aggregation boosts rare-disorder face matching up to 15 points","Inference-time aggregation improves rare-disorder facial retrieval by up to 15 points","No retraining: aggregating patient and disorder evidence lifts face retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that averaging embeddings — both across a patient's images and across a disorder's patients — produces a meaningful, stable representative of the phenotype in cosine-distance space, so that distance to the average behaves like the evidence the individual images provide.","fun_headline_variants_meta":{"raw":{"variants":["Averaging face evidence lifts rare-disorder retrieval by up to 15 points","Patient-weighted centroids boost rare-disorder facial retrieval by up to 15 points","Multi-level evidence aggregation boosts rare-disorder face matching up to 15 points","Inference-time aggregation improves rare-disorder facial retrieval by up to 15 points","No retraining: aggregating patient and disorder evidence lifts face retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001103,"raw_usage":{"total_tokens":4655,"prompt_tokens":1055,"completion_tokens":3600,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":3498}},"tokens_in":671,"tokens_out":3600,"duration_ms":22315,"temperature":1.0,"reasoning_tokens":3498,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:32:43.843538+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same aggregation pipeline with a different facial-phenotype encoder or on a future GMDB release and compare against the single-image baseline; if patient-weighted centroids or embedding-level fusion reduce top-1 accuracy for rare disorders, the geometric premise fails. A more direct check is to test a cohort of disorders with age-dependent facial presentations and inspect whether the disorder centroid falls in a low-density region of the embedding space, in which case nearest-neighbor distances should beat centroid distances and the mean-embedding representative is the wrong summary.","supporting_citations":[{"cited_title":"u ger, E. , author K \\","cited_arxiv_id":null,"evidence_quote":"Establishes GestaltMatcher retrieval in the Clinical Face Phenotype Space and introduces the GMDB evaluation setup for rare-disorder prioritization."},{"cited_title":", author Hustinx, A","cited_arxiv_id":null,"evidence_quote":"Documents GMDB v1.1.4, the controlled-access database used for all galleries, evaluation subsets, and the multi-image growth statistics."},{"cited_title":", author Steinberg, J","cited_arxiv_id":null,"evidence_quote":"Provides the founding evidence that ordinary facial photographs contain diagnostically relevant gestalt information and can be embedded in a clinical face phenotype space."}],"review_version":1}