Accurate and Private Diagnosis of Rare Genetic Syndromes from Facial Images with Federated Deep Learning

Ali Burak \"Unal; Cem Ata Baykara; Mete Akg\"un; Peter Krawitz

REVIEW 2 major objections 4 minor 8 references

A federated facial-dysmorphology service keeps 90% of centralized diagnostic accuracy while keeping patient images and encodings private.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 17:43 UTC pith:2KAFV54L

load-bearing objection Useful federated accuracy results for GestaltMatcher, but the privacy claim in Section 6.3 is an assertion, not a proof, and the aggregator can infer the very diagnoses the service is built to provide. the 2 major comments →

arxiv 2509.10635 v1 pith:2KAFV54L submitted 2025-09-12 cs.LG cs.CRcs.CV

Accurate and Private Diagnosis of Rare Genetic Syndromes from Facial Images with Federated Deep Learning

Ali Burak \"Unal , Cem Ata Baykara , Peter Krawitz , Mete Akg\"un This is my paper

classification cs.LG cs.CRcs.CV

keywords federated learningrare genetic syndromesfacial dysmorphologyprivacy-preserving machine learningkernel matrix computationsecure aggregationnon-IID datanext-generation phenotyping

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that hospitals can jointly build and query a GestaltMatcher-style facial phenotyping system without centralizing patient images or exposing their learned face encodings. It proposes a cross-silo horizontal federated learning setup in which a global ensemble feature extractor is trained by federated averaging with secure aggregation, after which a masking-based kernel computation lets a central aggregator derive pairwise cosine distances among patients without seeing plaintext encodings. The claim is that this federated service retains at least 90% of the centralized model's Top-k matching accuracy and stays stable as the number of hospital silos grows and as data become highly non-IID. If true, rare-syndrome diagnosis and discovery could proceed across institutions that currently cannot share data for legal or ethical reasons.

Core claim

The central claim is that federated learning plus a privacy-preserving Gram-matrix computation is sufficient to reproduce the diagnostic utility of a centralized facial dysmorphology service. Specifically, the paper shows that an ensemble feature extractor trained under federated averaging with secure aggregation maps patient faces from different silos into a common latent space; silos then send masked versions of those latent vectors to an aggregator, which computes cosine distances and responds to diagnostic queries using plaintext syndrome labels. The authors report that this federated service achieves at least 90% of the centralized service's Top-k accuracy across frequent-rare gallery/t

What carries the argument

The load-bearing machinery is two-part: (1) a globally shared convolutional ensemble feature extractor trained by federated averaging, with non-zero-sum masking so neither local nor aggregated model weights are visible to the aggregator; and (2) a privacy-preserving kernel matrix computation in which each silo masks its latent feature matrix with a shared random mask and a source-specific left inverse, sends the masked data to the aggregator, and the aggregator computes the Gram matrix of all patients' features. Cosine distances follow from the Gram entries via D(p,q) = 1 - G(p,q)/sqrt(G(p,p)G(q,q)), and those distances drive both syndrome inference and discovery of unusually similar patient

Load-bearing premise

The whole privacy argument depends on the aggregator learning nothing from the complete pairwise cosine-distance matrix plus all syndrome labels; the paper asserts this without analyzing what can be inferred from the Gram matrix itself, which determines latent vectors up to an orthogonal transformation.

What would settle it

Run a reconstruction attack in which a decoder is trained on pairs of Gram or distance-matrix rows and syndrome labels (not on latent vectors) to produce recognizable faces or identifiable attributes such as gender; any successful reconstruction from the aggregator's view would refute the claim that masked encodings reveal nothing.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Hospitals can jointly train and query a facial dysmorphology model without ever sharing patient images or plaintext latent encodings.
New participants can join after training by adopting the global feature extractor and the shared random seed, immediately using inference and discovery services.
The federated service retains at least 90% of centralized Top-k matching accuracy across frequent-rare gallery and test combinations.
Performance remains stable as the number of silos scales from 4 to 16 and under highly non-IID data distributions.
The cosine distance matrix enables detection of exceptionally similar small patient groups that could indicate undiscovered ultra-rare syndromes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The privacy guarantee rests on the aggregator learning nothing from the Gram matrix plus plaintext syndrome labels; since a Gram matrix determines latent vectors up to an orthogonal transformation, a stricter analysis would be needed to rule out face reconstruction from the aggregator's view.
The same masked-kernel infrastructure could be applied to other medical imaging tasks that require pairwise similarity, such as dermatology or histopathology, with a similar federated setup.
The robustness findings suggest that frequent aggregation together with secure masking may serve as a template for other rare-disease phenotype models where data are scarce and siloed.
A testable extension is to evaluate whether the accuracy and privacy claims hold at much larger scale, such as hundreds of silos or partially overlapping syndrome distributions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful federated accuracy results for GestaltMatcher, but the privacy claim in Section 6.3 is an assertion, not a proof, and the aggregator can infer the very diagnoses the service is built to provide.

read the letter

Colleague, here's my read on arXiv:2509.10635.

The genuinely new thing here is the assembly: taking the GestaltMatcher ensemble from Hustinx et al., training it federated with FedAvg plus a non-zero-sum masking secure aggregation, and then using FLAKE to compute the Gram matrix for cosine-distance inference without sharing latent vectors. That combination hasn't been published before, and the experimental section is solid evidence that the accuracy half works. They vary silo counts (4/8/16), aggregation intervals, and Dirichlet heterogeneity, with five runs for the key comparisons, and the central tendency is clear: the federated model stays within a few points of centralized, often above 90% relative performance. That's a real, useful result for the federated medical imaging community.

The soft spot is exactly where the stress-test note lands: Section 6.3's claim that the aggregator 'cannot infer anything about the patients' facial images or their latent representations' is not established. The aggregator gets the full Gram/cosine distance matrix (or the masked data to compute it) plus plaintext syndrome labels. The syndrome of any query patient is literally the output of the service, so the claim that labels are 'not considered private' is a value judgment, not a security argument, and for rare genetic syndromes the diagnosis is the most sensitive attribute in the system. Moreover, G = XX^T determines latent vectors up to an orthogonal transformation, and Section 6.3 itself shows those latent vectors enable recognizable face reconstruction. The authors never explain why the unknown rotation, combined with the no-collusion assumption, blocks reconstruction in the aggregator's setting. The security guarantee is delegated to their own FLAKE paper [19], which is fine as a basis, but they don't do the analysis for this cosine-distance deployment, and the proof in FLAKE was for SVM training, not nearest-neighbor inference from the full Gram matrix. So the privacy half needs a rewrite: either a proper analysis or a modest claim like 'the aggregator does not receive plaintext latent vectors; it receives masked data that resist reconstruction under the stated assumptions.'

Minor issues: the code URL has spaces and won't resolve as printed, and the aggregation-interval results (Table 1) don't report repeats, so variance there is unknown.

Verdict: this deserves a serious referee. The accuracy experiments are reproducible and the engineering is useful; the privacy claim needs to be reworded and ideally analyzed before anyone deploys this as a 'private' clinical service. If you're working on federated medical imaging, it's worth a quick read and a cite for the performance data, with a caveat about the privacy guarantee.

Referee Report

2 major / 4 minor

Summary. The paper proposes a federated GestaltMatcher service for rare-syndrome diagnosis from facial images. Hospitals collaboratively train a global ensemble feature extractor via cross-silo horizontal federated learning with secure aggregation, then compute pairwise cosine distances between patient latent representations using a masked kernel-matrix framework called FLAKE. The authors evaluate Top-k matching accuracy on the GestaltMatcher dataset under varying numbers of silos (4/8/16), aggregation intervals (1/5/10/25/50 epochs), and Dirichlet-based non-IID partitions, reporting stable accuracy that stays close to the centralized baseline. A central advertised property is that the protocol is privacy-preserving: the aggregator sees only masked encodings and the Gram/cosine-distance matrix, and, according to Section 6.3, 'cannot infer anything' about patient images or latent representations.

Significance. If the accuracy results are taken at face value, the paper makes a useful empirical contribution: it demonstrates that a federated version of a state-of-the-art dysmorphology pipeline can retain most of its diagnostic accuracy under realistic distributed and heterogeneous conditions. The measurements are credible: five runs are reported for the silo-count and Dirichlet-heterogeneity experiments, and the Top-k trends are internally consistent. The authors also provide code and describe dataset access. However, the paper's headline claim has two halves, accuracy and privacy, and the privacy half is not established. The assertion in Section 6.3 that the aggregator 'cannot infer anything' is contradicted by the protocol itself, since the aggregator receives the full Gram/cosine-distance matrix and plaintext syndrome labels. This is a load-bearing weakness, not a presentation issue, because 'private diagnosis' is central to the title, abstract, and motivation.

major comments (2)

[Section 6.3, final paragraph] The claim that 'the aggregator cannot infer anything about the patients' facial images or their latent representations' is not supported by the protocol described in Sections 5.2 and 5.3. The aggregator receives the full Gram matrix G and hence the cosine-distance matrix D(p,q) via Eq. (1), together with all gallery patients' syndrome labels in plaintext. It can therefore compute the same nearest-neighbor ranking that the service itself provides and label every query patient's syndrome. The statement that syndrome labels are 'not considered private' is asserted without argument; in this medical application the syndrome is the most sensitive attribute. Either the privacy claim must be narrowed to exclude inference of diagnoses, or a concrete analysis must be provided for what the aggregator can and cannot compute from (G, labels).
[Section 6.3; Eq. (1)] Even the weaker claim that patient images and latent representations are protected is not a consequence of FLAKE as presented. A Gram matrix G = X X^T determines the latent matrix X only up to an orthogonal transformation. The paper's own Section 6.3 and Figure 11 demonstrate that access to latent representations of this form permits recognizable face reconstruction and, in extreme phenotypes, potential identification. No argument is given that the unknown orthogonal transformation or the absence of auxiliary data prevents a similar reconstruction from the Gram matrix. Delegating to the FLAKE paper [19] is insufficient because [19] does not analyze this cosine-distance inference protocol or the additional leakage from plaintext syndrome labels.

minor comments (4)

[Section 3.4.4] The text refers to 't-SNE visualizations (Figure 9)' but the t-SNE panels are in Figure 10. Please correct the cross-reference.
[Section 1 / Section 3.2] Minor typographical issues include 'cross-siloed' and 'siloes' (should be 'silos') in multiple places. Also, the URL in Section 8 contains a space: 'Federated GestaltMatcher Service' should be percent-encoded or replaced with the actual repository URL.
[Section 5.1] The non-zero-sum masking description is informal. For a reader to verify that the aggregator indeed sees only a masked average, the protocol should be written with explicit equations for the masks and the final unmasking operation.
[Table 1 / Section 3.4.2] The aggregation-interval experiments appear to be single runs, while the silo and heterogeneity experiments use five runs. A sentence clarifying whether Table 1 reports a single run or an average would help interpretation.

Circularity Check

1 steps flagged

Privacy guarantee for inference is imported wholesale from the authors' own FLAKE paper [19] without deriving it from the protocol's actual outputs, making the central privacy claim partially self-citational.

specific steps

self citation load bearing [Section 6.3, final paragraph (privacy analysis of cosine-distance computation)]
"As the silos only share their masked encodings with the aggregator, the aggregator cannot infer anything about the patients’ facial images or their latent representations. The aggregator only obtains the Gram matrix of the latent representations of patients’ facial images, which is the desired output for the silos. Since the labels are not considered private and cannot be used to deduce information about the latent representations, the silos share them with the aggregator in plaintext. Thus, our federated GestaltMatcher service enables privacy preserving cosine distance computation and inferen"

The paper's headline privacy claim is the load-bearing premise of the 'private diagnosis' contribution. Instead of proving that the aggregator learns nothing from the actual protocol outputs—the full Gram/cosine-distance matrix computed from masked encodings plus plaintext syndrome labels (§5.2–5.3)—the paper asserts 'cannot infer anything' and outsources the analysis to [19], a prior paper co-authored by two of the present authors. Under the protocol the aggregator can compute the same Top-k syndrome match it returns to the querying silo, because it holds every pairwise distance and every gallery label in plaintext; the Gram matrix also determines the latent vectors up to an orthogonal transformation. The claimed guarantee is therefore imported from the self-citation rather than derived f

full rationale

The accuracy half of the paper (Section 3.4) consists of direct Top-k matching-accuracy measurements against a centralized baseline; no fitted parameter is renamed as a prediction and the >90% retention claim is an empirical ratio, not a derivation from the input distributions. The robustness and heterogeneity results are likewise measured. The only load-bearing circular step is the privacy guarantee in Section 6.3: the statement that the aggregator 'cannot infer anything' about patient images/latents is justified by a reference to FLAKE [19], which overlaps with the current authorship, rather than by an analysis of what the aggregator can compute from the Gram/cosine matrix and the plaintext labels it receives. Because this affects a central advertised contribution but the empirical content of the paper is independent, the score is 4 rather than 6-8. If one treats [19] as a fully external, accepted security proof, the circularity burden would be lower; under the stated rules, however, the target privacy conclusion is imported from the self-citation without adaptation, so it is flagged.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The performance claims are measured quantities with no fitted parameters anchoring them. The privacy claims rest on the semi-honest non-collusion model, the inherited FLAKE analysis, and the contested assumption that a Gram matrix plus plaintext labels leak nothing. The non-zero-sum masking adapts a published zero-sum mask but is a protocol detail, not an invented entity.

free parameters (3)

Aggregation interval (epochs between federated averaging) = values {1,5,10,25,50}; 5 chosen best in 8 of the reported settings
Hand-selected experimental sweep (Section 3.3, Table 1); the best interval depends on setting, so it is a design choice, not a fitted constant of the method.
Rare vs frequent syndrome threshold = 7 images
Definitional cutoff: syndromes with fewer than seven images are labeled data-scarce (Section 3.2). This split drives the R-R, F-FR, R-FR evaluation settings.
Dirichlet concentration alpha for non-IID simulation = {0.5, 1, 5, 10}
Simulation parameter controlling client heterogeneity (Section 3.3, Figure 8); not fitted to data, but the heterogeneity robustness claim is only tested at these four values and with 4 clients.

axioms (5)

domain assumption Semi-honest, non-colluding adversary model
All security claims assume corrupted parties follow the protocol and do not collude (Section 6.1). The paper itself notes active adversaries are out of scope.
domain assumption Shared seed and agreed model architecture across all silos
Silos must share a random seed for mask generation and agree on output layer size and architecture (Sections 5.1, 6.1). Any silo knowing the seed can reproduce the common mask basis, so security rests on the non-collusion assumption.
domain assumption FLAKE security guarantee inherits unchanged to this setting
The paper does not analyze the cosine-distance setting; it refers to the self-authored FLAKE paper (Section 6.3, "we refer the reader to the corresponding paper [19]").
domain assumption Syndrome labels are not private
Section 6.3 states labels are shared with the aggregator in plaintext because they "cannot be used to deduce information about the latent representations." The premise that a diagnosis is not sensitive is questionable in a healthcare context.
ad hoc to paper Observing the full Gram matrix reveals nothing about latent codes or images
Section 6.3 asserts the aggregator "cannot infer anything" from masked encodings and the resulting Gram matrix. A Gram matrix determines latent vectors up to orthogonal transformation, and the paper's own Section 6.3 shows latent vectors suffice for recognizable reconstruction, so the assertion requires proof the paper does not provide.

pith-pipeline@v1.3.0-alltime-deepseek · 15919 in / 14861 out tokens · 151646 ms · 2026-08-04T17:43:07.243748+00:00 · methodology

0 comments

read the original abstract

Machine learning has shown promise in facial dysmorphology, where characteristic facial features provide diagnostic clues for rare genetic disorders. GestaltMatcher, a leading framework in this field, has demonstrated clinical utility across multiple studies, but its reliance on centralized datasets limits further development, as patient data are siloed across institutions and subject to strict privacy regulations. We introduce a federated GestaltMatcher service based on a cross-silo horizontal federated learning framework, which allows hospitals to collaboratively train a global ensemble feature extractor without sharing patient images. Patient data are mapped into a shared latent space, and a privacy-preserving kernel matrix computation framework enables syndrome inference and discovery while safeguarding confidentiality. New participants can directly benefit from and contribute to the system by adopting the global feature extractor and kernel configuration from previous training rounds. Experiments show that the federated service retains over 90% of centralized performance and remains robust to both varying silo numbers and heterogeneous data distributions.

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 1 linked inside Pith

[1]

The Lancet371(9629), 2039–2041 (2008)

Schieppati, A., Henter, J.-I., Daina, E., Aperia, A.: Why rare diseases are an important medical and social issue. The Lancet371(9629), 2039–2041 (2008)

2039
[2]

American journal of medical genetics Part A179(6), 885–892 (2019)

Ferreira, C.R.: The burden of rare diseases. American journal of medical genetics Part A179(6), 885–892 (2019)

2019
[3]

The Journal of Clinical Pharmacology62, 38–55 (2022) 24

Liu, J., Barrett, J.S., Leonardi, E.T., Lee, L., Roychoudhury, S., Chen, Y., Trifillis, P.: Natural history and real-world data in rare diseases: applications, limitations, and future perspectives. The Journal of Clinical Pharmacology62, 38–55 (2022) 24

2022
[4]

Orphanet journal of rare diseases17(1), 436 (2022)

Santos Vieira, B., Bernab´ e, C.H., Zhang, S., Abaza, H., Benis, N., C´ amara, A., Cornet, R., Le Cornec, C.M., Hoen, P.A., Schaefer, F.,et al.: Towards fairification of sensitive and fragmented rare disease patient data: challenges and solutions in european reference network registries. Orphanet journal of rare diseases17(1), 436 (2022)

2022
[5]

Nature Methods20(6), 803–814 (2023)

Banerjee, J., Taroni, J.N., Allaway, R.J., Prasad, D.V., Guinney, J., Greene, C.: Machine learning in rare disease. Nature Methods20(6), 803–814 (2023)

2023
[6]

Nature genetics54(3), 349–357 (2022)

Hsieh, T.-C., Bar-Haim, A., Moosa, S., Ehmke, N., Gripp, K.W., Pantel, J.T., Danyel, M., Mensah, M.A., Horn, D., Rosnev, S.,et al.: Gestaltmatcher facilitates rare disease matching using facial phenotype descriptors. Nature genetics54(3), 349–357 (2022)

2022
[7]

https://eur-lex.europa.eu/eli/reg/ 2016/679/oj

General Data Protection Regulation (GDPR). https://eur-lex.europa.eu/eli/reg/ 2016/679/oj. Accessed: 2024-10-30 (2016)

2016
[8]

arXiv preprint arXiv:1610.02527 (2016)

Koneˇ cn` y, J., McMahan, H.B., Ramage, D., Richt´ arik, P.: Federated optimiza- tion: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527 (2016)

Pith/arXiv arXiv 2016

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.

[1] [1]

The Lancet371(9629), 2039–2041 (2008)

Schieppati, A., Henter, J.-I., Daina, E., Aperia, A.: Why rare diseases are an important medical and social issue. The Lancet371(9629), 2039–2041 (2008)

2039

[2] [2]

American journal of medical genetics Part A179(6), 885–892 (2019)

Ferreira, C.R.: The burden of rare diseases. American journal of medical genetics Part A179(6), 885–892 (2019)

2019

[3] [3]

The Journal of Clinical Pharmacology62, 38–55 (2022) 24

Liu, J., Barrett, J.S., Leonardi, E.T., Lee, L., Roychoudhury, S., Chen, Y., Trifillis, P.: Natural history and real-world data in rare diseases: applications, limitations, and future perspectives. The Journal of Clinical Pharmacology62, 38–55 (2022) 24

2022

[4] [4]

Orphanet journal of rare diseases17(1), 436 (2022)

Santos Vieira, B., Bernab´ e, C.H., Zhang, S., Abaza, H., Benis, N., C´ amara, A., Cornet, R., Le Cornec, C.M., Hoen, P.A., Schaefer, F.,et al.: Towards fairification of sensitive and fragmented rare disease patient data: challenges and solutions in european reference network registries. Orphanet journal of rare diseases17(1), 436 (2022)

2022

[5] [5]

Nature Methods20(6), 803–814 (2023)

Banerjee, J., Taroni, J.N., Allaway, R.J., Prasad, D.V., Guinney, J., Greene, C.: Machine learning in rare disease. Nature Methods20(6), 803–814 (2023)

2023

[6] [6]

Nature genetics54(3), 349–357 (2022)

Hsieh, T.-C., Bar-Haim, A., Moosa, S., Ehmke, N., Gripp, K.W., Pantel, J.T., Danyel, M., Mensah, M.A., Horn, D., Rosnev, S.,et al.: Gestaltmatcher facilitates rare disease matching using facial phenotype descriptors. Nature genetics54(3), 349–357 (2022)

2022

[7] [7]

https://eur-lex.europa.eu/eli/reg/ 2016/679/oj

General Data Protection Regulation (GDPR). https://eur-lex.europa.eu/eli/reg/ 2016/679/oj. Accessed: 2024-10-30 (2016)

2016

[8] [8]

arXiv preprint arXiv:1610.02527 (2016)

Koneˇ cn` y, J., McMahan, H.B., Ramage, D., Richt´ arik, P.: Federated optimiza- tion: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527 (2016)

Pith/arXiv arXiv 2016