REVIEW 3 major objections 3 minor 1 cited by
Privacy-enhancing Sclera Segmentation Benchmarking Competition: SSBC 2025
T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Competition results show synthetic-only sclera segmentation models can match or approach real-data-trained models, with top F1 scores above 0.8.
desk verdict Useful competition report with code/data, but the headline claim of synthetic-only competitiveness is under-anchored without baselines and per-domain test splits. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the two-track competition design: one track restricts teams to synthetic training data only, while the other permits mixing synthetic with a limited amount of real data. All models are evaluated on the same three datasets, making it possible to compare the effect of data source against the effect of training methodology, with F1 score as the primary performance measure.
What would settle it
Take the winning synthetic-track model and evaluate it on a larger, independent set of real ocular images collected from different environments, devices, and demographics than those used in the competition; if its F1 score drops materially below the mixed-track or real-data-trained baselines, the conclusion that synthetic-only training is competitive would be weakened.
Extended reading notes
Core claim
The paper's central claim is that models trained entirely on synthetic data can achieve competitive sclera segmentation performance, provided dedicated training strategies are used. Evidence comes from the 2025 Sclera Segmentation Benchmarking Competition, where top synthetic-track models reached F1 scores above 0.8 on evaluation datasets containing both synthetic and real-world images. The paper also claims that, in the mixed track, performance differences were driven more by methodological choices than by the inclusion of a limited amount of real data. Together these findings support the promise of synthetic data as a privacy-preserving alternative for biometric model development.
Load-bearing premise
The three evaluation datasets are representative enough of real-world sclera segmentation conditions that the measured performance of synthetic-only models will hold outside the benchmark.
Editorial extensions
If this is right
- Synthetic-only training can be a viable route for sclera segmentation, with top F1 scores above 0.8 on benchmark test sets.
- Dedicated training strategies—not simply the volume of synthetic data—are key to closing the gap with real-data-trained models.
- In the mixed track, methodological choices appear to matter more than the presence of a limited amount of real data.
- Privacy-preserving biometric development becomes more feasible if synthetic ocular images can substitute for real patient imagery.
- The public release of code and data enables other groups to reproduce and build on the competition results.
Reading between the lines
- If this result transfers to deployment, synthetic data could replace real biometric imagery in early-stage model development, reducing consent and privacy burdens in a domain where eyes are considered identifiable.
- A natural next step would be to ablate the mixed-track results by fixing the training methodology and varying only the amount of real data, to isolate the data-source contribution more cleanly.
- Because the benchmark test sets are fixed and public, leaderboard-driven optimization may overfit to their particular characteristics; an independent deployment-style test set would test robustness more convincingly.
- The finding suggests that data generation strategy—diversity, domain randomization, and augmentation—may matter more than whether the source is real or synthetic, which could generalize to other biometric segmentation tasks.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript summarizes the 2025 Sclera Segmentation Benchmarking Competition (SSBC 2025), which evaluated privacy-preserving sclera segmentation models trained on synthetic ocular images. Two tracks are reported: one using only synthetic data and one mixing synthetic data with limited real data. Nine research groups submitted models spanning transformer-based, lightweight, and generative-guided architectures. Evaluation was performed on three datasets containing both synthetic and real images. The central claims are that synthetic-only training can achieve competitive performance, with top F1 scores over 0.8 in the synthetic track, and that performance gains in the mixed track were driven more by methodological choices than by the inclusion of real data. The code and data are publicly available.
Significance. If fully substantiated, the results would be a useful contribution to privacy-aware biometric development, providing evidence on the feasibility of synthetic data for sclera segmentation while making code and data openly available for reproduction. The competition format itself is a strength: it provides a concrete, externally sourced benchmark rather than a single-team evaluation. However, the abstract alone leaves the headline claims insufficiently supported, so the significance depends critically on the complete paper's protocol and analysis.
major comments (3)
- [Abstract, claim 1] The claim that synthetic-only models are 'competitive' at F1 > 0.8 is unanchored. No comparator is specified: competitive relative to what? Real-data-trained models, prior state of the art, or an absolute threshold? Without the corresponding real-data baseline scores for the same test sets, the F1 figure does not support the 'competitive' wording. Please report per-track and per-dataset results for real-trained baselines alongside the synthetic-track results.
- [Abstract, evaluation datasets] The three evaluation datasets contain 'both synthetic and real-world images.' If the synthetic test images are generated from the same distribution or generator as the synthetic training data, the F1 > 0.8 figure may largely reflect training-test overlap rather than transfer to real-world conditions. The abstract does not report per-domain scores. Please provide separate F1 scores for synthetic and real subsets of each test set, and state whether the synthetic test images are from a held-out generator or held-out generation parameters.
- [Abstract, mixed-track claim] The statement that performance gains in the mixed track were 'driven more by methodological choices rather than by the inclusion of real data' is a causal attribution that cannot be identified from competition outcomes alone. Participants simultaneously chose architectures, training schedules, and data strategies, so the effect of data composition is confounded with methodological differences. To support this claim, the paper should include a controlled analysis (e.g., fixed architecture/training schedule with only data composition varied) or at minimum a multivariable analysis that separates these factors.
minor comments (3)
- [Abstract] No error bars, confidence intervals, or statistical significance tests are reported for the F1 scores. As a benchmark comparison, even summary statistics such as mean and standard deviation over multiple runs or participants would help.
- [Abstract] The phrase 'limited amount of real-world data' is vague; please quantify the amount of real data allowed in the mixed track.
- [Abstract] The term 'privacy-enhancing' appears in the title but is not defined or measured in the abstract. Since synthetic data is used as a proxy for privacy, the paper should clarify whether privacy is evaluated (e.g., through re-identification risk) or simply assumed from the use of synthetic data.
Circularity Check
No circularity: benchmark findings are empirical and externally evaluated.
full rationale
The paper is an abstract-only summary of a benchmarking competition. Its claims—that synthetic-only training can achieve F1 scores over 0.8 and that mixed-track gains were driven more by methodology than by real data—are empirical observations from a competition with external evaluation data. There is no derivation chain that reduces to its own inputs. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The evaluation uses held-out test datasets that include real-world images, providing an external benchmark. The absence of an explicit baseline for the word 'competitive' is a reporting/validity concern, not a circularity. The mixed-track inference is potentially confounded because methodological choices and data composition were not independently controlled, but that is an experimental-design limitation, not circular reasoning. Therefore the circularity score is 0.
Assumptions & free parameters
assumptions (2)
- domain assumption Synthetic ocular images are a suitable proxy for real sclera images in training segmentation models
- domain assumption F1 score is an appropriate measure of segmentation quality for the stated conclusions
Cite this review
Pith. "Pith review of Privacy-enhancing Sclera Segmentation Benchmarking Competition: SSBC 2025." pith.science (2026). https://pith.science/paper/HN2OTRSN
@misc{pith2026250810737,
author = {Pith},
title = {Pith review of: Privacy-enhancing Sclera Segmentation Benchmarking Competition: SSBC 2025},
year = {2026},
howpublished = {\url{https://pith.science/paper/HN2OTRSN}},
note = {Machine review of arXiv:2508.10737}
}
abstract
This paper presents a summary of the 2025 Sclera Segmentation Benchmarking Competition (SSBC), which focused on the development of privacy-preserving sclera-segmentation models trained using synthetically generated ocular images. The goal of the competition was to evaluate how well models trained on synthetic data perform in comparison to those trained on real-world datasets. The competition featured two tracks: $(i)$ one relying solely on synthetic data for model development, and $(ii)$ one combining/mixing synthetic with (a limited amount of) real-world data. A total of nine research groups submitted diverse segmentation models, employing a variety of architectural designs, including transformer-based solutions, lightweight models, and segmentation networks guided by generative frameworks. Experiments were conducted across three evaluation datasets containing both synthetic and real-world images, collected under diverse conditions. Results show that models trained entirely on synthetic data can achieve competitive performance, particularly when dedicated training strategies are employed, as evidenced by the top performing models that achieved $F_1$ scores of over $0.8$ in the synthetic data track. Moreover, performance gains in the mixed track were often driven more by methodological choices rather than by the inclusion of real data, highlighting the promise of synthetic data for privacy-aware biometric development. The code and data for the competition is available at: https://github.com/dariant/SSBC_2025.
Forward citations
Cited by 1 Pith paper
-
Step-Attention Refinement of DINOv3 Features for Efficient Anterior Eye Segmentation
Step-attention refinement of multi-level DINOv3 features plus a light conv decoder yields 85.55% mIoU and best domain-shift robustness on seven-class clinical anterior-eye segmentation.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.