REVIEW 3 major objections 7 minor 21 references
Improved Allergy Wheal Detection for the Skin Prick Automated Test Device
T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that combining all 32 differently lit images from the Skin Prick Automated Test (SPAT) device detects and outlines allergy wheals more accurately than the same automated pipeline using a single uniformly lit image.
desk verdict A fair internal ablation shows the 32-light SPAT input helps, but the headline accuracy gain is a single-run number without error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the channel-concatenated lighting stack: the 32 color images are simply concatenated along the channel dimension to form a 96-channel input tensor, so ordinary two-dimensional convolution layers can fuse the complementary shadow patterns from every lighting angle in a single forward pass. The second mechanism is the rigid-transform post-processing step, where a grid search over translation in x and y and rotation aligns the candidate wheal regions with the device's 12 known prick locations, followed by a greedy one-to-one matching. This two-step design converts raw pixel probabilities into clinically interpretable wheal detections and makes the single-image baseline a clean ablation, since only the number of input channels differs.
What would settle it
Retrain the baseline on 32 copies of the same full-light image, keeping the 96-channel input and all other settings equal: if detection accuracy at IoU threshold 0.5 climbs to the 91.8 percent level, the reported gain comes from more copies or parameters rather than from distinct lighting angles, whereas if it stays near the 85.2 percent baseline, the diversity of illumination is what carries the improvement.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that using the SPAT device's full stack of 32 images, each lit by a different light source, considerably improves allergy wheal detection compared with using a single conventional full-light image. The two pipeline arms are identical in architecture, training procedure, and post-processing; only the input channel count changes, from 3 channels for the baseline to 96 channels for the 32-image model. On the 217-patient validation set, the multi-image model attains Dice 0.787 versus 0.717 and detection accuracy 91.8 percent versus 85.2 percent at IoU threshold 0.5, and the paper reports that its advantage persists across the full range of IoU thresholds. The authors attribute the gain to the distinct shadow lines cast at wheal edges under different lighting angles, which are absent under uniform lighting, and their gradient-based analysis identifies light angles from the top center and the bottom sides as the most informative.
Load-bearing premise
The load-bearing premise is that the arm's shift between the prick procedure and image capture is only a translation and a rotation in the image plane; if the skin deforms, bends, or tilts out of the camera plane, the matching step can pair a detected wheal with the wrong prick location and lower the reported IoU accuracies for both methods.
Editorial extensions
If this is right
- On the validation cohort, a 91.8 percent detection accuracy at IoU threshold 0.5 with 32 images versus 85.2 percent with one full-light image means about 6 more clinically relevant wheals per 100 are correctly delineated.
- Because the two pipeline arms share the same segmentation network, training procedure, and post-processing, the measured gain is attributable to the multi-light input itself rather than to model changes.
- The gradient-based image-contribution analysis points to top-center and bottom-side light angles as the largest sources of the model's decisions, identifying the lighting directions that matter most for device design.
- The interpretable second step discards isolated false-positive pixel blobs that cannot be matched to a prick location, so the final wheal list carries a built-in sanity check appropriate for a medical setting.
- If the reported accuracies hold in wider deployment, the SPAT device combined with this pipeline could replace manual wheal reading with an automated, standardized first read-out that a physician then verifies.
Reading between the lines
- My inference, not tested in the paper: the same channel-concatenation input design could transfer to other multi-angle or multi-illumination medical imaging tasks, such as dermoscopy or wound assessment, wherever complementary shadows encode surface elevation.
- My inference: the steep drop in accuracy at stricter thresholds, from 74.3 percent at IoU 0.7 to 38.8 percent at IoU 0.8, suggests that remaining errors are mostly boundary-shape errors, so a non-rigid refinement after the rigid matching step could yield larger gains than any further improvement in pixel classification.
- My inference: because the saliency scores identify a few highly informative lighting angles, a smaller subset of images might retain most of the 32-image benefit, which the paper does not investigate; if confirmed, that could simplify the device and speed up capture.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a two-step pipeline for automatic wheal detection and delineation in skin prick tests performed with the Skin Prick Automated Test (SPAT) device. Step 1 is a U-Net segmentation model whose input is the 32 distinct-illumination images captured by the device, concatenated into a 96-channel tensor. Step 2 is an interpretable, algorithmic post-processing stage that thresholds the segmentation map, extracts connected components, and greedily matches the resulting wheal candidates to known prick locations using a per-test rigid transformation optimized by grid search. The paper reports Dice coefficients and a threshold-based IoU 'accuracy' on a hold-out set of 217 patients, comparing the 32-image input against a baseline that uses a single full-light image with otherwise identical architecture and post-processing. The headline results are Dice 0.787 vs 0.717 and accuracy 91.8% vs 85.2% at IoU threshold 0.5, leading to the conclusion that the 32-image modality offers considerably higher accuracy.
Significance. If the reported comparison is reliable, the paper offers a practical demonstration that multi-illumination imaging can substantially improve segmentation of skin test reactions, combined with an interpretable detection stage. Strengths include a large manually annotated dataset (10,416 wheals), a hospital-stratified train/validation split, and a clean ablation design: the baseline differs only in the number of input images, while the architecture, training setup, and step-2 post-processing are shared. The central weakness is that the headline comparison rests on a single random split and a single training run, with no confidence intervals, significance tests, or repeated-seed analysis. In addition, the reported 'accuracy' metric is recall-like and does not penalize false-positive detections at clinically negative sites. With the current evidence, the claim of 'considerably higher accuracy' is not yet statistically established, although the observed effect is plausible and the methodological setup is appropriate for a paired comparison.
major comments (3)
- [Section 4.2, Table 1] The reported improvements (Dice 0.787 vs 0.717, accuracy 91.8% vs 85.2% at tIoU=0.5) are based on a single random split (Section 2) and a single training run for each condition. The 474 wheals used for the IoU analysis are clustered within 217 patients, so the effective sample size for the 6.6-point accuracy difference is much smaller than 474. No confidence intervals, statistical tests, repeated seeds, or multiple cross-validation folds are reported. Because the central claim of the paper is that the 32-image input offers 'considerably higher accuracy,' this lack of uncertainty quantification is load-bearing. I recommend adding patient-level bootstrap confidence intervals for the paired difference, reporting performance across multiple training seeds and/or split repetitions, and reporting per-patient variance of the Dice and accuracy metrics.
- [Section 4.2] The 'accuracy' metric is recall-like: the denominator is the number of clinically relevant ground-truth wheals (474), and a wheal is counted as accurate when its IoU exceeds tIoU. Predicted wheals that are not matched to a ground-truth wheal, or that appear at clinically negative prick sites, do not reduce the accuracy score. This means a model that over-segments at every prick site is not penalized in the reported metric. The paper should report the number of false-positive wheals per test (or specificity and positive predictive value) for both the 32-image and single-image conditions, and should explicitly describe the metric as recall-based rather than 'accuracy' to avoid overstatement of the clinical benefit.
- [Sections 4.1 and 4.2, Table 1] The two headline metrics are computed on different underlying populations. The Dice coefficient is computed on all ground-truth annotations, including wheals below the 15.9 mm^2 threshold, while the IoU accuracy excludes those small non-elevated marks and uses only 474 of the 2604 validation wheals. Since the abstract and conclusion emphasize 'accuracy,' the table may give a misleading impression that the Dice gain and the accuracy gain reflect the same set of cases. The authors should compute Dice on the same clinically relevant subset (or separately report Dice for the excluded wheals) so that the comparison is consistent, and should explicitly state in the conclusion whether the improvement is driven by the clinically relevant wheals.
minor comments (7)
- [Section 1] The phrase 'off-the-shelve' should be 'off-the-shelf'.
- [Section 3.1] The architecture description is not fully reproducible: the number of double-block layers per resolution, the number of groups in group normalization, and the exact down/up-sampling schedule are not specified. Adding these details, or a code release, would strengthen the paper.
- [Section 3.2] The grid-search range and step size for the rigid transformation parameters (translation x, translation y, rotation) are not given. Without these details, the matching step cannot be reimplemented exactly.
- [Section 4.3] The gradient-based sensitivity scores in Figure 6 are descriptive; the text does not connect them to detection accuracy or to a pruning of the 32 images. Clarify whether these scores are intended to inform future image selection or are only qualitative.
- [Figure 5 (right)] The caption does not state whether the qualitative wheal examples are from the 32-image method or the baseline. Please specify.
- [Section 4.2] The term 'accuracy' is non-standard for a recall-like metric. Consider renaming it to 'recall at IoU threshold' or defining the term in the abstract and conclusion to avoid ambiguity.
- [Section 3.1] The manuscript does not state whether the 32 images are acquired simultaneously or sequentially, and whether they are co-registered. Since the input tensor is created by channel-wise concatenation of raw images, any patient movement between captures would create misalignment. Please clarify the acquisition timing and registration status.
Circularity Check
No significant circularity; the 32-image versus single-image comparison is an empirical hold-out evaluation with shared post-processing, not a reduction to its own inputs.
full rationale
The paper's central claim is that using 32 SPAT lighting-condition images yields higher wheal detection accuracy than one full-light image. This is supported by training a U-Net with 96-channel input versus a 3-channel baseline and evaluating both on a 217-patient hold-out set (Table 1, Section 4.2). The comparison is not circular: the multi-image and baseline models differ only in input channel configuration, and the algorithmic Step 2 detection and rigid-transform matching are identical for both conditions, so no fitted parameter is being renamed as a prediction. Self-citations to SPAT device reliability papers (e.g., refs [7], [17], [18]) provide background and clinical context rather than the load-bearing derivation of the accuracy improvement. The reported improvements are empirical observations, and any concerns about statistical significance or evaluation subset selection are correctness or robustness issues, not circularity. Thus no specific circular step can be quoted, and the appropriate score is 0.
Assumptions & free parameters
free parameters (2)
- Area threshold for clinically relevant wheals =
15.9 mm² (≈4.5 mm diameter)
- Segmentation probability threshold =
0.5
assumptions (4)
- domain assumption Manual polygon annotations by study personnel are accurate ground truth for wheal boundaries.
- domain assumption A wheal area threshold of 15.9 mm² corresponds to clinically relevant wheals of diameter 4.5 mm.
- domain assumption Rigid transformation (translation and rotation) is sufficient to account for arm repositioning between prick and image capture.
- domain assumption The 32 lighting images provide complementary information that improves segmentation.
Cite this review
Pith. "Pith review of Improved Allergy Wheal Detection for the Skin Prick Automated Test Device." pith.science (2026). https://pith.science/paper/AOYK6OI2
@misc{pith2026250605862,
author = {Pith},
title = {Pith review of: Improved Allergy Wheal Detection for the Skin Prick Automated Test Device},
year = {2026},
howpublished = {\url{https://pith.science/paper/AOYK6OI2}},
note = {Machine review of arXiv:2506.05862}
}
abstract
Background: The skin prick test (SPT) is the gold standard for diagnosing sensitization to inhalant allergies. The Skin Prick Automated Test (SPAT) device was designed for increased consistency in test results, and captures 32 images to be jointly used for allergy wheal detection and delineation, which leads to a diagnosis. Materials and Methods: Using SPAT data from $868$ patients with suspected inhalant allergies, we designed an automated method to detect and delineate wheals on these images. To this end, $10,416$ wheals were manually annotated by drawing detailed polygons along the edges. The unique data-modality of the SPAT device, with $32$ images taken under distinct lighting conditions, requires a custom-made approach. Our proposed method consists of two parts: a neural network component that segments the wheals on the pixel level, followed by an algorithmic and interpretable approach for detecting and delineating the wheals. Results: We evaluate the performance of our method on a hold-out validation set of $217$ patients. As a baseline we use a single conventionally lighted image per SPT as input to our method. Conclusion: Using the $32$ SPAT images under various lighting conditions offers a considerably higher accuracy than a single image in conventional, uniform light.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Annals of allergy, asthma & immunology 100(3), S1–S148 (2008)
Bernstein, I.L., Li, J.T., Bernstein, D.I., Hamilton, R., Spector, S.L., Tan, R., Sicherer, S., Golden, D.B., Khan, D.A., Nicklas, R.A., et al.: Allergy diagnostic testing: an updated practice parameter. Annals of allergy, asthma & immunology 100(3), S1–S148 (2008)
work page 2008
-
[2]
In: Image Pro- cessing: Machine Vision Applications VII
Bulan, O.: Improved wheal detection from skin prick test images. In: Image Pro- cessing: Machine Vision Applications VII. vol. 9024, pp. 138–147. SPIE (2014)
work page 2014
-
[3]
Journal of allergy and clinical immunology 116(2), 341–346 (2005)
Carr, W.W., Martin, B., Howard, R.S., Cox, L., Borish, L., et al.: Comparison of test devices for skin prick testing. Journal of allergy and clinical immunology 116(2), 341–346 (2005)
work page 2005
-
[4]
Wiener klinische Wochenschrift71, 551–554 (1959)
Ebruster, H.: The prick test, a recent cutaneous test for the diagnosis of allergic disorders. Wiener klinische Wochenschrift71, 551–554 (1959)
work page 1959
-
[5]
https://doi.org/10.5281/zenodo.3828935
Falcon, W., The PyTorch Lightning team: PyTorch Lightning (Mar 2019). https://doi.org/10.5281/zenodo.3828935
-
[6]
Theoretical Computer Science154(2), 165–181 (1996) 10 R
Fiorio, C., Gustedt, J.: Two linear time union-find strategies for image processing. Theoretical Computer Science154(2), 165–181 (1996) 10 R. Daemset al
work page 1996
-
[7]
Allergy78(5), 1366–1368 (2023)
Gorris, S., Uyttebroek, S., Backaert, W., Jorissen, M., Schrijvers, R., Thompson, M.J., Loeckx, D., Seys, S.F., Van Gerven, L., Hellings, P.W.: Reduced intra-subject variability of an automated skin prick test device compared to a manual test. Allergy78(5), 1366–1368 (2023)
work page 2023
-
[8]
Clinical and Translational Allergy13(2), e12220 (2023)
Gureczny, T., Heindl, B., Klug, L., Wantke, F., Hemmer, W., Wöhrl, S.: Allergy screening with extract-based skin prick tests demonstrates higher sensitivity over in vitro molecular allergy testing. Clinical and Translational Allergy13(2), e12220 (2023)
work page 2023
Show all 21 references
-
[9]
Clinical and translational allergy3, 1–10 (2013)
Heinzerling, L., Mari, A., Bergmann, K.C., Bresciani, M., Burbach, G., Darsow, U., Durham, S., Fokkens, W., Gjomarkaj, M., Haahtela, T., et al.: The skin prick test–european standards. Clinical and translational allergy3, 1–10 (2013)
2013
-
[10]
Allergy71(8), 1095–1102 (2016)
Justo, X., Díaz, I., Gil, J., Gastaminza, G.: Prick test: evolution towards automated reading. Allergy71(8), 1095–1102 (2016)
2016
-
[11]
arXiv e-prints pp
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv e-prints pp. arXiv–1412 (2014)
2014
-
[12]
Journal of Imaging Informatics in Medicine pp
Lee, Y.H., Shim, J.S., Kim, Y.J., Jeon, J.S., Kang, S.Y., Lee, S.P., Lee, S.M., Kim, K.G.: Allergy wheal and erythema segmentation using attention u-net. Journal of Imaging Informatics in Medicine pp. 1–9 (2024)
2024
-
[13]
Annals of allergy, asthma & immunology89(4), 368–371 (2002)
McCann, W.A., Ownby, D.R.: The reproducibility of the allergy skin test scoring and interpretation by board-certified/board-eligible allergists. Annals of allergy, asthma & immunology89(4), 368–371 (2002)
2002
-
[14]
In: Advances in Neural Information Processing Systems 32, pp
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperativ...
2019
-
[15]
In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18....
2015
-
[16]
World Allergy Organization Journal11, 1–3 (2018)
Sánchez-Borges, M., Martin, B.L., Muraro, A.M., Wood, R.A., Agache, I.O., An- sotegui, I.J., Casale, T.B., Fleisher, T.A., Hellings, P.W., Papadopoulos, N.G., et al.: The importance of allergic disease in public health: an icaall statement. World Allergy Organization Journal11...
2018
-
[17]
Frontiers in Allergy4, 1289031 (2023)
Seys, S.F., Gorris, S., Uyttebroek, S., Backaert, W., Jorissen, M., Schrijvers, R., Daems, R., Loeckx, D., Van Gerven, L., Hellings, P.W.: Evaluation of skin prick location on the forearm using a novel skin prick automated test device. Frontiers in Allergy4, 1289031 (2023)
2023
-
[18]
Rhinology62(2), 216–222 (2024)
Seys, S.F., Roux, K., Claes, C., Van Cappellen, L., Werpin, L., Thompson, M., Loeckx, D., Sebrechts, H., Gorris, S., Van Gerven, L.: Skin prick automated test device offers more reliable allergy test results compared to a manual skin prick test. Rhinology62(2), 216–222 (2024)
2024
-
[19]
In: Proceedings of the International Conference on Learning Representations
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. In: Proceedings of the International Conference on Learning Representations. ICLR (2014)
2014
-
[20]
Springer Nature (2022)
Szeliski, R.: Computer vision: algorithms and applications. Springer Nature (2022)
2022
-
[21]
In: Proceedings of the European conference on computer vision (ECCV)
Wu, Y., He, K.: Group normalization. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)
2018
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.