{"id":"bfb911c1-d281-40f6-84b4-bb00776f25c1","arxiv_id":"2502.04748","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Contrastive learning with in-painting as a self-supervised pre-training task slightly improves capsule network accuracy on PICCOLO polyp classification (0.40 vs 0.38) but gives worse AUROC than ImageNet pre-training.","lead":"This paper tests whether self-supervised pre-training can help capsule networks classify colon polyps when medical data is scarce. The authors report a small accuracy gain from contrastive pre-training, but the result is not consistent across all evaluation metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'surpassing ImageNet' claim rests on 0.01–0.02 absolute metric gaps with no error bars, and Table 1's balanced-accuracy values are arithmetically implausible for a 3-class problem; the comparison is not established as real.","rationale":"The paper has a clear and honest experimental setup: a modified CapsNet, two SSL pretext tasks, the PICCOLO dataset, and a common fine-tuning protocol across initialization strategies. The qualitative SSL reconstructions and t-SNE plots suggest the pretext tasks are actually learned, which gives the paper some independent support. However, the headline claim in Section 5 is built entirely on Table 1. The SSL-contrastive advantages are 0.01–0.02 absolute; no variance estimates are given, and ImageNet is better on MCC and AUROC, so the 'surpassing' wording is selective. The balanced-accuracy values are even more concerning: with three classes, standard balanced accuracy cannot be as low as 0.12–0.16 while accuracy is 0.38–0.40, given the reported test distribution. This indicates a metric definition or implementation error, making the claimed 6.67% balanced-accuracy improvement uninterpretable. The reader's weakest assumption about small weight updates is real and relevant, but the more decisive issue is empirical: even if SSL moves weights only slightly, the claimed advantage could still be real if the differences were statistically reliable—but the paper does not show that. My concern is testable with repeated seeds and a careful recomputation of metrics. The reader's conditional verdict remains appropriate: the paper is a plausible exploratory study, but the central quantitative claim needs correction and validation before it can be accepted as stated.","tokens_in":9072,"tokens_out":11616,"duration_ms":118747,"concrete_test":"Run all four initialization strategies for 10 seeds (or bootstrap the single test set if compute-limited), and report mean ± std and paired significance tests (e.g., McNemar or paired bootstrap) for accuracy, balanced accuracy, MCC, AUROC, and specificity. Independently recompute balanced accuracy from the test confusion matrix using the standard 3-class average recall and compare against Table 1; if the 95% confidence interval for any SSL-vs-ImageNet gap contains 0, or if the recomputed balanced accuracy does not match the table, the Section 5 superiority claim should be withdrawn or restricted to 'comparable performance within noise.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 1 gives SSL-contrastive vs ImageNet: accuracy 0.40 vs 0.38, balanced accuracy 0.16 vs 0.15, specificity 0.70 vs 0.69—absolute gaps of 0.01–0.02. No standard deviations, confidence intervals, repeated-seed runs, or significance tests are reported, so the Section 5 'surpassing' increments (5.26%, 6.67%, 1.45%) cannot be distinguished from run-to-run noise. The same table shows ImageNet ahead on MCC (0.25 vs 0.22) and AUROC (0.71 vs 0.62), which the conclusion omits. Moreover, the tabled balanced-accuracy values (0.12–0.16) are below the 0.333 random baseline for a three-class problem and, with test prevalences roughly 0.38/0.34/0.28, are arithmetically incompatible with the tabled accuracies if balanced accuracy is the standard average recall. The headline superiority claim therefore rests on metrics that are either unquantified or internally inconsistent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript studies self-supervised pre-training for a modified capsule network on the PICCOLO polyp dataset, a small three-class medical imaging benchmark with class imbalance and distribution shift. The authors compare four initialization strategies: Kaiming/Xavier, ImageNet-initialized first convolutional layers, SSL colorisation, and SSL contrastive learning combined with in-painting. They report that the SSL-contrastive model achieves accuracy 0.40 versus 0.38 for ImageNet pre-training and conclude, in Section 5, that it surpasses ImageNet pre-training by 5.26% in accuracy, 6.67% in balanced accuracy, and 1.45% in specificity. The paper also discusses CapsNet training dynamics, including very small weight updates and weak engagement of class capsules during SSL, and proposes longer training and layer-specific learning rates as future work.","tokens_in":9364,"tokens_out":7249,"duration_ms":76939,"significance":"The practical motivation is solid: CapsNets lack off-the-shelf pretrained weights, and target-domain SSL could be a cheap alternative to ImageNet pre-training for small medical datasets. The paper contributes a modified CapsNet architecture for 224x224 inputs, two SSL pretext-task variants, and a candid discussion of optimization and imbalance difficulties. However, the headline claim that SSL-contrastive pre-training surpasses ImageNet pre-training is not currently supported by the evidence: the comparison is single-run, the reported balanced-accuracy values are arithmetically inconsistent with the reported accuracies, and the same table shows ImageNet ahead on MCC and AUROC, metrics the authors themselves identify as primary. With corrected metrics, multi-seed statistics, and a fairer comparison, this would be a useful empirical data point.","major_comments":[{"comment":"The reported balanced-accuracy values are arithmetically incompatible with the reported accuracies under the standard definition of balanced accuracy as mean recall. For a three-class problem, random balanced accuracy is 0.333. Given the test-set class proportions reported in Section 3.1 (0.38, 0.34, 0.28), an overall accuracy of 0.38 requires a mean recall of at least about 0.333, and an accuracy of 0.40 requires at least about 0.353; the tabled values of 0.12-0.16 are therefore below what is possible. Unless 'balanced accuracy' is defined in a nonstandard way, Table 1 cannot be correct as reported. Please provide the confusion matrix, per-class recall, and explicit definitions of balanced accuracy, AUROC, and specificity for this three-class problem.","section":"Section 3.1, Table 1"},{"comment":"The conclusion that SSL-contrastive pre-training 'surpasses' ImageNet pre-training is not supported by the reported evidence. The absolute differences are 0.01-0.02 on a 333-frame test set, corresponding to a handful of frames, and no confidence intervals, repeated-seed runs, or significance tests are reported. Moreover, Table 1 itself shows ImageNet pre-training ahead on MCC (0.25 vs 0.22) and AUROC (0.71 vs 0.62), the two metrics the paper in Section 4 identifies as primary, yet these differences are omitted from the Section 5 summary. Please report multiple seeds with variability measures, a pre-specified primary metric, and a discussion of all metrics in the table.","section":"Section 5, Table 1"},{"comment":"The proposed mechanism of SSL pre-training is not established by the reported training dynamics. The text reports that primary-capsule and routing-weight updates during SSL training vary only from 10^-6 to 10^-5 and that class capsules 'barely engage' during SSL. If the pre-training changes the weights so little, the downstream accuracy differences may be attributable to data augmentation, the weighted sampler, early stopping, or random variation rather than to the initialization itself. Please quantify the causal effect of the pre-training, for example by reinitializing the class capsules before fine-tuning, ablating the SSL phase, and reporting convergence curves and gradient-norm statistics for each initialization strategy.","section":"Section 4"},{"comment":"The comparison between initialization strategies is confounded. ImageNet weights are applied to only the first four convolutional layers, while SSL weights initialize all layers; SSL models are trained for 200 epochs whereas supervised models are trained for a minimum of 50 epochs with early stopping; and hyperparameters were selected separately for each condition on a validation subset. These differences make it impossible to attribute the observed performance differences to the pre-training objective. Please use identical layer coverage, identical training budgets and schedules, a common hyperparameter-selection protocol, and multiple seeds across all four conditions.","section":"Sections 3.4 and 4"}],"minor_comments":[{"comment":"The validation-set description appears to contain a typo: it says the dominant class remains NICE Type 1 at 66%, then refers to Types 1 and 3 making up 15% and 19%; presumably the dominant class is Type 2. Please correct the class labels.","section":"Section 3.1"},{"comment":"The contrastive loss equations contain formatting errors, including 'P2N' in the denominator and unclear vector notation in the cosine-similarity expression. Please rewrite these equations cleanly and define all symbols consistently.","section":"Equations (2), (7), (8)"},{"comment":"The modified architecture is described only through a figure and general statements. For reproducibility, please provide a table with the exact layer configuration: kernel sizes, strides, channel counts, capsule dimensions, routing iterations, and decoder/transposed-convolution layout.","section":"Section 3.3"},{"comment":"The abstract states that the findings suggest contrastive learning and in-painting are suitable auxiliary tasks, while Section 4 reports that colorisation performs at near-random level. Please clarify in the abstract that colorisation was the underperforming pretext task.","section":"Abstract and Section 4"},{"comment":"Please report metrics with more than two decimals, since the relative improvements in Section 5 (e.g., 5.26% from 0.38 to 0.40) are computed from rounded values and may not be stable. Also specify how AUROC is computed for a three-class problem.","section":"Table 1"},{"comment":"The manuscript does not state whether code, trained model weights, or experiment logs will be made available; given that the central comparison is single-run, releasing these artifacts would substantially increase confidence in the results.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like a workshop contribution than a full journal paper, and the headline claim currently rests on an internally inconsistent table. I would be willing to reconsider a revised version that corrects the metric definitions, adds multi-seed statistics, and removes the confounding between initialization strategy and training budget. The distinctive capsule-network focus is a valid reason to keep it under consideration if the authors can address these issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate first exploration of SSL pre-training for capsule networks on a small medical dataset, but the headline claim of surpassing ImageNet pre-training does not survive contact with the paper's own Table 1. The gaps are 0.01–0.02 in accuracy and balanced accuracy, there are no error bars, and ImageNet wins on MCC and AUROC. Worse, the balanced-accuracy numbers (0.12–0.16) are below the 0.333 random baseline for a three-class problem and are arithmetically incompatible with the reported accuracies given the test-set class proportions. That is a red flag for the reliability of the rest of the table.\n\nWhat is genuinely new: applying contrastive + inpainting and colourisation as pretext tasks to a modified CapsNet, with a reasonable set of baselines (Kaiming/Xavier, ImageNet transfer). The architecture changes (extra conv layers, transposed-conv decoder, skip connections, spread loss) are described in enough detail to reproduce, and the paper is honest about the tiny weight updates (10^-6 to 10^-5) and slow convergence. The discussion of class capsules implicitly learning dataset imbalance is interesting and worth following up.\n\nThe soft spots are substantial. First, the central claim is a single-run comparison with no confidence intervals, repeated seeds, or significance tests; the observed \"improvements\" are within run-to-run noise for a 333-image test set. Second, the paper itself reports that primary capsule and routing weight updates during SSL are minuscule and that class capsules \"barely engage\" — which undercuts the idea that SSL pre-training is moving the weights into a better basin. The 5.26% accuracy improvement could equally be explained by the augmented training or the weighted sampler. Third, the conclusion selectively cites accuracy, balanced accuracy, and specificity while ignoring the MCC and AUROC results where ImageNet is better. Fourth, no code or data are released.\n\nWho is this for? Researchers working on CapsNets or SSL for small medical datasets, as a preliminary data point. The paper deserves a serious referee because the question is valid and the experiments are not trivial, but it needs major revision: rerun with multiple seeds, report error bars, fix the balanced-accuracy computation, and tone down the conclusion to \"comparable to ImageNet pre-training at best.\" I would not cite it as evidence of SSL surpassing ImageNet, but I'd point to it as an example of the kind of careful validation that small-dataset SSL comparisons need.","headline":"A plausible first exploration of SSL pre-training for CapsNets whose headline claim is undercut by the paper's own table: no error bars, impossible balanced-accuracy values, and ImageNet ahead on MCC and AUROC.","tokens_in":9878,"tokens_out":3067,"would_cite":false,"duration_ms":29339,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-supervised pre-training on a small, imbalanced medical dataset can initialize a capsule network as well as or better than ImageNet pre-training.","keywords":["self-supervised learning","capsule networks","medical imaging","transfer learning","polyp classification","contrastive learning","inpainting","colourisation"],"falsifier":"Re-run the fine-tuning stage from a random initialization using the exact same augmentation pipeline, weighted sampler, loss weights, and training schedule as the SSL-contrastive branch; if that run matches or exceeds 0.40 accuracy on the PICCOLO test set, the central claim is falsified. A second check: train the SSL branch for only one epoch so the weights barely move, then fine-tune; if downstream accuracy is unchanged, the pre-training step is not the cause of the gain.","tokens_in":8889,"feed_emoji":"🩺","tokens_out":7943,"duration_ms":66924,"temperature":0.7,"pith_summary":"This paper tries to establish that self-supervised pre-training on a small, imbalanced medical dataset can initialize a capsule network as well as or better than the standard practice of pre-training on ImageNet. The testbed is polyp classification on the PICCOLO colonoscopy dataset, which has only 3,433 frames and strong distribution shifts between splits. The authors compare four weight initializations for their modified capsule network: random (Kaiming/Xavier), ImageNet-pretrained ResNet features, SSL via colourisation, and SSL via contrastive learning combined with in-painting. They report that the contrastive pre-trained model matches ImageNet pre-training and edges ahead on accuracy, balanced accuracy, and specificity. If the claim holds, small medical imaging projects could pre-train on their own unlabelled target data instead of relying on large external datasets.","feed_headline":"SSL pre-training beats ImageNet for capsule networks on small data","feed_subtitle":"On a 3,433-image polyp dataset, contrastive pre-training edges out ImageNet in accuracy, balanced accuracy, and specificity.","key_machinery":"The central object is the modified capsule network plus the self-supervised pretext tasks used to pre-train it. The architecture adds five convolutional layers before the primary-capsule layer with smaller kernels and strides, replaces the original margin loss with spread loss, and uses a decoder built from transposed convolutions with down-weighted skip connections. The mechanism that is claimed to do the work is contrastive learning with an in-painting auxiliary task: augmented pairs of the same image are pulled together by the NT-Xent loss while the reconstruction branch fills masked patches, and the combined loss is meant to teach colour and texture features that matter for polyp diagnosis. The paper also compares this against a colourisation pretext task and against ImageNet-pretrained ResNet features applied to the first four convolutional layers.","core_discovery":"The central claim is that contrastive self-supervised learning combined with an in-painting task is a viable weight-initialization strategy for capsule networks on small medical datasets, achieving results comparable to ImageNet pre-training without requiring a large external corpus. On the PICCOLO test set, the SSL-contrastive pre-trained model reaches 0.40 accuracy against 0.38 for the ImageNet-initialized model, with higher balanced accuracy (0.16 vs 0.15) and higher specificity (0.70 vs 0.69), while the ImageNet model retains higher MCC (0.25 vs 0.22) and AUROC (0.71 vs 0.62). The paper reports these as increases of 5.26%, 6.67%, and 1.45% respectively in the conclusion. The colourisation pretext task performs poorly, near random, so the paper's positive result is specific to the contrastive-with-in-painting task, not SSL in general.","pith_inferences":["The reported gains are relative improvements on a single test split; the paper does not report repeated-seed variance, so it remains open whether the 5.26% accuracy edge is stable across runs.","The observed weight-update magnitudes of $10^{-6}$ to $10^{-5}$ during SSL pre-training raise the possibility that the downstream gains come from the augmented training and weighted sampling rather than from the SSL initialization itself; this competing explanation is not isolated in the paper.","If the mechanism is domain-relevant feature learning, the same contrastive-plus-in-painting recipe could plausibly transfer to other small medical imaging tasks with unlabelled target frames, though the paper only tests colon polyp classification.","A cheaper substitute, longer fine-tuning from random initialization with the same augmentations and weighting, is not compared, so the specific value of the SSL pre-training step over extended supervised training is unmeasured."],"forward_implications":["SSL pre-training on the target medical dataset can substitute for ImageNet pre-training when initializing capsule networks, removing the need for a large external dataset.","Contrastive learning combined with in-painting is the productive SSL recipe for this domain; colourisation alone is not.","Because capsule networks learn slowly, extending SSL training from 200 toward 500 epochs and using layer-specific learning rates are the paper's own suggested next steps for improving the gains.","SSL pre-training on imbalanced data teaches the network the dataset's class biases, adding 5-10 fine-tuning epochs before the bias is overcome.","Capsule networks are sensitive to initialization, and a self-supervised initialization that captures domain-specific features is a way to reduce that sensitivity."],"supporting_citations":[{"why":"Supplies the capsule network architecture and dynamic routing that the modified model builds on.","marker":"[22]"},{"why":"Provides the contrastive learning framework and NT-Xent loss used for the contrastive pretext task.","marker":"[3]"},{"why":"Provides the in-painting auxiliary task combined with contrastive learning.","marker":"[20]"},{"why":"Provides colourisation as the alternative SSL pretext task that performs poorly in the comparison.","marker":"[17]"},{"why":"Provides the PICCOLO dataset used for pre-training and evaluation.","marker":"[24]"},{"why":"Provides ImageNet, the large-scale dataset whose pre-trained weights serve as the baseline for comparison.","marker":"[6]"},{"why":"Provides the spread loss used for supervised classification in the modified CapsNet.","marker":"[13]"},{"why":"Provides the Kaiming/He initialization used as one of the four compared initializations.","marker":"[12]"}],"fun_headline_variants":["Contrastive SSL pre-training edges ImageNet for capsule nets","SSL contrastive pre-training beats ImageNet on small data","Contrastive SSL pre-training boosts capsule net accuracy on small medical data","SSL contrastive pre-training improves capsule nets on small data by 5.26%","Contrastive SSL pre-training edges out ImageNet for small medical datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim assumes that the self-supervised pre-training actually moves the capsule network's weights into a better starting point for fine-tuning; if the tiny weight changes observed during SSL training do not meaningfully affect downstream performance, the reported gains could instead come from the augmented training, the weighted sampling, or chance.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive SSL pre-training edges ImageNet for capsule nets","SSL contrastive pre-training beats ImageNet on small data","Contrastive SSL pre-training boosts capsule net accuracy on small medical data","SSL contrastive pre-training improves capsule nets on small data by 5.26%","Contrastive SSL pre-training edges out ImageNet for small medical datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000809,"raw_usage":{"total_tokens":3572,"prompt_tokens":990,"completion_tokens":2582,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2488}},"tokens_in":606,"tokens_out":2582,"duration_ms":17724,"temperature":1.0,"reasoning_tokens":2488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:37:18.526166+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the fine-tuning stage from a random initialization using the exact same augmentation pipeline, weighted sampler, loss weights, and training schedule as the SSL-contrastive branch; if that run matches or exceeds 0.40 accuracy on the PICCOLO test set, the central claim is falsified. A second check: train the SSL branch for only one epoch so the weights barely move, then fine-tune; if downstream accuracy is unchanged, the pre-training step is not the cause of the gain.","supporting_citations":[{"cited_title":"International Conference on Machine Learning 1, 1597–1607 (7 2020)","cited_arxiv_id":null,"evidence_quote":"Provides the contrastive learning framework and NT-Xent loss used for the contrastive pretext task."},{"cited_title":"2017 IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR) pp","cited_arxiv_id":null,"evidence_quote":"Provides colourisation as the alternative SSL pretext task that performs poorly in the comparison."},{"cited_title":"Interna- tional Conference on Learning Representations (2 2018)","cited_arxiv_id":null,"evidence_quote":"Provides the spread loss used for supervised classification in the modified CapsNet."},{"cited_title":"2015 IEEE International Conference on Computer Vision (ICCV) pp","cited_arxiv_id":null,"evidence_quote":"Provides the Kaiming/He initialization used as one of the four compared initializations."}],"review_version":1}