{"id":"7c29a154-63e5-4f61-8b14-49cfad5c106c","arxiv_id":"2502.07276","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors define a contrastive relationship gap and use a one-tailed T-test to detect whether a suspect encoder was pre-trained on a target dataset.","lead":"This paper introduces a method to verify whether a black-box contrastive-learning model was pre-trained on a specific dataset, without modifying the dataset. It could help dataset owners detect unauthorized use of their data in self-supervised models, a growing area of machine learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"p-values are not calibrated because the amplification hyperparameter a in Eq. (11) is tuned per dataset (CIFAR10 a=10000, ImageNette a=0.1, ImageNet a=1), so the reported p<0.05 does not validate the central claim as a hypothesis test.","rationale":"I identify the amplification hyperparameter as the single most load-bearing concern rather than the shadow-model null. The shadow concern is genuine but partially mitigated by the paper's legal-case experiments: across SimCLR/BYOL/SimSiam/MoCo v3/DINO and VGG16/ResNet18/ViT architectures, legal suspects on CIFAR10-2, SVHN, ImageNette-2, and ImageWoof all yield p>0.05, so the default shadow is at least empirically useful for the tested range. The a-tuning issue is not mitigated: the paper changes a per dataset and Appendix A.9 explicitly says the result depends on the choice. Since the headline claim is phrased as a p-value, invalid p-values directly undercut the claim as stated. The method could still work as a practical detector after fixing a or calibrating a threshold, but the current paper does not provide that. This leaves the reader's CONDITIONAL verdict unchanged: the core observation may be true, but the statistical evidence needs re-running with fixed a and/or a calibration study.","tokens_in":27096,"tokens_out":15664,"duration_ms":145561,"concrete_test":"Fix a to its stated default value of 1 for every dataset and rerun all configurations behind Tables 12 and 15 (and the ImageNet application if feasible), keeping all other settings identical. If every illegal case still has p<0.05 and every legal case still has p>0.05, the per-dataset tuning is not the cause of the reported separation and the concern is largely resolved. If any outcome flips, the claimed p-values are artifacts of the tuned a, and the paper must fix a before testing or report false-positive rates from a pre-specified a-selection rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central evidence is the one-tailed paired T-test producing p<0.05 for illegal suspects and p>0.05 for legal suspects. That evidence is undermined by the construction of the test statistic. In Eq. (11), the function I(·) is not an indicator: it returns the hyperparameter a when the stated inequality holds and 1 otherwise. Thus each positive unary or binary gap is weighted by a before being summed, making a a free multiplicative knob in d. Appendix A.2 uses a=10000 for CIFAR10, a=0.1 for ImageNette, and a=1 for ImageNet; Appendix A.9 shows the choice of a changes validation outcomes and states that the defender 'needs to set appropriate a based on his actual situation.' Because a is selected per dataset after seeing the data, the null distribution of the test statistic is not fixed in advance. A p-value computed after data-dependent selection of the statistic is not a valid frequentist p-value: Figure 7 shows the results move with a, and there is no pre-registered rule for choosing it. Consequently, the reported 'p-value markedly below 0.05' is not a calibrated significance test; it is a tuned decision rule. Sensitivity/specificity at the p=0.05 threshold inherits the same problem. The shadow-model null concern raised by the Reader is real, but the a-tuning issue is more immediate: even with a perfect shadow null, the p-values would still be invalid because the test definition changes with a.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DOV4CL, a dataset ownership verification method for self-supervised contrastive pre-trained models in a black-box setting. The method is based on two empirical observations: contrastive encoders produce more similar representations for augmentations of training samples than for unseen samples (unary relationship), and pairwise similarities among training samples are more stable under augmentation (binary relationship). The defender computes a 'contrastive relationship gap' between a suspect encoder and a shadow encoder using the defender's public and private datasets, then runs a one-tailed paired t-test with a 0.05 threshold to decide whether the suspect was pre-trained on the protected dataset. Experiments on CIFAR10, ImageNette, and ImageNet with SimCLR, BYOL, SimSiam, MoCo v3, and DINO report p-values below 0.05 for illegal suspects and above 0.05 for legal suspects, along with sensitivity, specificity, and AUROC comparisons against DI4SSL and EncoderMI. The paper also reports time cost, ablation studies, and an honest limitation section noting that the method does not work for masked image modeling. The code is publicly available.","tokens_in":27430,"tokens_out":3382,"duration_ms":31360,"significance":"If the statistical claims are valid, this would be a useful first step toward dataset ownership verification for contrastive pre-trained models, with the appealing properties of not modifying the dataset and requiring only a small fraction of the data. The paper is thorough in its empirical coverage across multiple architectures and contrastive algorithms, and it explicitly acknowledges limitations (e.g., MIM encoders) and provides code. However, the central evidence rests on p-values from a hypothesis test, and the test statistic depends on a hyperparameter that is tuned per dataset after seeing the data; this directly affects the validity of the headline claims. The paper also relies on a single shadow model as the null reference without establishing when that null assumption holds. The contribution is significant if these statistical issues can be resolved, but the current manuscript does not yet support the strong claim of calibrated hypothesis testing.","major_comments":[{"comment":"The amplification hyperparameter a invalidates the reported p-values as frequentist significance tests. In Eq. (11), I(·) is described as returning a when the condition holds and 1 otherwise, so a is a multiplicative weight on each gap component rather than an indicator. Appendix A.2 sets a = 10000 for CIFAR10, a = 0.1 for ImageNette, and a = 1 for ImageNet, and Appendix A.9 shows that the p-values change with a and states that the defender 'needs to set appropriate a based on his actual situation.' Since a is chosen after seeing the data and changes the test statistic, the null distribution is not fixed in advance, and the reported p-values in Tables 10-15 and Table 2 are not valid out-of-sample significance levels. The paper should either fix a before data inspection, report a sensitivity analysis across a without calling the results p-values, or use a proper calibration procedure.","section":"Eq. (11), Appendix A.2, Appendix A.9"},{"comment":"The shadow-model null assumption is not justified. The method uses a single defender-chosen shadow encoder (default ResNet18 with SimCLR trained on SVHN) as the reference in the paired t-test, and the null hypothesis is that the suspect's gap is not larger than the shadow's gap. If an innocent suspect has a different architecture, training algorithm, or training data, its gap on Dpub versus Dpvt may differ systematically from the shadow's, producing false positives. The paper tests only a limited set of combinations (mainly VGG16/ResNet18 with five contrastive methods on CIFAR10 and ImageNette), and does not provide a characterization of when the shadow is a valid null distribution. Additional experiments with more diverse innocent suspects, or a null constructed from multiple shadows, are needed to support the method's black-box claim.","section":"Section 3.2.3 and Tables 12, 15"},{"comment":"The definition of the contrastive relationship gap is ambiguous and does not clearly map to the t-test. Eq. (1) defines d as a set of differences, while Eq. (11) defines d as a 2-element set containing sums over unary and binary components; it is not clear whether d is a scalar, a vector, or a set, and how the paired t-test is applied to a collection of such objects after K samplings. The paper should specify the exact construction of the paired samples used in the t-test, including how the vector/set-valued d is reduced to a testable quantity.","section":"Eqs. (1) and (11), Section 3.2.3"},{"comment":"The reported p-values are averaged over three trials, but averaging p-values by arithmetic mean is not statistically meaningful. p-values from repeated experiments should be combined using meta-analytic methods or reported as ranges; the current presentation can mislead readers about the stability of the test outcomes. The sensitivity/specificity metrics in Table 1 are computed from these p-value classifications, so the issue propagates to the claimed AUROC of 1.0.","section":"Tables 10-15 and Table 2"}],"minor_comments":[{"comment":"There is a typo in the first sentence: 'the the availability' should read 'the availability.'","section":"Section 1"},{"comment":"Eq. (11) uses the symbol I(·) for a function that is not an indicator; this is confusing because standard notation for an indicator is 1{·}. Renaming this function, e.g., to w(·), would improve clarity.","section":"Section 3.2.2"},{"comment":"Figure 7 shows that the choice of a changes the verification outcome, and the text says this effect 'is related to the distributions of Dpub and Dsdw and is not fixed.' This is an honest statement, but it underscores the need for a principled selection rule rather than a per-dataset search.","section":"Appendix A.9"},{"comment":"The experimental setup uses a random split 'into two subsets of equal size' for Dpub and Dalt, but the paper does not describe how many random splits were used or whether the reported results are averaged across splits; this should be clarified.","section":"Section 4.1 and Appendix A.2"},{"comment":"The paper says the method uses only 0.1% of ImageNet data for verification, which is a strength, but the time comparison in Table 8 reports a single run; reporting variance across multiple runs would strengthen the claim.","section":"Section 4.3"},{"comment":"The watermark-based baseline 'DOV-CTRL' is adapted from a backdoor attack, and the comparison is useful; however, the paper does not discuss whether the watermark method modifies the dataset, which is a key difference from the proposed method and should be stated explicitly in the comparison.","section":"Appendix A.11"}],"recommendation":"major_revision","confidential_remarks":"The paper has appeared at ICLR 2025, and the core idea of using contrastive relationship gaps for DOV is interesting and timely. The statistical validity of the p-values is the main concern: the amplification parameter a is tuned per dataset after data inspection, so the headline 'rejects the null hypothesis with p<0.05' is not a calibrated test. This is fixable in principle (e.g., by fixing a a priori, reporting a grid of a, or using a calibration set), but it is a load-bearing issue that the current revision must address. The shadow-model null assumption is a second area that needs more empirical support. I do not recommend reject, because the method could be rescued with a more careful evaluation protocol, and the authors are transparent about limitations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper identifies a real signal, builds a verification method around it, and runs a lot of experiments, but the headline statistical claim does not hold up. The p-values are computed after per-dataset tuning of the amplification factor a, so they are not calibrated significance tests.\n\nWhat is actually new: the binary relationship stability observation is new, and combining it with the unary overfitting signal into a 'contrastive relationship gap' is a sensible construction for black-box verification of SSL encoders. The experimental coverage is broad—five contrastive algorithms, several architectures, CIFAR/ImageNette/ImageNet—and the authors release code and honestly report that the method fails on MAE. Showing how a changes results in Appendix A.9 is more candid than most papers in this area.\n\nThe soft spot is not minor. Eq (11) defines I(·) as returning a when the gap is positive, so a is a multiplicative weight on every positive unary/binary gap. The defense chooses a per dataset (10000 for CIFAR10, 0.1 for ImageNette, 1 for ImageNet), and Appendix A.9 says the defender 'needs to set appropriate a based on his actual situation.' That makes p<0.05 a tuned decision rule rather than a valid frequentist test. The shadow-model null assumption is a second real concern: a single ResNet18/SimCLR/SVHN shadow is used to represent innocent models of arbitrary architecture and training data, and the paper only tests a few combinations. The 'first DOV method' claim is also overstated, since DI4SSL already handles SSL encoders, though their setup differs.\n\nIf the authors reframe the method as a calibrated decision rule rather than a hypothesis test, and add a proper shadow-model sensitivity analysis, this could become a solid contribution. As written, the central evidence doesn't support the significance claim.\n\nThis is worth a serious referee—the method is novel and the flaw is fixable—but the paper needs major revision before acceptance.","headline":"A useful empirical signal and broad experiments, but the headline p-value claim is undermined by per-dataset tuning of the test statistic's amplification factor.","tokens_in":27984,"tokens_out":3110,"would_cite":false,"duration_ms":25638,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A one-tailed paired T-test on embedding similarity gaps can decide, at p<0.05, whether a black-box contrastive encoder was pre-trained on a particular unlabeled dataset.","keywords":["dataset ownership verification","contrastive learning","self-supervised learning","black-box encoder","hypothesis testing","membership inference","embedding similarity","unlabeled dataset protection"],"falsifier":"Train a set of innocent encoders that vary in architecture (for example ViT-B/16), training algorithm (for example DINO), training budget, and dataset similarity to Dpub, then run the published one-tailed paired T-test against the standard ResNet18-SimCLR-on-SVHN shadow; the central claim collapses if more than the nominal 5% of those innocent models yield p<0.05.","tokens_in":26885,"feed_emoji":"🔍","tokens_out":9226,"duration_ms":74269,"temperature":0.7,"pith_summary":"The paper tries to establish that dataset theft in self-supervised learning can be detected by querying the suspect model's embedding similarities alone. It claims that a contrastive pre-trained encoder leaves a measurable trace of any dataset it was trained on: compared with a private, unseen dataset, images from the training set have more mutually similar augmented views and a pairwise similarity structure that is more stable under augmentation. The method, called the contrastive relationship gap, computes this trace at global and local scales for the suspect model and for a defender-trained shadow model, then runs a one-tailed paired T-test; a p-value below 0.05 flags the suspect as having been trained on the protected dataset. The paper reports this test succeeds across SimCLR, BYOL, SimSiam, MoCo v3, and DINO, on datasets up to ImageNet with as little as 0.1% of the training set, and declines to work on masked-image-modeling encoders such as MAE.","feed_headline":"Paired T-test spots stolen datasets inside contrastive models","feed_subtitle":"Using only black-box embeddings, it flags stolen pre-training at p<0.05 across five contrastive methods.","key_machinery":"The central object is the contrastive relationship gap $d(D,\\hat{D},M,T)$, defined as the collection of differences $s_i - \\hat{s}_i$ between similarity sets computed on the protected dataset $D$ and a private unseen dataset $\\hat{D}$ for encoder $M$ under augmentation $T$. It is built from two similarity families: the unary relationship similarity set $S_U$, which averages cosine similarities among multiple global and local augmented views of the same image, and the binary relationship similarity set $S_B$, which measures how much the all-pairs similarity distribution of a set of images changes under augmentation. Multi-scale random cropping (global $T^g$ and local $T^l$ augmentations) activates the gap from both coarse and fine spatial scales. The decision procedure is a one-tailed paired T-test comparing the suspect's gap samples $d_{sus}$ with the shadow encoder's gap samples $d_{sdw}$; rejecting the null $H_0$ at $\\alpha=0.05$ declares the dataset stolen.","core_discovery":"The paper's central claim is that the embedding geometry of a contrastive pre-trained encoder contains a forensic signature of its unlabeled training set. The signature is formalized as the contrastive relationship gap: the difference between similarity statistics computed on the defender's public dataset and on a private dataset the suspect never saw. Two statistics define the gap: unary relationship similarity, measuring how much closer augmented views of the same image get when the image was in the training set, and binary relationship similarity, measuring how much less the pairwise similarity structure among images changes under augmentation when those images were in the training set. The paper argues that this gap is dramatically larger for an encoder trained on the protected dataset, and that comparing the suspect's gap to the gap of an innocent shadow encoder lets a one-tailed paired T-test reject the null hypothesis with p<0.05 for stolen datasets while failing to reject it for innocent ones. The authors validate this across five contrastive algorithms, CNN and ViT architectures, and datasets from CIFAR-10 to ImageNet, including robustness checks with DP-SGD training, downstream fine-tuning, and early stopping.","pith_inferences":["A natural extension the paper leaves implicit: the gap test should transfer to non-visual contrastive encoders (text, audio, graphs), because the unary and binary relationships only assume augmentation invariances, not image-specific structure.","The single-shadow null distribution is the fragile part: a suspect trained with much more data, a much longer schedule, or a much larger architecture may show a systematically larger gap even when innocent; an ensemble of shadow models or per-architecture calibration could harden the test.","The gap is plausibly a probe of memorization, not just membership, so varying augmentation intensity, dataset size, or regularization may make the reported p-values move in predictable ways that could be measured.","Because each paired T-test uses K subsampled gap estimates, the effective sample size is K; a practitioner should confirm the K samples are independent draws before interpreting p<0.05 as a sharp bound."],"forward_implications":["A data owner can verify theft of an unlabeled dataset from a pre-trained encoder without modifying the dataset and with only black-box access to the model's embeddings.","The same p-value test transfers across contrastive training algorithms (SimCLR, BYOL, SimSiam, MoCo v3, DINO) and across CNN and ViT architectures, so the defender need not know the suspect's training recipe.","On ImageNet-scale models, verification succeeds using 0.1% of the protected training data and takes far less compute than prior dataset-inference baselines.","The signal survives privacy-preserving training (DP-SGD), full fine-tuning for downstream tasks, and early stopping, so the verification is not limited to freshly pre-trained encoders.","The method does not generalize to masked image modeling encoders such as MAE, which the paper identifies as the boundary of its approach."],"supporting_citations":[{"why":"Supplies SimCLR, the contrastive objective whose unary and binary embedding relationships motivate the gap definition.","marker":"Chen et al. (2020)"},{"why":"SimSiam, a no-negative contrastive method the paper validates on.","marker":"Chen & He (2021)"},{"why":"BYOL, another validation target and the algorithm used for the t-SNE observations in Figure 1.","marker":"Grill et al. (2020)"},{"why":"MoCo v3, another validation method across CNN and ViT backbones.","marker":"Chen et al. (2021a)"},{"why":"DINO, the self-distillation method whose ViT encoders are included in the validation and ImageNet experiments.","marker":"Caron et al. (2021)"},{"why":"The one-tailed paired T-test is taken from this textbook; the whole verdict depends on it.","marker":"Hogg et al. (2013)"},{"why":"DI4SSL, the dataset-inference baseline for self-supervised encoders that the method is compared against.","marker":"Dziedzic et al. (2022)"},{"why":"EncoderMI, the membership-inference baseline adapted to dataset ownership verification.","marker":"Liu et al. (2021a)"},{"why":"DP-SGD, the privacy-training defense used to test the method's robustness.","marker":"Abadi et al. (2016)"},{"why":"MAE, the masked image modeling encoder used in the limitation study.","marker":"He et al. (2022)"}],"fun_headline_variants":["Contrastive models leak training data via embedding geometry","Relationship gap exposes stolen datasets in contrastive models","Black-box test detects if your dataset was used for pre-training","Paired T-test on embeddings uncovers dataset ownership theft"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The test treats one defender-selected shadow model (by default a ResNet18 trained with SimCLR on SVHN) as a faithful stand-in for what any innocent suspect's contrastive relationship gap would look like, so an innocent model whose training recipe produces systematically different gaps could be falsely accused.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive models leak training data via embedding geometry","Relationship gap exposes stolen datasets in contrastive models","Black-box test detects if your dataset was used for pre-training","Paired T-test on embeddings uncovers dataset ownership theft"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1411,"prompt_tokens":991,"completion_tokens":420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":355}},"tokens_in":607,"tokens_out":420,"duration_ms":4303,"temperature":1.0,"reasoning_tokens":355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T13:15:28.545935+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a set of innocent encoders that vary in architecture (for example ViT-B/16), training algorithm (for example DINO), training budget, and dataset similarity to Dpub, then run the published one-tailed paired T-test against the standard ResNet18-SimCLR-on-SVHN shadow; the central claim collapses if more than the nominal 5% of those innocent models yield p<0.05.","supporting_citations":[{"cited_title":"An empirical study of training self-supervised vision transformers","cited_arxiv_id":null,"evidence_quote":"SimSiam, a no-negative contrastive method the paper validates on."},{"cited_title":"Deep learning with differential privacy","cited_arxiv_id":null,"evidence_quote":"DP-SGD, the privacy-training defense used to test the method's robustness."}],"review_version":1}