{"id":"3e8446d3-ada2-4ddd-91ea-b63ec648de23","arxiv_id":"1908.04951","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper reports near-perfect OOD detection by fine-tuning a two-head CNN on the test data itself, which contaminates the evaluation.","lead":"This paper proposes a two-head neural network that uses unlabeled data to detect out-of-distribution images by maximizing the disagreement between two classifiers. The key caveat is that the unlabeled data are the test images themselves, so the reported near-perfect results may reflect overfitting to the test set rather than a general detector.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported AUROC is inflated because the model is fine-tuned on the same test images used for evaluation; the comparison with methods that never see test data is invalid until Xul is disjoint from the evaluation set.","rationale":"The paper's strongest claim is a large margin over prior OOD detection methods, with AUROC near 100%. For that claim to be trustworthy, the fine-tuning procedure that produces the detector must not have access to the same images that are later scored. The paper's own text makes clear that it does: Section 3.4 states 'we use the test data as the unlabeled data,' and Section 4.1.3 states that after holding out 1,000 ID and 1,000 OOD images for validation, 'the remaining test images containing unlabeled ID or OOD samples were used as Xul for unsupervised training and evaluation.' Thus the unlabeled set used to maximize discrepancy is exactly the union of the ID test set and the OOD test set that yields the AUROC in Table 2. Fine-tuning on those images allows the feature extractor and both classifiers to adapt to the specific test-time distribution, including the particular OOD images; evaluating on the same images is therefore not a measure of how well the detector generalizes to new OOD inputs. This is not a quibble about metric choice: the method is being compared against baselines that were trained only on labeled ID training data and never saw any test image. The Table 3 ablations vary the sizes of Xul but still use the same images for evaluation. Table 4 changes the OOD dataset used for fine-tuning relative to testing, which is a step in the right direction, but the ID test images are still present in Xul, so the ID side is contaminated. A clean test would use a completely disjoint fine-tuning set (e.g., training-split images) and evaluate only on the untouched test split. Until that is done, the claimed state-of-the-art results are not supported. This is a soundness/protocol failure, not a disagreement about the potential of discrepancy-based methods.","tokens_in":12581,"tokens_out":5970,"duration_ms":59003,"concrete_test":"Re-run Table 2 with a strictly disjoint unlabeled set: build Xul from the training splits of CIFAR-100 and TinyImageNet (e.g., 9k ID train images + 9k TIN train images), fine-tune for 10 epochs with the same LR=0.1 and margin=1.2, and evaluate AUROC only on the original CIFAR-100 test split and TIN test split. If AUROC stays above 99, the leakage concern is refuted; if it falls materially below the reported numbers (e.g., near or below ELOC's 96.2 on CIFAR-100/TINr), the headline result is an artifact of test-set contamination.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—state-of-the-art OOD detection via maximum classifier discrepancy on unlabeled data—rests on an evaluation that uses the test data itself for fine-tuning. Section 3.4 says 'we use the test data as the unlabeled data,' and Section 4.1.3 specifies that the remaining test images are used 'as Xul for unsupervised training and evaluation.' The exact ID and OOD images scored in Table 2 are therefore the same images used to adapt the two classifiers, so the reported 99.6–100.0 AUROC is a transductive, test-set-fitted number, not an estimate of generalization to new OOD inputs. The comparison to ODIN/ELOC (which never see test data) is invalid. Table 3 still evaluates on the same images as Xul, and Table 4, while changing the OOD source, shares ID test images between Xul and evaluation, leaving ID-side leakage. The paper's phrase 'totally unseen samples were included during evaluation' is misleading: those samples were seen, albeit without labels, during fine-tuning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-head CNN with a shared feature extractor and two classifiers for out-of-distribution (OOD) detection. The method first trains the network on labeled in-distribution (ID) data, then fine-tunes it using unlabeled data while maximizing the discrepancy between the two classifiers' softmax outputs, defined as H(p1) - H(p2). At inference, a sample is declared OOD when the L1 distance between the two classifiers' probability vectors exceeds a threshold. The authors report near-perfect AUROC values on CIFAR-10/CIFAR-100 benchmarks, real-world food and fashion simulations, and ablation studies. The central claim is that this 'unsupervised' fine-tuning on test data is both legitimate and highly effective, significantly outperforming prior methods that only use labeled ID training data.","tokens_in":12800,"tokens_out":3048,"duration_ms":31624,"significance":"If the empirical evaluation were sound, the idea of using two classifiers and maximizing their discrepancy on unlabeled data would be a meaningful contribution, and the reported AUROC values of 99.6-100.0 on several benchmarks would be a large improvement over prior art. The paper also provides a clear description of the method, extensive ablations, and two real-world simulation cases. However, the evaluation protocol is fundamentally invalid: the unlabeled data used for fine-tuning is the same test data used for evaluation, including the OOD test images. The loss in Eq. (5) explicitly maximizes the discrepancy on those exact test images, so the results are fitted, not predicted. This invalidates the head-to-head comparison with ODIN and ELOC, which do not see the test data during training, and makes the reported performance uninterpretable as a measure of OOD detection generalization.","major_comments":[{"comment":"The evaluation is contaminated because the unlabeled set Xul used for fine-tuning is the same set on which the method is evaluated. Section 4.1.3 states that the remaining test images, including OOD images, were used as Xul for unsupervised training and evaluation, and Section 3.4 says 'we use the test data as the unlabeled data.' Since Step B (Eq. (5)) directly maximizes the discrepancy on these exact test images, the AUROC values in Table 2 are fitted values, not measures of detection performance on unseen OOD inputs. The comparison with ODIN and ELOC, which do not use test data during training, is therefore not a fair comparison.","section":"Section 4.1.3 and Section 3.4"},{"comment":"The ablation studies do not resolve the leakage problem. In Table 3, the 9,000 ID and 9,000 OOD samples used for testing are the same samples that constitute Xul, so the claim that 'totally unseen samples were included during evaluation' is misleading. In Table 4, even when the OOD dataset used for testing is different from the OOD dataset in Xul, the ID test images are still the same as the ID images in Xul, so there remains ID-side leakage. No setting in the paper evaluates on a test set disjoint from Xul.","section":"Section 4.1.7, Table 3 and Table 4"},{"comment":"The inference rule uses an L1-distance threshold delta, but the paper does not explain how delta is selected or whether it is tuned on the validation split. If delta is tuned on the validation set, it is not independent of the fine-tuned model because the model was trained on Xul, which shares the same distribution as the validation set. More importantly, the evaluation on Xul means that the threshold and the model are both adapted to the evaluation data, further inflating the reported metrics.","section":"Section 3.5 and Eq. (6)"},{"comment":"The histogram in Figure 5a, showing that a simple threshold of 1.0 separates ID and OOD samples, is computed on Xul, the same data used for fine-tuning. This visualization therefore demonstrates that the model has fitted the specific OOD test set, not that it has learned a generalizable separation. The same issue applies to the claim that the two classifiers' maximum softmax scores for ID and OOD are cleanly separated after fine-tuning.","section":"Section 4.1.5 and Figure 5"}],"minor_comments":[{"comment":"The discrepancy term H(p1) - H(p2) is asymmetric with respect to the two classifiers, but the training procedure treats F1 and F2 symmetrically in Eq. (2). The paper should explain why this asymmetry is justified and whether the roles of F1 and F2 matter.","section":"Section 3.3, Eq. (1)"},{"comment":"The description of the fair-comparison modification for ODIN and ELOC is unclear: the paper says 'we used two classifiers and calculated the average score of these two classifiers as final output in the other methods,' but it is not specified how ODIN's temperature scaling and input preprocessing are applied in this ensemble setting.","section":"Section 4.1.1"},{"comment":"Figure 4 is described as a histogram of 'the discrepancy (L1 distance)' between the two classifiers' outputs, but Section 3.3 defines discrepancy as H(p1) - H(p2). The paper should clarify which quantity is plotted in Figure 4.","section":"Figure 4"}],"recommendation":"reject","confidential_remarks":"The core problem is a transductive-evaluation fallacy: the model is fine-tuned on the test set, including the exact OOD images that are later scored. This is not a minor methodological slip; it invalidates every headline number in Tables 2, 3, and 5, and the comparison to methods that never see the test set is unfair. Even if the authors re-ran experiments with a disjoint held-out set, the paper's framing as 'unsupervised' training on test data would still need major revision. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper proposes a two-head classifier network for out-of-distribution detection: after supervised pretraining on labeled ID data, it fine-tunes with an unsupervised loss that maximizes the discrepancy between two classifiers on unlabeled data. That unlabeled data is the test split itself. The authors are upfront about this: Section 3.4 says \"we use the test data as the unlabeled data,\" and Section 4.1.3 says the remaining test images (ID plus OOD) were used \"as Xul for unsupervised training and evaluation.\" This means the model is adapted directly on the images it is then scored on. The AUROC numbers in Table 2 (99.6–100.0) are fitted numbers, not estimates of how the detector behaves on new OOD inputs. The comparison to ODIN and ELOC, which never see the test set, is invalid.\n\nThat is the main problem, and it is not a subtle one. The paper's own description makes it explicit. The phrase \"totally unseen samples were included during evaluation\" in the ablation section is inaccurate: samples were unseen in the sense of having no labels, but they were seen by the optimiser during fine-tuning. Table 4 varies the OOD source, but ID test images are still shared between Xul and evaluation, so ID-side leakage remains.\n\nThere is something worth building on. Using unlabeled data for OOD detection is a sensible direction, and borrowing the maximum classifier discrepancy from Saito et al. is a legitimate adaptation. The writing is clear, the ablation on Xul size and composition is informative, and the real-world simulations are a nice extra. But the central evidence does not survive a re-run under a proper protocol: you would need to fine-tune on one unlabeled batch and evaluate on a disjoint batch (or at least hold out part of the unlabeled set). The 5% classification accuracy drop noted in the limitation section also deserves attention.\n\nFor a peer review decision: I would not accept these results as evidence of a state-of-the-art OOD detector. The paper needs a major revision with a clean evaluation protocol before it can be fairly judged. If the transductive framing is intentional, the authors should argue why a detector that has already seen the test batch is useful, and then evaluate on a different batch.\n\nThis belongs in the \"interesting idea, invalid execution\" pile. I would not cite the numbers as they stand.","headline":"The proposed transductive OOD detector is clearly explained and the idea is worth exploring, but the reported near-perfect results are an artifact of fine-tuning on the same test images used for evaluation.","tokens_in":13314,"tokens_out":3662,"would_cite":false,"duration_ms":35171,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a two-head classifier fine-tuned on unlabeled test data, including out-of-distribution images, separates OOD from in-distribution inputs with AUROC above 99% on most benchmarks.","keywords":["out-of-distribution detection","maximum classifier discrepancy","unsupervised fine-tuning","two-head classifier","softmax discrepancy","transductive learning","CIFAR-100","TinyImageNet"],"falsifier":"Evaluate on OOD classes that never appear in the unlabeled fine-tuning set (e.g., use only half the TinyImageNet classes in $X_{ul}$ and test on the other half); if AUROC drops substantially from the near-perfect values, the detector is exploiting the specific OOD images rather than a general notion of being out-of-distribution.","tokens_in":12350,"feed_emoji":"🔍","tokens_out":6795,"duration_ms":56072,"temperature":0.7,"pith_summary":"The paper tries to establish that out-of-distribution (OOD) inputs can be detected almost perfectly by fine-tuning a two-head classifier on unlabeled test data, maximizing the disagreement between the two heads. The authors propose a network with one shared feature extractor and two classifiers that agree on in-distribution (ID) images but are pushed to disagree on anything else. The driving idea is that the discrepancy between the two classifiers' softmax outputs marks whether an input lies outside the ID support. On benchmarks such as CIFAR-100 as ID with TinyImageNet, LSUN, or iSUN as OOD, the method reports AUROC values of 99.6–100.0, far above prior detectors. The practical stake is that OOD detection could be improved cheaply by exploiting unlabeled data available at test time.","feed_headline":"Unlabeled test data fine-tunes two classifiers to catch OOD images","feed_subtitle":"Fine-tuning on test images, including OOD ones, separates unfamiliar inputs from in-distribution data nearly perfectly.","key_machinery":"The central object is a two-head CNN: a shared feature extractor plus two classifier heads, $F_1$ and $F_2$, trained with a margin-based discrepancy loss $d(p_1, p_2) = H(p_1) - H(p_2)$. The margin $m$ in the unsupervised loss prevents overfitting by stopping optimization once the average discrepancy exceeds $m$. The machinery works by using the unlabeled set $X_{ul}$ (the test split) to maximize the gap between the decision boundaries of the two heads, effectively relocating OOD samples outside the ID manifold. At test time, the L1 distance between the two softmax outputs is the OOD score.","core_discovery":"The central discovery is that maximizing the discrepancy between two classifiers on unlabeled data, most of it drawn from the test split, separates ID from OOD samples. The paper defines the discrepancy loss as the difference in entropy between the two softmax outputs, $H(p_1) - H(p_2)$, and during fine-tuning alternately trains the network to classify labeled ID images correctly and to maximize this discrepancy on unlabeled images. This pushes OOD images toward high-entropy predictions from one head and low-entropy from the other, while ID images remain tightly classified by both. At inference, an input is flagged OOD if the L1 distance between the two softmax vectors exceeds a threshold. The paper demonstrates near-perfect separation on standard benchmark pairs and on simulated food and fashion applications, with a simple interpretable threshold near 1.0.","pith_inferences":["A natural test is to fine-tune on an unlabeled set containing only ID images (no OOD at all) and then evaluate on OOD test images; a large AUROC drop would indicate the method relies on seeing OOD samples during fine-tuning.","The transductive setup means the reported near-perfect AUROC should be read as an upper bound for a system that must generalize to future OOD inputs; a fair comparison with methods that do not see test data would require a separate unlabeled set disjoint from the evaluation OOD set.","The discrepancy principle might transfer to non-image modalities where unlabeled data is abundant, though the paper only evaluates images; this remains an open empirical question."],"forward_implications":["OOD detection can be performed without any labeled OOD samples, using only unlabeled data, which is easy to obtain.","The method works even when the unlabeled set contains few OOD samples (e.g., 500 OOD among 9,000 ID), as shown in ablation studies.","The method generalizes when the OOD dataset used in fine-tuning differs from the OOD dataset used for evaluation (e.g., fine-tune on TinyImageNet-crop, test on LSUN-crop).","The approach can be dropped onto existing architectures (DenseNet, Wide ResNet) with minimal modification.","A single threshold near 1.0 separates ID from OOD, unlike earlier methods requiring careful threshold tuning."],"supporting_citations":[{"why":"Supplies the two-head architecture idea from maximum classifier discrepancy in unsupervised domain adaptation, adapted here for OOD detection.","marker":"[22]"},{"why":"Provides the softmax-confidence baseline that the paper improves upon and motivates the observation that OOD samples tend to have lower prediction confidence.","marker":"[9]"},{"why":"ODIN is a main comparison baseline and the source of the benchmark datasets and evaluation metrics used in the experiments.","marker":"[16]"},{"why":"ELOC is the state-of-the-art baseline that the paper outperforms, and its benchmark protocol is followed.","marker":"[26]"},{"why":"CIFAR-10 and CIFAR-100 are used as the in-distribution datasets for the main experiments.","marker":"[13]"},{"why":"TinyImageNet is a primary out-of-distribution test set, with crop and resize variants.","marker":"[4]"},{"why":"LSUN is another out-of-distribution test set with crop and resize variants.","marker":"[31]"},{"why":"SUN database is the source of the iSUN out-of-distribution test set.","marker":"[28]"}],"fun_headline_variants":["Train two heads to disagree, then spot OOD","Max discrepancy on unlabeled data detects OOD","Two classifiers, one trick: disagree to detect OOD","Learn to disagree: a two-head CNN for OOD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method uses the test split, including the OOD test images, as the unlabeled data for fine-tuning, and then evaluates on that same split, so the near-perfect scores may reflect seeing the test OOD data rather than a generalizable OOD detector.","fun_headline_variants_meta":{"raw":{"variants":["Train two heads to disagree, then spot OOD","Max discrepancy on unlabeled data detects OOD","Two classifiers, one trick: disagree to detect OOD","Learn to disagree: a two-head CNN for OOD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000867,"raw_usage":{"total_tokens":3736,"prompt_tokens":901,"completion_tokens":2835,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":2771}},"tokens_in":517,"tokens_out":2835,"duration_ms":18543,"temperature":1.0,"reasoning_tokens":2771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:26:42.776549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate on OOD classes that never appear in the unlabeled fine-tuning set (e.g., use only half the TinyImageNet classes in $X_{ul}$ and test on the other half); if AUROC drops substantially from the near-perfect values, the detector is exploiting the specific OOD images rather than a general notion of being out-of-distribution.","supporting_citations":[{"cited_title":"Maximum classifier discrepancy for unsuper- vised domain adaptation","cited_arxiv_id":null,"evidence_quote":"Supplies the two-head architecture idea from maximum classifier discrepancy in unsupervised domain adaptation, adapted here for OOD detection."},{"cited_title":"A baseline for detect- ing misclassified and out-of-distribution examples in neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the softmax-confidence baseline that the paper improves upon and motivates the observation that OOD samples tend to have lower prediction confidence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ODIN is a main comparison baseline and the source of the benchmark datasets and evaluation metrics used in the experiments."},{"cited_title":"Out-of- distribution detection using an ensemble of self supervised leave-out classifiers","cited_arxiv_id":null,"evidence_quote":"ELOC is the state-of-the-art baseline that the paper outperforms, and its benchmark protocol is followed."}],"review_version":1}