{"id":"4eca94f6-8465-48e1-a840-26990f3f85c8","arxiv_id":"2508.00922","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CaliMatch adaptively calibrates classifier and OOD detector confidences in safe semi-supervised learning, improving classification accuracy on CIFAR-10/100, SVHN, TinyImageNet and ImageNet.","lead":"This paper introduces CaliMatch, a safe semi-supervised learning method that adaptively calibrates both the classifier and the out-of-distribution detector to handle unlabeled data containing unseen classes. It reports accuracy and calibration gains over existing safe SSL baselines on several image benchmarks, including ImageNet.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CaliMatch's online calibration consumes a held-out labeled validation set; the reported gains over baselines may be confounded by unequal access to validation labels during training.","rationale":"The reader's weakest assumption concerns the representativeness of the validation set; my concern is that the validation set is used online and asymmetrically, which is a more direct threat to the central empirical claim. Even if the validation split is perfectly representative, CaliMatch's per-epoch access to validation labels during training is an information advantage not available to the baselines unless explicitly controlled. The paper does not state whether the same 10% validation split is used for all methods, nor whether baselines receive any equivalent online calibration signal. The ablations in Table S-6 use OpenMatch with fixed label smoothing, mixup, MbLS, and RankMixup, but none of these consume validation labels in the same way. The circularity in Lemma 1 (S-3.2) reinforces the concern: since the paper's theoretical support is effectively a tautology, the empirical comparisons must carry the argument, and those comparisons are exactly where the validation-set confound sits. I therefore retain the reader's CONDITIONAL verdict; the paper should not be accepted until the confound is ruled out by a controlled experiment. My proposed check isolates the effect of validation-label access from the effect of adaptive calibration itself.","tokens_in":23835,"tokens_out":9192,"duration_ms":107954,"concrete_test":"Re-run the CIFAR-10 kappa=60% comparison under two identical data splits: (i) all methods train on the same 90% labeled subset, with the 10% validation split used only for early stopping for every method; (ii) CaliMatch additionally uses the validation labels for online bin-accuracy estimation while OpenMatch is given an equivalent online calibration signal (e.g., a fixed label-smoothing parameter re-tuned on the validation set each epoch). If CaliMatch's margin over OpenMatch (currently 1.43 percentage points) shrinks below one standard deviation under (i), or disappears under (ii), the reported advantage is attributable to validation-label access rather than to adaptive calibration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the experimental protocol around the validation set. Algorithm 1 updates the bin-accuracy reference values Gamma and Delta 'using validation dataset at every epoch,' and Table S-2 reports that 10% of the training dataset was used for validation. If this validation split is removed from the labeled training set for CaliMatch while comparison methods train on the full labeled split, then CaliMatch has exclusive online access to validation labels—an extra supervision signal that baselines do not receive. The reported improvements (e.g., 87.62 vs 86.19 on CIFAR-10 with kappa=60%) could then reflect this extra label access rather than the adaptive calibration mechanism. The ablations in Table S-6 do not control for this: 'OpenMatch with label smoothing' is not given the same online validation-based adaptation. Additionally, the theoretical justification (Lemma 1, S-3.2) is circular—epsilon is defined as the probability of the bad event, so the lemma's conclusion is tautological—leaving the empirical comparison as the main support for the central claim. This makes the validation-set confound decisive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"CaliMatch proposes a safe semi-supervised learning method that calibrates both the multiclass classifier and the one-versus-rest OOD detector during training. It uses adaptive label smoothing, where the smoothing degree is set by binning validation-set predictions and computing per-bin accuracies, and learnable temperature parameters for both heads. The method is built on FixMatch and uses the calibrated scores for pseudo-label selection and OOD rejection. Experiments on CIFAR-10, CIFAR-100, SVHN, TinyImageNet, and ImageNet report accuracy improvements over existing safe SSL methods, together with improved expected calibration error (ECE). The paper also includes a theoretical analysis (Lemma 1 and Theorem 1 in Supplementary Section S-3) intended to justify why calibration of both heads helps safe SSL.","tokens_in":2122,"tokens_out":1904,"duration_ms":75022,"significance":"If the reported gains are genuine, calibrating both the multiclass classifier and the OOD detector is a valuable and potentially generalizable contribution to safe SSL: the method is simple, the ablations are extensive, and the authors provide source code. The paper also makes a useful empirical point that overconfidence in both heads degrades pseudo-label quality and OOD rejection. However, the central empirical claim is confounded by the training-time use of a held-out labeled validation set, and the theoretical justification is tautological; these issues must be resolved before the contribution can be assessed. The breadth of datasets (including ImageNet) and the inclusion of learning-curve and threshold analyses are strengths, but they do not compensate for the unfair comparison induced by extra validation-label access.","major_comments":[{"comment":"The method's calibration targets Γ and Δ are computed at every epoch from a held-out labeled validation set, which is 10% of the training dataset (Table S-2). These targets enter the training loss through Eqs. (4) and (5). This constitutes training-time use of labels that the comparison methods do not receive: the baselines train only on the labeled training set, whereas CaliMatch additionally uses validation labels to shape its per-bin accuracy estimates and therefore its loss. The paper does not state whether the baselines also train on the full labeled split or on the remaining 90%, but in either case CaliMatch has access to strictly more labeled information. The reported gains (e.g., CIFAR-10 with κ=60%: 87.62 vs 86.19; Table S-8: ResNet-50 accuracy 72.73 vs 58.72) could therefore reflect this extra supervision rather than the calibration mechanism. The ablation in Table S-6 does not control for this, because the OpenMatch variants are not given the same online validation-based target computation. Please add an experiment in which every method is given the same number of labeled examples for training (with CaliMatch's validation set drawn from that budget) and report the resulting accuracies, or, alternatively, compute Γ and Δ on a separate validation set that is not used by any method and show the gains persist. This is the load-bearing issue: the main claim of improved accuracy is confounded by unequal access to validation labels.","section":"Algorithm 1 and Table S-2"},{"comment":"The theoretical justification is circular. Lemma 1 defines ε as the probability of the bad event (a selected sample being OOD or incorrectly pseudo-labeled) and then concludes that the probability of the bad event is at most ε; the bound ε ≤ 1 - min{τ1,τ2} + η follows directly from the calibration assumption that accuracy equals confidence, not from any property of the proposed adaptive smoothing or temperature scaling. Moreover, the condition min{τ1,τ2} ≥ 1-η is inconsistent with the experimental setting τ1 = 0.5 (main text, Section 4) and small calibration error η. Theorem 1 relies on Lemma 1 and therefore inherits its circularity. Since the abstract and contributions claim a theoretical justification for improving calibration of both heads, this claim is not supported. I recommend either rewriting the theory to make a non-tautological statement (e.g., a bound that explicitly depends on the proposed losses' effect on calibration error) or removing the theoretical claim from the abstract and contributions.","section":"Supplementary Section S-3.2, Lemma 1 and Theorem 1"},{"comment":"The claim that adaptive label smoothing 'eliminates the need to manually tune the smoothing degree' is overstated. The method still requires choosing the number of bins M, the thresholds τ1 and τ2, the coefficients λO, λOCal, and λS, and the initial temperatures TM and TO. The sensitivity analysis in Table S-5 covers only λO and λOCal; there is no sensitivity analysis for M or for the initialization of the temperatures. Furthermore, Table S-6 compares CaliMatch against OpenMatch with fixed label smoothing, mixup, MbLS, and RankMixup, but those baselines are not given the same validation-based adaptive targets, so the comparison conflates the validation-access effect with the calibration mechanism. Please provide an ablation that varies M and the temperature initialization, and add a control where the alternative calibration methods receive the same kind of validation-based adaptation (if feasible) or at least explicitly state the training-label budget for each row of Table S-6.","section":"Section 3.2 and Table S-6"}],"minor_comments":[{"comment":"In the TinyImageNet row, the CaliMatch F1 value is printed as '0.691 0.001' without parentheses around the standard deviation; please format consistently with the other entries.","section":"Table S-4"},{"comment":"The line 'Update Γ and ∆ using validation dataset at every epoch' appears after the parameter update, but it is unclear whether the update happens before or after the epoch's training iterations; please specify the order explicitly, especially because the calibration loss in Eqs. (4) and (5) uses Γ and ∆ from the previous epoch or from the current epoch.","section":"Algorithm 1"},{"comment":"The initial values of the scaling parameters TM and TO are set to 1.5 without any justification or sensitivity analysis; please provide a brief explanation or a small study showing the effect of this choice.","section":"Section 3.2"},{"comment":"The OOD calibration loss in Eq. (5) contains a min over k of the second term; the notation is confusing because the min is applied to a sum over k of per-class terms. Please clarify whether the min is inside the sum or over the whole second term, and specify the exact operation that defines the OOD smoothing target.","section":"Equation (5)"},{"comment":"The reliability diagrams in Figure 2 are described with blue and red bars, but the figure appears in grayscale; please ensure the figure is reproducible in color or add patterns to distinguish the bars.","section":"Section 4.2 and Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the validation-set confound: CaliMatch uses 10% of the labeled data as a validation set and updates its calibration targets from that set at every epoch, so it effectively receives more labeled supervision than the baselines. This is a fundamental protocol issue that must be addressed with a matched-label-budget experiment. If the gains persist under a fair comparison, the paper would be a solid contribution to safe SSL. The theoretical section is currently tautological and should be either rewritten or removed from the claims. I would not reject the paper outright because the empirical methodology is otherwise extensive and the proposed losses are sensible, but the current comparison is not convincing without the matched-budget control."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper to know about: CaliMatch applies adaptive label smoothing and learnable temperature scaling to both the multiclass classifier and the OvR OOD detector in safe SSL, with calibration targets derived from validation-set accuracy. The combination is new and the experiments are extensive, but I have a serious concern about the evaluation protocol.\n\nWhat's genuinely useful: the idea of using measured bin accuracies to set per-sample smoothing degrees, and applying it to the OOD detector as well as the classifier, is a legitimate extension that appears to improve pseudo-label quality and OOD rejection in practice. The ablation study is thorough, and the learning curves give a clear picture of why it works.\n\nThe soft spot: Algorithm 1 updates the calibration targets using the validation dataset at every epoch, and the paper reports using 10% of the training set for validation. It is not explained whether the comparison methods also get access to that held-out label information. If CaliMatch's validation split is subtracted from its training set while baselines train on the full labeled split, then CaliMatch receives an extra supervision signal—validation labels used online for calibration—that baselines do not have. The ablations in Table S-6 do not control for this: OpenMatch with label smoothing or mixup is not given the same online validation-based adaptation. This makes the reported gains, such as 87.62 versus 86.19 on CIFAR-10, hard to attribute to the calibration mechanism.\n\nThe theory section is mostly definitional. Lemma 1 defines epsilon as the bad-event probability and then concludes that the good-event probability is 1-epsilon; the bound that follows is just a restatement of calibration plus high thresholds. Theorem 1 is a standard bounded-gradient argument that does not depend on the specifics of CaliMatch. I would not put weight on this part.\n\nThese are fixable. For the empirical claim, run a fair comparison where either all methods use the same labeled set for training and CaliMatch's calibration uses a separate held-out set, or baselines receive the same validation statistics. For the theory, either sharpen it or drop it.\n\nWho this is for: safe SSL researchers. The method is likely useful when a small labeled validation set is available. It deserves a serious referee—the core idea is sound, but the comparison is not currently clean. Send it to review with instructions to focus on the validation protocol.","headline":"Adaptive calibration scheme for safe SSL, but the empirical comparison may be confounded by CaliMatch's online use of a validation set that baselines do not get.","tokens_in":24528,"tokens_out":4674,"would_cite":false,"duration_ms":53677,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CaliMatch argues that overconfidence, not the unseen-class mismatch itself, breaks semi-supervised learning, and that calibrating the classifier and OOD detector during training fixes it, beating seven methods on five image benchmarks.","keywords":["safe semi-supervised learning","out-of-distribution detection","confidence calibration","adaptive label smoothing","temperature scaling","pseudo-labeling","label distribution mismatch","overconfidence"],"falsifier":"On a benchmark where the true labels of all unlabeled in-distribution samples are known, record throughout training the per-batch gradient distance between the FixMatch surrogate loss and the ideal cross-entropy loss on the correctly pseudo-labeled ID subset; the paper's Theorem 1 predicts this distance shrinks as calibration error drops, so if accuracy gains appear while that distance stays flat, the claimed mechanism would be falsified. A second check: shrink the labeled validation holdout (for example from 10% to 2%) and see whether the bin-accuracy references stop tracking true per-bin accuracy and whether CaliMatch's advantage over OpenMatch collapses.","tokens_in":23676,"feed_emoji":"🎯","tokens_out":14819,"duration_ms":142420,"temperature":0.7,"pith_summary":"CaliMatch argues that in safe semi-supervised learning — the setting where the unlabeled pool contains classes absent from the labeled set — the main saboteur is overconfidence, not the mismatch itself: deep networks produce wrong pseudo-labels and missed out-of-distribution samples while sounding certain, and every threshold-based selection rule trusts those numbers. The proposed fix is to calibrate both the multiclass classifier and the one-versus-rest OOD detector during training, using label smoothing whose strength adapts per sample to the model's measured accuracy in that confidence bin, together with learnable logit temperatures. On CIFAR-10, CIFAR-100, SVHN, TinyImageNet, and ImageNet, the method reports the best accuracy and calibration among eight compared methods at unseen-class ratios of $\\kappa = 30\\%$ and $\\kappa = 60\\%$. The paper also gives a formal argument that calibrated confidence lowers the probability of selecting bad samples and pulls the training gradient toward the fully supervised ideal. If the claim is right, calibration is not a post-hoc nicety but a load-bearing component of any threshold-based SSL method.","feed_headline":"Calibration makes semi-supervised learning safer under unseen classes","feed_subtitle":"CaliMatch adaptively smooths labels and rescales logits so pseudo-labels and OOD rejection trust what the model knows.","key_machinery":"The carrying mechanism is the pairing of adaptive label smoothing with learnable logit scaling on both heads. The unit interval of model confidence is split into $M = 30$ equal bins; for each bin the model's accuracy on a held-out validation set is computed, and that per-bin accuracy becomes the reference confidence ($\\gamma$ for the multiclass classifier, $\\delta$ for the OOD detector) that defines the smoothed label targets for any sample whose confidence lands in that bin. The scaled probabilities $p_k^s(x) = \\mathrm{softmax}_k(f_\\theta(x)/T_M)$ and $q_k^s(x) = \\sigma_k(g_\\theta(x)/T_O)$ are trained against those instance-specific targets with $T_M$ and $T_O$ initialized at 1.5 and learned by gradient descent, so the model is pushed to be exactly as confident as it is accurate. These calibrated probabilities form the seen-class score $s_i^u = \\sum_{k\\in\\mathcal{Y}} p_k^s(x_i^u)\\,q_k^s(x_i^u)$ and the OOD score $u_i^u = 1 - s_i^u$; an unlabeled sample is consumed by the FixMatch consistency loss only if $s_i^u > \\tau_1$ and $\\max_k p_k^s(x_i^u) > \\tau_2$.","core_discovery":"CaliMatch's central claim is that safe SSL fails less from the label-distribution mismatch itself than from overconfident confidence estimates: deep networks assign high probability to wrong pseudo-labels and to unseen-class samples, and the two heads a safe-SSL model depends on — the multiclass classifier $f_\\theta$ and the one-versus-rest OOD detector $g_\\theta$ — are both miscalibrated. The paper proposes to calibrate both heads during training with adaptive label smoothing, where the smoothing degree for each sample is set by the validation accuracy of the confidence bin the sample falls into, plus learnable temperature parameters $T_M$ and $T_O$ (initialized at 1.5) that rescale the logits and are optimized by gradient descent. A seen-class score $s_i^u = \\sum_{k \\in \\mathcal{Y}} p_k^s(x_i^u)\\, q_k^s(x_i^u)$ and a calibrated max-confidence $c_i^u$ gate which unlabeled samples enter the FixMatch consistency loss, and the OOD score is $u_i^u = 1 - s_i^u$. Lemma 1 bounds the probability of selecting an OOD or mislabeled sample by $1 - \\min\\{\\tau_1, \\tau_2\\} + \\eta$ where $\\eta$ is the calibration error, and Theorem 1 bounds the gradient gap between the surrogate FixMatch loss and the ideal loss on clean in-distribution data by a term proportional to $|B_t^u|\\varepsilon$, so better calibration provably steers optimization toward the supervised ideal. Across the five benchmarks the method reports top or near-top accuracy and the lowest expected calibration error, including 63.04% top-1 on ImageNet at $\\kappa = 60\\%$.","pith_inferences":["A testable extension the paper does not run: the same bin-accuracy smoothing losses could be attached to FixMatch itself — without any OOD head — and should measurably reduce its vulnerability to unseen classes in the unlabeled pool.","The fragile input is the held-out validation estimate: with very small labeled budgets the per-bin accuracy references become noisy, so one natural variant is to estimate bin accuracy from the training labels themselves or from a cross-fit of the validation split; the paper leaves this unexplored.","The learnable temperatures can be read as an online, train-time analogue of temperature scaling, which suggests tracking whether $T_M$ and $T_O$ converge to values that minimize validation ECE — a diagnostic the paper does not report.","Because the seen-class score multiplies classifier and OOD probabilities, each head caps the other: if either is miscalibrated the combined score inherits the worse head, consistent with the paper's finding that on TinyImageNet calibrating only the OOD detector gives the lowest OOD-detection ECE."],"forward_implications":["Selecting pseudo-labels on calibrated confidence $c_i^u$ instead of raw max-softmax reduces the number of wrong labels that enter consistency regularization, which the paper's learning curves identify as the main accuracy driver.","Calibrating the OOD detector shrinks the fraction of unseen-class samples among high-confidence selections, so fewer out-of-distribution instances are absorbed into training and the OOD detection ECE drops accordingly.","The ablations indicate the two calibrations play complementary roles: multiclass calibration produces most of the classification gain, OOD calibration most of the OOD-detection calibration gain, and the shared encoder benefits from both.","The gains persist when the unseen-class ratio rises from 30% to 60% and on ImageNet at 60% mismatch, where CaliMatch reaches 63.04% top-1 accuracy and beats OpenMatch by 1.24 percentage points, so the mechanism is not limited to small-scale datasets.","CaliMatch stays ahead of OpenMatch across a grid of the two thresholds $\\tau_1$ and $\\tau_2$, so the improvement is not an artifact of a lucky threshold choice."],"supporting_citations":[{"why":"The safe-SSL scaffold CaliMatch is built on: its one-versus-rest OOD detector, seen-class score, and soft open-set consistency loss are adopted and recalibrated.","marker":"[24]"},{"why":"Supplies the FixMatch pseudo-label thresholding and consistency-regularization loss that consumes CaliMatch's reliably selected unlabeled subset.","marker":"[26]"},{"why":"Establishes the overconfidence of deep networks and the standard temperature-scaling remedy, which form the premise and baseline technique CaliMatch extends.","marker":"[7]"},{"why":"Sourced the calibration behavior of label smoothing that CaliMatch makes adaptive per confidence bin.","marker":"[18]"},{"why":"SafeStudent is a comparison baseline that calibrates unseen-class probabilities toward a uniform distribution via energy discrepancy.","marker":"[9]"},{"why":"IOMatch is a comparison baseline using a joint inlier-outlier OvR classifier and projection head that CaliMatch is measured against.","marker":"[14]"},{"why":"MTC, the multi-task curriculum safe-SSL baseline pairing MixMatch with an OOD score, is a core comparison CaliMatch must beat.","marker":"[32]"},{"why":"Wide ResNet-28-2, the backbone used for the four small-benchmark experiments, so the reported gains are tied to this architecture.","marker":"[33]"}],"fun_headline_variants":["CaliMatch: Calibration fixes overconfident semi-supervised learning","Adaptive calibration makes SSL safer against unseen classes","Calibrate both heads to reduce SSL overconfidence","CaliMatch: Don't trust overconfident pseudo-labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"CaliMatch's per-bin accuracy references are computed on a held-out labeled validation set (10% of the labeled training split on the four small benchmarks), so the calibration targets and smoothing degrees are only as trustworthy as that validation set's representativeness of the labeled distribution.","fun_headline_variants_meta":{"raw":{"variants":["CaliMatch: Calibration fixes overconfident semi-supervised learning","Adaptive calibration makes SSL safer against unseen classes","Calibrate both heads to reduce SSL overconfidence","CaliMatch: Don't trust overconfident pseudo-labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000878,"raw_usage":{"total_tokens":3890,"prompt_tokens":1133,"completion_tokens":2757,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":749,"completion_tokens_details":{"reasoning_tokens":2688}},"tokens_in":749,"tokens_out":2757,"duration_ms":24922,"temperature":1.0,"reasoning_tokens":2688,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:39:49.147931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a benchmark where the true labels of all unlabeled in-distribution samples are known, record throughout training the per-batch gradient distance between the FixMatch surrogate loss and the ideal cross-entropy loss on the correctly pseudo-labeled ID subset; the paper's Theorem 1 predicts this distance shrinks as calibration error drops, so if accuracy gains appear while that distance stays flat, the claimed mechanism would be falsified. A second check: shrink the labeled validation holdout (for example from 10% to 2%) and see whether the bin-accuracy references stop tracking true per-bin accuracy and whether CaliMatch's advantage over OpenMatch collapses.","supporting_citations":[{"cited_title":"Open- match: Open-set semi-supervised learning with open-set consistency regularization","cited_arxiv_id":null,"evidence_quote":"The safe-SSL scaffold CaliMatch is built on: its one-versus-rest OOD detector, seen-class score, and soft open-set consistency loss are adopted and recalibrated."},{"cited_title":"Fixmatch: Simpli- fying semi-supervised learning with consistency and confi- dence","cited_arxiv_id":null,"evidence_quote":"Supplies the FixMatch pseudo-label thresholding and consistency-regularization loss that consumes CaliMatch's reliably selected unlabeled subset."},{"cited_title":"On calibration of modern neural networks","cited_arxiv_id":null,"evidence_quote":"Establishes the overconfidence of deep networks and the standard temperature-scaling remedy, which form the premise and baseline technique CaliMatch extends."},{"cited_title":"When does label smoothing help? Advances in Neural In- formation Processing Systems, 32, 2019","cited_arxiv_id":null,"evidence_quote":"Sourced the calibration behavior of label smoothing that CaliMatch makes adaptive per confidence bin."},{"cited_title":"Safe-student for safe deep semi-supervised learn- ing with unseen-class unlabeled data","cited_arxiv_id":null,"evidence_quote":"SafeStudent is a comparison baseline that calibrates unseen-class probabilities toward a uniform distribution via energy discrepancy."},{"cited_title":"Iomatch: Simplifying open-set semi-supervised learning with joint inliers and outliers utilization","cited_arxiv_id":null,"evidence_quote":"IOMatch is a comparison baseline using a joint inlier-outlier OvR classifier and projection head that CaliMatch is measured against."},{"cited_title":"Multi- task curriculum framework for open-set semi-supervised learning","cited_arxiv_id":null,"evidence_quote":"MTC, the multi-task curriculum safe-SSL baseline pairing MixMatch with an OOD score, is a core comparison CaliMatch must beat."},{"cited_title":"Wide residual net- works","cited_arxiv_id":null,"evidence_quote":"Wide ResNet-28-2, the backbone used for the four small-benchmark experiments, so the reported gains are tied to this architecture."}],"review_version":1}