{"id":"6f3cbd9b-4d7f-4cd1-a2b6-9918797230a0","arxiv_id":"2502.02202","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Multiple projection heads, each trained with SupCon on a different label or hierarchy level, improve accuracy in hierarchical and multi-label classification, especially with few training samples.","lead":"This paper adds multiple projection heads to supervised contrastive learning, one for each label or hierarchy level, so the model learns separate notions of similarity. On CIFAR-100, DeepFashion, TripAdvisor, and BeerAdvocate the method beats single-head baselines, with the largest gains on small training sets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set-based hyperparameter selection and unreported text hyperparameters leave the SOTA claim unverified; a validation-based re-run with error bars is needed before acceptance.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the reported gains depend on hyperparameters chosen without a validation split, and key text hyperparameters are missing. I agree with that assessment. I also considered the mathematical derivation in Section 3.5, where the tau->0 limit is presented as a finite expression but is really an asymptotic statement that drops the log of the number of maximizers; however, that issue is secondary because the empirical claim is the central one. The decisive issue is that the reported improvements may be inflated by test-set selection and uncontrolled variance. The proposed concrete test would settle the concern by re-running the experiments under a validation-based protocol with confidence intervals; if the gains persist, the central claim would be substantially strengthened. Until then, the appropriate verdict remains conditional, matching the reader's assessment.","tokens_in":13769,"tokens_out":8252,"duration_ms":83364,"concrete_test":"Hold out a validation split from each CIFAR-100 training subset (e.g., 10K of the 50K set, 1K of the 5K set, 2K of the 10K set), tune tau1, tau2, the alpha weights, and all text hyperparameters (Jaccard threshold t, head temperatures) on validation only, then report mean and standard deviation over at least 5 seeds on the original test set. If MLCL no longer beats SupCon/CE by the reported margins (at least 9% at 5K/10K and at least 1% on full data), the SOTA claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical case for 'outperforms state-of-the-art contrastive learning' rests on numbers whose hyperparameters are selected on the test set and, for text, partly unreported. Section 4.2 fixes the superclass temperature at tau2=0.5 because Figure 4 shows the best test accuracy at that value on a training subset; no validation split is mentioned. For the text experiments, the Jaccard threshold t defining positive pairs for the global head, the per-head temperatures, and the per-head alpha values are not all reported, so the TripAdvisor and BeerAdvocate results cannot be reconstructed or checked for test-set leakage. Tables 2 and 3 report no error bars, standard deviations, or number of seeds, so the 1% full-data gains and even the 9-10% small-data gains could reflect selection bias plus run-to-run variance. Because the headline claim depends on the magnitude and reliability of these margins, the central claim is not yet established at the level asserted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Multi-level Supervised Contrastive Learning (MLCL), a supervised contrastive learning method that attaches multiple projection heads to an encoder, each trained with a temperature-scaled SupCon loss using a different similarity criterion: one per hierarchy level for hierarchical classification (CIFAR-100, DeepFashion) and one per label plus a global Jaccard-based head for multi-label classification (TripAdvisor, BeerAdvocate). The final loss is a weighted sum of per-head losses, optionally combined with cross-entropy. The authors report accuracy gains over SupCon and cross-entropy baselines, especially in low-data regimes, and provide temperature-limit and gradient analyses in the main text and appendix.","tokens_in":13979,"tokens_out":6820,"duration_ms":64008,"significance":"If the empirical claims hold, MLCL is a simple and general extension of SupCon that could be useful for hierarchical and multi-label tasks, with no change to the encoder and only a modest increase in parameters from the extra projection heads. The paper also provides a derived gradient formula and temperature-limit analyses, which are useful pedagogical contributions even if the τ→0 derivation has a technical flaw. However, the empirical case is currently under-supported because of test-set-based hyperparameter selection, missing error bars, absent multi-label contrastive baselines, and unreported text hyperparameters; these issues are load-bearing for the stated 'outperforms state-of-the-art' claim.","major_comments":[{"comment":"The superclass temperature τ2=0.5 is selected based on accuracy that appears to be test-set accuracy. Section 4.4 says the figure uses 'a subset of training samples,' but the y-axis is reported as accuracy and all other accuracy numbers in the paper are test accuracy. If the sweep is evaluated on the test set, the choice of τ2 is made by peeking at the test set, which biases the subsequently reported results. Please either perform the sweep on a validation split or carefully describe the split used; if the sweep is on a training subset, clarify and confirm the same value was chosen without test-set information.","section":"Section 4.4, Figure 4"},{"comment":"No error bars, standard deviations, or number of seeds are reported for the CIFAR-100 and DeepFashion experiments. The full-data gains over SupCon are about 1 percentage point, and even the large small-data gains (9-10 points at 5K/10K) could be affected by run-to-run variance and by the test-set-based hyperparameter choice. Please report mean ± standard deviation over at least five seeds for all image experiments, including the baseline numbers, so the reader can assess whether the improvements are significant.","section":"Tables 2 and 3"},{"comment":"The multi-label text experiments compare only against cross-entropy fine-tuning and an ablation without the global head. However, Section 2 cites multiple multi-label contrastive baselines (Zaigrajew and Zieba 2022; Zhang and Wu 2024; Sajedi et al. 2023; Małkiński and Mandziuk 2022), and the abstract and introduction claim outperformance over state-of-the-art contrastive methods. Without including at least one of these baselines under the same BERT encoder and training protocol, the multi-label claim is unsupported. Please add the most directly comparable multi-label contrastive baseline(s).","section":"Table 4 and Section 2"},{"comment":"The hyperparameters for the text experiments are incompletely reported. The Jaccard threshold t for the global head, the temperature for each per-label head and the global head, and the exact per-head α values are not all given; for TripAdvisor the stated α_i values (7×0.03 + 0.1 = 0.31) imply a cross-entropy weight of 0.69, not the claimed 0.7. Without these values, the TripAdvisor and BeerAdvocate results cannot be reproduced or checked for tuning on the test set. Please report all hyperparameters and any validation procedure used.","section":"Section 4.2"},{"comment":"The derivation of the τ→0+ limit in Equation (1) is mathematically incorrect. The step that drops the term log(1 + Σ_{a∈A(i)\\p} exp((z_i·z_a - z_i·z_max)/τ)) is not valid: for any p that is not the maximizer, this term tends to log(1 + m) where m is the multiplicity of the maximum, not to zero. The qualitative conclusion that the loss becomes dominated by the hardest negative is consistent with the prior literature (Wang and Liu 2021), but the written derivation does not establish it. Please correct the derivation or cite a valid one.","section":"Section 3.5"}],"minor_comments":[{"comment":"The method is referred to as 'MSCL' in the appendix, while the main text uses 'MLCL'. Please unify the name throughout.","section":"Appendix, Sections 8.1-8.2 and Tables 5-6"},{"comment":"Equation (2) states that Σ_h α_h = 1, but Equation (3) relaxes this by adding a cross-entropy term with weight (1 - Σ_h α_h). The constraint should be stated consistently, e.g., by writing Σ_h α_h ≤ 1 for the combined loss.","section":"Equation (2) and (3)"},{"comment":"The gradient analysis in the appendix assumes |P(i)| = |P(t)| for t ∈ P(i), which is true for class-based positive sets but not necessarily for the Jaccard-threshold global head, where the positive set is not guaranteed to be symmetric. The appendix should state this assumption and its limitation.","section":"Section 3.4, global head gradient"},{"comment":"The caption says 'Accuracy of MLCL as a function of the superclass projection head temperature' but does not specify whether this accuracy is on the training, validation, or test split. Please make this explicit.","section":"Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is simple and potentially useful, but the empirical validation falls short of the stated claims. The test-set-based hyperparameter selection and the absence of error bars are the most serious issues; even the large small-data gains need variance estimates to be credible. The text experiments also lack the multi-label contrastive baselines that are cited in related work. These problems are fixable with a re-run using a validation split, seed reporting, and additional baselines, so I recommend major revision rather than rejection. The mathematical error in the τ→0+ derivation is also a concern but is secondary to the empirical gaps."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: this is a simple, sensible extension of SupCon — one projection head per label or hierarchy level, plus a Jaccard-weighted global head, with a weighted sum of losses. The appendix math (temperature limits and full gradient) is correct as far as I can tell, and the low-data numbers are eye-catching: +9-10 points on CIFAR-100 with 5K-10K samples. The full-data gains are modest (1-2 points), which is honest.\n\nWhat is actually new: the specific combination of per-label/per-level heads with a Jaccard-weighted global head in a supervised setting. Multiple heads have been done before (Wang et al. 2023), and multi-label contrastive losses exist, but the unified hierarchical + multi-label framing with temperature analysis is a legitimate small contribution.\n\nSoft spots, in order of severity. First, the empirical case is underreported. Tables 2 and 3 have no error bars or seed counts; Table 4 says \"average over ten seeds\" but no standard deviations. The superclass temperature tau2=0.5 is justified by Figure 4, which appears to be a test-set accuracy sweep — no validation split is mentioned — and then the temperature analysis is used to explain why 0.5 works. That's circular leakage, or at best an underspecified tuning protocol. For the text experiments, the Jaccard threshold t, per-head temperatures, and per-head alphas are not all reported, so the results cannot be reproduced or checked for leakage. Second, the paper claims to outperform state-of-the-art contrastive methods, but the multi-label tables compare only against CE fine-tuning; the cited multi-label contrastive baselines (Zaigrajew and Zieba, Zhang and Wu, Sajedi et al.) are absent from the experiments. That overstates the claim. Third, novelty is incremental — multiple heads already exist in Wang et al. — but this is a minor caveat, not a fatal one.\n\nThe math holds up, the idea is coherent, and the low-data gains are large enough to be practically interesting. The paper deserves a serious referee, but the referee should require proper error bars, a validation-based tuning protocol, released hyperparameters/code, and the missing baselines before the SOTA claim can be accepted.","headline":"Simple multi-head SupCon extension with solid math and eye-catching low-data gains, but the empirical reporting (no error bars, test-set tuning, missing baselines) keeps the SOTA claim from being verified.","tokens_in":14541,"tokens_out":3908,"would_cite":true,"duration_ms":33109,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-level contrastive learning, with one projection head per label level, outperforms single-head contrastive learning and delivers its largest gains when training data are scarce.","keywords":["supervised contrastive learning","multi-level representation learning","projection heads","hierarchical classification","multi-label classification","temperature scaling","low-data learning","representation learning"],"falsifier":"Train MLCL on the 5K and 10K CIFAR-100 subsets with the superclass temperature chosen by a validation split, for example a grid search over $\\tau_2 \\in \\{0.1, 0.3, 0.5, 0.7, 1.0\\}$, and compare against SupCon under identical epochs and batch size; if the gap shrinks to near zero or reverses, the low-data advantage is driven by test-set selection rather than the multi-level objective.","tokens_in":13579,"feed_emoji":"🧠","tokens_out":7333,"duration_ms":66125,"temperature":0.7,"pith_summary":"The paper proposes MLCL, a supervised contrastive learning framework that replaces the single projection head of standard contrastive learning with several heads, one per level of label structure (subclass and superclass, or each aspect label plus a global head). Each head runs the standard supervised contrastive loss with its own temperature, and the final loss is a weighted sum. The central claim is that this lets one encoder represent multiple notions of similarity at once, and that this improves accuracy over single-head contrastive learning, especially when training data are scarce. The authors report gains of roughly 1 percentage point on full CIFAR-100 and DeepFashion, 9 to 10 points when CIFAR-100 is cut to 5K or 10K training samples, and consistent gains on two aspect-based sentiment datasets. If true, this means contrastive learning can exploit label hierarchies and multi-label structure without task-specific architectures, with the largest benefits in data-limited settings.","feed_headline":"A head per label level beats single-head contrastive learning on scarce data","feed_subtitle":"Adding projection heads for each label level gives up to 10-point accuracy gains on small CIFAR-100 subsets.","key_machinery":"The load-bearing object is the set of $H$ projection heads $\\{g_h\\}$, each producing a projection $z^h_i = g_h(f(\\tilde{x}_i))$ and contributing a supervised contrastive loss $\\mathcal{L}_h$ with its own temperature $\\tau_h$; the total loss is $\\sum_h \\alpha_h \\mathcal{L}_h$ (plus cross-entropy for text). Positive pairs are defined separately per head by the relevant similarity criterion, so one head sees same-subclass pairs, another sees same-superclass pairs, and a global head sees pairs with high Jaccard label overlap. The temperature analysis is what carries the argument: as $\\tau \\to 0^+$ the loss concentrates on the hardest negative and the hardest positive, while as $\\tau \\to +\\infty$ all negatives contribute uniformly, so the paper sets low temperature for fine subclass separation and high temperature for coarse superclass grouping. This lets one encoder integrate multiple similarity notions through a single weighted objective.","core_discovery":"The central claim is that multiple projection heads, each specialized to one level of a label hierarchy or one aspect label, produce better encoders than a single supervised contrastive head. In the hierarchical setting, a low-temperature head on subclass labels separates fine classes while a high-temperature head on superclass labels keeps coarse groupings together; in the multi-label text setting, one head per aspect plus a global head that weights pairs by Jaccard similarity acts as a regularizer. After training, the heads are discarded and a linear classifier is trained on the frozen encoder, so the gains come from the representation itself. The paper presents this as a generalization of supervised contrastive learning that subsumes the single-head case and is applicable to both hierarchical and multi-label classification.","pith_inferences":["A test the paper leaves implicit: whether the superclass temperature of 0.5 remains optimal when chosen on a held-out validation split rather than the test set; if not, part of the low-data gain may be selection noise.","The multi-head objective can be read as a form of loss ensembling, where each head supplies a different hardness profile over negatives and the weighted sum reshapes the gradient contributions of easy versus hard pairs; this could be tested by ablating the head weights $\\alpha_h$.","The global Jaccard-weighted head suggests a general recipe for ordinal or multi-label data: define a soft positive weight from label overlap and use it to modulate the contrastive loss, which could transfer to other multi-label domains such as medical coding or scene understanding.","The full-data improvement is modest, so the practically decisive claim is the low-data improvement; re-running the low-data experiments with multiple seeds and reported variance would clarify how much of the gap is stable."],"forward_implications":["If MLCL is right, supervised contrastive learning can exploit class hierarchies and multi-label structure with no change to the encoder, only additional projection heads and per-head temperatures.","The reported 9 to 10 point gains on 5K and 10K CIFAR-100 samples suggest the method's main value is in low-data regimes, where a single-head loss overfits or fails to structure the space.","Because the heads are removed at inference, MLCL adds no inference cost and yields a single representation usable by any downstream linear classifier.","The global projection head's stronger tolerance to label noise, up to 7 points in the appendix, implies the method can act as a regularizer in noisy-label settings.","The framework is stated to generalize standard contrastive learning through multiple projection heads, so the same recipe applies whenever label structure has more than one level."],"supporting_citations":[{"why":"Supplies the supervised contrastive loss that each projection head uses and serves as the main single-head baseline.","marker":"Khosla et al., 2020"},{"why":"Supplies the SimCLR baseline and the temperature-based discussion of hard negative behavior.","marker":"Chen et al., 2020a"},{"why":"Provides the analysis that higher temperatures spread penalty uniformly across negatives and lower temperatures concentrate on hard negatives.","marker":"Wang and Liu, 2021"},{"why":"Supplies the Jaccard similarity weighting used by the global projection head.","marker":"Zaigrajew and Zieba, 2022"},{"why":"Supplies the hierarchical multi-label contrastive baseline HiMulConE.","marker":"Zhang et al., 2022"},{"why":"Supplies the class-distance prototype baseline Guided.","marker":"Landrieu and Garnot, 2021"},{"why":"Supplies the pretrained transformer encoder used for the text classification experiments.","marker":"Devlin et al., 2018"},{"why":"Supplies the practice of combining supervised contrastive loss with cross-entropy when fine-tuning language models.","marker":"Gunel et al., 2020"}],"fun_headline_variants":["Per-label-level heads boost contrastive learning on small data","Multiple contrastive heads capture finer similarities for better encoders","A head for each hierarchy level: contrastive learning gains up to 10 points","Multi-head supervised contrastive learning trumps single-head on scarce data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rest on the assumption that the manually chosen temperature for the superclass head, and the similarity threshold for the global head in the text experiments, would still be good choices on a separate validation set rather than having been tuned to the test data.","fun_headline_variants_meta":{"raw":{"variants":["Per-label-level heads boost contrastive learning on small data","Multiple contrastive heads capture finer similarities for better encoders","A head for each hierarchy level: contrastive learning gains up to 10 points","Multi-head supervised contrastive learning trumps single-head on scarce data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1488,"prompt_tokens":840,"completion_tokens":648,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":574}},"tokens_in":456,"tokens_out":648,"duration_ms":6757,"temperature":1.0,"reasoning_tokens":574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:58:12.988107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MLCL on the 5K and 10K CIFAR-100 subsets with the superclass temperature chosen by a validation split, for example a grid search over $\\tau_2 \\in \\{0.1, 0.3, 0.5, 0.7, 1.0\\}$, and compare against SupCon under identical epochs and batch size; if the gap shrinks to near zero or reverses, the low-data advantage is driven by test-set selection rather than the multi-level objective.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the supervised contrastive loss that each projection head uses and serves as the main single-head baseline."},{"cited_title":"and Liu, H","cited_arxiv_id":null,"evidence_quote":"Provides the analysis that higher temperatures spread penalty uniformly across negatives and lower temperatures concentrate on hard negatives."},{"cited_title":"and Zieba, M","cited_arxiv_id":null,"evidence_quote":"Supplies the Jaccard similarity weighting used by the global projection head."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the hierarchical multi-label contrastive baseline HiMulConE."},{"cited_title":"and Garnot, V","cited_arxiv_id":null,"evidence_quote":"Supplies the class-distance prototype baseline Guided."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the practice of combining supervised contrastive loss with cross-entropy when fine-tuning language models."}],"review_version":1}