{"id":"3a622f3e-173c-48a9-9f45-d598985523bc","arxiv_id":"2412.00101","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A broad empirical study of multi-label contrastive losses, plus a new regularized loss, LREG, that improves Macro-F1 on large-label datasets.","lead":"This paper compares contrastive and classical losses for multi-label classification on six vision and NLP datasets, and introduces a contrastive loss with a gradient regularization term. The authors find contrastive losses help most on datasets with many labels and on Macro-F1, while ranking metrics such as mAP remain better served by ZLPR.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eqs. (8)–(9) misstate the gradient direction: differentiating Eq. (6) gives ∂ℓ/∂z_k ∝ z_i, not z_k; the §4.1 repulsion narrative and Eq. (13) inherit the error, so the theoretical justification for LREG needs correction.","rationale":"The strongest claim is that LREG improves multi-label classification by canceling an unwanted gradient component from close positive pairs. For that claim to be true, the gradient analysis motivating the regularization must be correct. It is not: Eqs. (8)–(9) have the wrong vector direction. This is not a cosmetic issue, because §4.1 uses the incorrect direction to argue that a positive pair 'moves in the same direction as its representation z_k'; the correct derivative moves along the anchor z_i. The threshold condition survives, so the method may still perform as advertised; the implemented regularizer (Eq. 11) produces gradient -w z_i for z_k, which cancels the repulsive term (σ-Λ) z_i. Thus the empirical claims are not obviously falsified. However, the paper's central theoretical contribution—a novel loss 'based on gradient analysis'—is expressed through incorrect formulas, and the claim of shared minima is under-supported. A conditional verdict is appropriate: require corrected derivations and a proof or disproof of the shared-minimum claim before accepting the explanation, while the empirical tables remain checkable. The reader identified the same gradient-direction issue; I agree on the mathematical error but differ on its consequence, since the sign condition still justifies the regularizer.","tokens_in":20105,"tokens_out":12488,"duration_ms":111686,"concrete_test":"Implement Eq. (6) in PyTorch/JAX on random normalized vectors and use autograd to compute ∂ℓ/∂z_k for a positive and a negative k; compare with Eqs. (8)–(9). If the gradients align with z_i rather than z_k, the printed derivation is wrong. Then compute the gradient of the implemented ℓ_reg from Eq. (11) (including its detach) and verify that at a point where σ>Λ the total gradient for z_k vanishes; this distinguishes a harmless typo from a broken method and determines whether the Sec. 4.1 narrative can be repaired.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is the gradient derivation in §3.4 and its use in §4. For ℓ(zi) = -Σ_{j∈P} Λ_j log p_j with p_j = exp(z_j·z_i)/Σ_{k∈A} exp(z_k·z_i), differentiating with respect to a positive representation z_k gives ∂ℓ/∂z_k = (σ_{k,i} - Λ_k) z_i, not (-Λ_k + σ_{k,i}) z_k as printed in Eq. (8); for a negative k it is σ_{k,i} z_i, not σ_{k,i} z_k. Direct computation: ∂p_j/∂z_k = p_j(δ_{jk} z_i - p_k z_i), so ∂ℓ/∂z_k = -Λ_k z_i + p_k z_i. Thus the geometric statement in §4.1 that a close positive pair's gradient 'moves in the same direction as its representation z_k' is incorrect. The sign condition (-Λ_k + σ_{k,i}) > 0 is nevertheless the correct condition for the positive-pair gradient to point along +z_i, which under gradient descent repels z_k from z_i; so the regularizer in Eq. (11) can still cancel the unwanted repulsion when implemented with autodiff. But Eq. (13) repeats the same direction error, and the paper's explanation of why LREG works is not supported by the printed derivation. A second, related gap: the claim in §4.2 that ℓ and ℓ_reg share the same minimum is not established by Eq. (10) alone; ℓ_reg can create new stationary points in the σ>Λ region, so the assertion that gains arise only from optimization behavior needs proof.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a broad empirical study of contrastive losses for multi-label classification in computer vision and NLP, across full-data and low-data regimes. It identifies a supposedly undesired gradient component in multi-label supervised contrastive loss and proposes a new loss, LREG, that adds a regularizer (Eq. 11) intended to cancel that component. The paper claims that LREG improves over prior contrastive and non-contrastive losses, particularly in Macro-F1 on datasets with many labels, and reports results on PASCAL, MS-COCO, NUS-WIDE, AAPD, RCV1, and BGC, with code released publicly.","tokens_in":20565,"tokens_out":7510,"duration_ms":67593,"significance":"If the theoretical story were correct, the paper would provide a useful systematic comparison of multi-label contrastive losses and a simple, practical regularizer with a clear geometric interpretation, backed by open-source code and a relatively broad set of benchmarks. The empirical finding that a gradient regularizer can improve Macro-F1 in large-label multi-label settings is potentially valuable. However, the central theoretical derivation in Sec. 3.4 is mathematically incorrect, and the claim in Sec. 4.2 that the regularized and original losses share the same minimum is not established. Since the proposed method's motivation and the paper's main narrative rest on these points, the contribution is currently conditional on correcting and re-validating that theory.","major_comments":[{"comment":"The derivatives in Eqs. (8) and (9) are not the correct derivatives of Eq. (6). For the loss ℓ(z_i) = -Σ_{j∈P(i)} Λ_j log p_j with p_j = exp(z_j·z_i)/Σ_{k∈A(i)} exp(z_k·z_i), direct differentiation gives ∂ℓ/∂z_k = (σ_{k,i} - Λ_k) z_i for a positive k and ∂ℓ/∂z_k = σ_{k,i} z_i for a negative k. The printed expressions instead have these gradients proportional to z_k. Consequently, the geometric statement in Sec. 4.1 that a close positive pair's gradient \"moves in the same direction as its representation z_k\" is incorrect. The sign condition (σ - Λ) > 0 is still the right condition for the positive-pair gradient to point along +z_i, which under gradient descent repels z_k from z_i, so the regularizer in Eq. (11) can still cancel the repulsive component when implemented with autodiff; but the paper's printed derivation and explanation are not valid as they stand and must be corrected.","section":"Sec. 3.4, Eqs. (8) and (9)"},{"comment":"Eq. (13) repeats the same direction error made in Eqs. (8) and (9): the gradient of ℓ_reg(z_i) with respect to a positive representation z_k is -max(0, -Λ_k+σ_{k,i}) z_i, not the printed expression proportional to z_k. Since the regularizer is actually evaluated through automatic differentiation in Algorithm 2, the implementation may well compute the correct cancellation, but the manuscript's mathematical description is inconsistent with its own algorithm. This needs to be fixed in both the main text and the algorithm description.","section":"Sec. 4.2, Eq. (13)"},{"comment":"The statement that ℂ(z_i) and ℂ(z_i)+ℓ_reg(z_i) \"share the same minimum, as established in Eq. (10)\" is not supported. Eq. (10) characterizes stationary points of the original loss ℓ, not of the regularized objective. In the region where σ_{k,i} > Λ_k, the regularizer exactly cancels the positive-pair gradient for that k, so the combined objective can have additional stationary points, plateaus, or modified basins. At minimum, the paper should prove that no new minima are introduced, or it should soften the claim and provide empirical evidence that the observed gains arise specifically from optimization dynamics rather than from altered stationary points.","section":"Sec. 4.2, claim about shared minima"},{"comment":"The claims that LREG \"outperforms previous methods in almost all metrics\" and \"establishes state-of-the-art performance\" are stronger than Table 2 supports. On PASCAL, LASY achieves higher Micro-F1 and Macro-F1 than LREG; on RCV1 and BGC, LREG's Micro-F1 and Hamming loss are worse than those of LZLPR. The more defensible reading of the tables is that LREG frequently improves Macro-F1 on datasets with many labels, while its advantages are mixed on Micro-F1 and Hamming. The abstract and Section 6 should be reworded to state this more precise conclusion.","section":"Sec. 6.1.2 and Contributions"}],"minor_comments":[{"comment":"The temperature hyperparameter is reported inconsistently: Sec. 5.3 says τ is set to 0.1, while Appendix A.2 states that \"in all our previous experiments on both vision and language datasets, the temperature parameter was set to 0.01.\" Please clarify which value was actually used, since temperature strongly affects contrastive losses and the reported PRR values.","section":"Sec. 5.3 vs. Appendix A.2"},{"comment":"The paper defines ℂ(z_i) in Eq. (16) without the label-overlap reweighting f, then introduces a reweighted version in Eq. (18) with a hyperparameter α. The main results refer to \"LREG (Eq. 15)\" without specifying whether the final loss includes Eq. (18). Since Table 6 reports α values, presumably the experiments use Eq. (18), but this should be stated explicitly.","section":"Eq. (16) vs. Eq. (18)"},{"comment":"Several entries in Table 3 appear to be malformed: \"26.13.\" in the RCV1 row, \"9.865\" in the BGC Hamming column, and \"86.57 3.099 97.77\" for the LREG row on RCV1, which should probably read \"73.09\" for Macro-F1. Please correct these transcription errors.","section":"Table 3"},{"comment":"The heading \"Rankinp Metrics\" contains a typo; it should be \"Ranking Metrics.\" In addition, the mAP results are relegated to the appendix even though they directly support the paper's claim that contrastive losses are weaker on ranking-based metrics; consider mentioning the mAP conclusion in the main text.","section":"Appendix A.6"},{"comment":"The reference to \"PASCAL VOC 2007 dataset [39]\" is incorrect: [39] is the MS-COCO paper. PASCAL is [38]. Please fix the reference.","section":"Appendix A.5"},{"comment":"There are many missing spaces in mathematical expressions, e.g., \"LP roto\", \"LM ulSupCon\", and \"LREG(Eq. 15)\". A careful proofread would improve readability.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The gradient-direction error is real and load-bearing, but it appears fixable: the sign condition and the regularizer's cancellation effect can be reformulated correctly in terms of z_i, and the implementation in Algorithm 2 is consistent with that corrected view. The shared-minimum claim, however, needs either proof or substantial softening. The empirical tables are reasonably broad but would benefit from variance estimates or significance tests, especially since several claims of superiority are based on differences of less than one percentage point. I recommend major revision rather than rejection because the core empirical contribution seems salvageable if the theory is corrected and the claims are scaled back to what the tables actually show."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main take: the empirical study is the real contribution; the theoretical motivation has a couple of load-bearing errors. The proposed LREG is a plausible loss, and the comparison across six datasets, low-data regimes, and vision/NLP backbones is genuinely useful. But Eqs. (8)-(9) differentiate incorrectly: the derivative of the log-softmax w.r.t. a paired representation is (σ_{k,i} - Λ_k) z_i, not (σ_{k,i} - Λ_k) z_k, and the regularizer gradients in Eqs. (12)-(13) have the same direction error. The sign condition is unchanged, so the cancellation idea survives, but the geometric story in Sec. 4.1 is wrong as printed.\n\nSecond, the claim that ℓ and ℓhat share the same minimum is not established. When the regularizer is active, the gradient cancellation can occur over a region, not just at the isolated stationary points of Eq. (10), so the regularized loss can have a different stationary set. The paper says any gain comes from optimization behavior, but that conclusion does not follow.\n\nWhat's good: the gradient regularization term is new; the broad empirical comparison of contrastive losses for multi-label classification is useful and ships code. The ablation shows the regularization helps Macro-F1 in several datasets, especially low-data settings. The prototype analysis is informative.\n\nWhere it's soft: the math errors sit in the core theoretical sections, and the \"state-of-the-art\" claim is stronger than the tables show—on NLP, Micro-F1 is often slightly lower than baselines, and on vision the gains are modest. There are no error bars or significance tests, despite claims about significant differences. The α study is cursory.\n\nRecommendation: send to peer review, but expect major revision. The empirical part can stand after cleanup; the theoretical sections need a rewrite or an explicit statement that the gradient-based justification is heuristic. A referee should check the code to confirm the implemented loss matches the intended cancellation, since the printed gradient formulas are unreliable.","headline":"Useful empirical study with a plausible new loss, but the gradient derivation in Eqs. (8)-(9) is wrong and the shared-minimum claim is unsupported; worth reviewing but needs major revision.","tokens_in":21010,"tokens_out":4663,"would_cite":true,"duration_ms":42138,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a regularized multi-label contrastive loss, LREG, that cancels a repulsive gradient component from close positive pairs, and reports consistent Macro-F1 gains over existing losses across vision and text benchmarks.","keywords":["multi-label classification","contrastive learning","supervised contrastive loss","gradient regularization","Macro-F1","representation learning","low-data regime","label interactions"],"falsifier":"Take Eq. (6) for a batch of one anchor, one positive, and one negative, and compute $\\partial \\ell / \\partial z_k$ numerically with automatic differentiation. Compare the resulting vector to $z_k$ and to $z_i$. If it is proportional to $z_i$ in every positive case, the sign condition that activates $\\ell_{\\mathrm{reg}}$ (Eq. 11) would not follow from Eqs. (8)-(9) as written.","tokens_in":1778,"feed_emoji":"","tokens_out":3859,"duration_ms":92855,"temperature":0.7,"pith_summary":"This paper tries to establish that contrastive losses are a strong alternative to standard losses for multi-label classification when the label set is large, and that a specific gradient pathology limits them. It reports that existing supervised contrastive losses underperform on few-label datasets and ranking metrics, but excel on Macro-F1 when labels are numerous and interactions are dense. To fix the pathology, the authors derive the gradient of the multi-label contrastive loss and propose LREG, a regularized contrastive loss that cancels the contribution of positive pairs whose gradient direction mimics that of negative pairs. Empirically, LREG improves over prior contrastive losses on COCO, NUS-WIDE, AAPD, RCV1 and BGC, with the largest margins on Macro-F1 and in low-data regimes. The paper also argues that contrastive learning succeeds partly because of its optimization scheme, not only because it models label interactions.","feed_headline":"Regularized contrastive loss wins on multi-label Macro-F1","feed_subtitle":"Adding a gradient gate to the supervised contrastive loss boosts large-label datasets and low-data regimes.","key_machinery":"The central object is the regularized multi-label supervised contrastive loss $L_{\\mathrm{REG}} = \\frac{1}{|B|} \\sum_{z_i \\in B} \\hat{\\ell}(z_i)$, where each $\\hat{\\ell}$ is the weighted SupCon-style log-softmax over positive pairs plus the regularization $\\ell_{\\mathrm{reg}}(z_i) = -\\sum_{j \\in P(i)} \\max(0, -\\Lambda^i_j + \\sigma_{j,i}.\\mathrm{detach()}) \\, z_j \\cdot z_i$. Here $\\Lambda$ is the normalized positive-pair weight and $\\sigma$ is the softmax score; the max gate activates only when a positive pair's gradient would otherwise point like a negative pair's. The mechanism doing the work is gradient surgery: it removes one component of the positive-pair gradient without changing the location of the loss minima, so any gains come from the optimization trajectory rather than from a different objective.","core_discovery":"The central claim is that in multi-label contrastive learning, once a positive pair becomes very close, the softmax score term $\\sigma_{k,i}$ can exceed the normalized positive weight $\\Lambda^i_k$, and then the gradient contribution of that positive pair points along the paired representation in the same direction as a negative pair's contribution, pushing the pair apart. On this reading, attraction and repulsion are asymmetric and the loss can contain an unwanted repulsive component. The paper's LREG loss adds a $\\max(0, -\\Lambda + \\sigma) \\cdot z_j \\cdot z_i$ regularization with a detached $\\sigma$, so that the offending component is removed while the global optimum of the contrastive loss is unchanged. The authors report that LREG matches or beats BCE, asymmetric loss, ZLPR and previous contrastive losses across six datasets, and that it improves representation alignment and Macro-F1, especially with 20% of the training data.","pith_inferences":["The paper's gradient mechanism is stated for derivatives with respect to the paired representation $z_k$; under the usual chain rule for the same log-softmax with a fixed anchor $z_i$, the derivative is proportional to $z_i$. The empirical gains would still stand, but the repulsion narrative would need re-expression under that convention.","The strong results of prototype-only contrastive loss suggest that part of the gap between contrastive and BCE-based methods may come from an optimizer-friendly landscape rather than from label correlations; ablating label interactions while keeping the gradient gate would isolate that factor.","The reported mAP gap between ZLPR and all contrastive losses points to an untested complement: combining LREG with a ranking-oriented loss could yield a single objective that is strong on both clustering and ranking metrics."],"forward_implications":["If LREG's characterization is right, contrastive losses can be used as drop-in representation learners for multi-label text and image data, particularly when the label set is large.","Macro-F1 gains on large-label datasets should transfer to tasks where rare labels dominate the evaluation, since the regularization stabilizes close positive pairs.","Because LREG and its ablation share the same minima, the reported improvement is attributable to the optimization trajectory, supporting methods that focus on gradient dynamics rather than only objective design.","The regularization can be attached to the standard SupCon loss without hurting multi-class accuracy, so it appears safe to apply in the multi-label setting while leaving single-label performance unchanged."],"supporting_citations":[{"why":"Defines the supervised contrastive loss that LREG extends with multi-label weighting and the gradient gate.","marker":"[24]"},{"why":"Introduces MulSupCon, whose per-label instance weighting and label-frequency reweighting LREG adopts.","marker":"[19]"},{"why":"Introduces the MSC loss and the attraction-repulsion imbalance idea that motivates reweighting and prototypes.","marker":"[18]"},{"why":"Shows that a prototype-only contrastive loss can be strong, the observation behind the paper's robustness-versus-interactions analysis.","marker":"[20]"},{"why":"Proposes the Jaccard-weighted base contrastive loss LBase that the paper studies and improves.","marker":"[10]"},{"why":"Provides the asymmetric loss baseline that dominates on small-label datasets in the comparison.","marker":"[16]"},{"why":"Provides the ZLPR ranking-based baseline that LREG must match or beat, especially on mAP.","marker":"[17]"},{"why":"Supplies the alignment and uniformity metrics used to test the paper's claim that LREG improves representation geometry.","marker":"[47]"}],"fun_headline_variants":["LREG loss removes contrastive repulsion, boosts Macro-F1","Contrastive loss fix lifts multi-label Macro-F1","Multi-label contrastive study reveals repulsive flaw","New regularization stops contrastive loss pushing wrongly","LREG wins on large-label sets without repulsion"],"cache_read_input_tokens":23040,"weakest_assumption_plain":"The load-bearing assumption is that, in the log-softmax contrastive loss, the gradient of a positive pair is directed along the paired representation $z_k$; if standard differentiation with the anchor held fixed instead gives a direction along the anchor $z_i$, the claimed repulsion mechanism needs to be re-derived.","fun_headline_variants_meta":{"raw":{"variants":["LREG loss removes contrastive repulsion, boosts Macro-F1","Contrastive loss fix lifts multi-label Macro-F1","Multi-label contrastive study reveals repulsive flaw","New regularization stops contrastive loss pushing wrongly","LREG wins on large-label sets without repulsion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000537,"raw_usage":{"total_tokens":2594,"prompt_tokens":975,"completion_tokens":1619,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1542}},"tokens_in":591,"tokens_out":1619,"duration_ms":12493,"temperature":1.0,"reasoning_tokens":1542,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:56:37.334734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take Eq. (6) for a batch of one anchor, one positive, and one negative, and compute $\\partial \\ell / \\partial z_k$ numerically with automatic differentiation. Compare the resulting vector to $z_k$ and to $z_i$. If it is proportional to $z_i$ in every positive case, the sign condition that activates $\\ell_{\\mathrm{reg}}$ (Eq. 11) would not follow from Eqs. (8)-(9) as written.","supporting_citations":[{"cited_title":"Supervised contrastive learning, 2021","cited_arxiv_id":null,"evidence_quote":"Defines the supervised contrastive loss that LREG extends with multi-label weighting and the gradient gate."},{"cited_title":"Multi-label supervised contrastive learning","cited_arxiv_id":null,"evidence_quote":"Introduces MulSupCon, whose per-label instance weighting and label-frequency reweighting LREG adopts."},{"cited_title":"Exploring contrastive learning for long-tailed multi-label text classification","cited_arxiv_id":null,"evidence_quote":"Introduces the MSC loss and the attraction-repulsion imbalance idea that motivates reweighting and prototypes."},{"cited_title":"Class prototypes based contrastive learning for classifying multi-label and fine-grained educational videos","cited_arxiv_id":null,"evidence_quote":"Shows that a prototype-only contrastive loss can be strong, the observation behind the paper's robustness-versus-interactions analysis."},{"cited_title":"An effective deploy- ment of contrastive learning in multi-label text classification","cited_arxiv_id":null,"evidence_quote":"Proposes the Jaccard-weighted base contrastive loss LBase that the paper studies and improves."},{"cited_title":"Asymmetric loss for multi-label classification","cited_arxiv_id":null,"evidence_quote":"Provides the asymmetric loss baseline that dominates on small-label datasets in the comparison."},{"cited_title":"ZLPR: A Novel Loss for Multi-label Classification","cited_arxiv_id":"2208.02955","evidence_quote":"Provides the ZLPR ranking-based baseline that LREG must match or beat, especially on mAP."}],"review_version":1}