{"id":"8ce5d811-a7ee-460a-84fd-30506bdce1e2","arxiv_id":"2412.18231","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A new loss and memory update improve Macro-AUC in imbalanced multi-label continual learning, backed by theoretical bounds of questionable validity.","lead":"This paper presents a replay-based method for multi-label continual learning, combining a reweighted margin loss (RLDAM) and a memory-updating rule (WRU) to improve Macro-AUC under label imbalance. It also claims the first theoretical generalization analysis for this setting, with experiments on three benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 2(3) sets the Lipschitz constant of a shifted hinge as 1/Δ; the true constant is 1, so the claimed margin-aware bound and optimal margins do not follow.","rationale":"The reader's weakest assumption is exactly the load-bearing error. Assumption 2(3) is mathematically false: shifting a 1-Lipschitz function by a constant does not change its Lipschitz constant, and the derivative argument is immediate. Since ρ_k appears as ρ_k^++ρ_k^- in Theorem 2 and Lemma 2, replacing ρ by 1 removes all margin dependence from the bound. Proposition 1's minimization then has no Δ-dependent objective, so the claimed optimal margins Δ_k=λ/|D_k^+|^{1/4} and the asserted superiority over the RU loss are unsupported. This invalidates the paper's central theoretical contribution, including the claim of being the first MLCL generalization analysis with a margin-aware advantage. A separate concern reinforces this: in Theorem 1 the bound is expressed through empirical risks on the full datasets D_i for past tasks, while the algorithm only optimizes with replay buffer subsets M_i. The remark that WRU keeps n_i τ_i 'stable and unbiased' does not account for the difference between \\hat R_{D_i} and \\hat R_{M_i}, so even if the Lipschitz issue were repaired, the continual-learning theorem as written would not establish the benefit of WRU. The empirical results may still be of interest, but the stated contributions—the first MLCL theoretical analysis and the superiority of RLDAM+WRU—are not supported. I therefore keep the reader's REJECT verdict unchanged.","tokens_in":25020,"tokens_out":7585,"duration_ms":68386,"concrete_test":"Analytically recompute the Lipschitz constant of ℓ(z−Δ)=max(0,1−z+Δ): for any z1≠z2, |ℓ(z1−Δ)−ℓ(z2−Δ)| ≤ |z1−z2|, with equality on the active interval, so the constant is 1, not 1/Δ. Then substitute ρ_k^+=ρ_k^-=1 into Theorem 2 and Proposition 1. If the complexity term no longer contains Δ and the optimization over margins under Δ_k^++Δ_k^-=β has no Δ-dependent objective, the margin-aware superiority claim fails. If the authors intended a scaled hinge ℓΔ(z)=max(0,1−z/Δ), then Eq. (3) and all experiments must be re-run with that loss, since the reported results use the standard shifted hinge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Assumption 2(3) (Section 4) asserts that ℓ(z−Δ_k^+) and ℓ(z−Δ_k^-) are (1/Δ_k^+)- and (1/Δ_k^-)-Lipschitz for the hinge loss. This is false. For ℓ(z)=max(0,1−z), the shifted function z↦ℓ(z−Δ)=max(0,1+Δ−z) has derivative in {0,1} everywhere it is differentiable, so its Lipschitz constant is exactly 1 for every Δ>0, not 1/Δ. This incorrect ρ=1/Δ relation is load-bearing: it enters the complexity term of Theorem 2 via Lemma 2's contraction inequality, and Proposition 1's optimal-margin derivation minimizes an objective that contains 1/Δ_k. With the correct ρ_k^+=ρ_k^-=1, the bound becomes independent of the margins, so RLDAM's guarantee is no tighter than the RU bound of Wu et al. (2023), and the formula Δ_k=λ/|D_k^+|^{1/4} is not derived. The claimed theoretical superiority of RLDAM over RU, and hence the basis for the RLDAM+WRU contribution, is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses Macro-AUC optimization in multi-label continual learning (MLCL). It proposes RLDAM, a reweighted label-distribution-aware margin loss that combines the reweighted univariate (RU) loss of Wu, Li, and Yin (2023) with the LDAM margin idea of Cao et al. (2019), and WRU, a memory-updating strategy intended to preserve class-wise positive/negative ratios in the replay buffer. The authors claim the first theoretical generalization analyses for MLCL, with batch and continual bounds showing superiority over RU, and they report experiments on C-PASCAL-VOC, C-MSCOCO, and C-NUS-WIDE.","tokens_in":25334,"tokens_out":8341,"duration_ms":73162,"significance":"If the theoretical claims were correct, the paper would make a useful contribution: it targets an important metric (Macro-AUC) in an underexplored setting, proposes a concrete and simple method, releases code, and includes ablation and memory-size studies. The empirical results are suggestive, with large gains over several baselines in Table 1. However, the central theoretical premise is invalid as stated: Assumption 2(3) misstates the Lipschitz constant of a shifted hinge loss, and this error propagates into Theorem 2, Proposition 1, and the MLCL bound. The claimed margin-dependent generalization guarantee and the derived optimal margins therefore do not follow. The MLCL bound also does not specifically depend on WRU. These are load-bearing issues for the paper's main contribution as framed.","major_comments":[{"comment":"The statement that the shifted hinge losses ℓ(z − Δ_k^+) and ℓ(z − Δ_k^-) are (1/Δ_k^+)- and (1/Δ_k^-)-Lipschitz is false. For ℓ(z) = max(0, 1 − z), the function z ↦ ℓ(z − Δ) has derivative 0 or −1 wherever differentiable, so its Lipschitz constant is exactly 1 for every Δ > 0. The footnote to Assumption 2(3) even acknowledges that the hinge loss is 1-Lipschitz. This error is load-bearing: the margin-dependent ρ terms enter Theorem 2 through Lemma 2's contraction inequality, and with ρ_k^+ = ρ_k^- = 1 the complexity term no longer depends on the margins. Consequently, the claimed superiority of the RLDAM bound over the RU bound of Wu et al. (2023) is not established.","section":"Section 4, Assumption 2(3)"},{"comment":"Even within the paper's own algebra, Proposition 1 does not derive the claimed optimal margins. The minimization is over 2Λr(1/Δ_1 √(1/|D_1^+|) + 1/Δ_2 √(1/|D_2^+|)) subject to Δ_1 + Δ_2 = β, and the reported minimizer is Δ_1^* = β|D_2^+|^{1/4}/(|D_1^+|^{1/4}+|D_2^+|^{1/4}). The proposition then asserts that the optimal margins are Δ_k^* = λ/|D_k^+|^{1/4}, which is not equivalent to the derived expression for any constant λ. The further extension to more than two labels and to the equal-margin conclusion Δ_+ = Δ_- for single-label tasks is asserted without proof. Since this margin schedule is used in the experiments, the schedule lacks theoretical justification.","section":"Section 4, Proposition 1"},{"comment":"The MLCL bound is stated with empirical risks hat R_{D_i}(f^{t,i}) on the full previous datasets D_i, but in replay-based continual learning only memory subsets M_i are available at task t. The proof of Lemma 4 introduces ñ_i points per previous task without defining the corresponding empirical risk on memory or a memory-to-full-data gap term. Moreover, none of the terms in Theorem 1 depends on the particular memory-updating rule; the bound would hold for any replay strategy that satisfies the stated sample-size and imbalance quantities. Thus the claimed theoretical support for WRU specifically is absent.","section":"Theorem 1, Lemma 4"},{"comment":"The limitations stated by the authors themselves apply directly to the central claim: Appendix F notes that finding optimal margins for all tasks remains challenging and that analyzing the actual deep networks via NTK is left to future work. In addition, the margin scale λ is a per-dataset fitted hyperparameter (λ = 1.0, 3.5, 4 for C-VOC, C-COCO, C-NUS), and all reported results come from a single run without error bars. Therefore the 'optimal margin' formula is not a falsifiable prediction, and the reported empirical superiority over baselines is not quantified against run-to-run variability.","section":"Section E.2 and Appendix F"}],"minor_comments":[{"comment":"The footnote states that the hinge and logistic losses are 1-Lipschitz, which directly contradicts Assumption 2(3); the manuscript should either correct the assumption or remove the contradictory statement.","section":"Section 4, footnote 4"},{"comment":"The text 'prove that minimizing the RLADM loss' contains a typo: it should read RLDAM.","section":"Section 3.1, footnote 3"},{"comment":"The caption says boldface denotes the best results, but only Macro-AUC rows are bolded; the Forgetting rows should be handled consistently, and the discussion should clarify whether lower forgetting is intended to be a strength of the method.","section":"Section 6.2, Table 1"},{"comment":"The ER results on C-COCO are non-monotonic in memory size (e.g., 62.68 at size 500, 63.97 at size 1000, 62.53 at size 1500), so the claim that the method is 'less sensitive to changes in memory size' should be supported by repeated runs and ideally confidence intervals.","section":"Section 6.4, Table 3"},{"comment":"The input list uses T for both the set of tasks and the task length, and the for loop over t ∈ (1, T) is ambiguous; distinct symbols would improve readability.","section":"Algorithm 1"}],"recommendation":"reject","confidential_remarks":"The central theoretical error in Assumption 2(3) is decisive: it invalidates the margin-dependent bound, the optimal-margin derivation, and the claimed theoretical superiority over RU. Fixing it would require rewriting the theoretical core, not local revision. The empirical part might be salvageable if the paper were reframed as purely empirical, but the current single-run results and per-dataset hyperparameter tuning would need substantial strengthening."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know up front: this paper's empirical recipe is interesting, but its theoretical core is wrong. The paper proposes RLDAM, a reweighted label-distribution-aware margin loss for Macro-AUC, and WRU, a greedy memory update that preserves class-ratio statistics. It reports large gains over existing methods on three multi-label continual learning benchmarks, and code is released. That part is real work, and the gap it targets—Macro-AUC in multi-label continual learning—is genuinely under-served.\n\nThe problem is Assumption 2(3). The paper asserts that the shifted hinge losses ℓ(z−Δ_k^+) and ℓ(z−Δ_k^-) are (1/Δ)-Lipschitz. The standard hinge loss ℓ(z)=max(0,1−z) is 1-Lipschitz, and shifting the argument by a constant does not change that. The correct constant is 1, for every Δ>0. This is not a pedantic point: the reciprocal relation is exactly what makes the margin appear in the complexity term of Theorem 2, and it is what Proposition 1's ”optimal margin” formula Δ_k=λ/|D_k^+|^{1/4} minimizes. With ρ=1 the bound becomes margin-independent, so the claimed superiority of RLDAM over the plain RU loss of Wu et al. (2023) is unsupported, and the optimal-margin derivation collapses.\n\nThe continual-learning bound (Theorem 1) inherits the same issue and adds more: it is assembled from existing machinery (Shi and Wang 2023; Mansour et al. 2009), and its key quantities—ψ_i, the discrepancy disc(D_i,D_t)—are neither computed nor controlled, so it does not actually demonstrate that WRU helps. The authors say the proof shows WRU stabilizes the ni·τk terms, but that claim is made in the remark, not in the math.\n\nThe experiments are the strongest part: the gains over baselines are large, and the ablations show each component contributes. But the results are single-run, the λ hyperparameter is fit per dataset (1.0, 3.5, 4), and some baselines were re-implemented. This is enough to be suggestive, not enough to be confirmatory.\n\nNet: the paper is a decent empirical study in a small niche, but it is not the first theoretical analysis of MLCL in any meaningful sense, because the analysis is built on a false assumption. My recommendation: send it to peer review if the venue is tolerant of heavy revision, but insist that the theoretical claims be corrected or removed. If the authors keep only the empirical contribution and modest claims for WRU, it could become a citable paper. As it stands, I would not cite the theory.\n\nTake care.","headline":"The empirical recipe (RLDAM + WRU) is promising, but the paper's theoretical contribution rests on a false Lipschitz assumption and should not be taken as established.","tokens_in":25834,"tokens_out":4242,"would_cite":false,"duration_ms":35308,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","68Q32"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a reweighted, label-distribution-aware margin loss plus a count-preserving memory update improves Macro-AUC in imbalanced multi-label continual learning, with the first generalization bounds for this setting.","keywords":["multi-label continual learning","Macro-AUC","class imbalance","replay-based memory","label-distribution-aware margin","reweighted univariate loss","generalization bound","class-incremental learning"],"falsifier":"Compute the Lipschitz constant of $\\ell(z-\\Delta)=\\max(0,1-(z-\\Delta))$ directly: for any shift $\\Delta$, the function has slope 0 or $-1$, so its Lipschitz constant is 1, not $1/\\Delta$. Replacing $\\rho=1/\\Delta$ in Theorem 2 with $\\rho=1$ removes the advertised benefit of choosing larger margins; checking whether the reported gains persist when the theory is recomputed with the true constant would settle the claim.","tokens_in":24842,"feed_emoji":"🎯","tokens_out":6312,"duration_ms":53727,"temperature":0.7,"pith_summary":"The paper tries to establish that Macro-AUC, a suitable measure when multi-label classes are imbalanced, can be optimized in continual learning by a replay-based method built from two ingredients: a loss that combines per-class margin control with positive/negative reweighting, and a memory-update rule that preserves each class's positive and negative counts in the buffer. It argues that this combination gives better generalization guarantees for Macro-AUC than the existing reweighted univariate loss, and it claims these bounds are the first theoretical generalization analyses in multi-label continual learning. If the arguments hold, the method would be a practical recipe for keeping high overall multi-label accuracy under class-incremental training, and a template for theory-guided loss and memory design.","feed_headline":"Count-aware replay raises Macro-AUC in multi-label continual learning","feed_subtitle":"A margin loss plus memory that preserves class counts beats standard replay on three multi-label continual benchmarks.","key_machinery":"The central object is the RLDAM loss, a univariate per-label surrogate that avoids pairwise computation while inheriting reweighting by reciprocal class counts and label-distribution-aware margins $\\Delta_k = \\lambda/|D_k^+|^{1/4}$. The theoretical core is a fractional Rademacher complexity bound for a kernel hypothesis class, a contraction inequality that reduces loss-space complexity to hypothesis-space complexity, and a margin-optimality calculation that sets margins inversely with the fourth root of positive-class size. WRU is the second mechanism: a greedy selection that keeps $|D_k^+|/|D_k^-|$ in memory close to the task's ratio, plus explicit storage of those counts, so that the replay risk inherits the original reweighting factors.","core_discovery":"The core claim is that the RLDAM loss, defined as $\\ell(f_k(x^+) - \\Delta_k^+) + \\ell(-f_k(x^-) - \\Delta_k^-)$ with per-class margins, yields a Macro-AUC generalization bound whose complexity term scales with $(\\rho_k^+ + \\rho_k^-)$ and can be made smaller than the corresponding term of the RU loss by choosing class-aware margins. In the continual setting, the paper claims that combining RLDAM with WRU, which stores samples so that each class's stored positive/negative ratio matches the original task, makes the replay risk an approximate unbiased estimator of the task risk and keeps the imbalance factors in the bound equal to joint-training values. The main theorem decomposes the total risk into task complexity, empirical risk, forgetting gap, and cross-task discrepancy, and the paper argues this decomposition supports both the loss and the memory-updating strategy.","pith_inferences":["If the shifted-hinge Lipschitz assumption is corrected to the true constant 1, the theoretical advantage of larger margins may disappear; empirical gains would then need to justify themselves without the bound.","The margin schedule $\\Delta \\propto |D^+|^{-1/4}$ could be turned into an online rule: recompute margins from streaming positive counts, making the method applicable when task boundaries are unknown.","RLDAM is a univariate surrogate, so it could be combined with regularization- or architecture-based continual learning frameworks without pairwise sampling; the paper's appendix experiment with EWC hints in that direction.","Applying the same reweighting-plus-margin idea to other multi-label metrics, such as F1 or mAP, would test whether the intuition generalizes beyond Macro-AUC."],"forward_implications":["On the three benchmarks considered, the full method reaches Macro-AUC of 88.69, 77.93, and 79.77, above the compared replay baselines, so practitioners could adopt RLDAM plus WRU as a drop-in replay loss and memory rule.","The loss alone improves Macro-AUC in batch multi-label learning over BCE and RU on the same datasets, so it can be used outside continual learning.","The bound suggests margins should grow as $|D_k^+|$ shrinks, making the loss a principled choice for any imbalanced multi-label dataset, not only continual streams.","Under WRU, memory maintains the original per-class positive/negative ratios, so replay training should not drift from joint-training imbalance factors."],"supporting_citations":[{"why":"Supplies the reweighted univariate loss and Macro-AUC generalization framework that RLDAM extends, plus the RU baseline it must beat.","marker":"Wu, Li, and Yin 2023"},{"why":"Gives the label-distribution-aware margin loss and the optimal-margin derivation $\\Delta \\propto n^{-1/4}$ that RLDAM adapts to multi-label learning.","marker":"Cao et al. 2019"},{"why":"Provides the model-based continual-learning bound technique that Theorem 1 largely follows.","marker":"Shi and Wang 2023"},{"why":"Defines the discrepancy distance used to bound the gap between different tasks' distributions.","marker":"Mansour, Mohri, and Rostamizadeh 2009"},{"why":"Introduces the multi-label continual learning setting and the PRS memory-strategy baseline.","marker":"Kim, Jeong, and Kim 2020"},{"why":"Provides the OCDM baseline that also optimizes class distribution in memory and is compared experimentally.","marker":"Liang and Li 2022"},{"why":"Provides the KRT baseline and is the source of the repeated-sample-across-tasks assumption in MLCIL.","marker":"Dong et al. 2023"},{"why":"Supplies the reservoir sampling baseline memory scheme that WRU is contrasted with.","marker":"Vitter 1985"}],"fun_headline_variants":["Count-aware memory lifts Macro-AUC in multi-label continual learning","New loss and memory rule boost Macro-AUC for imbalanced MLCL","Fixing class imbalance in continual multi-label learning via Macro-AUC","Macro-AUC optimization for imbalanced multi-label continual learning","Reweighted loss and count-preserving memory for Macro-AUC in MLCL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole margin-based improvement rests on the assumption that shifting the hinge loss by a margin $\\Delta$ makes it $(1/\\Delta)$-Lipschitz; in fact a shifted hinge loss is still 1-Lipschitz for any shift, so the margin-dependent shrinking of the bound is not established.","fun_headline_variants_meta":{"raw":{"variants":["Count-aware memory lifts Macro-AUC in multi-label continual learning","New loss and memory rule boost Macro-AUC for imbalanced MLCL","Fixing class imbalance in continual multi-label learning via Macro-AUC","Macro-AUC optimization for imbalanced multi-label continual learning","Reweighted loss and count-preserving memory for Macro-AUC in MLCL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1298,"prompt_tokens":989,"completion_tokens":309,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":216}},"tokens_in":605,"tokens_out":309,"duration_ms":3039,"temperature":1.0,"reasoning_tokens":216,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:54:37.230951+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the Lipschitz constant of $\\ell(z-\\Delta)=\\max(0,1-(z-\\Delta))$ directly: for any shift $\\Delta$, the function has slope 0 or $-1$, so its Lipschitz constant is 1, not $1/\\Delta$. Replacing $\\rho=1/\\Delta$ in Theorem 2 with $\\rho=1$ removes the advertised benefit of choosing larger margins; checking whether the reported gains persist when the theory is recomputed with the true constant would settle the claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the model-based continual-learning bound technique that Theorem 1 largely follows."},{"cited_title":"D.; Jeong, J.; and Kim, G","cited_arxiv_id":null,"evidence_quote":"Introduces the multi-label continual learning setting and the PRS memory-strategy baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the KRT baseline and is the source of the repeated-sample-across-tasks assumption in MLCIL."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reservoir sampling baseline memory scheme that WRU is contrasted with."}],"review_version":1}