{"id":"3dd11d4e-4d0c-44cc-8951-fea3e2e22624","arxiv_id":"2505.03118","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A learnable per-label, per-instance threshold penalty that subtracts an IDF plus batch-label KNN signal from logits achieves 0.1712 macro-F1 on AmazonCat-13K, but the KNN signal leaks labels and no test-time procedure is given.","lead":"This paper proposes an adaptive thresholding penalty for multi-label loss, combining a global IDF rarity score with a local KNN signal computed from the batch, and reports macro-F1 of 0.1712 on AmazonCat-13K. The local signal is computed from ground-truth labels, and the paper does not specify how thresholds are obtained at inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never defines how KNN_l(x) is computed at test time; since Eqs. 4-6 build it from ground-truth Y, the 0.1712 macro-F1 claim is unverified and may be label leakage.","rationale":"The reader's weakest_assumption identifies the same load-bearing point: the local KNN signal in Eqs. 4-6 is built from ground-truth labels, and the paper does not say how it is computed at test time. I considered the unequal training budgets (150 vs. 1500 epochs) and the fact that external baselines are cited rather than rerun, but those are secondary. The central claim is an empirical number, and the reported number is not interpretable until the inference-time KNN protocol is specified and shown to be leakage-free. The released code could settle this, so the proposed test is to inspect and rerun the inference path. If the code applies a clean feature-space KNN without test labels, the objection is dropped; as written, the paper's strongest claim is unsupported. This does not change the reader's REJECT verdict.","tokens_in":5120,"tokens_out":6951,"duration_ms":67446,"concrete_test":"In the released GitHub repository, trace the AmazonCat-13K inference path and verify whether KNN_l for a test sample ever reads a label vector of that test sample or a training label matrix. Then rerun the evaluation with a leakage-free local signal: build the neighborhood graph on training TF-IDF features, compute KNN_l from the labels of training neighbors only, and exclude the diagonal self-term from both training and inference. If this protocol reproduces 0.1712 macro-F1, the concern is refuted; if the number falls materially (toward or below the reported ~0.11 pseudo-label baseline), the headline result is an artifact of label leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is the undefined inference-time computation of the local signal. In Eqs. 4-6, KNN_l(x) is obtained from the ground-truth binary label matrix Y of the current training batch. Because Eq. 4 contains the diagonal term (sample i shares all of its labels with itself), KNN_l(x) is dominated by the sample's own true label vector. This signal enters theta_l(x) in Eq. 1 and is then used inside the BCE term of Eq. 3, so the loss is optimized with access to the labels being predicted. The paper never states what replaces Y at test time. If predicted labels are used, the training objective (true-label KNN) and inference objective (predicted-label KNN) differ, so the evaluated threshold is not the one that was learned. If cached training neighborhoods are used, the self-term cannot be reproduced for test samples, and the model may have learned a self-label shortcut. Either way, the claimed macro-F1 of 0.1712 and the 'over 6 points' advantage over AttentionXML/DEPL are not supported by the description. The external baselines are also not rerun under the same protocol, but the KNN ambiguity alone is sufficient to make the central empirical claim unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptive thresholding mechanism for multi-label classification that fuses a global IDF-based label-rarity signal with a local KNN-based similarity signal to produce per-instance, per-label thresholds. Rather than applying hard cutoffs, the thresholds are used as differentiable penalty terms inside a composite BCE-plus-margin loss. The method is showcased on AmazonCat-13K, where the author reports a macro-F1 of 0.1712, claimed to substantially outperform tree-based and pretrained transformer baselines such as AttentionXML and DEPL-style methods. Ablation results on BibTeX and Delicious are mentioned as preliminary, and a GitHub repository link is provided for code.","tokens_in":5462,"tokens_out":4203,"duration_ms":44907,"significance":"If the reported results were supported by a sound experimental protocol, this would be a notable contribution: a lightweight, interpretable MLP-based model beating much heavier extreme multi-label classifiers via a differentiable thresholding loss. The idea of fusing global and local signals for per-label, per-instance thresholds is interesting and modular, and the interpretability and efficiency claims are attractive. However, the current manuscript contains a potentially load-bearing label-leakage problem in the KNN signal and an unequal-training-budget confound in the baseline comparison, so the central empirical claim is not currently credible.","major_comments":[{"comment":"The local signal KNN_l is computed from the ground-truth label matrix Y, and the diagonal of KNNnorm is 1 for each sample because a sample shares all of its labels with itself. Consequently, the threshold θ_l(x) in Eq. (1) is a function of the true value y_l that the loss in Eq. (3) is trying to predict, making the reported improvement potentially an artifact of label leakage. The paper does not specify what replaces Y at inference time. If predicted labels are used, the training objective (KNN built on true Y) and the inference objective (KNN built on predicted labels) differ; if cached training neighborhoods are used, the self-term cannot be properly reproduced for test samples. The authors must define the inference-time KNN computation and re-run all experiments under a leak-free protocol, for instance by using the model's own predictions to build the similarity graph.","section":"Section 3, Eqs. (4)-(6), and Section 5 (Table 1)"},{"comment":"The comparison is confounded by unequal training budgets. The training paragraph states that the static-threshold baseline and the IDF-only ablation were stopped at 150 epochs because they \"converged early\", while the adaptive model and the KNN-only ablation ran the full 1500 epochs. Since Figure 1 shows the adaptive and KNN-only curves still improving well beyond epoch 150, the final macro-F1 values in Table 1 (0.0035 and 0.0094) for the two stopped baselines are not comparable to the 0.1712 of the adaptive model. All models must be trained under the same budget, or early stopping must be applied uniformly based on a held-out validation metric.","section":"Section 4 and Section 5.1/Table 1"},{"comment":"The claim of surpassing AttentionXML and DEPL-style methods \"by over 6 points\" is based on previously published macro-F1 numbers, not on controlled re-implementations under the same evaluation protocol. The authors need to either rerun the baseline methods with the same features, code, and evaluation split, or provide a detailed table showing how the published numbers were converted into a like-for-like comparison. The reported positive ratio of 0.0006 for the adaptive model (roughly 8 positive predictions per sample out of 13,330 labels) is unusually low compared with typical AmazonCat-13K results and should be justified with precision/recall or label-level statistics.","section":"Section 5, \"Comparison to Prior Work\""},{"comment":"The experimental setup does not describe the data split (train/validation/test) or the model selection process. The text says the IDF-only and static baselines \"converged prematurely\", but no stopping criterion or validation metric is given, and the statement is unsupported by a figure or table. The authors should specify the split, the early-stopping rule, the number of random seeds, and report standard deviations or confidence intervals.","section":"Section 4 and Section 6"}],"minor_comments":[{"comment":"The margin loss is written for a single label but it is not explicit whether the sum in Eq. (3) runs over all labels with this term added for each l; please clarify the notation.","section":"Section 3, Eq. (7)"},{"comment":"The related-work section refers to \"informal discussions\" and \"early notions\" of local thresholding without citing concrete prior works; please replace these vague references with specific, citable publications.","section":"Section 2"},{"comment":"The axis labels in Figure 3 appear to have Greek letters rendered as blank or missing characters (e.g., \"α\" and \"β\" are invisible), making the plot hard to interpret; please fix the figure so that all symbols and legends are readable.","section":"Figure 3"},{"comment":"The description of the TF-IDF features as \"Version 1 from the AttentionXML repository\" lacks a specific download URL or version hash; providing these would materially improve reproducibility.","section":"Section 4"},{"comment":"The abstract states that \"full code\" is released, but the only pointer is a GitHub link in the references; consider adding a Code/Data Availability section with instructions, dependencies, and a model checkpoint.","section":"Section 4 and Abstract"},{"comment":"It is unclear whether the optional logit standardization in Eq. (2) is applied during inference, and whether the statistics µ and σ are computed on the training set only; this should be stated explicitly.","section":"Section 3, Eq. (2)"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early draft with an interesting core idea, but the load-bearing empirical claims rest on an undefined test-time KNN signal that is computed from ground-truth labels during training, which constitutes a form of label leakage, and the baseline comparison is further confounded by unequal training budgets. These are not merely presentation issues; they invalidate the reported numerical results as they stand. I would recommend rejection rather than major revision, because the central experimental contribution needs a fundamentally re-designed evaluation before the claims can be considered trustworthy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the central empirical claim is not supported as written. The local KNN signal in Eqs. 4–6 is built from the true label matrix Y of the batch, so the threshold θ_l(x) in Eq. 1 sees the label being predicted. The paper never states how this signal is computed at inference, so the 0.1712 macro-F1 comparison is unverified and, in all probability, a leakage artifact.\n\nCredit where it's due: the idea of fusing an IDF rarity prior with a label co-occurrence neighborhood as a differentiable threshold penalty is genuinely not present in the cited related work. The paper is clearly written, and the author is transparent about the early stopping of the IDF-only and static-threshold baselines, which accounts for part of the gap but is itself a confound.\n\nThe leakage issue is not minor. In Eq. 4, KNNraw = YY^T, and after row normalization in Eq. 5 the self-entry of each row is 1. Thus each KNN_l[i,l] includes Y[i,l] directly, plus a weighted sum of other samples' labels. The loss in Eq. 3 therefore optimizes with access to the very label it is predicting. At test time there is no ground truth, so the method is underspecified. Using predicted labels would make training and inference objectives differ; using cached training neighborhoods cannot reproduce the self-term for new samples. Either way, the stated advantage over AttentionXML and DEPL is not legitimate.\n\nSecond, the training budget is unequal: the static and IDF-only models ran 150 epochs while the proposed model ran 1500. That alone would make the gap suspect even without leakage. Third, the comparison to prior work relies on published numbers rather than a common protocol, which is risky for a 6-point claim. The ablations (KNN-only at 0.1456) also suffer from the same true-label dependence.\n\nOn the positive side, the method is simple and modular, and the author provides code. But because the main result is invalid, the broader conclusions about medical coding are premature.\n\nMy recommendation: the idea is worth a second look if the inference-time procedure is fixed—for instance, computing KNN once from the training set as a fixed feature, or using predicted labels with stop-gradient. As written, the paper should be rejected, but it deserves a serious referee to pin down the exact failure mode and give the author a chance to repair the evaluation.","headline":"The paper's 0.1712 macro-F1 claim rests on a threshold that is computed from ground-truth labels in training, with no defined test-time procedure, so the headline result is unverified and likely a leakage artifact.","tokens_in":5933,"tokens_out":3770,"would_cite":false,"duration_ms":39410,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned global-local threshold, applied as a differentiable loss penalty, lifts extreme multi-label macro-F1 to 0.1712 on AmazonCat-13K, more than six points above published tree- and transformer-based baselines.","keywords":["adaptive thresholding","multi-label classification","extreme multi-label classification","inverse document frequency","K-nearest neighbors","differentiable loss","label imbalance","macro-F1"],"falsifier":"Inspect the released evaluation code for AmazonCat-13K and determine the source of $\\mathrm{KNN}_l$ at inference. If the neighbor matrix is formed from ground-truth labels, recompute macro-F1 with neighbors formed from predicted labels only; a drop materially below 0.1712 would falsify the claim as stated.","tokens_in":4929,"feed_emoji":"🎯","tokens_out":11791,"duration_ms":103133,"temperature":0.7,"pith_summary":"This paper proposes replacing fixed or globally tuned classification thresholds with per-label, per-instance thresholds formed by blending a global inverse-document-frequency rarity score with a local K-nearest-neighbor agreement score. Rather than cutting predictions at the threshold, the threshold is subtracted from the logits and used as a differentiable penalty inside the training loss, with a margin term that sharpens decisions near the boundary. The author reports that on the AmazonCat-13K benchmark a shallow multilayer perceptron (MLP) with about 2.8 million parameters reaches a macro-F1 of 0.1712, surpassing published tree-based and pseudo-label guided results by more than six points. The motivation is to make multi-label prediction safer and more interpretable in high-stakes settings such as automated medical coding.","feed_headline":"Beat tree and transformer baselines: 0.1712 macro-F1","feed_subtitle":"Global rarity plus local neighbor agreement set per-label thresholds in a lightweight model.","key_machinery":"The load-bearing object is the adaptive threshold $\\theta_l(x) = \\lambda\\alpha_l\\,\\mathrm{IDF}_l + (1-\\lambda)\\beta_l\\,\\mathrm{KNN}_l(x) + b_l$, a per-label, per-instance scalar. The global term $\\mathrm{IDF}_l = \\log(N/f_l)$ measures label rarity across the dataset; the local term is built from the ground-truth label matrix of a batch by forming $\\mathrm{KNN}_{\\mathrm{raw}} = YY^\\top$, row-normalizing, and multiplying back by $Y$ to obtain soft label scores for each instance. This is a differentiable soft KNN entirely in label space. The threshold is not applied as a cutoff; it is subtracted from logits inside a composite loss, $\\mathrm{BCEWithLogits}(z_l - \\theta_l, y_l) + \\lambda_m\\,\\mathrm{MarginLoss}(z_l, \\theta_l, y_l)$, so the model is penalized for activating a rare label unless local evidence supports it.","core_discovery":"The central claim is that fusing a global signal of label rarity with a local signal of label co-occurrence among similar training instances produces thresholds that, when used as penalties rather than hard cutoffs, improve extreme multi-label classification beyond prior methods. Concretely, the adaptive model achieves macro-F1 0.1712 on AmazonCat-13K with a 2.8M-parameter multilayer perceptron, compared with roughly 0.07 for the tree-based AttentionXML baseline and 0.11 for pseudo-label guided generation, and does so without label trees or pretrained transformers. The paper further claims the learned blend weight shifts toward the local KNN signal for rare labels, and that the penalty formulation yields a more conservative positive-prediction rate with lower BCE loss, which the author interprets as improved calibration.","pith_inferences":["Because the paper leaves the inference-time computation of the local signal open, a reproducibility check on the released code is the first step; the validity of the six-point gain rests on that code using predicted, not ground-truth, labels at test time.","A direct way to separate the head's contribution from the MLP's capacity is to freeze a pretrained text encoder, train only the adaptive threshold head, and measure whether most of the six-point gain survives.","The paper's emphasis on rare labels predicts that macro-F1 gains concentrate in low-frequency label bands; reporting F1 binned by label frequency would test that prediction.","If inference-time KNN uses the model's own predictions, the approach is effectively self-refinement, and repeated threshold-update passes could be tested for convergence or drift."],"forward_implications":["On AmazonCat-13K, a 2.8M-parameter MLP with the adaptive threshold head reaches macro-F1 0.1712, more than six points above the AttentionXML and pseudo-label guided baselines.","Because the threshold head is modular, it can be attached to pretrained encoders such as BERT or ClinicalBERT and trained without retraining the backbone, carrying the same interpretable penalty to other label spaces.","The learned blend weight shifts toward the KNN signal for rare labels, so the method's benefit is expected to concentrate in long-tailed label distributions.","Using thresholds as penalties rather than cutoffs produces a conservative positive-prediction rate and lower BCE loss, which the author interprets as improved calibration for high-precision settings."],"supporting_citations":[{"why":"Supplies the AmazonCat-13K TF-IDF features used in experiments and the AttentionXML tree-based baseline with macro-F1 around 0.07 that the paper claims to beat.","marker":"[You et al., 2019]"},{"why":"Provides the pseudo-label guided long-tailed baseline reaching 0.11 macro-F1, the prior published score the paper says it surpasses by over six points.","marker":"[Zhang et al., 2022]"},{"why":"Presents the label-embedding alternative against which the paper positions its discrete, interpretable global-local threshold modulation.","marker":"[Srikumar and Manning, 2014]"}],"fun_headline_variants":["Global-local fusion for adaptive MLC thresholds","Adaptive thresholds via global-local signal fusion","0.1712 macro-F1 from adaptive threshold penalties","Adaptive thresholds beat tree and transformer MLC","Per-instance adaptive thresholds from global-local signals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The local KNN signal is computed from the ground-truth label matrix of the training batch, and the paper does not specify how $\\mathrm{KNN}_l(x)$ is obtained for test instances, so the reported result silently assumes a test-time definition of local agreement.","fun_headline_variants_meta":{"raw":{"variants":["Global-local fusion for adaptive MLC thresholds","Adaptive thresholds via global-local signal fusion","0.1712 macro-F1 from adaptive threshold penalties","Adaptive thresholds beat tree and transformer MLC","Per-instance adaptive thresholds from global-local signals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001362,"raw_usage":{"total_tokens":5467,"prompt_tokens":830,"completion_tokens":4637,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":4566}},"tokens_in":446,"tokens_out":4637,"duration_ms":32335,"temperature":1.0,"reasoning_tokens":4566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:58:52.875659+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released evaluation code for AmazonCat-13K and determine the source of $\\mathrm{KNN}_l$ at inference. If the neighbor matrix is formed from ground-truth labels, recompute macro-F1 with neighbors formed from predicted labels only; a drop materially below 0.1712 would falsify the claim as stated.","supporting_citations":[{"cited_title":"Long-tailed Extreme Multi-label Text Classification with Generated Pseudo Label Descriptions","cited_arxiv_id":"2204.00958","evidence_quote":"Provides the pseudo-label guided long-tailed baseline reaching 0.11 macro-F1, the prior published score the paper says it surpasses by over six points."},{"cited_title":"Learning distributed representations for structured output prediction","cited_arxiv_id":null,"evidence_quote":"Presents the label-embedding alternative against which the paper positions its discrete, interpretable global-local threshold modulation."}],"review_version":1}