{"id":"1fb6f0ef-3e3b-442b-851c-91013f87b60d","arxiv_id":"2412.16963","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A local-hierarchy-correlation-guided Mixup ratio applied to depth-level prompt tuning outperforms state-of-the-art baselines on WOS, NYT, and RCV1-V2.","lead":"This paper adds a data augmentation step, Mixup, to a hierarchical prompt tuning method for hierarchical text classification. A similarity-based rule decides how strongly to mix pairs of training texts, and the authors report small but consistent gains on three benchmark datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central novelty—the Eq. 4 correlation-guided Mixup ratio—is supported only by small single-run ablation gains with no significance tests; without paired tests vs +Mixup, the claim that the guided ratio is responsible is unverified.","rationale":"I agree with the reader that Eq. 4 is a hand-designed heuristic whose theoretical grounding is thin, but the more pressing issue is empirical: the paper never demonstrates that the guided ratio beats vanilla Mixup with statistical confidence. The ablation table is the only direct evidence for the central contribution, and its margins are small, particularly on WOS Micro-F1 and RCV1-V2 Micro-F1. The paper's own statistical section compares only to HPT and HBGL, not to the +Mixup variant that isolates the proposed mechanism. Because the verdict is already CONDITIONAL, this concern does not change the verdict; it sharpens the condition: the authors should supply significance tests for the ablation. The proposed check is feasible with existing code and seeds and would settle whether the core mechanism is real or an artifact of a single seed.","tokens_in":15782,"tokens_out":5347,"duration_ms":44094,"concrete_test":"Run Prompt+Mixup and Prompt+LH-Mix under the identical five-seed protocol used for Table 6 on WOS, NYT, and RCV1-V2, and perform paired t-tests on Macro-F1 and Micro-F1 between the two conditions. If the LH-Mix advantage over +Mixup is not significant at p<0.05 on the majority of the six dataset-metric pairs, the central claim that Eq. 4's correlation guidance is responsible for the improvements should be weakened or rejected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LH-Mix's adaptive ratio (Eq. 4) materially improves over plain Mixup in hierarchical prompt tuning. The direct evidence is Table 3: Prompt+Mixup vs Prompt+LH-Mix gains of Micro-F1/Macro-F1 are +0.14/+0.25 on WOS, +0.22/+0.63 on NYT, and +0.02/+0.54 on RCV1-V2. These are single-run numbers; no standard deviations or paired significance tests are reported for this comparison. Table 6 and Table 7 provide five-seed statistics and t-tests only against HPT and HBGL, not against Prompt+Mixup, which is the baseline that isolates the contribution of Eq. 4. The parameter analysis in Section 5.9 examines only NYT and RCV1-V2 Macro-F1, and the Micro-F1 trends are dismissed as unreliable. Thus the specific contribution of the correlation-guided ratio—the only new component beyond HPT+vanilla Mixup—is not statistically established. Compounding this, Eq. 4 is explicitly heuristic and is tuned by grid search over alpha and beta. Without significance testing against the vanilla Mixup baseline, the headline 'five out of six metrics' could reflect seed luck rather than the proposed mechanism. This is load-bearing because if the LH-Mix vs +Mixup gap is within noise, LH-Mix reduces to a known recipe (hierarchical prompt + Mixup) and the claimed novelty is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LH-Mix, a method for hierarchical text classification that represents a text-specific local hierarchy as a depth-level prompt sequence and then applies Mixup to the [MASK] hidden states at each depth. The Mixup ratio is not sampled from a Beta distribution; instead, it is computed by Eq. (4) from the normalized cosine similarity of local-hierarchy representations. The authors evaluate on WOS, NYT, and RCV1-V2, compare with hierarchy-aware, LLM, and BERT-based baselines, and report that LH-Mix achieves the best performance on five of six metrics, with larger gains on Macro-F1 and in low-resource settings. They also provide five-seed statistics and t-tests against HPT and HBGL in Appendix A.2.","tokens_in":16104,"tokens_out":8409,"duration_ms":73038,"significance":"If the reported gains are robust, the paper makes a useful and simple contribution: it shows that a hierarchy-correlation-guided Mixup ratio, on top of hierarchical prompt tuning, improves HTC over strong baselines, especially for rare and deep labels and in sparse training regimes. The gradient derivation in Eq. (8) is a correct chain-rule application, and the inclusion of five-seed runs and t-tests against SOTA baselines is a strength. However, the central novelty, namely the mechanism of Eq. (4), is not directly statistically validated, and the headline numbers in Table 2 are inconsistent with the multi-seed statistics in Table 6, so the strength of the empirical claim currently exceeds the evidence.","major_comments":[{"comment":"The main results in Table 2 do not match the five-seed means in Table 6. For example, WOS Macro-F1 is 82.07 in Table 2 but 81.82±0.18 in Table 6, and RCV1-V2 Macro-F1 is 71.93 versus 71.49±0.50. Since Table 2 is used to claim the best performance in five out of six metrics, the paper should either report the multi-seed mean in Table 2 or explicitly state that Table 2 reports a single run and identify the corresponding seed. This is not a cosmetic issue because several reported margins are smaller than the standard deviations in Table 6.","section":"Section 5.4 and Appendix A.2 (Tables 2 and 6)"},{"comment":"The only comparison that isolates the contribution of Eq. (4) is Prompt+Mixup versus Prompt+LH-Mix in Table 3, with single-run gains of +0.14/+0.25 on WOS, +0.22/+0.63 on NYT, and +0.02/+0.54 on RCV1-V2 in Micro-F1/Macro-F1. These numbers are not accompanied by standard deviations or paired significance tests. The five-seed statistics in Table 6 and the t-tests in Table 7 compare only against HPT and HBGL, not against the vanilla Mixup baseline. Given that Section 5.9 itself cautions that Micro-F1 changes are small and may not accurately reflect the final results, the claim that the correlation-guided ratio is responsible for the observed improvement is not statistically established. Please add multi-seed results and paired significance tests for LH-Mix versus the vanilla Mixup variant.","section":"Table 3 and Appendix A.2"},{"comment":"Eq. (4) is explicitly heuristic, and its two parameters alpha and beta are chosen from grids on the same datasets. The manuscript does not state the selected per-dataset values, the selection criterion, or whether the grid search was performed on a held-out split. Because the paper concedes in Section 4.2 that there is no well-established theoretical framework for the similarity-to-ratio map, the possibility that the reported gains reflect tuned hyperparameters rather than the proposed mechanism cannot be ruled out. Please report the chosen alpha and beta for each dataset, the validation procedure, and results for neighboring grid values, at least for the datasets shown in Figure 5.","section":"Section 4.2 and Section 5.2"},{"comment":"The guiding signal s is not validated. The paper assumes that the frozen BERT [CLS] embedding of the gold-label sequence is a trustworthy measure of local-hierarchy correlation, but no evidence is provided that this cosine similarity tracks actual hierarchy distance or that it is what drives the gains. Figure 6 is a qualitative t-SNE case study after training and does not isolate the similarity measure. A concrete test would be to compare s against a structural distance such as the depth of the lowest common ancestor on a sample of label pairs, or to ablate Eq. (4) by replacing s with a random or constant value. Without such a test, the correlation-guided interpretation remains an assumption.","section":"Section 4.1"}],"minor_comments":[{"comment":"The text says local hierarchy representations use the same pre-trained model encoder and then says the encoder used for similarity calculation is distinct from the one used for classification; please clarify whether the similarity encoder is frozen and separate from the fine-tuned prompt encoder.","section":"Section 4.1"},{"comment":"The sentence 'we will release code our the Github' should read 'we will release our code on Github.'","section":"Section 5.2"},{"comment":"'Noted, for conciseness and clarity' should be 'Note that, for conciseness and clarity.'","section":"Section 5.10"},{"comment":"The underlining of current SOTA entries is not visible in the manuscript text; please ensure the formatting is correct.","section":"Table 2"},{"comment":"These figures report curves without error bars or multi-seed aggregation; please state whether they are single-run results.","section":"Figures 4, 5, and 7"},{"comment":"The expression shown is the derivative of the mixed loss with respect to the mixed prediction score ep^d, not the full gradient with respect to model parameters; the latter also involves the chain rule through Eq. (5). Please state this explicitly to avoid confusion.","section":"Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible and the paper has several strengths, but the empirical evidence for the central novelty is incomplete. The most important missing experiment is a multi-seed statistical comparison of LH-Mix against the vanilla Mixup baseline, and the discrepancy between Tables 2 and 6 must be resolved. If the authors can supply those results and clarify the hyperparameter selection, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clean, small-scale method paper. The new piece is applying Mixup to hierarchical prompt tuning for HTC, with a heuristic that makes the Mixup ratio depend on similarity between local hierarchy representations. That combination hasn't been done before in HTC, and the empirical story is consistent: small gains (roughly 0.1–0.6 Macro-F1) over strong baselines on WOS, NYT, and RCV1-V2, with most differences significant in the five-seed t-tests.\n\nWhat it does well: the ablation table isolates the three contributions (prompt, vanilla Mixup, guided ratio), the sparse-data experiments are sensible, the gradient derivation in Eq. 8 is correct, and the comparison against GNN-based and other augmentation methods helps position the work. The writing is direct, and the authors are candid that Eq. 4 is a heuristic with no theoretical backing.\n\nSoft spots: the central claim that the guided ratio matters hangs on Table 3, where Prompt+Mixup is compared to Prompt+LH-Mix. Those numbers are single-run; the five-seed statistics and t-tests in Appendix A.2 only cover HPT and HBGL, not the vanilla-Mixup baseline that isolates the novel component. So we don't actually know whether the adaptive ratio beats plain Mixup beyond seed luck. The parameter analysis is also only on Macro-F1, and the authors dismiss Micro-F1 trends as unreliable. Hyperparameters alpha and beta are chosen on the same datasets used for final results, which is a mild form of selection-on-test. And the headline numbers in Table 2 don't match the five-seed means in Table 6, which suggests the main table reports the best seed rather than the average; that should be reconciled.\n\nNone of these sink the paper. The combination of HPT+Mixup is still a valid empirical recipe, and the gain over HBGL/HPT is plausible even if the marginal value of Eq. 4 is not proven. But as written, the novelty rests on a comparison that hasn't been properly tested.\n\nWho it's for: people working on hierarchical text classification who want a cheap regularization trick; not someone looking for theory. It deserves a serious referee, and in fact is already accepted at KDD '25. If I were handling it, I'd ask for paired tests against vanilla Mixup and the code release before trusting the specific claim.","headline":"A modest but sound empirical paper: HPT+Mixup is new in HTC, the gains are real yet small, and the one novel component (the guided ratio) lacks direct significance testing.","tokens_in":16627,"tokens_out":2250,"would_cite":true,"duration_ms":19526,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A similarity-guided Mixup ratio lifts hierarchical text classification on three benchmark datasets.","keywords":["hierarchical text classification","mixup","prompt tuning","local hierarchy","label correlation","data augmentation","multi-label classification","BERT"],"falsifier":"Run LH-Mix with a per-pair ratio that is randomly shuffled independently of $s$: if shuffled ratios perform as well as the correlation-guided ones, the guidance carries no information and the reported gains come from Mixup's regularization alone. A faster check is to construct a dataset where gold-label sequences are lexical twins (high $s$) but their true label sets are nearly disjoint — under Eq. 4 they would be mixed hard, and if performance drops versus vanilla Mixup on that data, the similarity proxy is misleading.","tokens_in":15562,"feed_emoji":"🔀","tokens_out":5620,"duration_ms":44778,"temperature":0.7,"pith_summary":"This paper claims that hierarchical text classification (HTC) can be improved by mixing not just a single label representation but the entire text-specific label path, with the strength of mixing set by how similar those paths are in a learned latent space. The proposed LH-Mix model encodes each text's local hierarchy (the labels it actually uses) as a depth-aligned prompt sequence, then interpolates hidden states at every depth between pairs of examples. The novel piece is a hand-designed function that maps the cosine similarity between two local-hierarchy embeddings to a Mixup ratio: highly similar paths are blended toward a 50/50 mix, dissimilar paths are barely mixed. On WOS, NYT, and RCV1-V2 the method reports the best score in five of six F1 metrics, with statistically significant gains over the two strongest baselines on most metrics, and the gains grow as training data is sparser and the hierarchy deeper.","feed_headline":"Hierarchy-similarity Mixup beats SOTA on three text benchmarks","feed_subtitle":"A similarity-based blend ratio lifts macro-F1 and keeps gains when training data is sparse.","key_machinery":"The load-bearing object is the local-hierarchy correlation-guided Mixup ratio, $\\lambda = -(\\beta-0.5)s^\\alpha + \\beta$, with $\\alpha > 0$ controlling how quickly the blend strength changes with similarity and $\\beta \\in (0.5, 1]$ capping the maximum $\\lambda$ (minimum mixing). Here $s$ is the normalized cosine similarity (Eq. 3) between the frozen [CLS] embeddings of two gold-label hierarchy sequences, e.g. '[CLS][Dth1]CS[Dth2]Machine Learning[SEP]'. This function carries the argument: it turns a static, uniformly sampled Beta ratio into a per-pair intensity that approaches 0.5 for highly correlated local hierarchies (strong mixing) and approaches $\\beta$ for dissimilar ones (light mixing), and when applied to the hidden [MASK] states at each depth it is what lets the model learn sibling/peer correlations that the parent-child prompt alone does not encode.","core_discovery":"On the paper's own terms, the central discovery is that treating the local hierarchy as a sequence and applying Mixup under a similarity-adapted ratio captures implicit label correlations that both vanilla Mixup and graph-encoder baselines miss. Concretely, LH-Mix builds on HPT's depth-level prompt, replaces the [MASK] token with the gold label to form a 'local hierarchy sentence' for each input, and takes the frozen BERT [CLS] embedding of that sentence as the hierarchy's representation. For a pair of inputs, the normalized cosine similarity s of these embeddings is plugged into $\\lambda = -(\\beta-0.5)s^\\alpha + \\beta$ to produce the Mixup ratio, with $\\alpha$ and $\\beta$ tuned from a small grid. The mixed hidden states are fed through the depth classifiers, and the zero-bounded multi-label cross-entropy (ZMLCE) loss is mixed in the same proportion, which the paper argues mirrors vanilla Mixup's gradient behavior. The authors report best-in-table results on five of six metrics across WOS, NYT, and RCV1-V2, with t-tests showing significance on all but one comparison against HPT and all but two against HBGL, and they show the advantage widens when training data is downsampled to 10% and for deep or rare labels.","pith_inferences":["Because the similarity measure uses a frozen encoder on gold-label sequences, LH-Mix is computationally cheap at train time but cannot adapt its correlation measure as the classifier learns; a learnable but still frozen-like encoder (e.g., a stop-gradient projection) is a natural next test.","The same ratio-law could transfer to other structured prediction tasks where labels form a tree, such as taxonomic image classification or hierarchical retrieval, wherever a text-like sequence can be built from the label path.","The paper's heuristic mapping suggests a testable hypothesis about Mixup generally: instance-pair similarity should control blend strength along a monotone family; the exact exponent $\\alpha$ may be a per-task free parameter, which the reported $\\alpha$ sensitivity on NYT versus RCV1-V2 supports.","If the gains indeed come from peer/sibling correlation rather than from extra regularization, then LH-Mix should pair well with graph-based global hierarchy encoders, which the paper does not combine with HBGL; an ensemble or hybrid remains untested."],"forward_implications":["Any hierarchical multi-label model that already uses depth-level prompts can add the same correlation-guided Mixup with a small grid search over $\\alpha$ and $\\beta$, and the paper's ablations suggest it should beat vanilla Mixup and graph encoders.","In low-resource settings, the relative gain of LH-Mix over both prompt-only and vanilla-Mixup training grows as training data shrinks to 50%, 25%, and 10%, so the method is a candidate for HTC with sparse labels.","Macro-F1 improves more than Micro-F1, which the authors attribute to better label-level correlation; if correct, this means Mixup-style regularization is a viable way to address rare and deep labels without changing the architecture.","The two-parameter family in Eq. 4 subsumes linear ($\\alpha=1$) and nonlinear ($\\alpha \\neq 1$) similarity-to-ratio mappings, so the choice of $\\alpha$ can be tuned per dataset; the paper's parameter study shows NYT and RCV1-V2 prefer different $\\alpha$, meaning the family has enough capacity to express the difference."],"supporting_citations":[{"why":"Introduces the Mixup interpolation that LH-Mix adapts for hierarchical labels.","marker":"[42]"},{"why":"Provides the depth-level hierarchical prompt and ZMLCE loss that LH-Mix builds on.","marker":"[35]"},{"why":"The global/local hierarchy baseline that LH-Mix must beat, and the source of the local-hierarchy motivation.","marker":"[16]"},{"why":"Justifies using the [CLS] embedding of a label sequence as a similarity measure for local hierarchies.","marker":"[25]"},{"why":"Motivates adaptive, instance-similarity-based Mixup ratios that LH-Mix follows.","marker":"[27]"},{"why":"Motivates correlation-based mixing intensities for generating informative in-between samples.","marker":"[43]"},{"why":"Establishes gradient equivalence between label mixing and loss mixing, supporting the loss-mixing choice.","marker":"[4]"},{"why":"Provides the pattern for hidden-state interpolation in text classification that LH-Mix applies at each depth.","marker":"[12]"}],"fun_headline_variants":["Similarity-guided Mixup beats graph encoders in hierarchical text classification","Local hierarchy as sentence: Mixup ratio from similarity lifts HTC","Similarity-adapted Mixup outperforms existing HTC baselines on three datasets","Hierarchy-similarity Mixup ratio improves sparse-data HTC","LH-Mix: cosine-similarity guided Mixup for local hierarchy in HTC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-designed mapping from local-hierarchy similarity to Mixup intensity — $\\lambda = -(\\beta-0.5)s^\\alpha + \\beta$ — correctly expresses how strongly any pair of label paths should be blended, and that the frozen BERT [CLS] embedding of the gold-label sequence is a trustworthy measure of that similarity; the paper itself notes there is no well-established theoretical framework for this relationship.","fun_headline_variants_meta":{"raw":{"variants":["Similarity-guided Mixup beats graph encoders in hierarchical text classification","Local hierarchy as sentence: Mixup ratio from similarity lifts HTC","Similarity-adapted Mixup outperforms existing HTC baselines on three datasets","Hierarchy-similarity Mixup ratio improves sparse-data HTC","LH-Mix: cosine-similarity guided Mixup for local hierarchy in HTC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000693,"raw_usage":{"total_tokens":3140,"prompt_tokens":954,"completion_tokens":2186,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":2102}},"tokens_in":570,"tokens_out":2186,"duration_ms":14330,"temperature":1.0,"reasoning_tokens":2102,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:56:12.313808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LH-Mix with a per-pair ratio that is randomly shuffled independently of $s$: if shuffled ratios perform as well as the correlation-guided ones, the guidance carries no information and the reported gains come from Mixup's regularization alone. A faster check is to construct a dataset where gold-label sequences are lexical twins (high $s$) but their true label sets are nearly disjoint — under Eq. 4 they would be mixed hard, and if performance drops versus vanilla Mixup on that data, the similarity proxy is misleading.","supporting_citations":[{"cited_title":"Dauphin, and David Lopez-Paz","cited_arxiv_id":null,"evidence_quote":"Introduces the Mixup interpolation that LH-Mix adapts for hierarchical labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The global/local hierarchy baseline that LH-Mix must beat, and the source of the local-hierarchy motivation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates adaptive, instance-similarity-based Mixup ratios that LH-Mix follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates correlation-based mixing intensities for generating informative in-between samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes gradient equivalence between label mixing and loss mixing, supporting the loss-mixing choice."}],"review_version":1}