{"id":"941f7a21-9798-48e2-9cf8-7bcac8e5d6c1","arxiv_id":"2602.08986","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A node-weighted loss combining inverse-frequency weighting and ensemble-uncertainty focal terms improves recall of rare classes in hierarchical multi-label models by up to ~5x.","lead":"This paper proposes a weighted training loss for hierarchical multi-label classification that up-weights rare classes and uncertain predictions, reporting up to 5x recall gains. It matters because rare fine-grained classes are often the scientifically important ones in medical and environmental datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Focal branch relies on a self-referential uncertainty signal and is only weakly supported; headline recall gains appear driven by imbalance weighting alone.","rationale":"The imbalance-weighting contribution is credible and consistently supported across many datasets, so I do not see a fatal flaw there. The fragile part is the focal branch, exactly as the reader identified. The paper's own ablations and tables show that the uncertainty-gated focal term adds little or nothing beyond imbalance weighting in several datasets, underperforms for epistemic uncertainty variants, and only reliably separates from baseline with large ensembles. This does not refute the paper, but it means the central claim as stated -- that the combination of imbalance and focal weighting improves rare-node detection -- is not fully established. A paired-seed reanalysis and diversity-collapse check would settle whether the focal term is load-bearing. This supports the existing CONDITIONAL verdict rather than a stronger action; no evidence of leakage or internal contradiction was found in the main imbalance results.","tokens_in":25978,"tokens_out":8500,"duration_ms":97043,"concrete_test":"Run the focal-vs-imbalance-only comparison on all 16 gene-product datasets with 10 seeds and ensemble size 10, computing paired per-seed F1 differences (GMU/bBMA minus baseline) with a paired bootstrap and multiple-comparison correction. In parallel, track mean pairwise KL divergence among ensemble members on a fixed validation set across training for focal and imbalance-only runs. If focal gains are not significant on a majority of datasets after correction, or if pairwise KL collapses toward zero in focal runs, the focal contribution should be reported as not established and the headline narrowed to imbalance weighting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim bundles node-wise imbalance weighting with an uncertainty-gated focal term. The imbalance branch is well supported: Table 1 shows recall improvements up to roughly 6x at w0=0.25 across FUN datasets, and similar trends on GO. The focal branch is not. It computes U from the very ensemble being trained; gradients through U are stopped, but the weighting still couples all members' objectives and can collapse diversity, a risk the paper itself acknowledges via Lee et al. The paper's own results show focal F1 gains over the imbalance-only baseline are small and inconsistent: Table 4 gives Cellcycle 7.39 vs 6.51, Derisi 2.54 vs 2.27, but Seq 8.67 vs 8.84 (negative), with precision and Bin. AP often lower (Cellcycle Bin. AP 10.46 vs 10.53). Epistemic JS/KL terms are not significantly better than baseline on most datasets in Tables 4 and 6. Figure 3 shows focal candidates separate from baseline only at ensemble size 10 or more; at size 5 they overlap. Thus the evidence is consistent with imbalance weighting doing the work and focal weighting adding a fragile, sometimes harmful term. The assumption that U is a reliable, non-collapsing emphasis signal is the least secure condition for the combined-method claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a weighted loss for hierarchical multi-label classification (HML) that combines a node-wise inverse-frequency imbalance weighting with a focal term driven by ensemble uncertainty. The imbalance branch (Eqs. 7–9) reweights positive annotations of rare nodes independently of observation frequency; the focal branch (Eqs. 10–17, used in Eq. 6) gates a focal exponent by ensemble-derived uncertainty measures (bBMA, GMU, epistemic JS/KL). The method is evaluated on 16 gene-product datasets (FUN and GO), a benthic imagery dataset, Enron, and Diatoms, with comparisons to C-HMCNN, LPROS, and HROS-PD. The imbalance weighting alone strongly improves recall and F1 (Table 1), and combining it with bBMA/GMU focal terms yields further F1 gains on several datasets (Tables 2, 4, 6), though the epistemic terms do not consistently help.","tokens_in":26342,"tokens_out":5979,"duration_ms":65091,"significance":"If the results hold, the node-wise imbalance weighting is a valuable and practical contribution to HML: it is simple, reproducible (code provided), evaluated with standard deviations across many benchmarks, and improves recall by up to roughly a factor of six on rare nodes that existing resampling methods miss. The focal branch is the more fragile half: only bBMA/GMU show benefits, gains are not universal (e.g., Seq), the uncertainty signal is computed from the very ensemble being trained, and benefits appear only at ensemble sizes of 10 or more. The paper is strongest as an empirical study of node-wise weighting; the combined-method claim needs to be recalibrated and supported with additional ablations.","major_comments":[{"comment":"The abstract and conclusion attribute the 'up to factor five' recall improvements to the combined weighted loss, but Table 1 shows those gains are produced by imbalance weighting alone (w0=0.25) with no focal term. The focal experiments (Tables 2/4) use imbalance-only as the baseline, and the extra F1 gains are inconsistent: e.g., Seq bBMA F1 is 8.67±0.12 vs baseline 8.84±0.07 (Table 4), and Expr GO bBMA is 5.20±0.10 vs 5.28±0.08 (Table 6). Add a focal-only ablation and attribute the recall gains to the imbalance branch in the abstract and conclusion.","section":"Abstract / §5.2 / Table 1"},{"comment":"The focal weight U(Θ(X)) is computed from the same ensemble that is being trained. Gradients through U are stopped, but the weighting still couples the ensemble members' objectives and can reduce diversity. The paper cites Lee et al. on ensemble-collapse risk but does not measure diversity or test a non-self-referential uncertainty source. Figure 3 shows no separation from baseline at ensemble size 5, consistent with an unreliable signal. Add diversity diagnostics over training and an experiment with a fixed/pretrained uncertainty estimator to break the feedback loop.","section":"§3.2, Eq. (6)"},{"comment":"The global hyperparameters w0=0.25, u0=0.25, k=1 are chosen from 'preliminary testing' (Appendix A), but Appendix H shows the optimal w0 for Expr (FUN) is below 0.2 and that u0 and k have 'dampened' effects. Since w0 is the main performance lever (Figure 2, Table 1), the paper should provide per-dataset sensitivity analyses or a principled selection rule; otherwise the reported results depend on test-selected hyperparameters, and the claim that w0=0.25 is 'reasonable' is not supported by the Appendix H finding.","section":"Appendix H / Appendix A"},{"comment":"The significance criterion used throughout is an ad-hoc '>2σ' rule based on a small number of runs, applied across 16 datasets and multiple metrics without multiple-comparison correction. For example, Table 2 shows bBMA on Expr at 10.53±0.35 vs baseline 10.27±0.24, which is within 2σ under a pooled standard error, yet the text emphasizes statistically significant gains. Provide exact p-values or corrected confidence intervals, or temper the significance language.","section":"§5.3 / Tables 2, 4, 6"}],"minor_comments":[{"comment":"The caption says 'compared for the FUN datasets' but the table reports GO results; should read 'GO datasets'.","section":"Table 5 caption"},{"comment":"The quantities wmin and wmax are used before being defined; clarify that they are the min/max of the raw wi vector.","section":"§3.1, Eq. (8)"},{"comment":"The double subscript in µmax(2) is confusing; consider renaming to something like µmargin for readability.","section":"Eqs. (13)–(15)"},{"comment":"The sentence 'switching toF1 appears to drop' appears to contain a typo; presumably 'switching to dropout'.","section":"Appendix F"},{"comment":"The terms 'Epistemic JS' and 'Epistemic KL' are used without definition in the table captions; add a sentence explaining these refer to Eq. (17) with JS or KL divergence.","section":"Tables 2, 4, 6"}],"recommendation":"major_revision","confidential_remarks":"The imbalance-weighting contribution is solid, well-evaluated, and reproducible; the focal branch is overclaimed and its self-referential uncertainty mechanism is insufficiently supported. The requested ablations and statistical corrections are feasible within the paper's scope and would substantially strengthen it. No concerns about novelty or attribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The node-wise imbalance weighting is the actual contribution here, and it looks solid. Across 16 FUN/GO gene datasets, plus Enron, Diatom, and benthic imagery, weighting nodes rather than observations roughly doubles F1 and multiplies recall several-fold over the unweighted C-HMCNN baseline. The effect is consistent, standard deviations are reported, and the comparison against LPROS and HROS-PD is fair. If you work on hierarchical multi-label learning, the imbalance branch alone is worth your time, and the code is provided.\n\nThe focal branch is where I part ways with the abstract. The paper presents imbalance weighting plus an uncertainty-gated focal term as a package, but the evidence says the focal term is not carrying its weight. The imbalance-only baseline is already strong; adding bBMA/GMU focal weighting gives small F1 gains on some datasets, but precision and binarized AP often drop, and on Seq the focal term actually hurts (Table 4). The epistemic uncertainty terms (JS, KL) basically never beat the imbalance-only baseline. Figure 3 shows focal candidates only separate from baseline at ensemble size 10 or more. The uncertainty signal is computed from the ensemble you are training with gradients stopped, which is a reasonable thing to try, but the paper’s own robustness plots show the u0 and k hyperparameters have a dampened and uneven effect. So my read is: the authors found a genuine effect with imbalance weighting, then bolted on a focal term that is occasionally useful but inconsistently so, and the abstract oversells the combination.\n\nThat said, the paper is honest in the details. The appendices report the negative results, the underperformance of epistemic uncertainty, the failed scheduling experiments, and the dropout trade-off. No fatal flaw, no circularity beyond the normal focal-style feedback loop. The hyperparameters w0, u0, k are empirically chosen, but the paper includes a sensitivity analysis and notes that w0 should be chosen per problem.\n\nWho is this for? Anyone doing HML in bioinformatics or ecology, especially with rare descendant nodes. A serious referee would not have wasted their time; the imbalance-weighting result is reproducible and useful. My recommendation: accept the paper’s core message about node-wise imbalance weighting, cite that part, and treat the focal weighting as an exploratory add-on rather than a validated improvement.","headline":"The imbalance-weighting branch is the real result; the focal/uncertainty branch is weak sauce that the paper’s own tables undercut.","tokens_in":26801,"tokens_out":1411,"would_cite":true,"duration_ms":16493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Node-wise frequency weighting plus an uncertainty-gated focal term lets hierarchical multi-label models detect rare descendant nodes they previously ignored, with recall gains up to fivefold and statistically significant F1 improvements.","keywords":["hierarchical multi-label classification","class imbalance","rare nodes","focal loss","uncertainty quantification","ensemble methods","node-wise weighting","gene ontology"],"falsifier":"On a fixed benchmark, compare imbalance-only training against the full focal loss at ensemble sizes 2, 5, 10, and 30. If recall and F1 gains never exceed the imbalance-only baseline beyond run-to-run noise, or if gains peak and then collapse as the ensemble grows, the uncertainty-gating branch is not doing the work and the imbalance branch alone would be the cause of the reported improvements.","tokens_in":25907,"feed_emoji":"🧬","tokens_out":6794,"duration_ms":71308,"temperature":0.7,"pith_summary":"The paper addresses why hierarchical multi-label models stop short of fine-grained classes: child nodes are almost always rarer than their parents, and existing fixes that resample whole observations end up re-weighting the already-common parent labels. The authors propose to weight each node by its own frequency, with a floor so common nodes still learn, and to add a focal term that increases the loss on nodes where an ensemble of models is uncertain. On gene-function benchmarks this raises recall of rare nodes by up to a factor of five over existing methods and gives statistically significant F1 gains; the benefit is strongest when encoders are weak or data is scarce. The paper argues that a node-based lens, not an observation-based one, is the right way to handle hierarchical imbalance.","feed_headline":"Rare-node recall rises fivefold with a reweighted hierarchical loss","feed_subtitle":"Weighting nodes, not data points, makes models predict fine-grained classes they previously ignored","key_machinery":"The carrying mechanism is a multiplicative reweighting of the C-HMCNN max-constraint loss. The imbalance branch computes, for each node, an inverse-frequency weight from the node's aggregate occurrence count (itself plus descendants), rescales these weights to a floor w0, and applies them only to positive annotations; this decouples emphasis from observation frequency, so a rare child node can be up-weighted without inflating its common parent. The focal branch multiplies in an uncertainty factor (U0 + U^k), where U is measured from a model ensemble without gradients, so the loss concentrates on nodes where the ensemble is unsure. The asymmetry of weighting only positive labels and the floor","core_discovery":"The central claim is that the loss L_focal = (W0 + W(Y,f)) (U0 + U(Theta(X))^k) LMC, built on top of the C-HMCNN max-constraint loss, lets a hierarchical multi-label model learn rare descendant nodes it would otherwise ignore. W is a node-wise inverse-frequency weight, rescaled to a minimum gate W0 and applied only to positive annotations; U is an ensemble-derived uncertainty, with bBMA and GMU as the recommended candidates, gated by U0 and sharpened by exponent k. The paper reports recall gains up to a factor of five on gene-function benchmarks, statistically significant F1 improvements, and similar trends on text and image hierarchies; gains are largest exactly where rare-node detection is","pith_inferences":["If the mechanism is right, the node-wise plus uncertainty-gated weighting should transfer to any hierarchical classifier with a coherent loss, not just the C-HMCNN base used here; wrapping a different hierarchical loss in the same two factors is a direct test.","The precision-for-recall trade-off at low w0 is a tunable knob: screening applications that tolerate false positives could push w0 toward zero to maximize rare-node recall, while higher floors suit precision-sensitive tasks; a per-node or per-depth schedule might beat the single global floor.","Because gains are largest in low-data and weak-encoder regimes, the uncertainty signal could double as an annotation-acquisition or active-learning cue, pointing at nodes where new labels would help most; the paper does not test this use."],"forward_implications":["Models trained with the combined loss begin predicting deep hierarchy nodes they never predicted before; rare-node recall rises by up to five times over unweighted and resampling baselines on gene-function datasets.","Node-wise imbalance weighting at a floor of 0.25 delivers statistically significant F1 gains, and combining it with resampling pushes F1 higher on most gene-function datasets while also improving recall.","The uncertainty-gated focal branch adds further F1 gains over imbalance weighting alone when the ensemble is large enough, experimentally ten members, while epistemic KL and JS uncertainty terms do not reliably help.","The benefit concentrates in hard regimes: partially trained or noisy encoders and severely limited training data; with a fully pretrained encoder, rare-node gains remain but are offset by small degradations on common nodes.","The same pattern transfers beyond gene annotation to a text hierarchy and two image hierarchies, suggesting the weighting is not tied to a particular input modality."],"fun_headline_variants":["Rare-node recall up 5x with hierarchy-aware reweighting","New loss makes models find rare classes in hierarchies","Hierarchical loss targets rare nodes, boosting recall 5x","Focal loss on rare nodes: 5x recall in hierarchies","Node-weighting, not data-weighting, lifts rare-class recall 5x"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the ensemble uncertainty computed from the very models being trained stays a reliable, non-collapsing signal for which nodes need emphasis while that same signal is being used to modify the loss; the paper's own results show epistemic uncertainty variants underperform and focal gains appear clearly only with ten or more ensemble members.","fun_headline_variants_meta":{"raw":{"variants":["Rare-node recall up 5x with hierarchy-aware reweighting","New loss makes models find rare classes in hierarchies","Hierarchical loss targets rare nodes, boosting recall 5x","Focal loss on rare nodes: 5x recall in hierarchies","Node-weighting, not data-weighting, lifts rare-class recall 5x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000487,"raw_usage":{"total_tokens":2209,"prompt_tokens":688,"completion_tokens":1521,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":1431}},"tokens_in":432,"tokens_out":1521,"duration_ms":11919,"temperature":1.0,"reasoning_tokens":1431,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:03:13.553558+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed benchmark, compare imbalance-only training against the full focal loss at ensemble sizes 2, 5, 10, and 30. If recall and F1 gains never exceed the imbalance-only baseline beyond run-to-run noise, or if gains peak and then collapse as the ensemble grows, the uncertainty-gating branch is not doing the work and the imbalance branch alone would be the cause of the reported improvements.","supporting_citations":[],"review_version":1}