{"id":"5929d811-73fb-4a80-a19c-47ad88f8882e","arxiv_id":"2506.02005","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A pruned mBERT+BiLSTM model reaches 78% accuracy on a new 200-sentence Konkani metaphor test set and 83% on idiom classification, though the test set is very small and no error bars are given.","lead":"This paper adds a small set of metaphor-labeled Konkani sentences and tests a standard BERT-based classifier with attention-head pruning on a 200-sentence subset. The pruned model reports 78% accuracy for metaphor detection and 83% for idiom detection, but the test set is only 40 sentences.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pruning-sensitivity asymmetry rests on a four-sentence accuracy drop in a single 40-sentence test split; without variance or cross-validation the central conclusion is not established.","rationale":"The paper makes two contributions: a new metaphor-annotated Konkani dataset, and a pruning analysis that claims to reveal structural differences between idiom and metaphor processing. The dataset contribution is genuine: three native speakers verified labels and the corpus is released. However, the pruning analysis is the headline finding, and its evidential basis is a single 40-sentence test set. Because 0.88 to 0.78 is exactly 35/40 to 31/40 correct, the entire effect is four sentences. At that sample size, random label noise, a different random seed, or a slightly different split can readily change the conclusion. The reader's weakest assumption identified this same fragility, and I agree with that assessment. A conditional verdict is appropriate: the methods and data are worth publishing, but the task-specific structural claim should not be accepted as established until either more annotated data, cross-validation, or explicit uncertainty estimates are supplied. I am not raising a separate correctness objection; the limitations section is honest about the single-split issue, and the released artifacts should allow others to test the result. The concrete cross-validation check proposed above would settle whether the differential pruning result is real or a 40-sample artifact.","tokens_in":6944,"tokens_out":6688,"duration_ms":61611,"concrete_test":"Re-run the original-versus-pruned comparison under 10 random 80/20 splits of the 200 balanced sentences, or better, 5-fold cross-validation over the full 500 annotated sentences, fixing hyperparameters and the pruning criterion exactly as described. For each split, compute test accuracy and a paired McNemar p-value for original versus pruned on each task, and report a bootstrap 95% confidence interval on the accuracy difference. If pruned metaphor accuracy is not at least 5 points below original in at least 8 of 10 splits while idiom accuracy stays within 2 points, the task-specific sensitivity claim is not supported and should be reframed as preliminary. Also release the exact confusion matrix for the original Table 2 split so the four-example movement can be inspected directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that metaphor classification relies on a broader, more distributed set of attention heads and is therefore more pruning-sensitive than idiom classification. The evidence is Table 2: metaphor accuracy drops from 0.88 to 0.78 after pruning while idiom accuracy is stable. With the stated 80/20 split of 200 balanced sentences, the test set contains 40 items; 0.88 and 0.78 are exactly 35/40 and 31/40 correct. The entire metaphor-sensitivity signal is therefore four sentences. A paired McNemar test on 40 examples with roughly four discordant pairs cannot reach conventional significance, and the paper reports no seed variance, no confidence intervals, and no repeated splits. Section 3 also says only that the 200-sentence set was 'curated' from 500 annotated sentences without describing the selection rule, so the single split may not be representative. If one or two test sentences sit near the decision boundary, the qualitative result that idioms are robust while metaphors are fragile could invert. This is load-bearing: the article's main scientific conclusion is a difference in sensitivity that the reported evaluation cannot statistically resolve. The limitations section candidly acknowledges the single-split issue, yet Sections 6 and 7 state the conclusion categorically. The pruning mechanism itself is also underexplained, especially which data D supplies the gradients for the zero-importance scores, but the four-example delta is the most direct threat to the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a metaphor-annotated Konkani dataset built on the Konidioms corpus, and proposes a hybrid mBERT+BiLSTM model for metaphor and idiom classification. It applies a gradient-based attention-head pruning method (Michel et al., 2019) to both tasks. The authors report that after removing all heads with zero gradient importance (retaining 132 of 144 heads), idiom classification accuracy is stable (0.82 to 0.83), while metaphor classification accuracy drops from 0.88 to 0.78. From these results and the attention-head importance heatmaps, the paper concludes that idiom classification relies on localized lower-layer heads, whereas metaphor classification relies on a broader, more distributed attention profile and is therefore more sensitive to pruning. The paper releases its code and dataset.","tokens_in":7242,"tokens_out":3456,"duration_ms":31125,"significance":"If the reported pruning-sensitivity asymmetry is real, the finding that idiom and metaphor classification have distinct attention-head dependencies would be a useful contribution to low-resource figurative-language NLP. The paper also provides a new manually annotated Konkani metaphor dataset, with labels verified by native speakers, and it publicly releases code and resources, which are concrete assets for a severely under-resourced language. The ablation study across mBERT, IndiC-BERT, and XLM-R variants is a useful engineering comparison. However, the headline scientific claim rests on a very small test set and a four-sentence accuracy difference, so the significance is currently conditional on additional statistical validation.","major_comments":[{"comment":"The central claim that metaphor classification is more pruning-sensitive than idiom classification rests on a single test set of 40 sentences. The reported metaphor accuracy drop from 0.88 to 0.78 corresponds to exactly 4 sentence-level errors (35/40 to 31/40 correct). No confidence intervals, seed variance, cross-validation, or repeated splits are reported, and the 200-sentence balanced subset is described only as \"curated\" from 500 annotations without specifying the selection procedure. Because the qualitative asymmetry could invert with a small number of borderline test sentences, the paper must provide a more robust evaluation: e.g., k-fold cross-validation, bootstrap confidence intervals, or a paired McNemar test, and a precise description of how the 200-sentence subset was chosen.","section":"Section 3, Table 2"},{"comment":"The gradient-based importance score I_h is defined as an expectation over a dataset D, but the paper never specifies whether D is the training set, the validation set, or the test set. This is not a minor implementation detail: using the test set to compute importance and then evaluating on the same test set would leak information, while using the training set is the more defensible choice. The paper should state the exact composition of D and justify its use.","section":"Section 5"},{"comment":"The mBERT+BiLSTM architecture is selected after inspecting the ablation results in Table 3, which report both original and pruned accuracy for the chosen model and its baselines. Selecting the best-performing architecture on the same split that is later used for the headline comparison risks optimism bias and weakens the claim that the pruning effect is intrinsic to the task rather than an artifact of model selection. The authors should clarify whether the ablations were conducted on a held-out validation set and how the final test evaluation was kept separated from the selection process.","section":"Section 6, Appendix D (Table 3)"},{"comment":"The paper consistently describes the work as improving \"efficiency,\" yet no runtime, latency, memory, or parameter-count measurements are reported. The pruning removes 12 of 144 attention heads (8.33% of heads), which is a small fraction of the mBERT architecture, and the paper does not report how many parameters this actually removes or how it affects inference speed. The efficiency claim should either be backed by direct measurements or explicitly softened to \"small head-count reduction\" rather than general efficiency.","section":"Title, Abstract, Section 4"}],"minor_comments":[{"comment":"The sentence \"The contrasting patterns observed in the two classification tasks, suggests fundamental differences...\" contains a subject-verb agreement error; it should be \"suggest.\"","section":"Section 5.1"},{"comment":"The phrase \"after pruning 8.33% of parameters\" is imprecise; the paper prunes 8.33% of attention heads, not 8.33% of the total parameters. This distinction matters because attention heads are only a subset of the model's parameters.","section":"Section 6"},{"comment":"The model names in the first column are inconsistent: \"XLM-R + BiLSTM + AttnIdiom\" and \"XLM-R + BiLSTM + AttnMetaphor\" appear to combine the task name into the architecture name. This should be separated into a Model column and a Task column for clarity.","section":"Appendix D, Table 3"},{"comment":"The importance formula I_h = E_{(x,y)~D} |∂L/∂h(h)| is written in a confusing way: the argument of the gradient should be a head-indexed quantity, and the notation h(h) is unclear. Please define the head output function h and write the derivative either as ∂L/∂h_i or with an explicit notation such as h^{(i)}(x).","section":"Section 5"},{"comment":"The reference \"Paul Michel, Omer Levy, and Graham Neubig. 2019a. Are sixteen heads really better than one?\" is missing venue and page information; the 2019b entry is complete, but 2019a should be completed for consistency.","section":"References"},{"comment":"The caption for Figures 6 and 7 states that numerical decimal values are displayed, but the referenced figures are not visible in the text and their heatmap values are not otherwise provided; please include the figures or make them available in a readable resolution.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is plausible and the dataset contribution is real, but the difference between original and pruned metaphor accuracy is only four sentences on a 40-sentence test set. The authors' own Limitations section acknowledges the single-split issue, yet the Discussion and Conclusion state the finding categorically. If the authors cannot provide a more statistically grounded evaluation in revision, the asymmetry claim should not be presented as established. This is a borderline case; the current evidence is insufficient for acceptance, but the issue is fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper contributes a genuinely new, manually verified metaphor-annotated dataset for Konkani, but its central scientific claim—that metaphor classification is more pruning-sensitive than idiom classification—is not supported by the reported evaluation. The entire signal is four sentences on a single 40-sentence test split.\n\nWhat is new and good: the paper provides the first metaphor annotations for Konkani, extending the Konidioms corpus, with three native-speaker validators. The code and data are released, the limitations section is honest, and the ablation table comparing mBERT, IndicBERT, and XLM-R+BiLSTM+Attn gives useful context. Applying Michel et al.'s gradient-based head pruning to a low-resource figurative language task is a reasonable extension, and the paper is transparent that the architecture was chosen after seeing ablation results.\n\nThe load-bearing problem: Table 2's metaphor accuracy drop is 0.88 to 0.78, which on a 40-item test set is 35/40 to 31/40 correct—four examples. There are no confidence intervals, no repeated runs, no seed variance, and no cross-validation. The 200-sentence balanced subset was 'curated' from 500 without describing the selection rule, so the single split may not be representative. The limitations section candidly says 'evaluation on a single test split necessitates further validation,' but Sections 6 and 7 state the conclusion categorically. Also, the importance metric's dataset D is never specified (training set? validation set?), which matters for reproducibility. The efficiency gain is only 8.33% of heads (12 of 144), so calling the model 'efficient' oversells it.\n\nFor a reader: this is a resource/application paper, not a methodological breakthrough. The dataset, while tiny, may be useful for Konkani NLP and for studying figurative language in low-resource settings. The pruning-sensitivity conclusion should be treated as a hypothesis, not a finding. The paper deserves a serious referee because the resource contribution is real and the fragility can be addressed with variance reporting and cross-validation.","headline":"New Konkani metaphor annotations are a real but small resource; the pruning-sensitivity conclusion rests on a four-sentence accuracy drop in a single 40-sentence test split.","tokens_in":7755,"tokens_out":1862,"would_cite":false,"duration_ms":15811,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pruning zero-importance attention heads keeps Konkani idiom accuracy at 83% while dropping metaphor accuracy from 88% to 78%, and the paper attributes the difference to how the two tasks distribute attention across layers.","keywords":["Konkani NLP","metaphor classification","idiom classification","attention head pruning","mBERT","low-resource languages","figurative language","BiLSTM"],"falsifier":"Re-run the same pruning experiment on the released dataset with multiple 80/20 splits and several seeds; if the pruned metaphor accuracy gap versus the original is not consistently near 10 points and often falls within a couple of sentences' errors, the claim that metaphor classification is inherently more sensitive to attention-head pruning is refuted.","tokens_in":6780,"feed_emoji":"✂️","tokens_out":11161,"duration_ms":78850,"temperature":0.7,"pith_summary":"This paper tries to establish that gradient-based attention head pruning is a viable efficiency trick for Konkani figurative-language models, but only when the pruning target is chosen per task. The authors introduce a 500-sentence metaphor-annotated extension of the Konidioms corpus and fine-tune an mBERT+BiLSTM classifier on it and on idiom classification. After removing the 12 attention heads with zero gradient importance, idiom accuracy holds at 83% (up from 82%), while metaphor accuracy drops from 88% to 78% on a 200-sentence balanced subset. The paper attributes the asymmetry to attention structure: idioms rely on localized lower-layer heads, metaphors on a distributed set across layers. If correct, this gives low-resource Konkani a cheaper idiom model, a first metaphor benchmark, and a warning that one-size-fits-all pruning will not work for figurative language.","feed_headline":"Removing 12 mBERT heads leaves idioms at 83%, metaphors at 78%","feed_subtitle":"Konkani idioms survive pruning; metaphors drop 10 accuracy points, so pruning must be task-specific.","key_machinery":"The load-bearing mechanism is a gradient-based attention-head importance metric: for each of the 144 heads in mBERT, the model computes $I_h = \\mathbb{E}_{(x,y)\\sim D}|\\partial L / \\partial h(h)|$, the expected absolute gradient of the loss with respect to the head's output. Heads with $I_h = 0$ are removed post hoc (12 heads, 8.33% of Transformer parameters), while the BiLSTM is left unchanged. The argument turns on the contrast between the two tasks' importance heatmaps: idiom classification shows high scores mainly in the lower layers, while metaphor classification shows salient heads across all layers; the paper interprets this as the reason one task tolerates pruning and the other does not.","core_discovery":"The central claim is that pruning all mBERT attention heads whose gradient importance is exactly zero, 12 of 144 heads, has task-specific consequences for Konkani figurative-language classification. For idioms, the pruned mBERT+BiLSTM matches or slightly beats the original: accuracy goes from 0.82 to 0.83, recall from 0.89 to 0.91, and F1 stays at 0.88. For metaphors, the same pruning costs ten accuracy points (0.88 to 0.78), with F1 falling from 0.86 to 0.74. The paper explains the difference structurally: idiom-relevant heads cluster in the lower layers, while metaphor-relevant heads are spread across all layers, so removing any heads disrupts a more diffuse circuit. It also claims to release the first metaphor-annotated Konkani dataset and to apply attention head pruning for the first time to any Konkani NLP task.","pith_inferences":["I would not equate a zero gradient with functional redundancy: the paper's own numbers show that removing zero-score heads can still cost 10 accuracy points, so in small-data settings gradient magnitude at convergence is a weak proxy for a head's contribution.","A natural extension the paper does not run is a threshold sweep: pruning heads with small but nonzero importance values and tracking accuracy would show whether the metaphor drop is specifically about zero-score heads or about any head removal.","If the structural contrast (early-layer idioms, diffuse metaphors) replicates in other low-resource languages, attention-head importance maps could become a cheap pre-pruning diagnostic for figurative-language tasks."],"forward_implications":["Konkani idiom classification can be compressed by 12 attention heads with no accuracy loss: the pruned model reaches 83% accuracy and 0.88 F1, so the released pruning recipe is directly usable for idiom tasks.","Metaphor classification should not be pruned with the same zero-importance threshold: the same 12-head removal costs about 10 accuracy points, meaning pruning policies need to be task-specific.","Attention-head importance maps can act as a diagnostic: early-layer heads carry idiom signals while metaphor signals are distributed, giving a cheap way to predict whether a task will tolerate compression.","The 500-sentence metaphor-annotated subset, once released with the framework, gives future work a first Konkani benchmark for metaphor detection and a direct comparison point for other low-resource models."],"supporting_citations":[{"why":"Supplies the Konidioms corpus of Devanagari-script Konkani sentences that the paper extends with metaphor annotations and uses for idiom classification.","marker":"Shaikh et al. (2024)"},{"why":"Provides the neural idiom-classification model that the paper treats as the prior baseline to extend.","marker":"Shaikh and Pawar (2024)"},{"why":"Gives the BERT-based idiom-identification approach with custom loss functions that the paper positions as the state to build on.","marker":"Yayavaram et al. (2024)"},{"why":"Defines the pretrained multilingual BERT model whose 144 attention heads are pruned.","marker":"Devlin et al. (2019)"},{"why":"Supplies the gradient-based attention-head importance scoring that the pruning procedure is built on.","marker":"Michel et al. (2019b)"},{"why":"Extends attention-head pruning to multi- and cross-lingual tasks, motivating its use for a multilingual model on Konkani.","marker":"Ma et al. (2021)"},{"why":"Comparison showing BiLSTMs can outperform BERT on very small corpora, used to justify the hybrid mBERT+BiLSTM architecture.","marker":"Ezen-Can (2020)"},{"why":"Establishes gradients as feature-importance signals, an underpinning of the head-importance metric used for pruning.","marker":"Feng et al. (2018)"}],"fun_headline_variants":["Pruning mBERT: idioms gain, metaphors lose 10 points","Pruning 12 heads: Konkani idioms up, metaphors down 10","Task-specific pruning: mBERT idioms beat baseline, metaphors fall","Pruning mBERT heads: Konkani idioms 83%, metaphors 78%","Diffuse metaphor heads: pruning costs 10 points, idioms gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported accuracy gap is real, but it rests on a single 40-sentence test set where exactly four errors separate 88% from 78%, so a different split could erase the claimed difference.","fun_headline_variants_meta":{"raw":{"variants":["Pruning mBERT: idioms gain, metaphors lose 10 points","Pruning 12 heads: Konkani idioms up, metaphors down 10","Task-specific pruning: mBERT idioms beat baseline, metaphors fall","Pruning mBERT heads: Konkani idioms 83%, metaphors 78%","Diffuse metaphor heads: pruning costs 10 points, idioms gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3039,"prompt_tokens":880,"completion_tokens":2159,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2062}},"tokens_in":496,"tokens_out":2159,"duration_ms":14345,"temperature":1.0,"reasoning_tokens":2062,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:29:48.180928+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same pruning experiment on the released dataset with multiple 80/20 splits and several seeds; if the pruned metaphor accuracy gap versus the original is not consistently near 10 points and often falls within a couple of sentences' errors, the claim that metaphor classification is inherently more sensitive to attention-head pruning is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the BERT-based idiom-identification approach with custom loss functions that the paper positions as the state to build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes gradients as feature-importance signals, an underpinning of the head-importance metric used for pruning."}],"review_version":1}