{"id":"e95e5839-248b-483c-a39f-06902cec9550","arxiv_id":"2502.06811","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Adding a human-attention similarity term to a transformer loss improved accuracy on small, imbalanced datasets in two NLP tasks, but the printed loss equation has the wrong sign for the stated goal.","lead":"This paper introduces HuMAL, a method that feeds human word-highlighting annotations into transformer models as an extra training signal. On sentiment and personality tasks, the best variant improved classification when training data was tiny or imbalanced, though the paper's core equations contain apparent sign and algebra errors.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 2's plus sign makes the printed HuMAL-AL objective minimize cosine similarity with human attention, contradicting Table V; the reported alignment gain cannot arise from the stated loss.","rationale":"The reader's weakest assumption identifies exactly the load-bearing defect: the printed loss in Eq. 2 has the wrong sign for alignment. I read the paper in good faith: the empirical program is coherent, the low-resource setting is a sensible place to expect human attention to help, and the reported improvements are not a priori impossible. What breaks the argument is the internal inconsistency between the stated objective and the reported outcome. Since Table V says the model's attention became closer to human attention (cosine similarity rose to ~0.45) while Eq. 2 would require the optimizer to push cosine similarity down, the experiments must have used a different loss than the one the paper defines. Without released code, no reader can determine which objective actually produced Tables III-VI. The same sign error appears in Eq. 4 for HuMAL-AP, and Eq. 3 for HuMAL-AN is algebraically suspect (it drops alpha and mixes an attention vector with hidden states). These are not stylistic or presentational complaints; they affect whether the central claim is well-defined. The paper could become publishable after correcting the equations, adding the missing baselines, and releasing code, but as submitted the central mechanism is not supported. I therefore see no reason to move away from the reader's REJECT verdict, though the rejection is for correctability rather than for fraud or implausibility.","tokens_in":15962,"tokens_out":6011,"duration_ms":54237,"concrete_test":"Obtain the authors' code or reconstruct the training script, then run the highest-gain condition (Yelp, training size = 250, 1% imbalance) for 20 bootstrap replicates under two losses: (a) exactly Eq. 2 with alpha = 2 (CE + alpha*cos), and (b) CE - alpha*cos. Under (a), if the attention cosine similarity fails to increase or the AUC does not reach the reported 0.84, while (b) reproduces Tables III and V, the published equations do not describe the implemented method. If neither reproduces, inspect the token-to-word aggregation and the choice of attention layer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism of HuMAL-AL is attention regularization, but Eq. 2 as printed is L = sum_j CE(y_j,p_j) + alpha * cos(A^h_j, A^m_{j,11}), with alpha = 2 fixed in Section IV-C. Minimizing this loss minimizes the cosine similarity between human and machine attention, driving cos toward -1 rather than toward alignment. Table V reports mean cosine similarity rising from 0.27/0.19 under BERT to 0.45-0.48 under HuMAL-AL; that is impossible under Eq. 2 for any positive alpha unless the implemented objective was different, e.g., CE - alpha*cos or CE + alpha*(1-cos). The same sign issue applies to HuMAL-AP's Eq. 4. Because no code is released, the reader cannot verify whether a corrected loss was used. If a corrected loss was used, the equations misstate the method; if not, the reported AUC and attention-similarity improvements in Tables III, V, and VI are unexplained. Either way, the paper's central claim that the stated cosine-similarity regularization produces the gains is not supported as written. This is an internal inconsistency between the optimization objective and the reported outcome, not a disagreement about external consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HuMAL, three strategies for injecting human word-level attention annotations into transformer fine-tuning, and evaluates them on Yelp sentiment analysis and myPersonality introvert/extrovert classification. The flagship variant HuMAL-AL adds a cosine-similarity term between human attention and BERT's last-layer attention to the cross-entropy loss; the paper reports that HuMAL-AL improves AUC over fine-tuned BERT, especially with 1–5% imbalance and 250–500 training instances, and increases human–machine attention cosine similarity from about 0.19–0.27 to 0.45–0.48. The other two variants, HuMAL-AN and HuMAL-AP, give smaller or no gains. The paper also presents data-efficiency analyses and GPT-2/XLNet replications.","tokens_in":16222,"tokens_out":6757,"duration_ms":61461,"significance":"If the results are valid, the work would offer a cost-effective way to leverage cheap self-reported highlights to improve low-resource classification, and it would contrast with earlier negative results from eye-tracking/EEG and lexicon-based guidance. Strengths include the use of existing human-attention data on Yelp (Sen et al., 2020), comparison across three transformer architectures, replication across two tasks, and bootstrapped significance testing. However, the central objective as printed cannot produce the reported attention-alignment results, so the significance is conditional on a substantial correction and on code release.","major_comments":[{"comment":"The loss is written as L = Σ_j CE(y_j, p_j) + α·cos(A^h_j, A^m_{j,11}), with α=2 stated in Section IV-C. Since training minimizes the loss, the plus sign makes the cosine-similarity term push cos toward −1, i.e., it actively penalizes alignment with human attention. This contradicts the design statement in Section III that the model is 'penalized for deviating from human attention,' and it makes the increases in Table V (from 0.27/0.19 to 0.45–0.48) impossible under the printed loss. If the implemented objective was instead CE − α·cos or CE + α·(1−cos), the equations misstate the method, and no code or supplementary material is provided to resolve the ambiguity. This is a load-bearing internal inconsistency for the central mechanism.","section":"Section III-A, Eq. (2) and Section III-C, Eq. (4)"},{"comment":"The first expression states E_{j,11}^{AN} = (A^m_{j,11} + α A^h_j) × E_{j,10}, but the second equality gives E_{j,11} + A^h_j × E_{j,10}, dropping both the α coefficient and the role of A^m_{j,11} in the decomposition. If A^m_{j,11} E_{j,10} is meant to be E_{j,11}, the α should still appear in the second term. The equation also does not specify whether A^h_j is normalized to match the scale of A^m_{j,11}. As written, Eq. (3) is algebraically incorrect.","section":"Section III-B, Eq. (3)"},{"comment":"The text claims the performance benefit is 'particularly pronounced' under high class imbalance and small datasets, but in the personality task the imbalance rows show no significant improvement for HuMAL-AL over BERT (0.59 vs 0.57 and 0.62 vs 0.60, neither starred). The claim is only supported by the sentiment task and by the balanced personality condition; the summary and abstract should be scoped accordingly.","section":"Section V-A, Table III(b)"},{"comment":"The attention similarity reported for HuMAL-AL is computed against the very same human annotations that were used as the training target in the HuMAL-AL loss. The increase from BERT to HuMAL-AL is therefore a direct consequence of the objective and does not constitute independent evidence of human-like attention. It should be described as a manipulation check, with the held-out label AUC in Tables III and VI serving as the primary evidence.","section":"Section V-E, Table V"}],"minor_comments":[{"comment":"The table uses 'HuMAL3-xlnet' in the header and rows, while the text and other tables use 'HuMAL-AL-xlnet'; please standardize the naming.","section":"Table VI"},{"comment":"The table has layout problems: Yelp-100 and Yelp-200 rows contain misplaced 'Not enough data' cells and missing or duplicated entries, making the results difficult to parse.","section":"Table IV"},{"comment":"The normalization formula writes 'Pi i=0' as a product symbol; it should be a summation over i, i.e., Σ_i.","section":"Eq. (5)"},{"comment":"The text states results were consistent 'regardless of whether human attention was considered binary (first labeler only) or continuous,' but no comparison of binary vs. continuous conditions is reported in any table; either add the analysis or remove the claim.","section":"Section V-A"},{"comment":"The test split and test-set size for the myPersonality task are not specified; Table II only lists training instances. Please clarify how the evaluation set was constructed and how the imbalance/balance categories map to it.","section":"Section IV-B"},{"comment":"There are several typographical errors, including 'Our goal in aligning machine attention ... to enhance' missing a verb in Section II and an extra period after 'performance..' in Section VI.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The sign error in Eq. (2) is severe enough that I would not accept the paper without the exact corrected loss and code release. If the authors cannot provide the precise objective used in the experiments, the paper should be rejected. The revision should also temper the imbalance-related claims for the personality task and treat Table V strictly as a manipulation check."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper has a promising premise and a large, careful evaluation, but the central math as printed contradicts the reported results. Equations 2 and 4 add alpha times cosine similarity to the cross-entropy. Minimizing that sum drives cosine similarity toward -1, i.e., away from human attention. Yet Table V reports cosine similarity increasing from 0.27/0.19 to 0.45-0.48 under HuMAL-AL. That is not possible under the stated objective. No code is released to show whether the experiments used a corrected loss, so the main mechanism is unsupported as written.\n\nWhat the paper does well: it systematically compares three distinct integration strategies (loss, normalizer, prior) for task-specific self-reported attention, across three architectures and two tasks. The low-resource and imbalanced framing is practical, and the extra analyses on text length and loss convergence are thoughtful. If the loss sign is a typo, this could become a genuinely useful empirical contribution.\n\nThe soft spots go beyond the sign error. Eq. 3 mixes attention vectors with the embedding vector in a way that isn't dimensionally consistent, and the 'simplification' drops alpha and the machine-attention term. The authors cite McGuire and Tomuro (2021) and Zou et al. (2018) but do not benchmark against them. Table V's similarity measure uses the same annotations as the training signal, so part of that increase is expected, not independent validation. And in the personality task, HuMAL-AL only improves over BERT in the balanced setting, which undercuts the headline about imbalance.\n\nThe central flaw is load-bearing. It is not a minor equation typo you can wave away, because the reported alignment gain would be impossible under the printed loss. That said, the rest of the paper looks like honest empirical work, and the fix is plausibly simple (change the sign or use 1-cos), with code and the missing baselines added.\n\nI would send this to peer review rather than desk reject — a referee can force the equations to match the implementation and require artifacts. For my own citing purposes, not in current form. Reading group? Maybe, if you want a concrete example of a results section and an optimization objective contradicting each other.","headline":"Equation 2 as printed optimizes against human attention, not toward it, so the paper's central claim is unsupported as written — though the empirical setup is extensive and the fix may be straightforward.","tokens_in":16745,"tokens_out":2860,"would_cite":false,"duration_ms":25314,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HuMAL-AL, a regularizer that aligns BERT's last-layer attention with human word highlights, improves text classification AUC when labels are sparse and imbalanced.","keywords":["human attention","machine attention","attention alignment","BERT fine-tuning","imbalanced text classification","sentiment analysis","personality classification","supervised learning"],"falsifier":"Train BERT on a small Yelp subset with the loss exactly as printed in Equation 2 and log the cosine similarity between human and machine attention at each epoch: if the equation is literal, similarity should decrease toward -1, contradicting the paper's reported rise from 0.27 to 0.45. Inspecting the released training code for the sign of the attention term would settle it directly.","tokens_in":15770,"feed_emoji":"🧠","tokens_out":7223,"duration_ms":58794,"temperature":0.7,"pith_summary":"The paper claims that adding a human-attention alignment term to the fine-tuning loss of a transformer improves classification, and that the gain is largest exactly where data is hardest: 1–5% class imbalance and only 250–500 labeled instances. On Yelp sentiment, the HuMAL-AL variant raises mean AUC from 0.64 to 0.84 at 250 instances and 1% imbalance, and from 0.75 to 0.94 at 500 instances; it also beats fine-tuned BERT on introvert–extrovert classification of myPersonality posts. The paper reads this as evidence that cheap, task-specific human highlights can stand in for larger labeled datasets in low-resource text classification. A sympathetic reader would care because the annotations are simple word-level highlights from three crowd workers, not eye-tracking or brain data, and the same recipe transfers to GPT-2 and XLNet.","feed_headline":"Human highlights lift BERT AUC to 0.84 on sparse sentiment","feed_subtitle":"Aligning attention with cheap word highlights needs hundreds fewer labeled reviews than fine-tuned BERT.","key_machinery":"The load-bearing object is the HuMAL-AL loss, written in the paper as $L = \\sum_j \\mathrm{cross\\_entropy}(y_j, p_j) + \\alpha \\cdot \\cos(A^h_j, A^m_{j,11})$, with $\\alpha=2$. The machine attention vector $A^m_{j,11}$ comes from averaging, over the 12 heads, the CLS token's attention to each token in BERT's last layer (Equation 1); the human vector $A^h_j$ is the sum of binary highlight decisions across annotators, normalized to sum to 1 (Equation 5). This couples the classification objective to where the model looks, rather than only what it predicts. The paper also defines two alternatives: HuMAL-AN blends human attention into the final embedding, and HuMAL-AP injects human attention as a prior on the first layer instead of the last.","core_discovery":"On the paper's own terms, the central discovery is that HuMAL-AL—the strategy that adds a cosine-similarity term between the human attention vector and BERT's last-layer attention to the classification cross-entropy—improves AUC over fine-tuned BERT, and that the gain is largest at 1–5% imbalance with 250–500 training instances. The authors report this advantage on Yelp sentiment and myPersonality introvert/extrovert classification, across BERT, GPT-2, and XLNet backbones. They also report that HuMAL-AL reaches baseline-level AUC with roughly 484 fewer labeled reviews at 5% imbalance, and that cosine similarity between human and machine attention rises from about 0.27 to 0.45. The explanation offered is that task-specific human attention marks sentiment- and personality-relevant words, and the model is regularized toward those marks.","pith_inferences":["If the sign of the attention term is corrected, the mechanism is a soft constraint that pulls the model's focus onto the same roughly 13–21% of words humans mark, suggesting that a three-highlighter annotation protocol could substitute for thousands of labels in other low-resource text tasks.","The paper fixes the alignment target to the last layer, and its own first-layer variant HuMAL-AP performs worse, so a natural extension is to test whether the last layer is the best alignment point or whether layer choice should depend on task depth.","Because human attention agreement drops on longer texts and HuMAL's gain drops too, the regularization weight $\\alpha$ could be made to decay with text length or annotator agreement, a design the paper does not test.","The personality results are noisy (AUC standard deviations around 0.1), so a follow-up with more instances and a pre-registered sign for the attention loss would clarify whether the reported effect is robust."],"forward_implications":["At 1% imbalance and 250 training instances on Yelp sentiment, HuMAL-AL raises mean AUC from 0.64 (BERT) to 0.84; at 500 instances it raises it from 0.75 to 0.94.","At 5% imbalance, HuMAL-AL matches BERT while needing about 484 fewer labeled reviews on average.","The gains transfer to GPT-2 and XLNet as base models, with the largest improvements again at 1% imbalance and small sample sizes.","On personality classification from myPersonality posts, HuMAL-AL improves balanced AUC from 0.68 to 0.74 with majority-vote labels and from 0.64 to 0.75 with self-reported labels.","The benefit shrinks with text length: for 101–200 word reviews, the gains at small sample sizes are smaller than for reviews up to 50 words."],"supporting_citations":[{"why":"Supplies the Yelp sentiment dataset and the crowdsourced human attention maps used in the sentiment experiments.","marker":"[Sen et al., 2020]"},{"why":"Provides the myPersonality Facebook posts and self-reported introvert/extrovert labels that the authors re-annotated for human attention.","marker":"[Stillwell and Kosinski, 2015]"},{"why":"Defines BERT, the base transformer whose layer-11 attention and CLS token HuMAL-AL regularizes.","marker":"[Devlin et al., 2018]"},{"why":"Introduces the transformer self-attention mechanism that the paper modifies to inject human attention.","marker":"[Vaswani et al., 2017]"},{"why":"Supports averaging attention over heads within a layer and the claim that lower-layer BERT attention is broadly distributed.","marker":"[Clark et al., 2019]"},{"why":"The closest cognitive-attention-supervision baseline, which supervised BERT attention with eye-tracking and EEG and found no classification gain.","marker":"[McGuire and Tomuro, 2021]"},{"why":"A lexicon-based attention alignment baseline that improved sentiment classification only slightly, which HuMAL-AL aims to beat.","marker":"[Zou et al., 2018]"}],"fun_headline_variants":["Aligning human and machine attention trims labeled data needs","Human attention cues sharpen BERT on sparse sentiment","Attention alignment lifts AUC with fewer labeled reviews","Human attention boosts BERT's sparse-data performance","Match human focus to cut labeled data needs in ML"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on the deployed attention loss actually maximizing similarity to human attention, whereas Equation 2 as printed adds the cosine term and would push similarity to -1.","fun_headline_variants_meta":{"raw":{"variants":["Aligning human and machine attention trims labeled data needs","Human attention cues sharpen BERT on sparse sentiment","Attention alignment lifts AUC with fewer labeled reviews","Human attention boosts BERT's sparse-data performance","Match human focus to cut labeled data needs in ML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000716,"raw_usage":{"total_tokens":3211,"prompt_tokens":933,"completion_tokens":2278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2215}},"tokens_in":549,"tokens_out":2278,"duration_ms":15766,"temperature":1.0,"reasoning_tokens":2215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T11:23:10.922156+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train BERT on a small Yelp subset with the loss exactly as printed in Equation 2 and log the cosine similarity between human and machine attention at each epoch: if the equation is literal, similarity should decrease toward -1, contradicting the paper's reported rise from 0.27 to 0.45. Inspecting the released training code for the sign of the attention term would settle it directly.","supporting_citations":[{"cited_title":"A lexicon-based supervised attention model for neural senti- ment analysis","cited_arxiv_id":null,"evidence_quote":"A lexicon-based attention alignment baseline that improved sentiment classification only slightly, which HuMAL-AL aims to beat."}],"review_version":1}