Pith. sign in

REVIEW 4 major objections 5 minor 17 references

Enhancing Health Mention Classification Performance: A Study on Advancements in Parameter Efficient Tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read The paper claims that parameter-efficient tuning, especially prompt tuning combined with part-of-speech information, improves health mention classification across three social media datasets while updating far fewer parameters than full…

desk verdict Useful internal PEFT comparison that overclaims state-of-the-art: no external baselines, self-made splits, and no code make the headline impossible to verify. read the letter →

arxiv 2504.21685 v1 pith:LJOQDNOZ submitted 2025-04-30 cs.CL cs.AI

classification cs.CLcs.AI
keywords healthmentionclassificationparameter-efficientfine-tuningprompttuningsoftpromptingprefixLoRApart-of-speechtaggingsocialmediapublic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that health mention classification on social media can be improved not by scaling up models but by tuning only a small set of parameters and by giving the model explicit part-of-speech information. The authors compare prompt tuning, soft prompting, prefix tuning, and LoRA-style adapters against full fine-tuning of BERT and RoBERTa on the PHM2017, RHMD, and Illness datasets. Their central result is that prompt tuning, in some cases combined with POS information, achieves the highest F1 scores on all three datasets while updating far fewer parameters than full fine-tuning. They further claim these results beat prior published systems for health mention classification. If correct, this would make public health monitoring from Twitter and Reddit cheaper and easier to deploy, because one small generic model could serve many tasks.

What carries the argument

The central mechanism is the reformulation of classification as masked language modeling through prompt templates: the text is wrapped in a prompt such as "What is the health mention class? [MASK]." and the model's prediction for the [MASK] token is mapped through a verbalizer to a label. This is combined with parameter-efficient tuning techniques, prompt tuning, soft prompting with learnable virtual tokens, prefix/P-tuning V2, and LoRA, that keep the pretrained transformer frozen except for small trainable components. A second mechanism is POS injection, either by intermediate-task fine-tuning, first training on POS tagging and then on HMC, or by representation fusion, concatenating representations from the original model and a POS-tuned model before classification. These pieces let the model exploit grammatical dependencies that often decide whether a disease word refers to a real personal health condition.

What would settle it

A reader could settle the central claim by taking the best configurations, prompt tuning on PHM2017 and Illness and intermediate POS with prompt tuning on RHMD, and running them on the standard or original splits used by the prior published systems, then comparing F1 to the numbers those systems reported; if the scores fall back to or below the prior results, the claim that PEFT outperforms state-of-the-art health mention classification would not survive.

Watch

Extended reading notes

Core claim

The paper's core claim is that parameter-efficient tuning with prompt-based inputs is a superior strategy for health mention classification than conventional full fine-tuning. In the reported experiments, prompt tuning reaches an F1 of 91.5 on PHM2017, intermediate POS with prompt tuning reaches 82.2 on RHMD, and prompt tuning reaches 95.5 on Illness, all above the fine-tuning baselines of 90, 79-80, and 91 respectively. The paper also claims that injecting part-of-speech knowledge, either by first fine-tuning on a POS task or by fusing POS-tuned representations, lifts performance by roughly 1 to 3 percent, and that two modifications of soft prompting, placing virtual tokens around the text and combining soft tokens with a hard prompt, stabilize training and improve on plain soft prompting. These findings are presented as evidence that health mention classification can be done accurately without updating large language model bodies.

Load-bearing premise

The load-bearing assumption is that the authors' self-made random 70/30 splits are interchangeable with the evaluation protocols used by the earlier systems they claim to beat, since no official splits are used.

Editorial extensions

If this is right

  • On the paper's reported numbers, prompt tuning is the strongest single technique, with F1 of 91.5 on PHM2017, 80.9 on RHMD, and 95.5 on Illness, beating full fine-tuning on every dataset.
  • Adding POS information in the best configuration raises F1 by about 1 to 3 percent, and on RHMD the best result, 82.2, comes from intermediate POS combined with prompt tuning.
  • Wrapping text with soft virtual tokens and mixing them with a hard prompt improves plain soft prompting by 2 percent or more on RoBERTa-based models.
  • Cross-dataset masked-language-model pre-training performs within about 0.5 percent of fully in-domain training, suggesting domain shift can be handled without large task-specific pretraining corpora.
  • Because the trainable parameters are only a small set of prompts, prefixes, or adapters, one pretrained model can serve many health-classification tasks with low memory and training cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported gains hold under the original evaluation splits of PHM2017, RHMD, and Illness, then combining prompt tuning with POS cues could become a cheap default for other noisy social-media classification tasks, such as detecting misinformation or adverse drug reactions.
  • The paper compares against prior published systems under its own random 70/30 splits; a stronger test would be a shared-benchmark evaluation where every method sees the same test folds, which would reveal whether the improvement is real or split-dependent.
  • The success of wrapping soft prompts around the text suggests that the position of virtual tokens matters, and a natural extension is to test whether the same two modifications help on other sentence-level classification tasks beyond health mentions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies parameter-efficient fine-tuning (PEFT) techniques, including prompt tuning, soft prompting, prefix tuning, and LoRA, for health mention classification on three datasets: PHM2017, RHMD, and Illness. It also explores two ways of incorporating part-of-speech information: intermediate-task fine-tuning on POS and representation fusion with a POS-tuned model. The main empirical result is that prompt-tuning reaches the highest F1-micro scores on all three datasets (91.5, 80.9, and 95.5), outperforming the in-house BERT/RoBERTa fine-tuning baselines by small to moderate margins. The paper claims in the abstract and conclusion that these results outperform state-of-the-art methods, but no previously published state-of-the-art system is included in the comparison table, and the evaluation uses custom random 70/30 splits rather than the original protocols used in prior work.

Significance. If validated under the standard evaluation protocols of the prior literature, the paper would provide a practically useful result: prompt tuning combined with POS information can beat full fine-tuning for health mention classification while updating far fewer parameters. The internal experiment design is coherent in that all variants share the same optimizer, scheduling, and cross-validation setup, and the exploration of two POS incorporation strategies and modified soft prompting is a reasonable empirical contribution. However, the central state-of-the-art claim is unsupported as presented because Table 5 contains no external prior system and the custom splits may not be comparable to the splits used in the papers cited as state of the art. The paper does not provide code or split files, so the reported numbers are not independently reproducible at present.

major comments (4)
  1. [Section 4.1, Table 5, Abstract] The abstract states that PEFT techniques 'significantly improves performance in terms of F1-score compared to state-of-the-art methods across all three datasets,' but Table 5 lists only in-house fine-tuning baselines (BERT and RoBERTa). No prior published HMC system from the Related Work section appears in the comparison. Section 4.1 further states: 'We created our own splits because the dataset providers did not supply them... random 70%/30% split.' Since the authors do not show that their random splits match the train/test partitions used by Karisani and Agichtein (2018a), Naseem et al. (2022), or Karisani et al. (2022), every 'outperforms state-of-the-art' claim in the abstract and conclusion is unverifiable as written. Please rerun the experiments on the original evaluation protocols of the cited datasets, or explicitly reimplement prior systems on the same splits, and compare against those numbers; otherwise the state-of-the-art claim must be removed or replaced with a claim about outperforming the in-house baselines.
  2. [Section 4.1, Tables 2-4] No variance estimates, confidence intervals, or significance tests are reported. Several design conclusions rest on differences of 0.1 to 0.5 F1 points, e.g., Table 2 reports 95.4 vs 95.5 for two verbalizers on the Illness dataset and Table 3 reports 80.8 vs 80.9 on RHMD. Such small gaps are within typical run-to-run noise for transformer fine-tuning, so the verbalizer and prompt selections are not yet supported without repeated runs or statistical testing.
  3. [Section 5, contribution 2] The paper claims that incorporating grammar-aware language models improves performance 'by a magnitude ranging from 1% to 3%.' This is not consistent with the numbers in Table 5: prompt tuning over fine-tuning gives gains of about 1.5 points on PHM2017, 0.9 points on RHMD, and 4.5 points on Illness. Please specify exactly which comparison supports the 1%-3% range and cite the corresponding table rows.
  4. [Section 3.2.2, Table 5] The domain-adaptation experiments are incompletely reported. Table 5 has missing entries ('-') for the PHM2017 dataset under 'pre-training then fine-tuning' and 'pre-training then prefix-tuning,' and the text does not state how much pre-training data is used or whether the two auxiliary datasets are concatenated. Consequently, the Section 4.2 claim that multiple-source domain adaptation achieves a 'performance drop of around 0.5%' cannot be verified from the presented results.
minor comments (5)
  1. [Abstract and Section 3.1] The dataset name is written as 'RHDM' in the abstract but as 'RHMD' in Section 3.1, Table 1, and elsewhere in the paper; please use one consistent spelling.
  2. [Section 2, Section 3.2, Table 1] There are several typos: 'a a new dataset' in Section 2, 'the a lack' in Section 3.2, 'NonNon-Health Mentions' in Table 1, and 'gives an insights' in Section 4.2. These should be corrected.
  3. [Sections 3.2.3 and 6] The abbreviation 'LoRa' should be 'LoRA' for consistency with the cited work by Hu et al. (2021).
  4. [Abstract and Section 3.2] The abstract calls the models 'biomedical natural language methods (NLP),' but the experiments use general-domain BERT and RoBERTa, which are not biomedical models. Please clarify the terminology or use biomedical PLMs such as BioBERT if that is intended.
  5. [References] References Karisani and Agichtein (2018a) and (2018b) appear to be the same paper with the same title but different venue formatting; please merge or clearly distinguish them.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical PEFT/POS comparisons are evaluated on held-out test splits and do not reduce to their inputs.

full rationale

The paper's claims are empirical: it trains PEFT variants (prompt-tuning, soft-prompting, prefix-tuning, LoRa) and POS-augmented variants on three HMC datasets and reports F1 on a held-out test split. The target scores are not defined in terms of the method hyperparameters or POS features, and no fitted parameter is relabeled as a prediction. The PEFT techniques are standard published methods (Lester et al. 2021; Li and Liang 2021; Liu et al. 2021; Hu et al. 2021), and the POS incorporation schemes are explicit pipelines (intermediate-task fine-tuning and representation fusion) rather than restatements of the evaluation metric. There are no load-bearing self-citations: the reference list contains no prior work by the present authors, so no uniqueness theorem or adopted ansatz is imported from the authors' own earlier papers. The principal weakness is that Section 4.1 says 'We created our own splits because the dataset providers did not supply them' with a 'random 70%/30% split', and Table 5 compares only to in-house fine-tuning baselines rather than published SOTA numbers. This makes the abstract's 'compared to state-of-the-art methods' claim unverified, but that is an evaluation-protocol and baseline-comparison validity concern, not circular reasoning. Under the provided rubric, non-circular overclaiming is scored as no circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on hand-selected prompts, verbalizers, architecture choices, and an ad hoc evaluation split. The BERT/RoBERTa models and dataset annotations are assumed from prior work; no code, splits, or full hyperparameters are released. The main unstated premise is that the self-created splits are comparable to the published evaluation protocols, which the paper does not justify.

free parameters (4)
  • Per-dataset prompt template = e.g., 'What is the health mention class? [MASK].' for RHMD and PHM; 'Is a person diagnosed with disease?
    Prompt templates are hand-selected per dataset and change F1 by up to 1% (Tables 2 to 4).
  • Per-dataset verbalizer set = e.g., 'Figure, Non, Health' vs 'FM, NM, HM'; 'Yes, No' vs 'Positive, Negative'
    Verbalizer choices shift F1 by 0.5 to 0.7% and were chosen after comparison.
  • Number of frozen layers per dataset = RHMD: 6, Illness: 2, PHM: embedding only
    Reported in Section 4.1; freezing choices are arbitrary and affect PEFT capacity.
  • Soft prompt length / LoRA rank = Not reported
    These are standard PEFT hyperparameters that materially affect results; omitting them makes reproduction impossible.
assumptions (4)
  • ad hoc to paper Custom random 70/30 splits are a valid substitute for the standard splits used in prior work.
    Section 4.1 introduces the splits; all SOTA comparisons in the paper depend on this premise.
  • ad hoc to paper BERT and RoBERTa are suitable 'biomedical natural language methods'.
    The abstract calls the models biomedical NLP methods, but BERT and RoBERTa are general-domain; no biomedical model (e.g., BioBERT, PubMedBERT) is used.
  • domain assumption The three datasets' annotations are used as-is without re-validation.
    The paper relies on existing labels for PHM2017, RHMD, and Illness.
  • domain assumption F1-micro on the test set is the appropriate metric and matches metric definitions in prior HMC work.
    Evaluation protocol in Section 4.1; comparability with prior numbers requires identical metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Health Mention Classification Performance: A Study on Advancements in Parameter Efficient Tuning." pith.science (2026). https://pith.science/paper/LJOQDNOZ

@misc{pith2026250421685,
  author       = {Pith},
  title        = {Pith review of: Enhancing Health Mention Classification Performance: A Study on Advancements in Parameter Efficient Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LJOQDNOZ}},
  note         = {Machine review of arXiv:2504.21685}
}
read the original abstract

Health Mention Classification (HMC) plays a critical role in leveraging social media posts for real-time tracking and public health monitoring. Nevertheless, the process of HMC presents significant challenges due to its intricate nature, primarily stemming from the contextual aspects of health mentions, such as figurative language and descriptive terminology, rather than explicitly reflecting a personal ailment. To address this problem, we argue that clearer mentions can be achieved through conventional fine-tuning with enhanced parameters of biomedical natural language methods (NLP). In this study, we explore different techniques such as the utilisation of part-of-speech (POS) tagger information, improving on PEFT techniques, and different combinations thereof. Extensive experiments are conducted on three widely used datasets: RHDM, PHM, and Illness. The results incorporated POS tagger information, and leveraging PEFT techniques significantly improves performance in terms of F1-score compared to state-of-the-art methods across all three datasets by utilising smaller models and efficient training. Furthermore, the findings highlight the effectiveness of incorporating POS tagger information and leveraging PEFT techniques for HMC. In conclusion, the proposed methodology presents a potentially effective approach to accurately classifying health mentions in social media posts while optimising the model size and training efficiency.

Figures

Figures reproduced from arXiv: 2504.21685 by the authors.

Figure 1
Figure 1. Prompt architecture for the RHMD dataset. ”SM” which refer to non-health mention, awareness, other mention, and self mention. Soft-Prompting Instead of adding a hard prompt, as in prompt-tuning, a soft prompt involves adding virtual learnable tokens (continuous prompts) with the input text. The input embedding sequence is ex￾pressed as [ h0,..., hi, e(x)], given the trainable continuous embeddings [h0,..., hi] repre… view at source ↗
Figure 2
Figure 2. Modified Soft-prompting architecture to include virtual tokens before and after the text. The second modification is to combine both prompt-tuning with soft prompting as shown in fig￾ure 3, so that the input sequence could be represented as [h0,..., hi, e(x), e(prompt), e(“[MASK]”)] given an input text x and a prompt [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages

  1. [1]

    Improving health mention classification through emphasising literal mean- ings: A study towards diversity and generalisation for public health surveillance

    Olanrewaju Tahir Aduragba, Jialin Yu, Alexandra Cristea, and Yang Long. Improving health mention classification through emphasising literal mean- ings: A study towards diversity and generalisation for public health surveillance. In ACM Web Con- ference 2023-Proceedings of the World Wide Web Conference, WWW 2023 , pages 3928–3936. ACM,

  2. [9]

    Prefix-tuning: Op- timizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Op- timizing continuous prompts for generation. arXiv preprint arXiv:2101.00190,

  3. [10]

    Parameter-efficient fine-tuning without introduc- ing new latency

    Baohao Liao, Yan Meng, and Christof Monz. Parameter-efficient fine-tuning without introduc- ing new latency. arXiv preprint arXiv:2305.16742 ,

  4. [11]

    P-tuning v2: Prompt tuning can be comparable to fine- tuning universally across scales and tasks

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine- tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602,

  5. [12]

    Sensitivity of adversarial perturbation in fast gradient sign method

    Yujie Liu, Shuai Mao, Xiang Mei, Tao Yang, and Xu- ran Zhao. Sensitivity of adversarial perturbation in fast gradient sign method. In 2019 IEEE sympo- sium series on computational intelligence (SSCI) , pages 433–436. IEEE,

  6. [13]

    Identification of disease or symp- tom terms in reddit to improve health mention clas- sification

    Usman Naseem, Jinman Kim, Matloob Khushi, and Adam G Dunn. Identification of disease or symp- tom terms in reddit to improve health mention clas- sification. In Proceedings of the ACM Web Confer- ence 2022, pages 2573–2581,

  7. [14]

    Tracking health re- lated discussions on reddit for public health appli- cations

    Albert Park and Mike Conway. Tracking health re- lated discussions on reddit for public health appli- cations. In AMIA annual symposium proceedings , volume 2017, page

  8. [16]

    Exploring Domain Shift in Extractive Text Summarization

    Danqing Wang, Pengfei Liu, Ming Zhong, Jie Fu, Xipeng Qiu, and Xuanjing Huang. Exploring do- main shift in extractive text summarization. arXiv preprint arXiv:1908.11664,

Show all 17 references
  1. [17]

    Multi-layer representa- tion fusion for neural machine translation

    9 PEFT for HMC Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yin- qiao Li, and Jingbo Zhu. Multi-layer representa- tion fusion for neural machine translation. arXiv preprint arXiv:2002.06714,

  2. [2013]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 ,

  3. [2018]

    Did you re- ally just have a heart attack? towards robust de- tection of personal health mentions in social media

    Payam Karisani and Eugene Agichtein. Did you re- ally just have a heart attack? towards robust de- tection of personal health mentions in social media. In Proceedings of the 2018 World Wide Web Con- ference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018 , pages 1...

  4. [2019]

    Lora: Low-rank adapta- tion of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adapta- tion of large language models. arXiv preprint arXiv:2106.09685,

  5. [2020]

    Rethinking why intermediate-task fine-tuning works

    Ting-Yun Chang and Chi-Jen Lu. Rethinking why intermediate-task fine-tuning works. arXiv preprint arXiv:2108.11696,

  6. [2021]

    Figurative usage detection of symptom words to improve per- sonal health mention detection

    Adith Iyer, Aditya Joshi, Sarvnaz Karimi, Ross Sparks, and Cecile Paris. Figurative usage detection of symptom words to improve per- sonal health mention detection. arXiv preprint arXiv:1906.05466,

  7. [2022]

    Practical transformer-based multilingual text classification

    Cindy Wang and Michele Banko. Practical transformer-based multilingual text classification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies: Industry Papers, pages 121–129,

  8. [2023]

    Leveraging sentiment distri- butions to distinguish figurative from literal health reports on twitter

    Rhys Biddle, Aditya Joshi, Shaowu Liu, Cecile Paris, and Guandong Xu. Leveraging sentiment distri- butions to distinguish figurative from literal health reports on twitter. In Proceedings of the web con- ference 2020, pages 1217–1227,

  9. [2025]

    Sep- arating fact from fear: Tracking flu infections on twitter

    Alex Lamb, Michael Paul, and Mark Dredze. Sep- arating fact from fear: Tracking flu infections on twitter. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 789–795,

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.