Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Ustnlp16 at SemEval-2025 Task 9: Improving Model Performance through Imbalance Handling and Focal Loss

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EDA applied to minority classes is the strongest imbalance remedy tested, lifting BERT's product-category accuracy from 0.22 to 0.55 and macro F1 from 0.03 to 0.30.

desk verdict A workmanlike SemEval system paper whose headline claim is not statistically secured because the same hand-made 20% split is used both to pick and to score the winning configuration. read the letter →

arxiv 2505.00021 v1 pith:RBBNVLCP submitted 2025-04-24 cs.CL cs.AI

Zhuoang Cai , Zhenghao Li , Yang Liu , Liyuan Guo , Yangqiu Song This is my paper · ORCID
classification cs.CLcs.AI
keywords classimbalancefoodhazarddetectionEasyDataAugmentationfocallossrandomoversamplingtransformerfine-tuningSemEval-2025Task9textclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is a system description for SemEval-2025 Task 9, a food hazard detection challenge in which short recall reports must be classified into product and hazard categories under severe class imbalance. Its central claim is that Easy Data Augmentation (EDA)—the recipe of synonym replacement, random insertion, random swap, and random deletion—applied to minority classes is the strongest imbalance-handling method among those tested. On product detection, EDA at sample rate 0.2 raises BERT from 0.22 accuracy / 0.03 macro F1 / 0.13 weighted F1 to 0.55 / 0.30 / 0.52, beating random oversampling, focal loss, and all combinations. On hazard detection, combining focal loss with EDA at rate 0.1 raises accuracy from 0.58 to 0.86 and macro F1 from 0.17 to 0.59. All comparisons are made on a 20% random holdout of the organizer data, not the official SemEval test split.

What carries the argument

The machinery is a two-stage imbalance intervention layered on a transformer backbone. First, EDA applies four per-instance lexical operations—synonym replacement, random insertion, random swap, and random deletion—each with a 50% chance per instance and, for the first three, a random operation count between 1 and the sentence length. Second, classes below a target fraction $r$ of the largest class are oversampled after tokenization. During training, optional focal loss $FL(p_t) = -\alpha_t(1-p_t)^\gamma \log(p_t)$ with $\alpha=1$, $\gamma=2$ reweights the gradient toward low-confidence examples. The workhorse is the factor $(1-p_t)^\gamma$, which shrinks the loss contributed by confident predictions and thereby steers training toward the rare and hard classes that dominate the macro-F1 score.

What would settle it

Re-run BERT with EDA at sample rate 0.2 on several random 20% splits or on the official SemEval-2025 test set: if EDA's macro F1 does not stay clearly above the random-oversampling value of 0.25 on most repeats, the paper's central claim about EDA's superiority is falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery, on its own terms, is that a purely lexical augmentation recipe is enough to overcome most of the observed class-imbalance penalty in this benchmark. BERT fine-tuned with cross-entropy and no augmentation scores 0.22 accuracy, 0.03 macro F1, and 0.13 weighted F1 on the product-category subtask. Applying EDA to all training examples with a target sample rate of 0.2 produces 0.55 accuracy, 0.30 macro F1, and 0.52 weighted F1; the other EDA rates (0.1, 0.5, 1.0) all land close to this, so the effect is stable across the tested hyperparameter range. The authors interpret this as evidence that EDA mitigates class imbalance by giving minority classes enough lexically varied training exposure. Focal loss and oversampling each improve over the baseline but never surpass EDA alone, and for hazard detection the best configuration is focal loss plus EDA at rate 0.1, reaching 0.86 accuracy.

Load-bearing premise

The load-bearing premise is that the single 20% random holdout used for all comparisons and for choosing the best sample rate is representative of the official test data, a reliability issue the paper's Limitations section does not examine.

Editorial extensions

If this is right

  • At sample rate 0.2, EDA alone is the best configuration tested for the product-category subtask, beating every oversampling rate and every focal-loss combination.
  • The EDA result is stable across the tested sample rates: 0.1, 0.5, and 1.0 all stay near 0.54–0.55 accuracy, so the method is not peaking at a single hyperparameter value.
  • On the hazard subtask, focal loss plus EDA at rate 0.1 lifts BERT from 0.58 accuracy / 0.17 macro F1 to 0.86 accuracy / 0.59 macro F1.
  • RoBERTa with the same hazard configuration reproduces BERT's scores, so the conclusion does not depend on a specific pretrained backbone.
  • The paper's conclusion is that imbalance handling in short food-safety text should put augmentation first, with oversampling and focal loss as complements rather than replacements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: a direct test the paper leaves to the official leaderboard is to run EDA r=0.2 on the SemEval-2025 test split; the internal holdout result cannot certify that outcome.
  • Editorial: if the finding transfers, the same four lexical operations are a near-zero-cost first baseline for any short, long-tail text classification benchmark, with contextual generation as a stronger but more expensive alternative.
  • Editorial: the persistent gap between EDA and EDA-plus-oversampling hints that after lexical augmentation, minority-class duplication saturates; sweeping oversampling while holding EDA fixed would isolate that effect.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a system for SemEval-2025 Task 9 (Food Hazard Detection) that combines random oversampling, Easy Data Augmentation (EDA), and focal loss with BERT and RoBERTa backbones. The authors report that EDA substantially improves product-detection performance (accuracy rising from 0.22 to 0.55 at sample rate 0.2) and that focal loss plus EDA improves hazard detection (accuracy 0.58 to 0.86). The central claim is that EDA effectively mitigates class imbalance and that combining it with focal loss further enhances robustness. The system ranked 13th on ST1 and 12th on ST2.

Significance. If the reported gains are robust, the result that a simple lexical augmentation technique (EDA) can more than double accuracy and raise macro-F1 from 0.03 to 0.30 on a severe-class-imbalance task would be a useful empirical finding for practitioners working on food hazard detection and similar low-resource, imbalanced text classification. The paper's systematic enumeration of configurations (Table 3) is a strength, and the authors honestly report the official SemEval ranks. However, the manuscript does not provide code, data, or a reproducibility package, and the experimental design does not statistically secure the headline claim. The paper's internal contradiction between the abstract and the results further weakens its conclusions as stated.

major comments (3)
  1. [§5.1, §5.2, Table 3] The central claim that EDA effectively mitigates class imbalance rests on a single 20% holdout split, and the best sample rate (r=0.2) is selected from results on that same holdout. The paper reports no error bars, random seeds, cross-validation, or stratification. The differences among EDA rates are tiny (accuracy 0.54, 0.55, 0.55, 0.54; macro-F1 0.29, 0.30, 0.30, 0.30), corresponding to a few examples on a holdout of a few hundred, so the 'best' configuration is a selected maximum rather than an unbiased estimate. The headline gain should be supported by multiple seeds or splits, or by scores on the official SemEval test set; reporting only ranks (13th and 12th) is insufficient.
  2. [Abstract, §5 (Combination paragraph), §6, Table 3] The abstract and conclusion state that 'combining focal loss with oversampling and EDA further enhances model robustness,' but the paper's own results contradict this. Table 3 shows that EDA alone outperforms every combination: for example, EDA0.1 achieves 0.54/0.29/0.50 accuracy/macro-F1/weighted-F1, while Focal loss + EDA0.1 achieves 0.53/0.29/0.50, and Oversampling + EDA 0.1 achieves 0.49/0.25/0.45. The text in §5 explicitly acknowledges that combinations 'did not achieve the same high levels of effectiveness as EDA alone.' The abstract and conclusion must be revised to match the evidence.
  3. [§4.1 vs §5] The pipeline order is described contradictorily. Section 4.1 says random oversampling is performed after tokenization and 'EDA ... is then applied,' implying EDA operates on tokenized text; Section 5 says 'EDA is applied before tokenization ... oversampling is applied after tokenization.' Since EDA operations (synonym replacement, insertion, swap, deletion) are word-level, the order determines what transformations are possible and directly affects the augmented training data. The authors must specify the exact pipeline order and the representation level at which each augmentation is applied, or the reported numbers cannot be reproduced.
minor comments (5)
  1. [§5.2] The text mentions 'data split: training/validation/test' but the experimental setup describes only a 20% test split; no validation set is defined. Please clarify whether a validation set was used and, if so, how it was created and used for hyperparameter selection.
  2. [Eq. (2), §5.1, Table 3 caption] The definition of sampling_strategy in Eq. (2) does not specify how target_count relates to the sample rate r. The text in §5.1 says classes smaller than r% of the most-frequent class are upsampled, while the Table 3 caption says '10% of the maximum sample'; unify these descriptions.
  3. [References] The citation for Adam is given as 'Kingma, 2014'; the correct reference is Kingma and Ba, 'Adam: A Method for Stochastic Optimization,' ICLR 2015.
  4. [Table 4] The table lists only two rows (BERT and RoBERTa with Focal loss + EDA 0.1), but the text says RoBERTa was investigated 'in a similar manner.' Report the full set of RoBERTa configurations or state explicitly that only these two configurations were run for the hazard task.
  5. [Table 3] The row 'Oversampling + EDA 0.1' appears after the focal-loss rows, separated from the other oversampling rows; reordering the rows by technique would improve readability.

Circularity Check

0 steps flagged · score 1.0 of 10

Empirical comparison without derivation; mild selection-on-holdout issue is a statistical validity concern, not circularity.

full rationale

This paper is an empirical systems description, not a derivation, so the classic circularity patterns do not apply. The central claim—that EDA improves ACC/F1 on the imbalanced product-detection subtask—is supported by direct comparisons in Table 3 across configurations run on a single 20% holdout described in Section 5.2. No equation in the paper defines EDA's performance in terms of the reported metric, and no fitted parameter is renamed as a prediction. The hyperparameters (sample rate r, alpha, gamma) are selected by inspecting results on the same holdout that is then used to report the chosen configuration's score; this is a model-selection/overfitting issue (the reported 'best' value is a selected maximum over roughly fifteen configurations, with adjacent EDA rates differing by only 0.01 in accuracy), but it is not circularity because the comparison itself remains an external empirical measurement. The contradictions between Section 4.1 (EDA applied after tokenization/oversampling) and Section 5 (EDA applied before tokenization) are reproducibility concerns rather than circular reasoning. The self-citations in the paper (Fan et al. 2024a, 2024b) appear only in related-work background and are not load-bearing for the experimental claim. The Limitations section acknowledges synthetic noise and generalizability but does not mention the selection-on-test-set issue; that omission is relevant to soundness, not to circularity. Overall, the derivation chain is self-contained as an empirical study: the reported gains are not equivalent by construction to any input or fitted quantity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its empirical claims depend on a few hand-chosen hyperparameters and on the representativeness of its custom data split.

free parameters (3)
  • sample rate r (oversampling and EDA) = 0.1, 0.2, 0.5, 1.0; best 0.2 for EDA, 0.1 for focal+EDA
    Controls how much minority classes are upsampled; selected after comparing Table 3 results on the test split.
  • focal loss alpha = 1
    Set based on 'initial experiments indicating improved recall for underrepresented hazards' (Section 4.2).
  • focal loss gamma = 2
    Set based on initial experiments, tuneable per dataset.
assumptions (4)
  • domain assumption The 20% random split of the SemEval dataset is representative of the task distribution.
    The paper evaluates on its own split rather than the official test set; if unrepresentative, all reported numbers are uninformative. Section 5.2.
  • domain assumption EDA-augmented sentences preserve the original label.
    The paper relies on EDA to create valid minority-class training examples, but its Limitations section concedes augmented samples may not preserve semantics.
  • domain assumption The organizer-provided BERT baseline is a valid baseline for comparison.
    The paper compares all configurations to 'BERT base' and treats the difference as the effect of the balancing methods. Section 5.2.
  • standard math Standard cross-entropy, focal loss, and transformer fine-tuning behave as described in the cited literature.
    Relies on standard machine learning theory; no proof needed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ustnlp16 at SemEval-2025 Task 9: Improving Model Performance through Imbalance Handling and Focal Loss." pith.science (2026). https://pith.science/paper/RBBNVLCP

@misc{pith2026250500021,
  author       = {Pith},
  title        = {Pith review of: Ustnlp16 at SemEval-2025 Task 9: Improving Model Performance through Imbalance Handling and Focal Loss},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RBBNVLCP}},
  note         = {Machine review of arXiv:2505.00021}
}
read the original abstract

Classification tasks often suffer from imbal- anced data distribution, which presents chal- lenges in food hazard detection due to severe class imbalances, short and unstructured text, and overlapping semantic categories. In this paper, we present our system for SemEval- 2025 Task 9: Food Hazard Detection, which ad- dresses these issues by applying data augmenta- tion techniques to improve classification perfor- mance. We utilize transformer-based models, BERT and RoBERTa, as backbone classifiers and explore various data balancing strategies, including random oversampling, Easy Data Augmentation (EDA), and focal loss. Our ex- periments show that EDA effectively mitigates class imbalance, leading to significant improve- ments in accuracy and F1 scores. Furthermore, combining focal loss with oversampling and EDA further enhances model robustness, par- ticularly for hard-to-classify examples. These findings contribute to the development of more effective NLP-based classification models for food hazard detection.

Figures

Figures reproduced from arXiv: 2505.00021 by the authors.

Figure 1
Figure 1. The structured pipeline for food hazard detec [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 canonical work pages

  1. [1]

    Adegbenjo and Michael O

    Adeyemi O. Adegbenjo and Michael O. Ngadi. 2024. https://doi.org/10.3390/foods13203300 Handling the imbalanced problem in agri-food data analysis . Foods, 13(20)

    work page doi:10.3390/foods13203300 2024

  2. [2]

    Jacob Devlin. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2019

  3. [3]

    Wei Fan, Haoran Li, Zheye Deng, Weiqi Wang, and Yangqiu Song. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.195 G old C oin: Grounding large language models in privacy laws via contextual integrity theory . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3321--3343, Miami, Florida, USA. Association f...

    doi:10.18653/v1/2024.emnlp-main.195 2024

  4. [4]

    Wei Fan, Weijia Zhang, Weiqi Wang, Yangqiu Song, and Hao Liu. 2024 b . https://arxiv.org/abs/2310.17922 Chain-of-choice hierarchical policy learning for conversational recommendation . Preprint, arXiv:2310.17922

    work page Pith review arXiv 2024

  5. [5]

    Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    arXiv 2014

  6. [6]

    Maureen Lyndel C Lauron and Jaderick P Pabico. 2016. Improved sampling techniques for learning an imbalanced data set. arXiv preprint arXiv:1601.04756

    work page Pith review arXiv 2016

  7. [7]

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll \'a r. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980--2988

    2017

  8. [8]

    Korbinian Randl, John Pavlopoulos, Aron Henriksson, Tony Lindgren, and Juli Bakagianni. 2025. S em E val-2025 task 9: The food hazard detection challenge. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Vienna, Austria. Association for Computational Linguistics

    work page 2025

Show all 13 references
  1. [9]

    Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2020. Fast wordpiece tokenization. arXiv preprint arXiv:2012.15524

    2020 arXiv

  2. [10]

    Jason Wei and Kai Zou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196

    2019 arXiv

  3. [11]

    Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. https://aclanthology.org/2021.ccl-1.108/ A robustly optimized BERT pre-training approach with post-training . In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218--1227, Huhhot, China...

    2021

  4. [12]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  5. [13]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.