{"id":"86e291d8-2c77-4f72-99e9-caabc016bd68","arxiv_id":"2505.00021","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On the SemEval-2025 food hazard detection task, Easy Data Augmentation outperformed oversampling and focal loss combinations, raising BERT's product-label accuracy from 0.22 to 0.55.","lead":"Food recall reports are imbalanced: rare hazards get few examples, so models miss them. This paper compares oversampling, text augmentation, and focal loss on a BERT classifier for a food safety shared task, finding that Easy Data Augmentation gives the biggest accuracy and F1 gains.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EDA benefit is measured on a single self-selected 20% holdout with the best sample rate chosen on the same split; without repeated splits or official-test scores the headline gain is not statistically secured.","rationale":"The reader's weakest assumption is the evaluation methodology: a single random 20% holdout, no cross-validation, and best sample rate selected on the same split. I agree that this is the most load-bearing point, because the paper's strongest quantitative claim, EDA0.2 improving product detection from 0.22/0.03/0.13 to 0.55/0.30/0.52, is exactly a selected maximum on that holdout. I add two specifics: (1) the 0.01-0.02 accuracy differences between EDA rates are within a few examples' noise on a few-hundred-example test set, so singling out r=0.2 is not meaningful without uncertainty; (2) the official SemEval result is reported only as a rank, so the holdout-based comparison cannot be checked against the true test distribution. The internal contradiction about EDA order (Section 4.1 vs Section 5) is real but secondary; it affects reproducibility more than the validity of the gain. The Limitations section does not flag the selection-on-test-set issue, which the stress-test instructions require me to surface. The recommended check, multi-split/seeded evaluation or official-test comparison, would settle whether the EDA advantage is robust. Because the reader already recommended conditional acceptance pending release and validation, my read does not change the verdict.","tokens_in":6464,"tokens_out":6608,"duration_ms":62659,"concrete_test":"Obtain the released code (or reimplement Section 5.2), then run the BERT product-detection pipeline on K=5 independent stratified 20% splits with at least 5 random seeds each, and on the official SemEval-2025 test split if available. For every Table 3 configuration (baseline, oversampling at each r, EDA at each r, and focal-loss combinations), report mean +/- std and paired McNemar tests against baseline and against EDA0.2. If EDA0.2's advantages over oversampling and focal loss do not replicate on most splits/seeds or on the official test, the headline claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that EDA effectively mitigates class imbalance rests on Table 3, which reports BERT product-detection results on a single 20% holdout described in Section 5.2. Section 5.1 fixes the oversampling/EDA sample rate r and Table 3 compares r=0.1, 0.2, 0.5, 1.0; the paper then selects EDA0.2 as the maximum (accuracy 0.55, macro-F1 0.30, weighted-F1 0.52 vs baseline 0.22/0.03/0.13). Because the same holdout is used both to choose the best configuration and to report its performance, the result is a selected maximum over roughly fifteen configurations, not an unbiased estimate. The differences among adjacent EDA rates are tiny (0.54, 0.55, 0.55, 0.54 accuracy), i.e. a few examples on a holdout of a few hundred, so the 'best' rate is likely noise. No error bars, seeds, cross-validation, or stratified split are reported; the official SemEval test is mentioned only as ranks (13th on ST1, 12th on ST2), not as scores. The paper's Limitations section discusses synthetic noise and generalizability but does not acknowledge the selection-on-test-set problem. A secondary reproducibility issue is the contradictory pipeline order: Section 4.1 describes EDA as applied after tokenization/oversampling, while Section 5 states EDA is applied before tokenization. That contradiction does not by itself refute the headline claim, but it makes re-running the exact experiment harder.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a system for SemEval-2025 Task 9 (Food Hazard Detection) that combines random oversampling, Easy Data Augmentation (EDA), and focal loss with BERT and RoBERTa backbones. The authors report that EDA substantially improves product-detection performance (accuracy rising from 0.22 to 0.55 at sample rate 0.2) and that focal loss plus EDA improves hazard detection (accuracy 0.58 to 0.86). The central claim is that EDA effectively mitigates class imbalance and that combining it with focal loss further enhances robustness. The system ranked 13th on ST1 and 12th on ST2.","tokens_in":6759,"tokens_out":4098,"duration_ms":37912,"significance":"If the reported gains are robust, the result that a simple lexical augmentation technique (EDA) can more than double accuracy and raise macro-F1 from 0.03 to 0.30 on a severe-class-imbalance task would be a useful empirical finding for practitioners working on food hazard detection and similar low-resource, imbalanced text classification. The paper's systematic enumeration of configurations (Table 3) is a strength, and the authors honestly report the official SemEval ranks. However, the manuscript does not provide code, data, or a reproducibility package, and the experimental design does not statistically secure the headline claim. The paper's internal contradiction between the abstract and the results further weakens its conclusions as stated.","major_comments":[{"comment":"The central claim that EDA effectively mitigates class imbalance rests on a single 20% holdout split, and the best sample rate (r=0.2) is selected from results on that same holdout. The paper reports no error bars, random seeds, cross-validation, or stratification. The differences among EDA rates are tiny (accuracy 0.54, 0.55, 0.55, 0.54; macro-F1 0.29, 0.30, 0.30, 0.30), corresponding to a few examples on a holdout of a few hundred, so the 'best' configuration is a selected maximum rather than an unbiased estimate. The headline gain should be supported by multiple seeds or splits, or by scores on the official SemEval test set; reporting only ranks (13th and 12th) is insufficient.","section":"§5.1, §5.2, Table 3"},{"comment":"The abstract and conclusion state that 'combining focal loss with oversampling and EDA further enhances model robustness,' but the paper's own results contradict this. Table 3 shows that EDA alone outperforms every combination: for example, EDA0.1 achieves 0.54/0.29/0.50 accuracy/macro-F1/weighted-F1, while Focal loss + EDA0.1 achieves 0.53/0.29/0.50, and Oversampling + EDA 0.1 achieves 0.49/0.25/0.45. The text in §5 explicitly acknowledges that combinations 'did not achieve the same high levels of effectiveness as EDA alone.' The abstract and conclusion must be revised to match the evidence.","section":"Abstract, §5 (Combination paragraph), §6, Table 3"},{"comment":"The pipeline order is described contradictorily. Section 4.1 says random oversampling is performed after tokenization and 'EDA ... is then applied,' implying EDA operates on tokenized text; Section 5 says 'EDA is applied before tokenization ... oversampling is applied after tokenization.' Since EDA operations (synonym replacement, insertion, swap, deletion) are word-level, the order determines what transformations are possible and directly affects the augmented training data. The authors must specify the exact pipeline order and the representation level at which each augmentation is applied, or the reported numbers cannot be reproduced.","section":"§4.1 vs §5"}],"minor_comments":[{"comment":"The text mentions 'data split: training/validation/test' but the experimental setup describes only a 20% test split; no validation set is defined. Please clarify whether a validation set was used and, if so, how it was created and used for hyperparameter selection.","section":"§5.2"},{"comment":"The definition of sampling_strategy in Eq. (2) does not specify how target_count relates to the sample rate r. The text in §5.1 says classes smaller than r% of the most-frequent class are upsampled, while the Table 3 caption says '10% of the maximum sample'; unify these descriptions.","section":"Eq. (2), §5.1, Table 3 caption"},{"comment":"The citation for Adam is given as 'Kingma, 2014'; the correct reference is Kingma and Ba, 'Adam: A Method for Stochastic Optimization,' ICLR 2015.","section":"References"},{"comment":"The table lists only two rows (BERT and RoBERTa with Focal loss + EDA 0.1), but the text says RoBERTa was investigated 'in a similar manner.' Report the full set of RoBERTa configurations or state explicitly that only these two configurations were run for the hazard task.","section":"Table 4"},{"comment":"The row 'Oversampling + EDA 0.1' appears after the focal-loss rows, separated from the other oversampling rows; reordering the rows by technique would improve readability.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like a workshop/system-description paper. The main concern for the journal is the selection-on-test-set problem: the headline EDA gain is a selected maximum over roughly fifteen configurations on a single 20% split. This is fixable within the manuscript's scope by adding multiple seeds/splits, reporting confidence intervals, or using the official test scores. The contradiction between the abstract and Table 3 must also be resolved. I recommend major revision rather than rejection because the qualitative conclusion (EDA helps over baseline) is likely robust, but the paper's current presentation is not."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhuoang, I read this system description. It does exactly what a shared-task paper should do: it states the task, describes a modest pipeline (BERT/RoBERTa plus oversampling, EDA, focal loss), and reports numbers. The most useful part is Table 3, where EDA takes product-detection accuracy from 0.22 to 0.55 and macro-F1 from 0.03 to 0.30 on their split. Those are real gains, and the paper is honest that combinations don't beat EDA alone, even though the abstract overstates the combination story.\n\nThe soft spot is the one you flagged. Everything rests on a single 20% holdout from the organizer data, not the official test split. The sample rate r is chosen by comparing r=0.1, 0.2, 0.5, 1.0 on that same holdout, and then the chosen configuration's score is reported from the same split. That's a selected maximum, not an unbiased estimate. Adjacent EDA rates give 0.54, 0.55, 0.55, 0.54 accuracy—a difference of a few examples—so the 'best' rate is plausibly noise. No error bars, no seeds, no repeated splits. On the hazard subtask, official ranks are mentioned (13th, 12th) but no official scores, so we can't compare against the reported 0.86. The Limitations section says sensible things about synthetic noise and generalizability but does not acknowledge the selection-on-test-set problem.\n\nThere is also a pipeline contradiction: Section 4.1 says EDA is applied after tokenization and oversampling, while Section 5 says EDA before tokenization and oversampling after. Either order can be made to work, but a reader cannot re-run the experiment as described. That is a reproducibility issue, not a fatal one.\n\nI agree with the reader's conditional verdict. The contribution is a legitimate application of standard tools to a new food-hazard benchmark; it is not a new technique and it does not change how we reason about imbalance handling. The citation pattern is normal—Wei & Zou, Lin et al., the task organizers—nothing suspicious.\n\nWho should read it: someone working on food-safety NLP who wants a quick configuration comparison; anyone else can skip. It deserves a serious referee because the empirical result, if reproduced on the official split, is useful, but the current write-up needs revision: release code, repeat runs/splits, report official-test scores, align the pipeline description, and correct the abstract.\n\nI'd send it back for a round of revision rather than desk-reject.","headline":"A workmanlike SemEval system paper whose headline claim is not statistically secured because the same hand-made 20% split is used both to pick and to score the winning configuration.","tokens_in":7300,"tokens_out":1732,"would_cite":false,"duration_ms":15824,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EDA applied to minority classes is the strongest imbalance remedy tested, lifting BERT's product-category accuracy from 0.22 to 0.55 and macro F1 from 0.03 to 0.30.","keywords":["class imbalance","food hazard detection","Easy Data Augmentation","focal loss","random oversampling","transformer fine-tuning","SemEval-2025 Task 9","text classification"],"falsifier":"Re-run BERT with EDA at sample rate 0.2 on several random 20% splits or on the official SemEval-2025 test set: if EDA's macro F1 does not stay clearly above the random-oversampling value of 0.25 on most repeats, the paper's central claim about EDA's superiority is falsified.","tokens_in":6233,"feed_emoji":"⚠️","tokens_out":10459,"duration_ms":90774,"temperature":0.7,"pith_summary":"The paper is a system description for SemEval-2025 Task 9, a food hazard detection challenge in which short recall reports must be classified into product and hazard categories under severe class imbalance. Its central claim is that Easy Data Augmentation (EDA)—the recipe of synonym replacement, random insertion, random swap, and random deletion—applied to minority classes is the strongest imbalance-handling method among those tested. On product detection, EDA at sample rate 0.2 raises BERT from 0.22 accuracy / 0.03 macro F1 / 0.13 weighted F1 to 0.55 / 0.30 / 0.52, beating random oversampling, focal loss, and all combinations. On hazard detection, combining focal loss with EDA at rate 0.1 raises accuracy from 0.58 to 0.86 and macro F1 from 0.17 to 0.59. All comparisons are made on a 20% random holdout of the organizer data, not the official SemEval test split.","feed_headline":"Simple text trick lifts BERT's rare-class F1 from 0.03 to 0.30","feed_subtitle":"On SemEval-2025 food hazard reports, Easy Data Augmentation beats oversampling and focal loss, reaching 0.55 accuracy.","key_machinery":"The machinery is a two-stage imbalance intervention layered on a transformer backbone. First, EDA applies four per-instance lexical operations—synonym replacement, random insertion, random swap, and random deletion—each with a 50% chance per instance and, for the first three, a random operation count between 1 and the sentence length. Second, classes below a target fraction $r$ of the largest class are oversampled after tokenization. During training, optional focal loss $FL(p_t) = -\\alpha_t(1-p_t)^\\gamma \\log(p_t)$ with $\\alpha=1$, $\\gamma=2$ reweights the gradient toward low-confidence examples. The workhorse is the factor $(1-p_t)^\\gamma$, which shrinks the loss contributed by confident predictions and thereby steers training toward the rare and hard classes that dominate the macro-F1 score.","core_discovery":"The paper's central discovery, on its own terms, is that a purely lexical augmentation recipe is enough to overcome most of the observed class-imbalance penalty in this benchmark. BERT fine-tuned with cross-entropy and no augmentation scores 0.22 accuracy, 0.03 macro F1, and 0.13 weighted F1 on the product-category subtask. Applying EDA to all training examples with a target sample rate of 0.2 produces 0.55 accuracy, 0.30 macro F1, and 0.52 weighted F1; the other EDA rates (0.1, 0.5, 1.0) all land close to this, so the effect is stable across the tested hyperparameter range. The authors interpret this as evidence that EDA mitigates class imbalance by giving minority classes enough lexically varied training exposure. Focal loss and oversampling each improve over the baseline but never surpass EDA alone, and for hazard detection the best configuration is focal loss plus EDA at rate 0.1, reaching 0.86 accuracy.","pith_inferences":["Editorial: a direct test the paper leaves to the official leaderboard is to run EDA r=0.2 on the SemEval-2025 test split; the internal holdout result cannot certify that outcome.","Editorial: if the finding transfers, the same four lexical operations are a near-zero-cost first baseline for any short, long-tail text classification benchmark, with contextual generation as a stronger but more expensive alternative.","Editorial: the persistent gap between EDA and EDA-plus-oversampling hints that after lexical augmentation, minority-class duplication saturates; sweeping oversampling while holding EDA fixed would isolate that effect."],"forward_implications":["At sample rate 0.2, EDA alone is the best configuration tested for the product-category subtask, beating every oversampling rate and every focal-loss combination.","The EDA result is stable across the tested sample rates: 0.1, 0.5, and 1.0 all stay near 0.54–0.55 accuracy, so the method is not peaking at a single hyperparameter value.","On the hazard subtask, focal loss plus EDA at rate 0.1 lifts BERT from 0.58 accuracy / 0.17 macro F1 to 0.86 accuracy / 0.59 macro F1.","RoBERTa with the same hazard configuration reproduces BERT's scores, so the conclusion does not depend on a specific pretrained backbone.","The paper's conclusion is that imbalance handling in short food-safety text should put augmentation first, with oversampling and focal loss as complements rather than replacements."],"supporting_citations":[{"why":"provides the EDA operations (synonym replacement, random insertion, random swap, random deletion) that the paper identifies as the source of the gains","marker":"(Wei and Zou, 2019)"},{"why":"provides BERT, the backbone on which the product-detection comparisons and the baseline are computed","marker":"(Devlin, 2019)"},{"why":"provides focal loss, the reweighting loss used in the best hazard-detection configuration","marker":"(Lin et al., 2017)"},{"why":"defines SemEval-2025 Task 9, its dataset, and the evaluation setup the paper addresses","marker":"(Randl et al., 2025)"},{"why":"supports combining resampling with augmentation for imbalanced text classification, motivating the paper's pipeline","marker":"(Lauron and Pabico, 2016)"},{"why":"provides RoBERTa, the second backbone used to show the hazard configuration transfers across models","marker":"(Zhuang et al., 2021)"}],"fun_headline_variants":["EDA boosts BERT's rare-class F1 tenfold on food hazard task","Text augmentation clears class imbalance in food hazard detection","Oversampling and focal loss lose to EDA in SemEval food task","Simple EDA lifts F1 from 0.03 to 0.30 in hazard classification","Focal loss plus EDA hits 0.86 accuracy on hazard detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single 20% random holdout used for all comparisons and for choosing the best sample rate is representative of the official test data, a reliability issue the paper's Limitations section does not examine.","fun_headline_variants_meta":{"raw":{"variants":["EDA boosts BERT's rare-class F1 tenfold on food hazard task","Text augmentation clears class imbalance in food hazard detection","Oversampling and focal loss lose to EDA in SemEval food task","Simple EDA lifts F1 from 0.03 to 0.30 in hazard classification","Focal loss plus EDA hits 0.86 accuracy on hazard detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1626,"prompt_tokens":965,"completion_tokens":661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":561}},"tokens_in":581,"tokens_out":661,"duration_ms":6729,"temperature":1.0,"reasoning_tokens":561,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:31:51.561114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run BERT with EDA at sample rate 0.2 on several random 20% splits or on the official SemEval-2025 test set: if EDA's macro F1 does not stay clearly above the random-oversampling value of 0.25 on most repeats, the paper's central claim about EDA's superiority is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines SemEval-2025 Task 9, its dataset, and the evaluation setup the paper addresses"},{"cited_title":"Improved Sampling Techniques for Learning an Imbalanced Data Set","cited_arxiv_id":"1601.04756","evidence_quote":"supports combining resampling with augmentation for imbalanced text classification, motivating the paper's pipeline"}],"review_version":1}