{"id":"d07e74e8-ef5d-469e-a0a3-4e69de5e9e0f","arxiv_id":"2509.04202","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"SED-Aug augments social media texts with LLM rewrites and adds tuned noise to structural embeddings, reporting large F1 gains over prior baselines, though the gain over its own unaugmented base is much smaller.","lead":"Social event detection finds real-world events in social media posts. The paper combines LLM-generated text rewrites with tuned noise on graph embeddings and reports large F1 gains, but the main comparison omits the framework's own unaugmented base model, so the headline improvement is not a clean measure of the augmentation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline improvement conflates base-model strength with augmentation benefit: unaugmented base already beats published baselines by ~10%, so the 17.67% claim overstates the augmentation effect.","rationale":"The reader's weakest assumption identifies exactly this issue: the experimental comparison does not isolate the augmentation effect because the unaugmented base model is not reported alongside the baselines. I agree that this is the most load-bearing concern for the paper's central claim. The paper's own Table 7 provides the key evidence: the base model already reaches 85.71 average F1 on Twitter2012, while the best baseline is 77.57; the augmentation adds only 5.57 points. The abstract's 17.67% improvement is therefore mostly attributable to the base model's strength, not the augmentation framework. This does not make the paper worthless—the augmentation does yield a positive gain over the base—but it means the headline overstates the contribution. The reader's CONDITIONAL verdict is appropriate: if the authors add the missing baseline row and the augmentation gain remains positive, the claim is weakened but not invalidated; if the gain disappears, the contribution is trivial. The test-set tuning concern is secondary but reinforces the need for careful re-analysis. I recommend no change to the verdict, as the reader has already correctly identified this issue.","tokens_in":15617,"tokens_out":5791,"duration_ms":55165,"concrete_test":"Re-run the main experiments (Table 1) adding a row for the exact same pipeline without any augmentation modules (no LLM rewrites, no feature-space perturbations), using identical data splits and hyperparameters. Compute the relative improvement of SED-Aug over this unaugmented base. If this augmentation-only gain is substantially smaller than the advertised 17.67% (e.g., ~6.5% as suggested by Table 7), the claim that SED-Aug outperforms baselines by 17.67% is misleading because most of the gap is pre-existing in the base model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the dual augmentation framework yields state-of-the-art results, with a 17.67% average F1 improvement over the best baseline on Twitter2012. Table 7 shows, however, that the framework's unaugmented base model (without any explicit or implicit augmentation) already achieves 85.71 average F1 on Twitter2012 at the full 70% training ratio, versus 77.57 for GraphHAM in Table 1. Thus the base model alone surpasses the best baseline by 10.5% (relative), and the augmentation itself adds only 5.57 points (6.5% relative over the base). The abstract and conclusion attribute the full 17.67% to SED-Aug, but the largest part is inherited from the base model's architecture or preprocessing. The paper never includes the unaugmented base model in Table 1, so readers cannot separate the augmentation's contribution from the base model's advantage. This is the most load-bearing concern because it directly challenges the paper's contribution claim. A secondary issue is that hyperparameters (α, σ, c, r, η) appear to be selected using test-set performance (Appendix A.4), which inflates the reported numbers, but the missing baseline alone undermines the headline comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SED-Aug, a dual augmentation framework for social event detection. Explicit augmentation uses LLM-based generation strategies (paraphrasing, adding context, style transfer, entity-preserving paraphrase, extract-and-rewrite) to create diversified text, while implicit augmentation applies five perturbation methods (GP, PGP, IDGP, CGP, FDP) to structural fused embeddings in feature space. The framework is evaluated on Kawarith6, Twitter2012, and Twitter2018 against a set of published baselines. The authors report state-of-the-art average F1 scores, with claimed improvements over the best baseline of about 17.67% on Twitter2012 and 15.57% on Twitter2018. Ablation studies, limited-data experiments, and distribution visualizations are provided to support the contribution.","tokens_in":15891,"tokens_out":4315,"duration_ms":44459,"significance":"If the augmentation effect were cleanly established, SED-Aug would be a practically useful and reproducible recipe for low-resource social event detection: it moves LLM cost to augmentation time, combines text-level and feature-level diversity, and includes public code. The paper also ships a relatively broad set of ablations and a candid limitations section. However, the central quantitative claim is currently confounded by the strength of the unaugmented base model, so the significance cannot be assessed until the augmentation-specific gain is isolated.","major_comments":[{"comment":"The headline claim that SED-Aug outperforms the best baseline by 17.67% on Twitter2012 is not attributable to the augmentation framework. Table 7 reports the framework's unaugmented base model at the 70% training ratio with Micro F1 87.72, Macro F1 83.69, and average 85.71. Table 1 lists GraphHAM as the best baseline with average F1 77.57 on Twitter2012. The base model alone therefore exceeds the best baseline by 8.14 points absolute (10.49% relative), while augmentation adds only 5.57 points absolute (6.50% relative) over the base. The abstract and conclusion credit the full 17.67% relative improvement to the dual augmentation, which overstates the augmentation effect. The unaugmented base model must be included in Table 1 (or a directly comparable table), and the text should separate the base-model advantage from the augmentation gain.","section":"§1, §4.2, Table 1 vs. Table 7"},{"comment":"The paper describes SED-Aug as 'plug-and-play' and claims it can be integrated into a SED model, but the base model used in all experiments is never identified. The pipeline mentions a pre-trained language model, social graph construction, and graph aggregation, but the specific SED classifier/backbone is not named or described. Without this, the reader cannot tell whether the strong unaugmented performance comes from the backbone architecture, preprocessing, or the augmentation. To support the plug-and-play claim, the authors should name the base model and run the augmentation on at least one additional existing SED model (e.g., GraphHAM or GraphMSE).","section":"§3, §4.1, Figure 1"},{"comment":"The probability threshold α and other implicit-augmentation hyperparameters (σ, c, r, η) appear to be tuned directly on test-set performance. Appendix A.4 reports test Micro/Macro F1 for α = 0.1, 0.3, 0.6 on all three datasets, and the final choice α = 0.6 coincides with the highest test scores per dataset. No validation-set-based selection or nested evaluation is described. This protocol can inflate the reported absolute numbers and weakens the comparison. The authors should either select hyperparameters on the validation split and report test results for the selected configuration, or explicitly discuss the resulting optimism.","section":"Appendix A.4"},{"comment":"The definition of FDP is not implementable as written. In Eq. (9), N is defined earlier as the number of message samples, but the slice F_i[N - r*N : N] is applied to the frequency-domain representation of a single embedding, whose length should be the embedding dimension D, not N. The high-frequency/low-frequency split must be specified with respect to the feature dimension. Eq. (10) also mixes a complex Gaussian notation with a scalar 'noise level η' without defining the elementwise operation. This is a reproducibility issue for one of the five proposed implicit augmentation methods.","section":"Eq. (9), §3.2"}],"minor_comments":[{"comment":"The claim that implicit augmentation 'consistently provides additional benefits' is not directly supported: Table 3 has no row for 'no implicit augmentation' within the same table. The baseline value is inferred from Table 2. Add an explicit 'w/o implicit' row to Tables 3 and 4.","section":"Table 3 and §4.3"},{"comment":"The zero/few-shot LLM results on Kawarith6 (few-shot Micro F1 98.1, Macro F1 98.0) are very close to SED-Aug (98.41/98.29). The paper should discuss more carefully why this does not undermine the claimed advantage of the augmentation framework over a direct LLM classifier, beyond the cost argument.","section":"Appendix A.5"},{"comment":"The notation n_PGP ∼ N(0, σ²)·G is ambiguous: G is a set of embeddings, not a scalar; the intended operation is presumably an elementwise product with the current embedding magnitude. Clarify the formula.","section":"§3.2, Eq. (5)"},{"comment":"The symbol α is used for the probability threshold in Eq. (2) and for the variance control parameter in Eq. (6). These are different hyperparameters; use distinct notation.","section":"§3.2, Eq. (6)"},{"comment":"The distribution visualizations are qualitative. Reporting a quantitative distribution dissimilarity measure (e.g., KL divergence or Wasserstein distance) before and after augmentation would strengthen the claim that the perturbation preserves the overall distribution.","section":"§4.7, Figures 3–4"},{"comment":"No standard deviations or significance tests are reported for any of the tables. Given the small gaps between some configurations (e.g., Twitter2012 Micro F1 92.89 vs. 93.03), error bars or repeated-run statistics would help assess robustness.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim needs to be reframed around the augmentation gain rather than the absolute lead over published baselines. The missing base-model comparison and test-set hyperparameter tuning are the two issues that most affect the paper's validity. If the authors can supply an honest decomposition of base-model versus augmentation contribution and re-run the hyperparameter selection on validation data, the paper could become acceptable; without those changes, the headline contribution is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's headline claim—17.67% better than the best baseline—is mostly inherited from the base model, not from the augmentation. Table 7 shows the framework without augmentation already hits 85.71 average F1 on Twitter2012 at the 70% split, versus 77.57 for GraphHAM in Table 1. So the base model alone is already 8.14 points ahead of the 'best baseline'; the augmentation adds another 5.57 points. The abstract and conclusion attribute the full 17.67% to SED-Aug, which overstates the augmentation's contribution.\n\nWhat's good: The paper does a thorough empirical job. Table 7 is the real contribution: across training ratios from 10% to 70%, adding augmentation consistently improves over the unaugmented base, with larger gains when data is scarce (10.29% relative at 10%). That pattern is credible and practically useful for crisis and market monitoring. The five explicit LLM strategies and five implicit perturbation methods are standard individually, but the combination is new for SED, and the ablation tables are reasonably complete. Code is released.\n\nSoft spots beyond the missing baseline: The probability threshold α is selected on the test set (Appendix A.4 reports three values, and the best is chosen for each dataset). No error bars or multiple runs. The FDP explanation in Section 4.5 contradicts itself—it says retaining high-frequency noise is best but then credits low-frequency components for the gain—and Eq. (5) for PGP and Eq. (6) for IDGP have murky notation about what is scaled by what. These are fixable but need attention.\n\nThe reader's take is correct on the main point: the missing unaugmented base model in Table 1 is the load-bearing flaw. I'd add that the paper never states which backbone SED-Aug wraps. If it's GraphHAM, the discrepancy with the published GraphHAM number is unexplained; if it's a different stronger model, that should be disclosed as the real baseline.\n\nVerdict: worth a serious referee, but only with major revision. The authors need to include the base model in Table 1, tune α on validation rather than test, report variance, and clean up the FDP wording. The empirical finding on low-resource settings is solid enough to justify the effort.","headline":"The augmentation effect is real but modest; the paper's 17.67% headline is mostly the base model's strength, not the augmentation's.","tokens_in":16406,"tokens_out":3458,"would_cite":false,"duration_ms":32080,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual augmentation framework combining LLM text rewrites with feature-space perturbations reports average F1 of 91.28 on Twitter2012 and 88.02 on Twitter2018, beating the best baselines by roughly 15–17%.","keywords":["social event detection","data augmentation","large language models","feature-space augmentation","graph neural networks","Twitter event benchmarks","low-resource learning","class imbalance"],"falsifier":"Run the exact SED-Aug pipeline with both augmentation stages switched off on the full Twitter2012 and Twitter2018 training sets and compare its F1 against the augmented version. If the unaugmented model already matches or beats the published baselines, the headline improvement is not attributable to augmentation; if the augmented model's gain over the same-base-model control equals the reported gain, the dual augmentation claim is confirmed.","tokens_in":15456,"feed_emoji":"🔄","tokens_out":6665,"duration_ms":58212,"temperature":0.7,"pith_summary":"Social event detection needs labeled tweets, which are expensive to annotate. This paper proposes SED-Aug, a plug-and-play augmentation framework that attacks label scarcity from two directions at once: explicit augmentation uses a large language model to rewrite each message in five ways, and implicit augmentation perturbs structure-fused message embeddings in feature space with five noise schemes. On the Twitter2012 and Twitter2018 benchmarks, the augmented model reports average F1 scores of 91.28 and 88.02, respectively, about 17.67% and 15.57% above the best published baselines. The paper further shows that both augmentation halves help, that gains are largest under class imbalance and limited data, and that the LLM cost is paid offline so the final detector needs no LLM at inference.","feed_headline":"Dual augmentation lifts event detection F1 by 15-17%","feed_subtitle":"LLM rewrites plus feature-space noise beat top social-event-detection baselines on Twitter data.","key_machinery":"The carrying mechanism is the dual augmentation loop on top of a graph-based event classifier. Explicit augmentation (one-stage: paraphrasing, adding context, style transfer, entity-preserving paraphrase; two-stage: extract keywords/entities/knowledge graph and rewrite) is applied to raw messages before embedding. Implicit augmentation then operates on the structure-fused message embeddings, using five perturbation schemes—Gaussian Perturbation, Proportional Gaussian Perturbation, In-Distribution Gaussian Perturbation, Clipped Gaussian Perturbation, and Frequency-Domain Perturbation—selected with probability threshold alpha. The point of the dual design is that LLMs handle text diversity whi","core_discovery":"On its own terms, the paper's central claim is that data scarcity in social event detection can be substantially relieved by combining two complementary augmentation types rather than choosing one. Explicit augmentation enriches the surface text with LLM-generated paraphrases, context, style shifts, entity-preserving rewrites, and extract-and-rewrite variants; implicit augmentation then adds controlled noise to the structure-fused message embeddings produced by a graph-based SED model. Together these transformations keep semantic and relational content intact while increasing training diversity, and the paper reports best-published average F1 scores on Kawarith6, Twitter2012, and Twitter2018","pith_inferences":["The paper's headline percentages compare SED-Aug to published baselines rather than to its own unaugmented twin; a direct matched-base-model test on the full Twitter splits would reveal how much of the stated gain is due to augmentation and how much to the base graph model and BERT embeddings.","The five feature-space perturbations could be combined explicitly rather than selected one at a time, since they act on different aspects (scale, distribution, clip bounds, frequency); an ensemble or learned mixture may yield further diversity gains.","If the plug-and-play claim holds, the same dual augmentation should transfer to streaming or open-set event detection, where labels are even scarcer and new event types appear over time; a natural test is to attach SED-Aug to an incremental SED model and measure forgetting.","The LLM-based explicit augmentation strategies should be sensitive to model choice; the paper validates GPT-4o-mini against stronger LLMs, but a broader sweep could show whether cheaper or domain-tuned models preserve the gains."],"forward_implications":["Plug-and-play augmentation works across three different event-detection datasets, including an Arabic crisis corpus, with no re-annotation cost.","LLM augmentation can be done in advance; inference remains cheap because the detector never calls the LLM.","Feature-space augmentation helps most when classes are imbalanced, as shown by the large Macro F1 gains on Twitter2018.","Dual augmentation is most valuable under data scarcity: using 10% of Twitter2012, it adds 10.29 points of average F1 over the unaugmented model.","All five explicit strategies beat the no-explicit baseline, with entity-preserving paraphrase the strongest overall, while keyword-based rewriting dominates on two of three datasets."],"supporting_citations":[{"why":"Supplies Twitter2012, the main benchmark for the headline 17.67% improvement claim.","marker":"(McMinn et al., 2013)"},{"why":"Supplies Twitter2018, the benchmark for the headline 15.57% improvement claim.","marker":"(Mazoyer et al., 2020)"},{"why":"Supplies Kawarith6, the third evaluation dataset used to show cross-dataset generality.","marker":"(Alharbi and Lee, 2021)"},{"why":"GraphHAM is the best baseline in Table 1, the yardstick against which the reported improvement percentages are computed.","marker":"(Qiu et al., 2024a)"},{"why":"Established feature-space augmentation, the lineage for the implicit perturbation methods.","marker":"(DeVries and Taylor, 2017)"},{"why":"KPGNN is a graph-based SED baseline that uses structure-fused embeddings, the kind of representation the implicit augmentation operates on.","marker":"(Cao et al., 2021)"},{"why":"ETGNN is a stronger graph baseline whose published Twitter2012 numbers set a comparison standard that the framework must exceed.","marker":"(Ren et al., 2022b)"},{"why":"BERT is the pretrained encoder that generates the message embeddings used throughout the pipeline.","marker":"(Kenton and Toutanova, 2019)"}],"fun_headline_variants":["SED-Aug boosts event detection F1 by up to 17.7% on Twitter","Dual augmentation lifts social event detection on two datasets","LLM rewrites plus feature noise improve event detection","Data augmentation framework beats baselines on Twitter2012 and Twitter2018"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper's central improvement claim treats the gap to published baselines as the effect of augmentation, although its own Table 7 shows the unaugmented base model on Twitter2012 already reaches 85.71 average F1—well above the 77.57 best baseline—so part of the headline gain is inherited from the base model rather than caused by augmentation.","fun_headline_variants_meta":{"raw":{"variants":["SED-Aug boosts event detection F1 by up to 17.7% on Twitter","Dual augmentation lifts social event detection on two datasets","LLM rewrites plus feature noise improve event detection","Data augmentation framework beats baselines on Twitter2012 and Twitter2018"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1095,"prompt_tokens":680,"completion_tokens":415,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":340}},"tokens_in":424,"tokens_out":415,"duration_ms":4791,"temperature":1.0,"reasoning_tokens":340,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:18:11.848740+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact SED-Aug pipeline with both augmentation stages switched off on the full Twitter2012 and Twitter2018 training sets and compare its F1 against the augmented version. If the unaugmented model already matches or beats the published baselines, the headline improvement is not attributable to augmentation; if the augmented model's gain over the same-base-model control equals the reported gain, the dual augmentation claim is confirmed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Twitter2012, the main benchmark for the headline 17.67% improvement claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Twitter2018, the benchmark for the headline 15.57% improvement claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Kawarith6, the third evaluation dataset used to show cross-dataset generality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Established feature-space augmentation, the lineage for the implicit perturbation methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KPGNN is a graph-based SED baseline that uses structure-fused embeddings, the kind of representation the implicit augmentation operates on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BERT is the pretrained encoder that generates the message embeddings used throughout the pipeline."}],"review_version":1}