{"id":"7fef5cfa-af06-4dac-a67c-3a2ec514e4fe","arxiv_id":"1908.07018","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new five-language disaster event extraction dataset shows that rule-based cues added to Bi-LSTM models improve rare tail event labels when training data is small.","lead":"This paper introduces a five-language dataset for disaster event extraction and tests neural models augmented with hand-written rules. The work suggests that dictionary-based rules can improve rare event labels when annotated data is scarce.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run point estimates without error bars or significance tests are insufficient to establish that rule augmentation reliably beats Bi-LSTM at small training sizes; some 20% cells are already below baseline.","rationale":"The reader's verdict is CONDITIONAL, and I agree with the overall assessment, but my emphasis differs from the reader's named weakest assumption. The dictionary-coverage/cost issue is real and directly affects the 'easily incorporated' claim, but it does not undermine the empirical comparison if the reported numbers are reliable. The more fundamental problem is that the empirical comparison itself lacks any measure of variance. Table 6 shows the core evidence for 'outperforms deep learning models on lesser annotated data': at 20% training, several rule-augmented cells are below baseline, and many reported gains are under 2 micro-F1 points. Table 5 tail-label gains are larger, but they are also inconsistent across languages and methods. Without error bars, seeds, or significance tests, and with no released code or data, these point estimates cannot be distinguished from seed-to-seed noise, which is known to be nontrivial for Bi-LSTM models on small datasets. This is load-bearing because the abstract, introduction, and conclusion explicitly advance a comparative claim; if the ordering flips under reseeding, the central claim fails even if dictionary extension is cheap. The reader did list 'results lack error bars and significance tests' in the rationale, but did not make it the weakest assumption, so I would say the agreement is partial. The proposed concrete test is straightforward and would settle whether the concern actually lands. If the results hold under multi-seed rerunning, the conditional concerns would be largely resolved and I would move toward acceptance; as it stands, CONDITIONAL is the right verdict.","tokens_in":8825,"tokens_out":7393,"duration_ms":77633,"concrete_test":"Release InDEE-2019 and the training code, then rerun every language × training-size (20% and 40%) × method (A, B, C, D) with at least 10 random seeds while keeping the same hyperparameters. Report mean and 95% confidence intervals for overall micro-F1, macro-F1, and tail-label micro-F1, and perform a paired bootstrap or McNemar significance test comparing each rule-augmented method against baseline A on the same test split. The central claim survives only if a majority of low-resource languages show a significant (p<0.05) advantage for at least one rule-augmented method at 20% and/or 40%, and if the tail-label advantage is also significant after correcting for multiple comparisons.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that rule-augmented methods outperform deep learning on lesser annotated data rests entirely on the point estimates in Tables 5 and 6, but the paper reports no variance, no random seeds, no confidence intervals, and no significance tests. At the critical 20% condition, the evidence is mixed and the margins are small. In Table 6, Bengali overall micro-F1 is A=64.26 versus B=63.90, C=63.16, D=63.28, i.e., all three rule-augmented variants are below baseline; Tamil D=66.25 is below A=67.44; Hindi B=48.44 is below A=48.56. At 40%, several cells are also below baseline. The tail-label micro-F1 gains in Table 5 are larger on some languages, e.g., Marathi 20% B=44.62 versus A=39.76, but other languages and methods at 40% show losses, e.g., Bengali 40% B=51.38 versus A=49.69 is a gain, but Hindi 40% D=42.84 versus A=42.68 is small and Tamil 40% D=50.93 versus A=55.94 is a loss. For Bi-LSTM models on small training sets, seed-to-seed variability is known to be nontrivial. Since the paper's headline comparative claim is the load-bearing contribution, the absence of any statistical analysis means the claimed advantage could be run-to-run noise. This concern is independent of the dictionary-coverage issue: even if dictionary construction is cheap and coverage is high, the empirical ordering must be stable to support the conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces InDEE-2019, a disaster-domain event extraction dataset in five Indic languages (Marathi, Hindi, English, Tamil, Bengali) with 43–48 labels per language, and proposes three rule-augmented Bi-LSTM variants for sequence labeling. The rule layer is a multi-hot vector per token built from hand-made positive and negative synonym dictionaries; it is incorporated either by concatenation with word embeddings, by a parallel Bi-LSTM whose hidden states are concatenated, or by distillation with a teacher distribution biased by the rule vector. Experiments train on 20%–100% splits and report micro/macro F1, with special attention to tail labels. The empirical claim is that rule-augmented methods outperform plain Bi-LSTM at small training sizes and that tail labels can be incorporated by creating new rules without large annotated data.","tokens_in":9154,"tokens_out":4906,"duration_ms":51806,"significance":"If substantiated, the paper would provide a useful resource (the InDEE-2019 dataset) and a practical recipe for low-resource event extraction: a simple, interpretable rule layer that compensates for scarce training data and handles rare labels. The framing as sequence labeling, the coverage of five languages, and the release of a new dataset are concrete strengths. However, the central comparative claim rests entirely on single-run point estimates without variance or significance testing, and the cost of constructing the rule dictionaries is not quantified. As written, the evidence is not strong enough to establish that rule augmentation reliably beats the Bi-LSTM baseline on lesser annotated data; the direction of the effect is plausible but the effect size is often within run-to-run noise.","major_comments":[{"comment":"Every F1 score in Tables 3–6 is a point estimate with no confidence intervals, standard deviations, number of random seeds, or significance tests. This matters directly because the headline claim (rule-augmented methods outperform deep learning on lesser annotated data) is supported by margins that are often small. For example, in Table 6 at 20% training, Bengali B/C/D are all below baseline A (63.90/63.16/63.28 vs 64.26), Tamil D is below A (66.25 vs 67.44), and Hindi B is below A (48.44 vs 48.56). At 40%, Tamil D is below A (50.93 vs 55.94 in Table 5; 70.03 vs 70.99 in Table 6). Without repeated runs and a paired significance test (e.g., bootstrap or approximate randomization over test sentences), the claimed advantage could be run-to-run noise. I recommend reporting mean±std over at least 5 seeds and a significance test for the key 20%/40% comparisons.","section":"§5.2, Tables 3–6"},{"comment":"The Tamil block in Table 6 shows exactly identical micro/macro F1 values for 80% and 100% training for all four models (e.g., A: 73.62/69.77; B: 74.3/69.1; C: 73.7/69.33; D: 73.69/69.2). It is implausible that training on 80% versus 100% of the data yields identical scores for every model unless the rows were copied or the experimental setup is misdescribed. This must be corrected or explained, since it undermines the trustworthiness of the reported table.","section":"Table 6, Tamil rows"},{"comment":"The definition of a tail label is ambiguous: 'We chose those tails labels whose sum forms 5% of total training set instances.' This could mean a cumulative-frequency threshold, a per-label count threshold, or something else, and it should be stated precisely (e.g., labels whose total number of training mentions is below a fixed cutoff). In addition, Table 5 is introduced with the phrase 'significant improvement,' but no significance testing is reported anywhere; several cells show tiny margins (e.g., Hindi 20% D: 29.87 vs A: 29.57; Tamil 20% B: 44.95 vs A: 44.88). Please clarify the tail-label criterion and qualify the word 'significant' accordingly.","section":"§5.2, tail-label definition and Table 5"},{"comment":"The claim that 'tail labels can be easily incorporated by creating new rules without the requirement of large annotated data' depends on the cost and coverage of the hand-built synonym and negative dictionaries, but the paper provides no dictionary sizes, per-label counts, or coverage rates. Algorithm 1 defaults an unmatched token to 'other' (ri[other]=1), so incomplete dictionary coverage could directly suppress rare-event recall. The manuscript should report how many dictionary entries were created per language and per label, evaluate rule-vector coverage on the test set, and discuss the manual effort involved; otherwise the 'without large annotated data' claim is not yet supported.","section":"§4.2, §4.3, Algorithm 1"}],"minor_comments":[{"comment":"The sentence 'Once we create robust dictionary for one language is easy to extend it to other languages' should be rephrased for grammar and, more importantly, should be supported by evidence or hedged as a hypothesis, since cross-lingual dictionary transfer is a nontrivial claim.","section":"§4.2.1"},{"comment":"The pseudocode is ambiguous at line 3: it checks whether the whole word sequence {w1,...,wn} is in {neg}, while the surrounding text says 'If any word wi in the sentence found in negative dictionary.' Please align the pseudocode with the intended per-word check and clarify the variable 'flag' initialization and scope.","section":"Algorithm 1"},{"comment":"Equation (1) applies exp(-C(1-ri)) where ri is a multi-hot vector; the reader needs an explicit statement that the exponent is taken elementwise over tags, and a definition of C is needed at the point of the equation (it is introduced later as 'C = 1'). Similarly, the imitation parameter pi is introduced in text but should be defined immediately before Eq. (2).","section":"§4.4.3, Eq. (1)"},{"comment":"The counts of 'labels improved' should state explicitly whether equal-score labels are excluded and what the denominator is (the total number of labels per language, 43–48). This will prevent misinterpretation of the improvement counts.","section":"Tables 3 and 4"},{"comment":"Figure 1 contains garbled Marathi text and an untranslated explanation line; the figure should be cleaned and the example dictionary entries presented in a readable transliteration or with a translation, since the reader cannot verify the dictionary construction from the current image.","section":"Figure 1"},{"comment":"The dataset release is a contribution, but no URL or accessibility statement is given beyond the name 'InDEE-2019' in a footnote. Please add a data availability link or repository reference.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The duplicate Tamil rows in Table 6 are the most serious data-integrity concern and should be resolved before the paper can be considered further. The paper would also benefit from an explicit reproducibility note about random seeds and dictionary artifacts. If the authors can supply the underlying experimental logs, the editor may wish to verify the reported point estimates during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things first: the dataset is a real contribution; the headline empirical claim is not proven. InDEE-2019 is a five-language disaster-event dataset with 43-48 labels per language, and the annotation effort looks genuine. The rule-augmentation variants are explicitly re-purposed from Hu et al. (2016), so the dataset is the genuinely new piece. What the paper does well: it targets a practical problem, evaluates across five languages and five training sizes, and reports micro/macro F1. The tail-label analysis is a useful diagnostic, even though the tail-label definition is ambiguous. The soft spots are real. Every F1 in Tables 3-6 is a single run with no variance, no seeds, no significance test. At 20% training the ordering is mixed: Bengali micro-F1 is B=63.90 vs A=64.26, Tamil D=66.25 vs A=67.44, and Hindi B=48.44 vs A=48.56. That directly contradicts the abstract's claim that rule-augmented methods outperform deep learning on lesser annotated data. The rule vector also leaks tag information: Algorithm 1 sets a bit whenever a synonym-dictionary word appears in the window, so tail gains partly reflect hand-built dictionary coverage. They assert dictionaries are easy to extend across languages but give no coverage numbers or construction cost. The paper also omits obvious baselines: a dictionary-only tagger and a weak-supervision comparison such as Snorkel, which they cite. The dataset and code are not available, making replication impossible from the text. None of this, though, destroys the paper's value. The dataset alone justifies referee attention, and the mixed results at larger training sizes are informative for the low-resource IE community. My recommendation: send to peer review, but require multiple seeds or confidence intervals, a clearer tail-label definition, and at least a dictionary-only baseline. If the authors can show stable gains with proper error bars, this becomes a solid resource paper. If not, the dataset still stands on its own.","headline":"The dataset is the real contribution; the rule-augmentation gains are plausible but statistically unproven, and the paper overclaims at larger training sizes.","tokens_in":740,"tokens_out":2492,"would_cite":true,"duration_ms":41863,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rule augmentation lifts rare event labels in low-resource event extraction.","keywords":["event extraction","sequence labeling","low-resource languages","rule augmentation","tail labels","InDEE-2019","disaster domain","Bi-LSTM"],"falsifier":"A direct test is to compute dictionary coverage on the gold tail-label triggers in the test split: run Algorithm 1 with the released dictionaries and check what fraction of gold trigger tokens have a dictionary match in their window. If coverage on tail labels is low (say, under 80%) while the reported tail F1 gains are large, the gains cannot be coming from the rules, and the 'tail labels can be easily incorporated' claim is falsified.","tokens_in":8650,"feed_emoji":"🌪️","tokens_out":6174,"duration_ms":61726,"temperature":0.7,"pith_summary":"The paper tries to establish that the performance ceiling for low-resource event extraction is set by annotator effort, not by model capacity, and that a deliberately small rule layer can break that ceiling. It introduces a new disaster-domain dataset across five languages, models event extraction as sequence labeling, and augments a Bi-LSTM with per-word rule vectors built from synonym dictionaries. On 20% and 40% training splits, the rule-augmented variants beat the plain Bi-LSTM on micro-F1 and macro-F1 for several languages, with the clearest gains on tail labels. If true, event types that occur rarely in text can be captured by writing a new rule rather than collecting thousands of new labeled sentences.","feed_headline":"Rules rescue rare event labels on little data","feed_subtitle":"Five languages: adding synonym rules to a Bi-LSTM beats plain deep learning when annotations are scarce.","key_machinery":"The carrying object is the rule vector: for each word, Algorithm 1 sets bits for every label whose synonym dictionary has a match inside a window of size 2l around the word, and sets 'other' when the word is in a negative dictionary or nothing matches. It is multi-hot because labels overlap, so an attack can be both a normal bombing and a terrorist attack. The paper feeds this vector into the Bi-LSTM in three ways: concatenated with fastText embeddings, passed through a parallel Bi-LSTM whose hidden states are concatenated, or distilled into the network by biasing the prediction distribution as q*(T|w_i) proportional to p_theta(T|w_i) exp(-C(1-r_i)). The rule vector does the work of transferring rare-label knowledge without needing many training examples.","core_discovery":"The central discovery is that tail labels in event extraction are handled more reliably when the model receives a rule-derived vector alongside learned word representations. The paper states that tail labels can be easily incorporated by creating new rules without large annotated data. In its experiments, rule-augmented methods outperform plain deep learning on lesser annotated data and low-resource languages; for example, many tail labels improve at 20% and 40% training sizes. The authors' own framing is that the value of rules is greatest exactly where deep learning is weakest.","pith_inferences":["The practical claim 'without large annotated data' hides the cost of dictionary construction; a careful reader would measure dictionary building time per label to assess the true trade-off.","Because the method's tail gain depends on lexical coverage, a natural improvement is to let the rule layer learn morphological variants from unlabeled text, reducing the need for exact dictionary entries.","The same rule-vector recipe likely transfers to other sequence labeling problems where labels have recognizable trigger lexicons, such as argument role labeling, slot filling, or intent detection.","If dictionary coverage correlates with tail F1 gain, the method becomes a diagnostic: labels with low coverage are exactly the ones that need more annotation, turning rule augmentation into a data-collection guide."],"forward_implications":["At 20% and 40% training data, rule-augmented variants improve micro- and macro-F1 over the Bi-LSTM baseline for several low-resource languages, with the gains concentrated among tail labels.","A new tail label can be added by writing a synonym dictionary and a rule, which the paper argues requires no large annotated corpus.","Even though plain Bi-LSTM catches up when all training data is available, the rule signal determines performance at the small-data sizes typical of low-resource settings.","Dropping CRF is justified in this regime: transition probabilities learned from skewed label distributions hurt rare tags, and rule vectors supply the missing signal."],"supporting_citations":[{"why":"Supplies the Bi-LSTM-CRF sequence-labeling architecture that the baseline and rule-augmented models extend.","marker":"(Huang et al., 2015)"},{"why":"Provides the modern Bi-LSTM named-entity-recognition design that motivates the contextual encoder used here.","marker":"(Lample et al., 2016)"},{"why":"Gives the pre-trained fastText subword embeddings used to handle out-of-vocabulary words in low-resource languages.","marker":"(Bojanowski et al., 2017)"},{"why":"Supplies the teacher-student rule-projection framework adapted for the distillation-based rule augmentation method.","marker":"(Hu et al., 2016)"},{"why":"Supports the premise that deep learning methods are data hungry, motivating the need for rule augmentation in small-data settings.","marker":"(Sun et al., 2017)"},{"why":"Provides the IOB tagging scheme that the paper adapts into the simpler TO (Tag-Other) annotation format.","marker":"(Ramshaw and Marcus, 1999)"},{"why":"Offers a weak-supervision baseline that motivates the paper's rule-based alternative for handling many labels with scarce data.","marker":"(Ratner et al., 2017)"}],"fun_headline_variants":["Rule-augmented deep learning handles tail event labels","Tail labels in event extraction need rule assistance","For rare event tags, rules beat pure deep learning","Low-resource languages gain from rule-augmented event models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands on the assumption that a hand-built synonym dictionary covers the trigger words that actually occur in test text, because any word outside the dictionaries is silently labeled 'other' and contributes no rule signal.","fun_headline_variants_meta":{"raw":{"variants":["Rule-augmented deep learning handles tail event labels","Tail labels in event extraction need rule assistance","For rare event tags, rules beat pure deep learning","Low-resource languages gain from rule-augmented event models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000791,"raw_usage":{"total_tokens":3388,"prompt_tokens":753,"completion_tokens":2635,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":369,"completion_tokens_details":{"reasoning_tokens":2572}},"tokens_in":369,"tokens_out":2635,"duration_ms":18350,"temperature":1.0,"reasoning_tokens":2572,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:28:20.278883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to compute dictionary coverage on the gold tail-label triggers in the test split: run Algorithm 1 with the released dictionaries and check what fraction of gold trigger tokens have a dictionary match in their window. If coverage on tail labels is low (say, under 80%) while the reported tail F1 gains are large, the gains cannot be coming from the rules, and the 'tail labels can be easily incorporated' claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the modern Bi-LSTM named-entity-recognition design that motivates the contextual encoder used here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the premise that deep learning methods are data hungry, motivating the need for rule augmentation in small-data settings."},{"cited_title":"Bach, Henry R","cited_arxiv_id":null,"evidence_quote":"Offers a weak-supervision baseline that motivates the paper's rule-based alternative for handling many labels with scarce data."}],"review_version":1}