{"id":"28ebbf61-88fd-401b-b691-c03b4b44231e","arxiv_id":"1909.02766","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":30,"one_line_summary":"Giveme5W1H is a publicly available rule-based system that extracts the 5W1H phrases of a news article's main event, with reported precision of 0.73 overall and 0.82 for the first four W questions.","lead":"This paper describes an open-source system, Giveme5W1H, that automatically extracts who, what, when, where, why, and how from English news articles to identify the main event. It reports expert-judged precision of 0.73 overall and 0.82 for the first four W questions, and it releases code and datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported MAgP values rest on post-hoc relevance ratings of the system's own outputs; without an independent gold-standard or same-protocol baseline, the 0.73/0.82 numbers may be inflated by rater anchoring.","rationale":"The reader's CONDITIONAL verdict is reasonable, but the weakest load-bearing assumption is not primarily the 80-article parameter-tuning sample. The parameters were tuned on a separate annotated corpus and the headline result is measured on an independent 120-article BBC set, so the reported numbers already reflect the selected configuration's performance on held-out data; overfitting would lower rather than inflate that independent measurement unless the training and test corpora share style in a way that is not demonstrated. The most direct threat to the abstract's precision numbers is the evaluation protocol itself: relevance is judged after seeing the system's outputs, with no independent gold-standard 5W1H annotation for the test set and no calibrated same-protocol baseline in this paper. High ICR shows the three raters are consistent, but consistency among raters who all view the same system outputs cannot rule out shared leniency or anchoring on partial-but-plausible phrases. The paper itself flags the absence of a score threshold (Section 5), which forces a candidate to be output for 'why' and 'how' even when no cause or method is explicitly stated, making partial-credit inflation especially plausible for those questions. The proposed independent-annotation check is feasible because the authors already used exactly that annotation protocol for the training set in Section 3.5. If the anchored MAgP is close to the reported values, the central usability claim is strengthened substantially; if it is not, the system may still be a useful open-source tool, but the headline numbers would need to be revised or rescoped. Because the concern is empirical and addressable, and because the system and dataset remain valuable conditional on such a check, UNCHANGED relative to the reader's CONDITIONAL verdict is appropriate.","tokens_in":13688,"tokens_out":13255,"duration_ms":139198,"concrete_test":"Select 30-50 randomly sampled articles from the 120-article BBC test set. Have two new annotators, blind to system output, independently annotate the best 5W1H phrase for each question using the same protocol as in Section 3.5 (read article, then write the single most suitable phrase per question). Then have a third new annotator score Giveme5W1H's outputs against these independent gold phrases using the same 0/0.5/1 rubric as in Section 4, resolving disagreements by the Section 3.5 combination rules. Compute MAgP on this subset and compare with the paper's 0.73 (all six) and 0.82 (first four W). If the independently anchored MAgP falls more than about 0.05 below the reported values, the display-and-rate evaluation overstates real extraction quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central numerical claim (Section 6: MAgP 0.73 all questions, 0.82 first four W) is supported solely by an evaluation in which three assessors read an article, view the phrases Giveme5W1H extracted, and rate each phrase as relevant, partially relevant, or non-relevant on a 0/0.5/1 scale (Section 4). The 120 BBC test articles have no independent gold-standard 5W1H annotation, and no baseline system is evaluated under the same protocol in this paper; the only comparison to Giveme5W is a single reported number from a prior evaluation. The high intercoder reliability in Table 6 (ICR >= 0.87) demonstrates that the raters agree with each other, but it does not demonstrate that the displayed phrases are actually the article's main-event 5W1H answers. Because the system always outputs a candidate whenever one is found (Section 5 explicitly notes there is currently no score threshold to suppress low-confidence 'why'/'how' answers), raters may anchor on plausible but non-central phrases and award partial credit. A moderate systematic leniency or anchoring effect would change the headline numbers materially. This is the more direct threat to the central claim than parameter overfitting: even a configuration that generalizes perfectly could receive inflated scores under this display-and-rate protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Giveme5W1H, an open-source system that extracts phrases answering the journalistic 5W1H questions (who, what, when, where, why, how) from English news articles to describe the main event. It describes a three-phase pipeline (preprocessing with CoreNLP, SUTime, Nominatim, and AIDA; four extraction chains for action, environment, cause, and method; and scoring functions with learned weights), and reports an expert evaluation on 120 BBC articles with three assessors, giving a mean average generalized precision (MAgP) of 0.73 across all questions and 0.82 for the first four W questions. The paper also introduces an annotated training dataset and states that the system is the only universal open-source 5W1H extractor.","tokens_in":14149,"tokens_out":4223,"duration_ms":41183,"significance":"If the reported evaluation is credible, this paper fills a genuine gap: it provides a modular, publicly available system and an annotated dataset, plus a sufficiently detailed algorithm description to support reimplementation and benchmarking. The release of the code and the dataset is a concrete contribution that should be credited. The headline numbers (MAgP 0.73/0.82) would demonstrate useful performance for news analysis use cases, provided the evaluation methodology is sound. The paper is clearly written in most technical sections, and the modular design is a strength. However, the evaluation protocol has weaknesses that directly affect the central claim, so the findings should be treated as promising but not yet fully substantiated.","major_comments":[{"comment":"The evaluation uses a display-and-rate protocol in which assessors see the phrases Giveme5W1H has extracted and judge them as relevant, partially relevant, or non-relevant, without an independent gold-standard annotation of the test articles and without a baseline system evaluated under the same protocol. Because Section 5 states that the system always outputs a 'why' or 'how' candidate when one is found, raters may anchor on plausible but non-central phrases and award partial credit, which can inflate MAgP. This is a direct threat to the central numerical claim of 0.73/0.82. The paper should either (a) provide an independent gold-standard annotation of the 120 test articles (or a subset) and report agreement between system output and that gold standard, or (b) run at least one baseline (e.g., Giveme5W, or a simple lead/headline heuristic) under the identical rating protocol and report the difference with a significance test. Reporting only inter-assessor agreement does not address the anchoring concern, since the raters are rating the same displayed phrases.","section":"Section 4, Table 6"},{"comment":"The scoring weights and heuristic thresholds (on the order of twenty parameters) are tuned on an 80-article training set using a semi-automated search, with only 20 articles held out for parameter validation and no cross-validation or stability analysis. The evaluation set of 120 BBC articles is separate, which is good, but the paper provides no variance estimates, confidence intervals, or significance tests for the reported MAgP values. The claim that the configuration generalizes from the training sample to news articles at large is therefore unquantified. The authors should report bootstrap or split-half variability of the MAgP on the 120 articles, and, if possible, show how MAgP varies when the parameter search is repeated on different training subsamples. Without this, the reader cannot assess how much of the reported performance is due to overfitting to the annotation scheme or to the specific training outlets.","section":"Section 3.5"},{"comment":"The paper acknowledges that for 'why' and 'how' the system always outputs a candidate as long as one is found, with no score threshold to suppress low-confidence answers. The 'why' MAgP is 0.48, which is well below the other questions. This conflation of ranking quality with the absence of abstention makes the practical utility of the output unclear: a user applying the system would receive 'why' and 'how' phrases that are frequently non-relevant, yet the reported aggregate MAgP treats them as part of the overall performance. The authors should provide a threshold-abstention analysis (e.g., MAgP as a function of a score cutoff, or precision/recall with an abstain option) to clarify whether the current always-return behavior is appropriate for the claimed use cases, or whether a threshold should be set in the released system.","section":"Section 5, 'why'/'how' output"}],"minor_comments":[{"comment":"The phrase 'ICR/44=0.81' appears to contain a typo or LaTeX artifact; this should read 'ICR = 0.81'.","section":"Section 3.5"},{"comment":"The workflow description says 'as shown in Figure 1', but Figure 1 is the news article example; the pipeline diagram is Figure 2, so the reference should be corrected.","section":"Section 3"},{"comment":"The table shows ICR values in the first numeric column without a caption note explaining this; separating the ICR row from the per-category columns would improve readability.","section":"Section 4, Table 6"},{"comment":"The formula for the weighted score sum is garbled in the rendering: 's;=∑wq,?sq,?@A\"?BC' should be typeset properly, e.g., s_q = sum_i w_{q,i} s_{q,i}.","section":"Section 3.3"},{"comment":"The comparison with Parton et al. states 'Giveme5W1H achieves a 0.12 higher MAgP}' with a malformed superscript; this should read MAgP_5W.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a system description with a potentially valuable open-source release and annotated dataset. The main barrier to acceptance is the evaluation methodology: the headline numbers are not backed by an independent gold standard or a same-protocol baseline, and the parameter tuning is not accompanied by variance or stability analysis. These issues are fixable within a revision (e.g., by adding a baseline, significance tests, and a threshold analysis), so major revision seems appropriate rather than rejection. The paper's claim to be the 'only universal open-source 5W1H extractor' is plausible but not validated by a systematic comparison; the authors should temper it or provide a more recent survey to justify it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: worth reading for the artifact, not for the headline numbers. The paper describes an improved open-source 5W1H extractor with a published annotated dataset, and it is honest about the weak spots. The evaluation, however, has a load-bearing flaw that should make you cautious about quoting 0.73 and 0.82.\n\nWhat is genuinely new: the system adds 'how' extraction, coreference-aware frequency scoring, canonicalized dates and locations, and a combined sentence-distance scorer. The code and training data are public, which is more than most related work offers. The system design is described in enough detail to reimplement or adapt, and the modular pipeline is a practical contribution for news-analysis pipelines, especially for social scientists who do not want to build extractors from scratch.\n\nThe evaluation uses three assessors on 120 BBC articles and reports per-category MAgP with high intercoder reliability. That is real work, and the authors are transparent that 'why' (0.48) and 'how' (0.61) are much weaker than the first four Ws. But the protocol is display-and-rate: assessors read the article, see the system's extracted phrases, and rate relevance. There is no independent gold-standard 5W1H annotation for the test set, no same-measure baseline, and no confidence intervals. High ICR only shows that raters agree with each other, not that the phrases are actually the main event. Because the system always outputs a candidate when one is found and has no score threshold, raters can anchor on plausible but non-central phrases and award partial credit. Even a moderate leniency effect would move the headline numbers materially. That is the more direct threat to the central claim than parameter overfitting; the stress-test note holds up.\n\nThe parameter-search concern is real but smaller. Roughly thirty weights and constants are tuned on 80 training articles, with a small validation split and no variance analysis. The 'universal' claim is also too broad for an English-only, five-category, mostly US/UK-outlet evaluation. Citation practice looks normal; the self-citation to Giveme5W is a baseline, not a circular dependency.\n\nBottom line: if someone needs a working 5W1H extractor and a starting benchmark, this is a useful system to build on. I would cite it for the code and dataset, not for the precision claims. If this comes to you as a referee, send it out, but require a same-measure baseline or an independent gold-standard evaluation before the numbers are accepted as evidence of generalizable performance.","headline":"The open-source system and dataset are the real contribution; the 0.73/0.82 evaluation numbers should be treated as upper-bound estimates until a same-protocol baseline or independent gold standard appears.","tokens_in":14668,"tokens_out":2333,"would_cite":true,"duration_ms":27811,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-source system extracts a news article's main event by answering the journalistic 5W1H questions, achieving 0.73 mean average generalized precision overall and 0.82 on the first four Ws.","keywords":["event extraction","5W1H","news analysis","question answering","main event detection","canonicalization","open-source software","natural language processing"],"falsifier":"Take a held-out set of more than 100 articles from outlets and beats not represented in the training data, such as local newspapers or financial wire services, have three annotators score the extracted phrases on the same graded relevance scale, and check whether mean average generalized precision on the first four W questions remains near 0.82; if it falls well below that, the tuned configuration is tied to the training distribution rather than being universal.","tokens_in":13499,"feed_emoji":"📰","tokens_out":11228,"duration_ms":99345,"temperature":0.7,"pith_summary":"Giveme5W1H is an open-source system that takes an English news article and returns short phrases answering who did what, when, where, why, and how; together these six answers describe the article's main event. The paper's central claim is that this universal, rule-based approach is accurate enough for real use: in an expert evaluation on 120 news articles it achieved a mean average generalized precision of 0.73 over all six questions and 0.82 over the first four Ws, which the authors argue can alone summarize an event. This matters because event extraction is a prerequisite for news aggregation, clustering, summarization, and media-bias analysis, and no equally general, publicly available tool existed. The paper also contributes an annotated training dataset and a modular pipeline, letting other researchers reuse, adapt, and benchmark against the system instead of reimplementing extraction from scratch.","feed_headline":"Giveme5W1H answers who, what, when, where, why, how at 73% precision","feed_subtitle":"An open-source pipeline turns any English news article into six phrases that describe its main event.","key_machinery":"The central machinery is the 5W1H question set used as an event schema, operationalized by four extraction chains. The action chain collects subject noun phrases and their sibling verb phrases for who and what; the environment chain uses canonicalized temporal and geographic candidates for when and where; the cause chain detects causal conjunctions, causative adverbs, and causative-verb patterns for why; and the method chain finds copulative-conjunction clauses and adjective/adverb phrases for how. Each candidate is scored by a weighted sum $s_q = \\sum_i w_{q,i} s_{q,i}$ of position, frequency, and type-specific factors, with weights tuned by a semi-automated search on the annotated dataset using semantic and temporal/geographic error distances; a combined scorer then adjusts method scores by the sentence distance to the chosen action. This combination of canonicalization, parse-tree rules, and learned weights carries the extraction from raw article text to a concise event description.","core_discovery":"The paper's central claim is that the journalistic 5W1H questions are a workable operational definition of a news article's main event, and that a carefully engineered pipeline of syntactic and domain-specific rules can answer them from explicit text with usable precision. The system canonicalizes temporal mentions to standardized timestamps, locations to geocoordinates, and other named entities to knowledge-graph concepts; extracts candidates through four dedicated chains; and selects final answers with per-question weighted scoring plus a combined sentence-distance adjustment, with roughly twenty scoring weights learned semi-automatically from a new annotated dataset. In the reported evaluation, mean average generalized precision is strong for who (0.92), what (0.79), when (0.78), and where (0.78), but weaker for why (0.48) and how (0.61). The authors further state that this is the only universal, open-source 5W1H extractor, and that it outperforms their earlier five-W system.","pith_inferences":["Inference: the scoring weights tuned on roughly 80 training articles from major US and UK outlets may not transfer to local, wire-service, or non-Western news styles; a cross-outlet generalization test would reveal how universal the 'universal' claim really is.","Inference: the weaker why and how scores suggest the bottleneck is not the 5W1H schema itself but the detection of explicitly signaled causes and methods; a learned or hybrid component for those two questions could be slotted into the modular pipeline and tested against the same benchmark.","Inference: if the 0.82 for the first four Ws holds across domains, main-event extraction for explicitly reported events becomes a nearly solved subtask, shifting research attention to implicit events, cross-document event coreference, and canonical event representation.","Inference: the reported numbers are not directly comparable to earlier precision figures because relevance scales and test sets differ; a shared evaluation corpus would let the community arbitrate between systems."],"forward_implications":["Researchers can use the system and its datasets directly, avoiding the redundant reimplementation of event extraction that motivated the paper.","Downstream tasks such as article clustering, summarization, and news aggregation can consume six standardized phrases, with temporal and geospatial information in canonical form.","Because the first four Ws alone reach 0.82 precision, applications that need only who, what, when, and where can treat that subset as sufficiently reliable for many analyses.","The modular architecture allows future work to replace individual components, such as the cause or method extractors, without discarding the rest of the pipeline.","The newly released annotated dataset gives the field a shared training and evaluation resource, making future system comparisons possible for the first time."],"supporting_citations":[{"why":"Defines the earlier five-W system that Giveme5W1H extends and must outperform, and supplies the evaluation procedure reused here.","marker":"[17]"},{"why":"Supplies the temporal normalization that turns natural-language dates into standardized time instances for 'when' candidates.","marker":"[7]"},{"why":"Supplies named-entity disambiguation that links entities to knowledge-graph concepts for canonical output and scoring.","marker":"[19]"},{"why":"Provides the causative-verb inventory and pattern constraints used by the cause extractor to find 'why' candidates.","marker":"[11]"},{"why":"Provides the cause-effect indicator lexicon (causal conjunctions and adverbs) used by the cause extractor.","marker":"[25]"},{"why":"Provides the semantic phrase-distance measure used as an error function when learning the scoring weights.","marker":"[27]"},{"why":"Supplies the news corpus from which the evaluation's 120 articles were sampled.","marker":"[12]"},{"why":"Defines the graded-relevance evaluation framework underlying the mean average generalized precision score.","marker":"[23]"},{"why":"Provides the strongest prior five-W precision figure that the paper compares against.","marker":"[48]"},{"why":"Provides the only other published evaluation covering all six 5W1H questions, used as a comparison point.","marker":"[24]"}],"fun_headline_variants":["Giveme5W1H: open-source news event extractor at 73% precision","Universal 5W1H system answers news main events with 73% precision","5W1H news tool: 82% precision on first four W's","Who, what, when, where: news event extraction at 82% precision","Extract main news events via 5W1H questions at 73% precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy assumes that the roughly twenty scoring weights tuned on an 80-article training sample from 13 major US and UK outlets will perform just as well on news articles generally, including other outlets, genres, and writing styles.","fun_headline_variants_meta":{"raw":{"variants":["Giveme5W1H: open-source news event extractor at 73% precision","Universal 5W1H system answers news main events with 73% precision","5W1H news tool: 82% precision on first four W's","Who, what, when, where: news event extraction at 82% precision","Extract main news events via 5W1H questions at 73% precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1550,"prompt_tokens":981,"completion_tokens":569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":459}},"tokens_in":597,"tokens_out":569,"duration_ms":5903,"temperature":1.0,"reasoning_tokens":459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:39:38.924943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of more than 100 articles from outlets and beats not represented in the training data, such as local newspapers or financial wire services, have three annotators score the extracted phrases on the same graded relevance scale, and check whether mean average generalized precision on the first four W questions remains near 0.82; if it falls well below that, the tuned configuration is tied to the training distribution rather than being universal.","supporting_citations":[{"cited_title":"Proceedings of the ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL) (Fort Worth, Texas, USA, 2018), 339–340","cited_arxiv_id":null,"evidence_quote":"Supplies named-entity disambiguation that links entities to knowledge-graph concepts for canonical output and scoring."},{"cited_title":"Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics-Volume 1 (1998), 571–577","cited_arxiv_id":null,"evidence_quote":"Provides the cause-effect indicator lexicon (causal conjunctions and adverbs) used by the cause extractor."},{"cited_title":"Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (2012), 540–555","cited_arxiv_id":null,"evidence_quote":"Supplies the news corpus from which the evaluation's 120 articles were sampled."},{"cited_title":"INTERSPEECH (2009), 2707–2710","cited_arxiv_id":null,"evidence_quote":"Provides the strongest prior five-W precision figure that the paper compares against."},{"cited_title":"Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (2005), 510–515","cited_arxiv_id":null,"evidence_quote":"Provides the only other published evaluation covering all six 5W1H questions, used as a comparison point."}],"review_version":1}