Pith. sign in

REVIEW 3 major objections 5 minor 17 references

Automated Journalistic Questions: A New Method for Extracting 5W1H in French

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A transparent rule-based pipeline matches GPT-4o at French 5W1H extraction.

desk verdict First French 5W1H corpus and pipeline, but the parity claim with GPT-4o rests on an in-sample comparison and a 1-point gap with no significance test. read the letter →

arxiv 2505.14804 v2 pith:D5HKREXE submitted 2025-05-20 cs.CL cs.LG

Maxence Verhaverbeke , Julie A. Gramaccia , Richard Khoury This is my paper
classification cs.CLcs.LG
keywords 5W1HextractionFrenchnewsrule-basedNLPexplainableAIGPT-4obaselinejournalisticquestionsannotationcorpusCamemBERT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces the first automated system for extracting answers to the six journalistic questions (who, what, when, where, why, how) from French-language news articles. The proposed pipeline is rule-based and fully explainable: it combines standard French NLP tools with a French language model to identify candidate answers, then ranks them with weighted, question-specific scoring factors. To test it, the authors built a new corpus of 250 Quebec news articles in which four human annotators marked the 5W1H answers. The central result is that the pipeline's answers agree with human annotators about as often as GPT-4o's do (31% versus 32% average agreement), while remaining transparent about how each answer was chosen. The paper argues that this makes an explainable alternative to black-box generative models viable for French news processing.

What carries the argument

The central mechanism is a three-phase extraction pipeline. In the preprocessing phase, standard French NLP tools and a pre-trained French language model (CamemBERT) provide sentence splitting, part-of-speech tagging, named entity recognition, coreference resolution, and an initial set of answers from a question-answering module. In the candidate extraction phase, four independent modules gather candidate answers: actions for who/what, the environment for when/where, causes for why, and methods for how. In the scoring phase, each candidate receives a score $S_{c,q} = \sum_i w_{i,q} s_{i,q,c}$, a weighted sum of question-specific factors such as position in the article, frequency of coreferences, presence of causal markers, and temporal precision. A per-question score threshold then selects which candidates are returned as answers. The weights and thresholds are the load-bearing parameters: they encode the assumption that journalistic articles put core information early, repeat it often, and mark causality and method with recognizable linguistic cues.

What would settle it

Recompute agreement after splitting the 250-article corpus into a tuning set used to choose thresholds and weights and a held-out test set used for evaluation; if the pipeline's held-out agreement falls materially below GPT-4o's, the claim of parity would not generalize.

Watch

Extended reading notes

Core claim

The paper claims that a transparent, rule-based extraction pipeline can match the performance of the large language model GPT-4o at answering the 5W1H questions on French news articles. The evidence is the average agreement between each system's answers and the answers of four human annotators on a new 250-article corpus: 31% for the pipeline versus 32% for GPT-4o. The pipeline performs better on the who, what, where, and when questions, while GPT-4o performs better on the harder why and how questions. The authors also provide the first annotated French 5W1H corpus and argue that, because the pipeline's decision process is explainable, it can be audited and improved more easily than a generative model.

Load-bearing premise

The parity claim rests on the assumption that the reported 31% agreement is not inflated by tuning the pipeline's score thresholds and weights on the same 250-article corpus that is later used for evaluation.

Editorial extensions

If this is right

  • French news summarization, clustering, and aggregation systems can now use a free, explainable 5W1H extractor rather than relying on a black-box generative model.
  • The public 250-article annotated corpus gives French NLP a benchmark for 5W1H tasks, something that previously did not exist.
  • The pipeline's success on who/what/where/when suggests that structured news leads contain most of the information needed for these questions, and that rule-based methods can capture it.
  • The gap on why/how indicates that the underlying linguistic signals for causality and manner are more varied and implicit; the paper states that better weights and extraction rules could close this gap.
  • The transparency of the system allows journalists and researchers to trace exactly which factors led to each answer, which is not possible with GPT-4o.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the parity report survives out-of-sample testing, it would suggest that for highly structured news genres, a small set of positional and lexical signals captures most of what a large language model extracts, which could inform lightweight 5W1H tools for low-resource languages.
  • The evaluation protocol tunes per-question thresholds and weights on the same 250-article corpus used for the reported agreement, so the 31% figure may be optimistic; a held-out split or cross-validation would test whether parity with GPT-4o is stable on new articles.
  • The 'rule-based' label obscures some neural components: the preprocessing phase uses a CamemBERT question-answering module and coreference classifier, so the transparency claim applies to the scoring and selection logic, not to every component.
  • The low inter-annotator agreement (average 0.42) suggests the human reference itself is subjective; agreement-based evaluation may therefore be noisy, and a finer-grained span-matching metric might rank the two systems differently.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes the first automated 5W1H extraction pipeline for French news articles, built as a modular rule-based system with preprocessing (SpaCy, CamemBERT, a Q&A module), candidate extraction for who/what/when/where/why/how, and weighted scoring over question-specific factors. It also introduces a corpus of about 250 Québec French news articles annotated by four independent annotators. The evaluation compares the pipeline's agreement with human annotators against GPT-4o's agreement with the same annotators, reporting an average agreement of 31% for the pipeline and 32% for GPT-4o, which is interpreted as evidence that the pipeline performs as well as GPT-4o on this task.

Significance. If the parity claim held, the paper would make a useful contribution: a transparent, explainable French 5W1H extractor plus the first annotated French 5W1H corpus would be a practical resource for news analysis, summarization, and aggregation. The paper also has the virtue of comparing a rule-based system against a frontier LLM in a controlled extraction task, which is an interesting and timely baseline study. However, the evaluation protocol has a load-bearing weakness: the headline agreement is an in-sample quantity, because per-question thresholds and scoring weights were selected on the same 250-article corpus used for the reported results. This makes the central claim currently unsupported and in need of a held-out evaluation.

major comments (3)
  1. [§3.5, §5.3, Figure 4] The headline 31% agreement is not an out-of-sample estimate. Section 3.5 states: "We tested multiple thresholds and picked the optimal one per question as the one that allows the algorithm to select the set of answer that best matches that returned by humans," and Section 5.3 states: "The optimal thresholds indicated in this figure are also those used to generate the results of Figure 4." Thus the thresholds are selected to maximize agreement with the human annotations on the same corpus that produces the reported 31%. The scoring weights in §3.4 are likewise described as "adjusted empirically" with no held-out data. GPT-4o is not tuned in any comparable way, so the comparison is between an in-sample optimized system and an untuned baseline. The authors should freeze all thresholds and weights on a development set, or report results from nested or held-out evaluation, and state explicitly which parameters were selected on which portion of the data.
  2. [§5.2, Figure 4] The parity claim rests on a 1-percentage-point difference between 31% and 32% average agreement, but no confidence intervals, per-article variance, or significance test are reported. With this sample size and the large annotator variability documented in §4.1, a 1-point difference is well within the range of noise. The authors should report per-article distributions and either bootstrap confidence intervals or a paired statistical test for the comparison with GPT-4o, and should also state how many GPT-4o runs were used, given that the model is stochastic.
  3. [§4.1, §3.3, Table 7] The evaluation metric itself is not validated. The agreement ratio is computed using "our previous measure for similar candidates," which depends on a word-ratio similarity threshold defined only as a "predefined threshold" in §3.3. No sensitivity analysis is given for this threshold, and the metric's behavior as a function of answer-set size is not characterized. This is relevant because Table 7 shows the algorithms return roughly twice as many answers per question on average as human annotators (2.27 and 2.59 versus about 1.16), and a larger response set can inflate chance overlap under a set-based ratio. The authors should report whether the 31% versus 32% conclusion is robust to the similarity threshold and to alternative evaluation metrics, such as token-level precision/recall or F1.
minor comments (5)
  1. [Abstract, §1] The phrase "Answering them is a crucial prerequisites" has a subject-verb agreement error, and the introduction contains the typo "pipelime algorithm" where "pipeline algorithm" is meant.
  2. [§1, §4] The corpus size is inconsistent: the introduction and methodology refer to 250 articles, while the contribution bullet in Section 1 says 249 articles. The count should be made consistent throughout.
  3. [§2, Reference [12]] Reference [12] is cited as a source that "generated POS and grammar tags for the text, then designed rules to map sentence components to 5W1H answers," but [12] appears to be the MLQA cross-lingual question-answering dataset paper. This citation mismatch should be corrected or the claim reassigned to the appropriate reference.
  4. [§3.3, §4.1] The "predefined threshold" for the word-ratio similarity between candidates is never given a numerical value, and the same similarity measure is used for both candidate deduplication and inter-annotator agreement. Please state the threshold explicitly and justify its choice.
  5. [§4.1] The worked agreement example is confusing: the sets in the numerator and denominator are not clearly the two annotators' answer sets, and the repetition of "Justin Trudeau" in the denominator obscures the computation. Rewriting this example with two unambiguous answer lists would improve clarity.

Circularity Check

1 steps flagged · score 5.0 of 10

The pipeline's headline 31% agreement is computed with per-question thresholds selected to maximize agreement with the same annotators on the same 250-article corpus, so the reported parity with GPT-4o rests on an in-sample fitted quantity.

  1. fitted input called prediction [Section 3.5 (Candidate Selection Phase) and Section 5.3 (Number of Answers)]
    "We tested multiple thresholds and picked the optimal one per question as the one that allows the algorithm to select the set of answer that best matches that returned by humans. ... The optimal thresholds indicated in this figure are also those used to generate the results of Figure 4."

    The reported headline result is the average agreement with human annotators: 31% for the pipeline vs 32% for GPT-4o (Section 5.2, Figure 4). That agreement is computed on the same 250-article corpus used to select the per-question score thresholds. The threshold directly controls how many candidate answers are returned, and the agreement metric is the ratio of similar answers to the total number of answers supplied by both parties, so changing the threshold mechanically changes the agreement. Choosing the threshold that maximizes agreement on the evaluation corpus and then reporting that maximized agreement as the system's performance makes the 31% figure an in-sample, fitted quantity rather than an out-of-sample prediction.

full rationale

The paper's derivation chain is otherwise self-contained: the pipeline adapts published components (Gimme5W1H, SpaCy, CamemBERT), the dataset is independently constructed, and the GPT-4o baseline is untuned. The one load-bearing circularity is that the per-question answer-selection thresholds are selected on the same 250-article corpus used for evaluation, specifically by maximizing agreement with the human annotators, and the resulting in-sample agreement is then presented as the system's performance in the abstract claim of parity with GPT-4o. This is a genuine fitted-input-called-prediction pattern: the reported 31% is, for each question, the maximum over the tested threshold values of the agreement on the evaluation data, not an unbiased estimate. The tuning space is small and the comparison to GPT-4o is not forced by construction, so the circularity is moderate rather than total; however, a held-out evaluation or threshold selection on a development set would be required to support the central claim as stated.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central contribution does not introduce new theoretical entities; the free parameters are the hand-tuned thresholds and weights, and the main axiomatic burden is the assumption that news structure and the off-the-shelf French models make the rule-based candidate extraction reliable.

free parameters (4)
  • Per-question answer score thresholds = Not given numerically; selected from Figure 5
    Chosen by testing multiple thresholds per question and keeping the one that maximizes agreement with human annotators on the evaluation corpus (Sections 3.5 and 5.3).
  • Scoring factor weights w_iq = Values in Tables 1-6
    Based on the weights of Gimme5W1H and 'adjusted empirically' on the corpus (Section 3.4), with no held-out validation.
  • Q&A module confidence threshold = 0.5
    Hand-chosen cutoff for retaining answers from the CamemBERT Q&A module (Section 3.2).
  • Candidate similarity word ratio threshold = Not specified
    A 'predefined threshold' used to decide when two candidates are equivalent; the value is not stated (Section 3.3).
assumptions (4)
  • domain assumption News articles follow the inverted pyramid structure, so position in the article is a reliable signal of importance.
    Invoked in the scoring factors for 'who', 'what', 'when' and 'where' (Section 3.4).
  • domain assumption The off-the-shelf French NLP components (SpaCy, CamemBERT NER and coreference, and the FQuAD/PIAF Q&A model) are accurate enough on Quebec news text for candidate generation.
    The pipeline depends on these components in the preprocessing phase (Section 3.2) without evaluating them on this corpus.
  • domain assumption The agreement ratio between two answer sets is a valid measure of 5W1H extraction quality.
    Used in place of precision and recall because annotations are subjective (Sections 4.1 and 5.2); its reliability is not validated.
  • domain assumption The 250-article corpus of Quebec news is representative of French news articles generally.
    The corpus is drawn from four Quebec outlets (Section 4), but generalization to other French-speaking regions and genres is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Journalistic Questions: A New Method for Extracting 5W1H in French." pith.science (2026). https://pith.science/paper/D5HKREXE

@misc{pith2026250514804,
  author       = {Pith},
  title        = {Pith review of: Automated Journalistic Questions: A New Method for Extracting 5W1H in French},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D5HKREXE}},
  note         = {Machine review of arXiv:2505.14804}
}
read the original abstract

The 5W1H questions -- who, what, when, where, why and how -- are commonly used in journalism to ensure that an article describes events clearly and systematically. Answering them is a crucial prerequisites for tasks such as summarization, clustering, and news aggregation. In this paper, we design the first automated extraction pipeline to get 5W1H information from French news articles. To evaluate the performance of our algorithm, we also create a corpus of 250 Quebec news articles with 5W1H answers marked by four human annotators. Our results demonstrate that our pipeline performs as well in this task as the large language model GPT-4o.

Figures

Figures reproduced from arXiv: 2505.14804 by the authors.

Figure 1
Figure 1. This pipeline begins by applying standard NLP preprocessing algorithms to clean up a news article’s text, then it uses four different phrase extraction algorithms to identify candidate answers to the six questions, and finally it ranks these candidates using question￾specific weighting schemes to return the best answer to each question. The authors of [6] used a similar approach but focusing more heavily on the key … view at source ↗
Figure 2
Figure 2. Our French-language 5W1H pipeline. 3.2. Preprocessing Phase Our preprocessing phase uses a mix of Spacy’s NLP tools and pre-trained CamemBERT Transformer models [15]. Specifically, we leverage SpaCy’s “fr_core_news_lg” model for ba￾sic NLP preprocessing operations, namely sentence splitting and tokenization, word lemma￾tization, part-of-speech (POS) tagging and syntactic parsing. Meanwhile, the CamemBERT￾based model… view at source ↗
Figure 3
Figure 3. Agreement between annotators per question. “who did what where and when” in the story. On the other hand, the agreement between annotators is much lower for the “why” and “how” questions, ranging between 0.14 and 0.28. This shows that determining the cause or manner of an event can be difficult for humans, a problem also observed in [5]. Moreover, these elements may be unclear, taken for granted, or implicitly descr… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Agreement between our algorithm and each annotator (left) and between GPT￾4o and each annotator (right). When we compare our algorithm and the baseline together, we can see that our algorithm agrees more with the annotators than GPT-4o for the easy W questions, but tha…
Figure 5
Figure 5. Figure 5: Comparison of average agreement per threshold value for each question type [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 14 canonical work pages

  1. [1]

    Evaluating the Inverted Pyramid Structure through Au- tomatic 5W1H Extraction and Summarization

    B. Keith, M. Horning, and T. Mitra. “Evaluating the Inverted Pyramid Structure through Au- tomatic 5W1H Extraction and Summarization”. In:Computational Journalism C+J(2020). Ed. by null.url: https://par.nsf.gov/biblio/10274096

    work page arXiv 2020

  2. [2]

    E. Goffman. Frame analysis: An essay on the organization of experience.Harvard University Press, 1974

    work page 1974

  3. [3]

    Deep semantic role labeling for tweets using 5W1H

    K. Chakma, A. Das, and S. Debbarma. “Deep semantic role labeling for tweets using 5W1H”. In: Computación y Sistemas23.3 (2019), pp. 751–763

    work page 2019

  4. [4]

    Evaluating the inverted pyramid structure through automatic 5w1h extraction and summarization

    B. Norambuena, M. Horning, and T. Mitra. “Evaluating the inverted pyramid structure through automatic 5w1h extraction and summarization”. In:Computational Journalism Sym- posium. 2020

    work page 2020

  5. [5]

    Giveme5w1h: A universal system for extracting main events from news articles

    F. Hamborg, C. Breitinger, and B. Gipp. “Giveme5w1h: A universal system for extracting main events from news articles”. In:arXiv preprint arXiv:1909.02766(2019)

    arXiv 2019

  6. [6]

    Chinese news event 5w1h elements extraction using semantic role labeling

    W. Wang, D. Zhao, and D. Wang. “Chinese news event 5w1h elements extraction using semantic role labeling”. In:2010 Third International Symposium on Information Processing. IEEE. 2010, pp. 484–489

    work page 2010

  7. [7]

    Extracting 5W1H event semantic elements from Chinese online news

    W. Wang, D. Zhao, L. Zou, D. Wang, and W. Zheng. “Extracting 5W1H event semantic elements from Chinese online news”. In:International Conference on Web-Age Information Management. Springer. 2010, pp. 644–655

    work page 2010

  8. [8]

    Martin-Lagardette

    J.-L. Martin-Lagardette. Le guide de l’écriture journalistique. 6th ed. Éditions La Découverte, 2005

    work page 2005

Show all 17 references
  1. [9]

    L. Ross. L’écriture de presse. 2nd ed. Gaëtan Morin Éditeur, 2005

    2005

  2. [10]

    The5wstructureforsentimentsummarization- visualization-tracking

    A.Das,S.Bandyaopadhyay,andB.Gambäck.“The5wstructureforsentimentsummarization- visualization-tracking”. In:International Conference on Intelligent Text Processing and Com- putational Linguistics. Springer. 2012, pp. 540–555

    2012

  3. [11]

    News event extraction using 5W1H approach & its analysis

    S. Sharma, R. Kumar, P. Bhadana, and S. Gupta. “News event extraction using 5W1H approach & its analysis”. In:International Journal of Scientific & Engineering Research4.5 (2013), pp. 2064–2068

    2013

  4. [12]

    MLQA: Evaluating cross-lingual extractive question answering

    P. Lewis, B. Oğuz, R. Rinott, S. Riedel, and H. Schwenk. “MLQA: Evaluating cross-lingual extractive question answering”. In:arXiv preprint arXiv:1910.07475(2019)

    2019 arXiv

  5. [13]

    Classification-based strategies for combining multiple 5-w question an- swering systems

    S. Yaman, D. Hakkani-Tür, G. Tur, R. Grishman, M. Harper, K. R. McKeown, A. Mey- ers, and K. Sharma. “Classification-based strategies for combining multiple 5-w question an- swering systems”. In:Tenth Annual Conference of the International Speech Communication Association. 2009

    2009

  6. [14]

    Who, what, when, where, why? comparing multiple approaches to the cross-lingual 5w task

    K. Parton, K. McKeown, R. E. Coyne, M. T. Diab, R. Grishman, D. Hakkani-Tür, M. Harper, H. Ji, W. Y. Ma, A. Meyers, et al. “Who, what, when, where, why? comparing multiple approaches to the cross-lingual 5w task”. In: (2009)

    2009

  7. [15]

    CamemBERT: a Tasty French Language Model

    L. Martin, B. Muller, P. J. O. Suárez, Y. Dupont, L. Romary, É. V. de la Clergerie, D. Seddah, and B. Sagot. “CamemBERT: a Tasty French Language Model”. In:CoRR abs/1911.03894 (2019). arXiv: 1911.03894. url: http://arxiv.org/abs/1911.03894

    2019 arXiv

  8. [16]

    what” score is≥ 0.2) • Why does who_answer act? (else if the “who

    W. G. Bleyer. Newspaper writing and editing. Houghton Mifflin, 1923. 13 Appendix Q&A Module Prompts who • Which person or company is the main subject of this event? what • What is happening to who_answer in this news article? The answer is in the opening sentences. (if the “wh...

    1923

  9. [17]

    who," "where,

    Who? 2. What? 3. Where? 4. When? 5. Why? 6. How? Here are the instructions for your task: - Provide direct quotations from the text for each question. - Do not translate or truncate the quotations. - You may give multiple answers for each question, but only include the most re...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.