Pith. sign in

REVIEW 3 major objections 4 minor 31 references

Benchmarking Agentic Newswriting via Journalistic Workflows

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read NEWSAGENT claims that current LLM agents can fetch relevant facts for news stories but cannot plan or integrate them into a coherent narrative.

desk verdict A useful new agentic benchmark for newswriting, but the function-wise F1 scores rest on an exact-match ground truth that conflates retrieval with stylistic divergence, and the printed numbers have at least one arithmetic inconsistency. read the letter →

arxiv 2509.00446 v2 pith:DYTJAT3H submitted 2025-08-30 cs.AI

classification cs.AI
keywords agenticnewswritingLLMagentsjournalismbenchmarktime-awareretrievalinformationworkflownarrativeplanningnewsgenerationmultimodalcontent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NEWSAGENT is a benchmark that turns real newswriting into an agentic task: given a news title, a release date, and firsthand material, an agent must search a historical database, edit a draft, and rephrase it into a finished article. The paper's central claim is that current agents handle the retrieval half of this workflow, finding relevant facts, but fail at the planning half: they rarely revise or remove content, they choose different facts than human journalists, and the resulting articles lag on narrative integration. To back this, the authors built 6,237 human-verified examples from BBC and AP News articles, splitting each into firsthand and historical information by publication date, and evaluated several LLMs under a ReAct-style loop. If the claim is right, NEWSAGENT gives the field a realistic testbed for measuring whether agents can turn web data into a coherent product, rather than merely summarizing pre-retrieved text.

What carries the argument

The unit of analysis is the 'object,' a semantically coherent piece of text (description, image caption, or transcript segment) extracted from a real article. Each task splits objects into firsthand information (available at release time) and historical information (published before the release date), stored in a separate database. Two functions carry the workflow: a time-aware Search that returns top-k historical objects by embedding cosine similarity, and an Edit function with Insert and Remove actions that mutate the draft. Agents loop through these actions under a ReAct framework until they issue Terminate, after which a rephrasing step produces the final article; a dimension-wise GPT-4

What would settle it

Give each agent the exact set of historical objects the human journalist used, let it write the article from those, and have professional editors blind-compare these forced-alignment articles against the agent's free-search output. If forced-alignment does not reliably win on quality, the benchmark's equation of human selection with the right information is cast into doubt.

Watch

Extended reading notes

Core claim

The paper's central discovery is a measurable gap between retrieval and narrative construction in agentic newswriting. Across all tested models, agents successfully locate relevant historical context when searching, but their editing behavior shows no self-correction: Remove is never invoked, and Insert operations rarely target the same information human journalists chose. Function-wise F1 scores against the human-authored ground truth are low for both search and edit, yet end-to-end pairwise evaluations find that human-written articles do not consistently win, and closed-source models do not consistently beat open-source ones. The authors interpret this as evidence that agent-human divergen

Load-bearing premise

The benchmark treats the historical objects that human journalists actually used in the published article as the correct answer, so an agent that picks different but equally valid background facts is counted as wrong.

Editorial extensions

If this is right

  • NEWSAGENT can serve as a testbed for whether new planning or self-correction mechanisms improve agentic content generation, with separate scores for search, edit, and final article.
  • The finding that agents never remove content suggests that without explicit error signals, LLM agents will not revise; adding revision incentives or feedback is a concrete next target.
  • Because closed-source models do not consistently outperform open-source ones on this workflow, model selection for editorial tasks should be based on workflow-specific evaluation rather than general reasoning benchmarks.
  • The weak correlation between F1 alignment with human content and end-to-end win rates implies that matching the human article and writing a good article are distinct evaluation axes that benchmarks should report separately.
  • The 2-step execution setting's precision-recall trade-off shows that interface design changes agent behavior measurably, so interaction design is part of the capability being measured.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper treats the human journalist's historical-object selection as ground truth; a natural extension is to collect multiple acceptable reference articles per event and score agents against the union, separating editorial divergence from capability deficit.
  • The temporal split, with historical information defined strictly by publication date, could transfer to other open-ended information work such as financial reporting or medical writing, where missing context must be actively discovered.
  • The observation that human articles lose on style and importance dimensions to LLM output suggests the GPT-4 evaluator may reward stylistic elaboration over concision; testing with professional journalist judges would show whether that preference is real.
  • A testable extension is to give agents an explicit revise action with feedback, such as a signal that the draft is redundant, since the current result that Remove is never invoked may be an artifact of the absence of such a signal rather than a fundamental limitation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces NEWSAGENT, a benchmark of 6,237 (also stated as 6,327) human-verified news-writing tasks derived from BBC and AP News articles. Each article is decomposed into firsthand and historical objects, and an agent is given a title, release date, and firsthand materials, then must issue time-aware searches, insert relevant historical objects into a draft, optionally remove content, and rephrase the final article. The authors evaluate GPT-4o, GPT-4o mini, Gemma-3, Qwen3-32B, and Llama-4-Scout under ReAct in 1-step and 2-step execution modes. They report function-wise precision/recall/F1 against human-written ground truth and end-to-end pairwise comparisons via a dimension-wise GPT-4 judge. The main claimed findings are that agents can retrieve relevant facts but diverge from human editorial choices, rarely self-correct (Remove is never invoked), and that closed-source models do not uniformly outperform open-source models.

Significance. If the benchmark and its evaluation are sound, NEWSAGENT fills a real gap: it models an iterative, time-constrained journalistic workflow rather than static retrieval-augmented generation. The human-verified dataset, the temporal leakage control, and the release of code are concrete strengths. The paper also makes a useful negative finding about current agents' limited self-correction and the divergence between search behavior and final article quality. However, the central quantitative claims currently rest on function-wise F1 scores that contain numerical inconsistencies, and on a ground-truth formulation that conflates exact lexical match with retrieval and planning ability. These issues must be addressed before the benchmark's headline conclusions can be trusted.

major comments (3)
  1. [§3.3, Table 3] The reported F1 values are inconsistent with the reported precision and recall. For example, GPT-4o 1-step Search with P=0.327 and R=0.292 has F1 = 2·0.327·0.292/(0.327+0.292) = 0.308, not 0.233. GPT-4o mini 1-step Search (0.282, 0.322) yields F1 = 0.301, not 0.231; the Rule-Based row (0.040, 0.174) yields 0.065, not 0.058. Since the paper's 'divergence' finding and the claim that agents 'struggle with planning' rest on these numbers, all function-wise metrics and their interpretations in §4.2.2 and the takeaways need to be recomputed and corrected. Also, the GPT-4o mini row contains '0322' instead of '0.322'.
  2. [§3.3, §4.2.2] The Search and Edit ground truth is defined as the exact historical objects used in the published human article, and Insert/Remove matching requires exact string equality after punctuation removal. Because the database contains objects from 31k articles, the same fact commonly appears in different wordings. An agent that retrieves the correct fact but not the exact sentence from the original article is penalized. Low F1 therefore conflates genuine retrieval or planning failures with legitimate lexical divergence. The paper itself concedes in §4.2.2 that divergence 'does not necessarily imply lower article quality,' yet the Abstract and §5 still interpret low F1 as 'struggling with planning and narrative integration.' Please either adopt a semantic-equivalence matching procedure (e.g., paraphrase-aware scoring) or report what fraction of misses are near-duplicate reformulations, and softe
  3. [§3.3, §4.3] The end-to-end judge is GPT-4, which was also used during curation to classify objects as firsthand vs historical. This same-model-family evaluation may systematically prefer certain stylistic or structural patterns, possibly explaining the result that human-written articles do not consistently win and that similarity to human outputs does not indicate quality. The 40-article validation (72% agreement) is useful but was performed on GPT-4o/GPT-4o-mini outputs, not on the full set of model comparisons used in the paper. Please report human agreement on a sample of actual pairwise comparisons among the evaluated agents, and ideally per-dimension agreement. This is needed to rule out judge bias as an alternative explanation for the central end-to-end findings.
minor comments (4)
  1. [Abstract, §1, §3.1.2] The number of examples is inconsistently reported: 6,237 in the abstract and introduction, but 6,327 in §3.1.2. Please reconcile.
  2. [§3.2.2, §4.1.3] The search top-k (5), similarity threshold (0.7), and rule-based baseline threshold (0.8) are fixed. A short sensitivity analysis or justification would help establish that the results are not threshold artifacts.
  3. [§4.1.3] The exact-match requirement after punctuation removal is not specified in enough detail: does it also normalize case, articles, or whitespace? Supply the exact matching transformation so that the procedure is reproducible.
  4. [Table 2] The table header arrangement is hard to parse: the 1-step/2-step columns apply to six different metrics, and the row for GPT-4o has 12 numeric entries. A clearer layout, with each metric grouped under its own sub-header, would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark's ground truth is external human-written articles, and no result reduces to its own inputs by construction.

full rationale

NEWSAGENT is an empirical benchmark construction and evaluation, not a formal derivation, so the relevant question is whether any conclusion is forced by definition, fitted parameter, or self-citation. The core ground truth for Search and Edit metrics (Section 3.3, Table 3) is the set of objects human journalists actually used in the published article; this is an external artifact collected from BBC/APNews via Fundus Scraper (Section 3.1), not a quantity fitted or generated by the paper's models. The divergence finding (Section 4.2.2) is an F1 measurement against that external set, and the paper explicitly states it does not necessarily imply lower article quality, so it does not define divergence as failure. The end-to-end judge is GPT-4, and GPT-4 was also used to label objects as firsthand/historical (Section 3.1.2 vs 3.3); this is a same-model-family overlap that could affect evaluation validity, but it is not a circular reduction: the object labels were independently reviewed by two annotators and agreement-filtered, the judge was validated against human preferences (72% agreement), and the benchmark's central claims about search/edit behavior rely on human ground truth rather than on GPT-4's own preferences. The only self-citation among the authors, [21], appears in related work as an example of journalism complexity and is not load-bearing for any benchmark construction or conclusion. No equation is defined in terms of its own output, no fitted parameter is renamed a prediction, and no uniqueness or ansatz is imported from the authors' prior work. Hence no circular step meets the evidence bar.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on the GPT-4-based decomposition of articles into firsthand and historical objects, the assumption that human-written selections are the correct ground truth for relevance, and the reliability of the GPT-4 evaluator. No new physical or scientific entities are postulated; the free parameters are engineering choices in the retrieval and evaluation setup.

free parameters (4)
  • Search retrieval threshold = 0.7
    Cosine similarity threshold above which search results are returned; affects precision/recall trade-off.
  • Search top-k = 5
    Number of retrieved objects returned per search, set to 5 in experiments.
  • Rule-based baseline threshold = 0.8
    Cosine similarity threshold used by the rule-based baseline to insert historical objects.
  • Operation limit = 20
    Maximum number of Search/Insert/Remove actions before termination.
assumptions (4)
  • domain assumption GPT-4's classification of article objects into firsthand vs historical information, filtered by human agreement, correctly represents the information available to a journalist.
    The benchmark's task formulation and ground truth depend on this decomposition (Sec 3.1.2).
  • domain assumption The set of historical objects used in the human-written article is a valid ground truth for measuring the relevance of agent-retrieved information.
    Search/Edit F1 is computed against the human-written article (Sec 3.3).
  • domain assumption A 72% agreement rate between the dimension-wise GPT-4 evaluator and human preferences, measured on 120 pairs, makes the evaluator a reliable proxy.
    The end-to-end ranking relies on this evaluator (Sec 3.3).
  • domain assumption Embedding-based cosine similarity with all-MiniLM-L6-v2 captures object-level relevance for news content.
    Used for retrieval and for the rule-based baseline (Sec 4.1.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Agentic Newswriting via Journalistic Workflows." pith.science (2026). https://pith.science/paper/DYTJAT3H

@misc{pith2026250900446,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Agentic Newswriting via Journalistic Workflows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DYTJAT3H}},
  note         = {Machine review of arXiv:2509.00446}
}
read the original abstract

Recent advances in autonomous digital agents from industry (e.g., Manus AI and Gemini's research mode) highlight their potential for structured tasks through autonomous decision-making and task decomposition, but it remains unclear how well such systems support real-world information-intensive workflows. We study this question in journalism, where newswriting requires iterative planning, contextual reasoning, and active discovery of missing background to produce a coherent article. We introduce NEWSAGENT, a benchmark for evaluating how agents search raw materials, select relevant information, and iteratively revise drafts through core journalistic functions. Given a writing instruction and partial firsthand materials, agents must identify narrative perspectives, issue keyword-based queries, retrieve historical context, and generate complete news articles. Unlike typical summarization or retrieval tasks, essential context is not directly available and must be actively discovered, reflecting real-world reporting constraints. NEWSAGENT consists of 6k human-verified examples derived from real news. We evaluate open- and closed-sourced LLMs with commonly-used agentic frameworks on NEWSAGENT, which shows that agents are capable of retrieving relevant facts but struggling with planning and narrative integration. We believe that NEWSAGENT serves a realistic testbed for iterating and evaluating agent capabilities in terms of web data manipulation to real-world productivity.

Figures

Figures reproduced from arXiv: 2509.00446 by the authors.

Figure 1
Figure 1. Comparison between one-time content generation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the NEWSAGENT. To construct the benchmark, we collect news articles, extract the news title and release [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Object distribution by type. any historical information were discarded. Two annotators indepen￾dently reviewed all GPT-4 classifications, and any object without agreement was removed, resulting in 6,327 validated articles. 3.1.3 Dataset Distribution. Across the dataset, 69% of objects are classified as firsthand information and 31% as historical informa￾tion [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Pairwise head-to-head win rates (%) for the end-to-end newswriting task. Each cell shows the percentage of cases in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Dimension-wise preference distributions for pairwise model comparisons across six evaluation dimensions. Each pie [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison between a BBC human-written article (left) and Qwen3-32B output (right) for the same news event. Blue [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 21 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

  2. [2]

    Anthropic. 2024. Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. https://www.anthropic.com/news/3-5-models-and-computer-use

  3. [3]

    Significant-gravitas/auto-GPT: An experimental open-source attempt to make GPT-4 fully autonomous

    autogpt 2023. Significant-gravitas/auto-GPT: An experimental open-source attempt to make GPT-4 fully autonomous. https://github.com/Significant- Gravitas/Auto-GPT/tree/master

  4. [4]

    Max Dallabetta, Conrad Dobberstein, Adrian Breiding, and Alan Akbik. 2024. Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Yixin Cao, Yang Feng, and Deyi Xiong (Eds.). Association for Computational Ling...

  5. [5]

    Jinyuan Fang, Zaiqiao Meng, and Craig MacDonald. 2025. KiRAG: Knowledge- Driven Iterative Retriever for Enhancing Retrieval-Augmented Generation. In ACL (1). Association for Computational Linguistics, 18969–18985

  6. [6]

    Google Team. 2025. Introducing Gemini Deep Research. https://gemini.google/ overview/deep-research/. Accessed: 2025-04-06

  7. [7]

    Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau,...

  8. [8]

    Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, Yu Liu, Chunyuan Li, and Hongsheng Li. 2025. MMSearch: Unveiling the Potential of Large Models as Multi-modal Search Engines. In ICLR. OpenReview.net

Show all 31 references
  1. [9]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In NeurIPS

  2. [10]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. Agent...

  3. [11]

    Meta. 2025. Llama 4 Scout 17B-16E Instruct. https://huggingface.co/meta- llama/Llama-4-Scout-17B-16E-Instruct

  4. [12]

    Nickerson, and Lydia B

    Savvas Petridis, Nicholas Diakopoulos, Kevin Crowston, Mark Hansen, Keren Henderson, Stan Jastrzebski, Jeffrey V. Nickerson, and Lydia B. Chilton. 2023. AngleKindling: Supporting Journalistic Angle Ideation with Large Language Models. In CHI. ACM, 225:1–225:16

  5. [13]

    Claudia Quinonez and Edgar Meij. 2024. A new era of AI-assisted journalism at Bloomberg. AI Mag. 45, 2 (2024), 187–199

  6. [14]

    Minjie Shen, Yanshu Li, Lulu Chen, and Qikai Yang. 2025. From Mind to Machine: The Rise of Manus AI as a Fully Autonomous Digital Agent. arXiv:2505.02024 [cs.AI] https://arxiv.org/abs/2505.02024

  7. [15]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In NeurIPS

  8. [16]

    Alexander Spangher, Yao Ming, Xinyu Hua, and Nanyun Peng. 2022. Sequentially Controlled Text Generation. InEMNLP (Findings). Association for Computational Linguistics, 6848–6866

  9. [17]

    Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze

  10. [18]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025)

  11. [19]

    Qwen Team. 2025. Qwen3-32B. https://huggingface.co/Qwen/Qwen3-32B

  12. [20]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  13. [21]

    Wei-Yao Wang, Yu-Chieh Chang, and Wen-Chih Peng. 2024. Style-News: In- corporating Stylized News Generation and Adversarial Verification for Neural Fake News Detection. In EACL (1). Association for Computational Linguistics, 1531–1541

  14. [22]

    Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023. Describe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task Agents. InNeurIPS

  15. [23]

    Zhao Wang, Sota Moriyama, Wei-Yao Wang, Briti Gangopadhyay, and Shingo Takamatsu. 2025. Talk Structurally, Act Hierarchically: A Collaborative Frame- work for LLM Multi-Agent Systems. arXiv:2502.11098 [cs.AI] https://arxiv.org/ abs/2502.11098

  16. [24]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS

  17. [25]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. CoRR abs/2308.08155 (2023)

  18. [26]

    Jialin Yang, Dongfu Jiang, Lipeng He, Sherman Siu, Yuxuan Zhang, Disen Liao, Zhuofeng Li, Huaye Zeng, Yiming Jia, Haozhe Wang, Benjamin Schneider, Chi Ruan, Wentao Ma, Zhiheng Lyu, Yifei Wang, Yi Lu, Quy Duc Do, Ziyan Jiang, Ping Nie, and Wenhu Chen. 2025. StructEval: Benchmar...

  19. [27]

    Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan

    Lili Yao, Nanyun Peng, Ralph M. Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-Write: Towards Better Automatic Storytelling. In AAAI. AAAI Press, 7378–7385

  20. [28]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In NeurIPS

  21. [29]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR. OpenReview.net

  22. [2023]

    In ACL (1)

    Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge- Intensive Multi-Step Questions. In ACL (1). Association for Computational Lin- guistics, 10014–10037

  23. [2024]

    In EMNLP

    Do LLMs Plan Like Human Writers? Comparing Journalist Coverage of Press Releases with LLMs. In EMNLP. Association for Computational Linguistics, 21814–21828

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.