Pith. sign in

REVIEW 3 major objections 5 minor 26 references

A two-stage language-model system tags nearly 300,000 SEC 8-K filings into 119 quote-grounded event types, with a dedicated quality score that raises precision from 12% to 96% and separates market reactions that share one coarse SEC item co

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 08:58 UTC pith:CVNXRY25

load-bearing objection Solid systems + measurement paper: releases 601k grounded 8-K tags, shows second-pass scoring is the real dial, and backs the taxonomy with a no-LLM event study. the 3 major comments →

arxiv 2607.08346 v1 pith:CVNXRY25 submitted 2026-07-09 cs.CL q-fin.GN

Grounded Event Extraction from SEC 8-K Filings with a Fine-Grained Taxonomy

Rian Dolphin , Joe Dursun , Jarrett Blankenship , Katie Adams , Quinton Pike This is my paper
classification cs.CL q-fin.GN
keywords SEC filingsevent extractionlarge language modelsevent studyLLM-as-judgeForm 8-Kquote groundingquality score
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

U.S. public companies report material events on Form 8-K, but the SEC's item codes are too coarse: one code mixes CEO departures with routine director retirements, and many price-moving disclosures land in a catch-all. This paper builds a two-stage extraction system that assigns each filing tags from a fixed 119-type economic taxonomy, forces every tag to cite a verbatim quote from the filing, and then re-grades that quote alone to produce a 1–5 quality score. Applied to 292,984 filings, it yields 601,088 grounded tags that the authors release. A stronger model acting as judge finds that precision climbs monotonically with the score while unsupported tags fall to near zero, and an event study that uses no language model shows the tags separate economically distinct moves that item codes lump together. The result is a usable filtering dial and a public corpus for fine-grained disclosure research.

Core claim

The paper establishes that schema-constrained extraction plus fuzzy n-gram quote grounding, followed by a dedicated second pass that re-grades each cited quote against its category definition, produces a calibrated quality score: judge-assessed precision rises from 12% at score 1 to 96% at score 5, unsupported tags fall from 8% to near zero, and an unsigned abnormal-return event study confirms that the resulting 119-type taxonomy separates market reactions within and across SEC item codes.

What carries the argument

The two-stage pipeline: stage one extracts taxonomy tags under schema validation and fuzzy four-gram quote grounding so every tag is a valid type anchored to filing text; stage two re-reads only the cited quote against the category definition to assign a 1–5 quality score that functions as a filtering dial.

Load-bearing premise

That a stronger language model judging a stratified sample of tags is a reliable enough stand-in for human labels that the reported precision-by-score curve can be trusted without large-scale human annotation.

What would settle it

Large-scale human annotation of the same 5,125 stratified tags showing that precision does not rise monotonically with the quality score, or an independent event study on high-score tags failing to separate reaction magnitudes within Item 5.02 (for example CEO departures versus routine officer appointments).

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Downstream users can threshold the quality score to trade tag coverage for higher precision without re-running either model stage.
  • Event studies can isolate economically distinct disclosures that currently share a single SEC item code, such as CEO departures versus routine officer appointments under Item 5.02.
  • The released corpus of 601,088 grounded, scored tags over 292,984 filings supports fine-grained disclosure measurement beyond item codes.
  • Content-based tags can track event classes (for example cybersecurity incidents) continuously across regulatory regime changes that introduce or alter item codes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same two-stage pattern—constrained extraction plus dedicated quote re-grading—should transfer to other structured financial disclosures such as earnings call transcripts or press releases.
  • Inline self-scoring saturating while second-pass re-grading calibrates points to a general design rule for confidence in constrained language-model extraction, not only for 8-Ks.
  • Stability of market-reaction rankings after high-confidence filtering suggests most of the economic signal lives in the high-precision subset, so low-score tags can often be discarded rather than repaired.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a two-stage LLM pipeline that tags Form 8-K filings against a three-tier taxonomy of 119 corporate event types. Stage 1 enforces schema-valid taxonomy entries and anchors every tag to a supporting span via fuzzy four-gram validation; stage 2 re-grades each cited quote against the category definition to produce a 1–5 quality score. Applied to 292,984 filings (2022–mid-2026), the system yields 601,088 grounded tags, which are released. Over a 5,125-tag stratified sample, an LLM judge reports precision rising monotonically from 12% (score 1) to 96% (score 5), with unsupported tags falling from 8% to near zero; an ablation shows that only dedicated second-pass scoring yields a usable dial. An event study on unsigned standardized abnormal returns over 182,174 cleanly attributable filings, using no language model, shows that the taxonomy separates economically distinct events that share the same SEC item code and that tags add more reaction-relevant variance than item codes add to tags.

Significance. If the results hold, the work supplies both a practical measurement instrument and a reusable extraction recipe for a core disclosure corpus. Strengths that should be credited explicitly include: (i) structural reliability mechanisms (schema constraints plus verbatim quote grounding) that make individual tags checkable without re-running the model; (ii) the clear methodological finding that second-pass re-grading, not inline self-rating, produces a calibrated quality dial; (iii) a non-circular economic validation that uses market reactions alone; and (iv) public release of 601,088 quality-scored, quote-grounded tags. Together these make fine-grained 8-K event labels usable for empirical finance and for further NLP work on grounded extraction, and they document the economic coarseness of SEC item codes with evidence that does not depend on language models.

major comments (3)
  1. Section 5.1–5.2: The precision-by-score curve and the claim that second-pass scoring is calibrated rest entirely on an Opus 4.8 LLM judge, with no human annotation sample reported. The design (full filing + single category definition + cited quote; separate correct / clearly-wrong / not-supported verdicts; reweighting to the corpus score distribution) is careful, and the economic rankings are shown to be robust under high-score filtering (Spearman 0.939). Even so, for the central claim that the quality score is a usable reliability dial, a modest human audit—e.g., a few hundred stratified tags with inter-annotator agreement—would substantially reduce dependence on the unvalidated judge axiom and should be added or clearly scoped as a limitation with a concrete plan.
  2. Section 3.1: The three-tier taxonomy of 119 tertiary types is a core invented entity of the paper, yet its construction process is under-specified (who drafted the definitions, what sources or prior event inventories were used, how near-duplicate or ambiguous types were resolved, and whether any external review was performed). Because every precision and event-study claim is relative to this fixed vocabulary, a short construction appendix or reproducibility note is load-bearing for others to interpret or extend the released tags.
  3. Section 3.2 (quote grounding): The acceptance rule—≥40% of overlapping four-word runs must appear verbatim—is a free parameter that directly bounds the hallucination claim. No sensitivity analysis is reported (e.g., 30%/50% thresholds, alternative n, or exact-match baselines). A brief ablation would show whether the reported unsupported rates and the second-stage score distribution are stable under reasonable alternatives, or whether the 40% cut is material to the results.
minor comments (5)
  1. Figure 2: The dual-panel precision / hallucination sweep is informative, but the overlapping ≥1…≥5 labels on both axes make the operating points hard to read at a glance; a small legend or annotated keep-thresholds would help.
  2. Section 6.1: The SAR construction (absolute market-adjusted return scaled by baseline σ, max of t0 and t0+1) is standard and well motivated, but the no-news benchmark of 1.0 for the two-day measure is stated without a short derivation or simulation check; one sentence would clarify why readers should treat points above 1.0 as larger-than-normal.
  3. Section 4 / data availability: The release links are given, which is excellent; please also state the exact license and whether the raw model outputs (pre-validation retries) or only the final validated tags are included, so downstream users know the audit surface.
  4. Related work (Section 2.2): The comparison to taxonomy-aligned 10-K risk extraction [3] is useful; a one-sentence contrast on event granularity (discrete events vs. risk factors) and on second-pass scoring vs. post-hoc judge filtering would sharpen the positioning.
  5. Throughout: Occasional phrasing such as “we use the word calibration loosely” (Section 5.2) is appropriately cautious; consider defining the score as an ordinal reliability rank rather than “calibration” in the abstract and introduction to avoid probabilistic connotations.

Circularity Check

0 steps flagged

No significant circularity: quality-score calibration is checked against an independent stronger judge and market reactions, not against the pipeline's own outputs by construction.

full rationale

This is an empirical systems-and-measurement paper, not a first-principles derivation. The two-stage extraction (schema constraints + fuzzy n-gram quote grounding, then a dedicated quote-only re-grade) produces tags and quality scores; those scores are then audited by a stronger external LLM judge (Opus 4.8) on a stratified sample and, separately, by an event-study design that uses only market returns and SEC item codes and explicitly 'makes no return-prediction claims' and 'uses no language model.' The paper itself flags that the second-stage grader belongs to the same model family as the extractor and is therefore a form of self-assessment, but immediately measures its value against the independent judge rather than against self-agreement; the economic rankings remain essentially unchanged after high-score filtering (Spearman 0.939). Self-citations to the authors' prior 10-K and news-extraction work appear only in Related Work and are not load-bearing premises or uniqueness theorems. No fitted parameter is renamed a prediction, no definition equates an output to its input, and no ansatz is smuggled in via citation. The residual dependence on LLM judges is an ordinary evaluation limitation already acknowledged in the paper, not a circular reduction of the central claims. Score 1 reflects only the mild, non-load-bearing self-assessment caveat; the derivation chain is otherwise self-contained against external market benchmarks.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central claims rest on a small set of domain and methodological assumptions rather than free parameters or new physical entities. The main free choices are engineering thresholds (n-gram overlap, quality scale, event-study windows) and the author-written taxonomy definitions. No fitted scientific constants drive the result; the quality score is an output, not an input parameter of a derivation.

free parameters (3)
  • fuzzy n-gram acceptance threshold = 40%
    Quote accepted only when ≥40% of overlapping 4-word runs appear verbatim in the filing; chosen by authors, not derived.
  • quality score scale (1–5) = 1–5 integer
    Integer rubric defined by the authors for the second-stage grader; the calibration claim is empirical, not forced by the scale itself.
  • event-study baseline and event windows = 60d / 11d gap / 2-day window
    60-day baseline ending 11 days before filing; two-day max reaction on t0 and t0+1; minimum 40 trading days, price >$1, etc. Standard but author-chosen.
axioms (4)
  • domain assumption SEC item codes are economically coarse labels that mix heterogeneous events and leave many material events in catch-all Item 8.01.
    Stated in Introduction and Related Work; treated as background motivation rather than a result to be proved.
  • ad hoc to paper A stronger LLM judge given the full filing, single category definition, and cited quote can serve as a reliable oracle for correctness and support.
    Section 5.1; no large human annotation set is provided to validate the judge itself.
  • domain assumption Unsigned, variance-standardized abnormal returns (SAR) over a two-day window measure the economic distinctness of event types without requiring directional prediction.
    Section 6.1; standard event-study practice adapted to the paper's unsigned design.
  • ad hoc to paper Schema validation plus fuzzy n-gram quote checks make fabricated quotes structurally impossible and thereby bound hallucination.
    Section 3.2; the 40% threshold and 4-word runs are design choices whose residual error is measured empirically.
invented entities (2)
  • three-tier taxonomy of 119 corporate event types independent evidence
    purpose: Provides the fixed vocabulary of economically motivated labels that replace or refine SEC item codes.
    Author-constructed hierarchy with primary/secondary/tertiary levels and short definitions; not taken from an external standard.
  • second-stage quality score (1–5) independent evidence
    purpose: Calibrated reliability dial attached to every tag so users can trade coverage for precision without re-running extraction.
    Defined and produced by the paper's own grader pass; validated against an external judge and market reactions.

pith-pipeline@v1.1.0-grok45 · 17857 in / 3396 out tokens · 28506 ms · 2026-07-10T08:58:30.602006+00:00 · methodology

0 comments
read the original abstract

Form 8-K filings are the primary channel through which U.S. public companies disclose material events, but the SEC item codes attached to them are coarse: a single item spans routine administrative changes and chief executive departures, and many of the most market-moving disclosures fall into a catch-all item. Large language models make fine-grained labelling feasible at corpus scale, but only if the labels can be traced to the source text and shown to be reliable. We present a two-stage system that tags 8-K disclosures against a three-tier taxonomy of 119 event types. The first stage constrains output to valid taxonomy entries and anchors every tag to a verbatim quote via fuzzy n-gram validation; the second re-grades each cited quote against the category definition to produce a quality score. Applying the system to 292,984 filings from 2022 to 2026 yields 601,088 grounded event tags, which we release. Over 5,125 stratified tags, an LLM judge finds precision rises monotonically with the quality score, from 12% to 96%, while unsupported tags fall from 8% to near zero. Ablation shows the score is calibrated only when assigned in a dedicated second pass. An event study on unsigned abnormal returns confirms, without any language model, that the taxonomy separates economically distinct events sharing an item code.

Figures

Figures reproduced from arXiv: 2607.08346 by Jarrett Blankenship, Joe Dursun, Katie Adams, Quinton Pike, Rian Dolphin.

Figure 1
Figure 1. Figure 1: Calibration of the quality score: judge-assessed [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Precision against the fraction of tags retained as the quality-score keep-threshold sweeps from 1 to 5. The second-stage [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mean reaction size (SAR) for filings whose only substantive item is 5.02, grouped by taxonomy tag, with 95% confidence intervals. The dashed line marks the no-news benchmark. least reactive ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Quarterly counts of filings tagged as cybersecurity [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Share of 8-K filers per quarter carrying financial [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Share of each sector’s 8-K filings carrying selected [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages · 5 internal anchors

  1. [1]

    Dogu Araci. 2019. FinBERT: Financial sentiment analysis with pre-trained language models.arXiv preprint arXiv:1908.10063(2019)

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  2. [2]

    Ekkehart Boehmer, Jim Musumeci, and Annette B. Poulsen. 1991. Event-study methodology under conditions of event-induced variance.Journal of Financial Economics30, 2 (1991), 253–272

    work page 1991

  3. [3]

    Rian Dolphin, Joe Dursun, Jarrett Blankenship, Katie Adams, and Quinton Pike

    work page

  4. [4]

    Taxonomy-Aligned Risk Extraction from 10-K Filings with Autonomous Improvement Using LLMs.arXiv preprint arXiv:2601.15247(2026)

    work page arXiv 2026

  5. [5]

    Rian Dolphin, Joe Dursun, Jonathan Chow, Jarrett Blankenship, Katie Adams, and Quinton Pike. 2024. Extracting structured insights from financial news: An augmented LLM driven approach.arXiv preprint arXiv:2407.15788(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. 2023. Grammar- constrained decoding for structured NLP tasks without finetuning. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 10932–10952

    work page 2023

  7. [7]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks.Proceedings of the National Academy of Sciences120, 30 (2023), e2305016120

    work page 2023

  8. [8]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning. 1321–1330

    work page 2017

  9. [9]

    Jing He and Marlene A. Plumlee. 2020. Measuring disclosure using 8-K filings. Review of Accounting Studies25, 3 (2020), 903–962

    work page 2020

  10. [10]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation.Comput. Surveys55, 12 (2023), 1–38

    work page 2023

  11. [11]

    Alina Lerman and Joshua Livnat. 2010. The new Form 8-K disclosures.Review of Accounting Studies15, 4 (2010), 752–778

    work page 2010

  12. [12]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words.Transactions on Machine Learning Research(2022)

    work page 2022

  13. [13]

    Alejandro Lopez-Lira and Yuehua Tang. 2023. Can ChatGPT forecast stock price movements? Return predictability and large language models.arXiv preprint arXiv:2304.07619(2023)

    work page arXiv 2023

  14. [14]

    Tim Loughran and Bill McDonald. 2011. When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks.Journal of Finance66, 1 (2011), 35–65

    work page 2011

  15. [15]

    Tim Loughran and Bill McDonald. 2016. Textual analysis in accounting and finance: A survey.Journal of Accounting Research54, 4 (2016), 1187–1230

    work page 2016

  16. [16]

    Lefteris Loukas, Manos Fergadiotis, Ion Androutsopoulos, and Prodromos Malakasiotis. 2021. EDGAR-CORPUS: Billions of tokens make the world go round. InProceedings of the Third Workshop on Economics and Natural Language Processing. 13–18. Grounded Event Extraction from SEC 8-K Filings with a Fine-Grained Taxonomy

    work page 2021

  17. [17]

    Craig MacKinlay

    A. Craig MacKinlay. 1997. Event studies in economics and finance.Journal of Economic Literature35, 1 (1997), 13–39

    work page 1997

  18. [18]

    Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. Automated anno- tation with generative AI requires validation.arXiv preprint arXiv:2306.00176 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM evaluators recognize and favor their own generations. InAdvances in Neural Information Processing Systems, Vol. 37

    work page 2024

  20. [20]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine- tuned with human feedback. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing...

    work page 2023

  21. [21]

    Securities and Exchange Commission

    U.S. Securities and Exchange Commission. 2023. Cybersecurity risk management, strategy, governance, and incident disclosure. Release No. 33-11216; 88 FR 51896

    work page 2023

  22. [22]

    Efficient Guided Generation for Large Language Models

    Brandon T. Willard and Rémi Louf. 2023. Efficient guided generation for large language models.arXiv preprint arXiv:2307.09702(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebas- tian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann

    work page

  24. [24]

    BloombergGPT: A large language model for finance.arXiv preprint arXiv:2303.17564(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  25. [25]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT- Bench and Chatbot Arena. InAdvances in Neural Information Processing Systems, Vol. 36

    work page 2023

  26. [26]

    Zhihan Zhou, Liqian Ma, and Han Liu. 2021. Trade the event: Corporate events detection for news-based event-driven trading. InFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2114–2124

    work page 2021