LLM-IE: A Python Package for Generative Information Extraction with Large Language Models

Enshuo Hsu; Kirk Roberts

REVIEW 5 major objections 6 minor 1 cited by

LLM-IE: A Python Package for Generative Information Extraction with Large Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper presents LLM-IE, a Python package that turns LLM-based clinical information extraction into modular building blocks, and reports that its sentence-by-sentence extractor gets the best F1 scores on i2b2/n2c2 benchmarks.

desk verdict A useful software package whose load-bearing benchmark ranking is not yet rigorous enough to trust, but which clearly deserves peer review rather than a desk rejection. read the letter →

arxiv 2411.11779 v1 pith:JQFHRDON submitted 2024-11-18 cs.LG

Enshuo Hsu , Kirk Roberts This is my paper · ORCID

classification cs.LG

keywords largelanguagemodelsinformationextractionnamedentityrecognitionrelationclinicalNLPpromptengineeringPythonpackagebiomedicalnaturalprocessing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's goal is to remove the engineering friction that keeps clinical NLP teams from using large language models for information extraction. It presents LLM-IE, a Python package whose building blocks cover the whole pipeline—task definition, prompt design, entity and attribute extraction, relation extraction, storage, and visualization—and whose central innovation is the Prompt Editor, an interactive LLM agent that turns a plain task description into a structured prompt template with schema and output format. The benchmark results are the paper's concrete evidence: on three i2b2/n2c2 clinical challenges, the sentence-based frame extractor achieved the highest F1 for named entity recognition and entity attributes, but at the cost of longer inference time. If the package works as described, it would let biomedical NLP practitioners build and inspect extraction pipelines without becoming prompt-engineering specialists.

What carries the argument

The load-bearing objects are frames and the Prompt Editor. A frame is the package's structured unit of extracted information, holding a frame ID, entity text, entity spans, and a set of attributes, produced when a FrameExtractor converts the LLM's JSON output through post-processing. The Prompt Editor is an LLM agent with access to stored prompt templates and writing guidelines; it generates prompt templates containing task description, schema definition, output format definition, and input placeholders, and users iterate with it until the template is final. The argument is carried by the extractors: BasicFrameExtractor prompts once, ReviewFrameExtractor prompts again to amend its own output, and SentenceFrameExtractor splits the document into sentences and prompts sentence by sentence, which the paper attributes to better recall and span detection.

What would settle it

Re-run the three frame extractors on the same three challenges with several different sets of eight example prompts and at least one additional open-weight model, and count how often each extractor produces outputs that must be discarded. If sentence-by-sentence prompting no longer holds the top F1, or if the ranking changes across prompt sets and models, the reported order of extractors is not settled.

Watch

Extended reading notes

Core claim

The central claim is that an integrated software layer for LLM-based information extraction is feasible and useful: one package can present different underlying LLM inference engines through a single interface, implement the prompting algorithms the field has published, and guide users through prompt and schema design with an agent. The reported discovery is that on the 2012 i2b2 temporal relations, 2014 i2b2 de-identification, and 2018 n2c2 adverse drug event benchmarks, the SentenceFrameExtractor—which prompts the model one sentence at a time—consistently beats the direct BasicFrameExtractor and the self-reviewing ReviewFrameExtractor in F1 for both entities and attributes, while the Review extractor improves recall over Basic on all NER tasks. The multi-class relation extractor reaches very high recall with lower precision. The package also demonstrates an end-to-end system that extracts drug, condition, and adverse drug event frames, links them with relations, and renders the result for human inspection.

Load-bearing premise

The load-bearing premise is that testing each extractor once on each challenge, with one model and one set of eight example prompts, and simply discarding outputs that do not follow the required format, is enough to rank the extractors fairly.

Editorial extensions

If this is right

A user can start from a short task description and end with a working NER, attribute, or relation extraction pipeline by chatting with the Prompt Editor and then applying an extractor through a uniform engine interface.
For clinical entity and attribute extraction, sentence-by-sentence prompting should be the default algorithm when inference time is acceptable, because it produced the best F1 on every benchmark task reported.
A review pass is a cheap way to raise recall over one-shot prompting without switching to sentence splitting, since the Review extractor outperformed Basic in recall on all NER tasks.
Relation extraction can be kept cheap by pre-processing frame pairs with decision rules that skip LLM calls when a relation is impossible, as demonstrated with drug and dosage frames.
Because all prompt templates, inputs, and outputs are stored and visualizable, error analysis on complex extraction schemas is part of the pipeline rather than an afterthought.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not report whether Prompt Editor–generated prompts were held fixed across extractors or compared against hand-written prompts, so the agent's contribution to the benchmark results remains untested; a natural experiment is to run the same extractors with and without Prompt Editor guidance.
The benchmark is a single run in an 8-shot setting with one model and discards malformed JSON outputs, so the extractor ranking in Table 2 should be read as provisional; re-running across few-shot exemplar sets, temperatures, and smaller open-weight models would show whether the sentence extractor's lead is stable.
The frame data model, with explicit spans and overlap validation, points toward nested or overlapping entity extraction, a capability the paper does not claim or test.
A sentence-splitting extractor that also does a review pass is an obvious combination implied by the paper's two best mechanisms, but it is not implemented, so its possible gain is an untested prediction.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

LLM-IE is a credible piece of software, and the paper's real contribution is the package itself: a modular, documented toolkit with a prompt-editing agent and support for several inference engines. The benchmark reruns published extractor algorithms and confirms the same broad ordering people have seen elsewhere, so the results have face validity. The i2b2/n2c2 numbers are a useful reference point for practitioners choosing among Basic, Review, and Sentence extractors. I also credit the authors for publishing code and data on GitHub, and for their honest limitation list.

The soft spots are real but mostly fixable. The benchmark claim that the Sentence extractor is best is the load-bearing result, yet the exact prompts, exemplar counts, and iteration history are not in the paper—they point to a GitHub repo. That makes the comparison vulnerable to the criticism that they tuned one prompt harder than the others. The Discussion admits "Inconsistent elements in the JSON list are discarded," which means parse-failure rates could be changing the F1 scores asymmetrically. No error bars or multiple runs are reported, so the gaps between extractors are not shown to be reliable. Those three issues together mean the ranking in Table 2 is not yet secure. And the paper's own Table SX is missing, which breaks the otherwise clean presentation.

The Prompt Editor is the genuinely new piece, but it is never evaluated in isolation. Is it any better than handwriting prompts? We do not know. That is a missed opportunity, but not a fatal flaw. The "no dedicated software available" claim reads as overstated, though the package appears to be one of few integrated toolkits in this specific niche.

Who is this for? Practitioners who want a ready-made LLM-IE pipeline and a baseline to cite. It is not a scientific breakthrough, and the empirical findings need revision. But it is a serious engineering paper with reproducible artifacts. I would send it to peer review—an experienced referee can demand the missing prompts, parse-failure rates, and error bars. It might be worth a reading group discussion on how to evaluate tool papers, but not for the benchmark numbers themselves.

Referee Report

5 major / 6 minor

Summary. The manuscript describes LLM-IE, a Python package for building LLM-based information extraction pipelines. It provides a uniform interface to different inference engines, three frame extractors (Basic, Review, Sentence), relation extractors, data management and visualization types, and a Prompt Editor agent intended to help users with schema definition and prompt design. The authors benchmark the extractors on three i2b2/n2c2 datasets using Llama-3.1-70B in an 8-shot setting and report that SentenceFrameExtractor achieves the highest F1 while consuming more GPU time. They also demonstrate an end-to-end system evaluation on a synthesized clinical note with a visualization.

Significance. If the results hold, LLM-IE would be a useful contribution as an integrated, modular toolkit that lowers the barrier to LLM-based clinical information extraction. The release on PyPI and GitHub, the coverage of multiple prompting algorithms, and the built-in Prompt Editor address a real practical gap. However, the paper's headline empirical claim is currently under-supported: the benchmark lacks error bars, the exact prompts are not reported, parse-failure rates are not given, and the key innovation (Prompt Editor) is never evaluated in isolation. These gaps must be closed before the software's value proposition is fully credible.

major comments (5)

[Benchmarking and System Evaluation / Table 2] Table 2 reports that SentenceFrameExtractor outperforms the other extractors in F1 by large margins (e.g., 0.78 vs 0.55 on the 2012 EVENT task), but the comparison does not control for prompt quality. The prompt templates generated with the Prompt Editor and the exemplar counts are not included in the manuscript; readers are only referred to a GitHub repository. If the Sentence prompt was more carefully engineered or contained more informative few-shot examples, the measured advantage could be a property of the prompt rather than the algorithm. Please provide the exact prompt templates for all extractors, the exemplar counts, and a controlled comparison (for example, a shared prompt backbone or a systematic variation of exemplars) to support the claim that the Sentence algorithm itself is the best.
[Benchmarking and System Evaluation / Table 2] The benchmark reports a single run per extractor with no variance estimate. Many differences are small (e.g., 0.555 vs 0.5755 F1 for Review vs Basic on the 2014 de-identification strict setting; 0.5505 vs 0.5567 on the relaxed setting), and without multiple seeds, bootstrap confidence intervals, or paired significance tests the ranking in Table 2 is not statistically grounded. At least three runs per condition with mean and standard deviation, or a paired test across documents, should be reported for the main comparisons.
[Discussion, limitation 3] The paper states that 'Inconsistent elements in the JSON list are discarded,' but no parse-failure rates are given per extractor. This is a potential source of bias: if one extractor produces more malformed JSON outputs, discarding those elements removes false positives and false negatives asymmetrically, which can inflate its F1 and recall. Please report the number and proportion of discarded elements for each extractor, and consider a sensitivity analysis that treats unparsed elements as errors rather than excluding them.
[Methods, System Design / Objective] The abstract identifies the Prompt Editor as the key innovation, yet no experiment or user study evaluates it. There is no evidence that prompts created with the Prompt Editor are better than manually written prompts, or that the agent's suggestions improve downstream extraction performance. Please add an evaluation of the Prompt Editor (for example, comparing user-authored prompts with and without its assistance, or a qualitative/quantitative quality assessment of generated templates).
[Methods, System Design] The text references 'Table SX' for implementation details, but no such table appears in the manuscript. This placeholder leaves the implementation details of the extractors, post-processing, and relation extraction incomplete. Please supply the table or remove the reference and provide the details in the main text or a proper supplement.

minor comments (6)

[Title page / affiliations] The affiliation line contains a typo: 'Enteprise' should be 'Enterprise'.
[System Evaluation section] In the sentence 'We utilized the LLE-IE package', 'LLE-IE' should be 'LLM-IE'.
[Table 2] The column header 'GPU time (s)/ Note' is confusing; please clarify the units and what 'Note' refers to.
[Methods, Prompt Editor] The system prompt for the Prompt Editor is truncated with an ellipsis; either show the full prompt or indicate that it is abridged.
[Methods, Prompt Editor] The template shown for the chat prompt includes placeholders such as {{prompt_guideline}}, but the text does not explain how these placeholders are populated; a brief explanation would improve readability.
[Figure 3 caption] The caption notes that only a few entity attributes are displayed for publication; the selection of displayed attributes should be described in the main text for transparency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark claims are evaluated against independent i2b2/n2c2 corpora, and the sole self-citation is a non-load-bearing qualitative comparison.

full rationale

LLM-IE is a software-description paper rather than a derivation: it makes no formal prediction, fits no parameters, and invokes no uniqueness theorem. The central empirical claim, that SentenceFrameExtractor gives the best Table 2 F1, is grounded in external gold standards (2012/2014 i2b2 and 2018 n2c2) with Llama-3.1-70B and independently cited prompting algorithms [3,8-14,17], so the ranking is not true by construction. The only self-citation, "The benchmark results are reasonable compared to our recent publication [22]," is a qualitative consistency remark in the Discussion and is not load-bearing for any result. The stated limitations, including "Inconsistent elements in the JSON list are discarded," the user's responsibility for finalizing prompt templates, and single-LLM evaluation, are genuine threats to benchmark reliability and reproducibility, but they are experimental-validity concerns rather than circular steps: no conclusion is presupposed by the package's own definitions or by a self-citation chain. Therefore no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central results are benchmark comparisons, not derivations, so the ledger holds no fitted constants. The largest hand-chosen inputs are the 8-shot exemplars and prompt templates; the core assumptions are about LLM output reliability and the representativeness of the benchmark setup.

free parameters (2)

8-shot exemplar configuration = not specified in the manuscript
All benchmark runs used an 8-shot setting, but the exact exemplars and their order are not reported; this hand-chosen setup can substantially affect few-shot extraction scores.
prompt templates for each extractor = not included in the manuscript
The prompt templates that define schema, output format, and examples are the core intervention; they are generated via the Prompt Editor but not shown or evaluated.

assumptions (4)

domain assumption LLM outputs can be parsed into JSON frames, and discarding malformed elements does not bias comparisons.
Invoked in Discussion limitation point 3; the benchmark and pipeline assume discarded outputs are not systematically related to extractor quality.
domain assumption i2b2/n2c2 gold standards are accepted measures of clinical information extraction performance.
The benchmarks use these shared tasks as evaluation, so their annotation guidelines define what counts as correct extraction.
ad hoc to paper The Prompt Editor's generated prompt templates are usable and helpful.
No evaluation is provided for the key innovation; the paper assumes the LLM agent writes useful prompts.
domain assumption Llama-3.1-70B with the vLLM engine is representative of current open-source LLM inference for these tasks.
All benchmark results come from a single model and engine, so generalizability to other LLMs is assumed rather than shown.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM-IE: A Python Package for Generative Information Extraction with Large Language Models." pith.science (2026). https://pith.science/paper/JQFHRDON

@misc{pith2026241111779,
  author       = {Pith},
  title        = {Pith review of: LLM-IE: A Python Package for Generative Information Extraction with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQFHRDON}},
  note         = {Machine review of arXiv:2411.11779}
}

read the original abstract

Objectives: Despite the recent adoption of large language models (LLMs) for biomedical information extraction, challenges in prompt engineering and algorithms persist, with no dedicated software available. To address this, we developed LLM-IE: a Python package for building complete information extraction pipelines. Our key innovation is an interactive LLM agent to support schema definition and prompt design. Materials and Methods: The LLM-IE supports named entity recognition, entity attribute extraction, and relation extraction tasks. We benchmarked on the i2b2 datasets and conducted a system evaluation. Results: The sentence-based prompting algorithm resulted in the best performance while requiring a longer inference time. System evaluation provided intuitive visualization. Discussion: LLM-IE was designed from practical NLP experience in healthcare and has been adopted in internal projects. It should hold great value to the biomedical NLP community. Conclusion: We developed a Python package, LLM-IE, that provides building blocks for robust information extraction pipeline construction.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Enhancing Generative Information Extraction with Two-step Validation: A Product Attribute Use Case
cs.CL 2026-07 conditional novelty 4.0 of 10

Reformulating generative product IE as LLM correction of PLM drafts improves F1 on weakly expressed attributes and lets mid-size local models approach larger ones.

Reference graph

Works this paper leans on

7 extracted references · 3 canonical work pages · cited by 1 Pith paper

[1]

2015;58:S11–9

J Biomed Inform. 2015;58:S11–9. doi: 10.1016/j.jbi.2015.06.007 20 Henry S, Buchan K, Filannino M, et al. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J Am Med Inform Assoc. 2019;27:3–12. doi: 10.1093/jamia/ocz166 21 Dubey A, Jauhri A, Pandey A, et al. The Llama 3 Herd of Models

doi:10.1016/j.jbi.2015.06.007 2015
[3]

https://dl.acm.org/doi/10.1145/3600006.3613165 (accessed 9 October

7 Efficient Memory Management for Large Language Model Serving with PagedAttention | Proceedings of the 29th Symposium on Operating Systems Principles. https://dl.acm.org/doi/10.1145/3600006.3613165 (accessed 9 October

arXiv
[4]

Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers

12 Jahan I, Laskar MTR, Peng C, et al. Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. Toronto, Canada: Association for Computational Linguistics 2023:326–36. 13 Wadhwa S, Amir S, Wallace B. Revisiting Relation Ex...

work page 2023
[5]

Large Language Models as Instructors: A Study on Multilingual Clinical Entity Extraction

17 Meoni S, De la Clergerie E, Ryffel T. Large Language Models as Instructors: A Study on Multilingual Clinical Entity Extraction. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. Toronto, Canada: Association for Computational Linguistics 2023:178–90. 18 Sun W, Rumshisky A, Uzuner O. Evaluating temporal relations in cli...

work page 2023
[13]

Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track

doi: 10.1136/amiajnl-2013-001628 19 Stubbs A, Kotfila C, Uzuner O. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track

doi:10.1136/amiajnl-2013-001628 2013
[2022]

Structured information extraction from scientific text with large language models

4 Dagdelen J, Dunn A, Lee S, et al. Structured information extraction from scientific text with large language models. Nat Commun. 2024;15:1418. doi: 10.1038/s41467-024-45563-x 5 Ollama. https://ollama.com (accessed 9 August

doi:10.1038/s41467-024-45563-x 2024
[2024]

Language Models are Few-Shot Learners

2 Brown T, Mann B, Ryder N, et al. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems. 2020;33:1877–

work page 2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

2015;58:S11–9

J Biomed Inform. 2015;58:S11–9. doi: 10.1016/j.jbi.2015.06.007 20 Henry S, Buchan K, Filannino M, et al. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J Am Med Inform Assoc. 2019;27:3–12. doi: 10.1093/jamia/ocz166 21 Dubey A, Jauhri A, Pandey A, et al. The Llama 3 Herd of Models

doi:10.1016/j.jbi.2015.06.007 2015

[2] [3]

https://dl.acm.org/doi/10.1145/3600006.3613165 (accessed 9 October

7 Efficient Memory Management for Large Language Model Serving with PagedAttention | Proceedings of the 29th Symposium on Operating Systems Principles. https://dl.acm.org/doi/10.1145/3600006.3613165 (accessed 9 October

arXiv

[3] [4]

Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers

12 Jahan I, Laskar MTR, Peng C, et al. Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. Toronto, Canada: Association for Computational Linguistics 2023:326–36. 13 Wadhwa S, Amir S, Wallace B. Revisiting Relation Ex...

work page 2023

[4] [5]

Large Language Models as Instructors: A Study on Multilingual Clinical Entity Extraction

17 Meoni S, De la Clergerie E, Ryffel T. Large Language Models as Instructors: A Study on Multilingual Clinical Entity Extraction. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks. Toronto, Canada: Association for Computational Linguistics 2023:178–90. 18 Sun W, Rumshisky A, Uzuner O. Evaluating temporal relations in cli...

work page 2023

[5] [13]

Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track

doi: 10.1136/amiajnl-2013-001628 19 Stubbs A, Kotfila C, Uzuner O. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track

doi:10.1136/amiajnl-2013-001628 2013

[6] [2022]

Structured information extraction from scientific text with large language models

4 Dagdelen J, Dunn A, Lee S, et al. Structured information extraction from scientific text with large language models. Nat Commun. 2024;15:1418. doi: 10.1038/s41467-024-45563-x 5 Ollama. https://ollama.com (accessed 9 August

doi:10.1038/s41467-024-45563-x 2024

[7] [2024]

Language Models are Few-Shot Learners

2 Brown T, Mann B, Ryder N, et al. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems. 2020;33:1877–

work page 2020