{"id":"b6cae4d3-e1ce-473a-8af1-edceeca243ed","arxiv_id":"2411.11779","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-IE packages LLM-based information extraction with an interactive prompt-editing agent, and benchmarks show the SentenceFrameExtractor performs best on i2b2/n2c2 NER and attribute tasks.","lead":"LLM-IE is a Python package that helps programmers build tools for pulling names, facts, and connections out of text using large language models, and it includes an interactive assistant for designing prompts. The authors tested it on medical records and found that processing one sentence at a time gave the best accuracy, though it was slower.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Table 2 extractor ranking is not yet secure: the exact prompt templates are not reported and malformed JSON outputs are silently discarded, so the SentenceFrameExtractor advantage may reflect prompt tuning or scoring artifacts rather than the algorithm itself.","rationale":"The reader's weakest-assumption analysis identifies essentially the same load-bearing risk: a single 8-shot configuration, prompts produced with the assistance of the Prompt Editor but not evaluated, and malformed-JSON discarding make the Table 2 ranking fragile. I agree, and I would not change the CONDITIONAL verdict. The paper is an engineering contribution rather than a new learning method, so the benchmark is the main evidence that the package is useful and that the sentence-based extractor is the right default choice. That evidence is undermined by the lack of prompt transparency and by an acknowledged post-processing step that can bias comparisons. Neither issue is fatal: the package may well be useful, and the benchmark may be correct. But the authors should be asked to release the exact prompts and raw outputs, quantify parse failures, and show that the ranking is stable under reasonable variations in prompt writing and under a conservative treatment of malformed outputs. The missing Table SX and the over-broad 'no dedicated software' claim are secondary; they do not affect the correctness of the software design claim, but they should be fixed in revision. The concern is not about the authors' integrity; it is about the evidence base for a specific comparative result.","tokens_in":5854,"tokens_out":2294,"duration_ms":27063,"concrete_test":"Audit the benchmark by requesting, for every row of Table 2, the exact prompt template, the number of few-shot exemplars, the raw LLM outputs, and per-extractor JSON parse failure rates. Then recompute precision, recall, and F1 under two changes: (a) treat every malformed JSON element as an empty prediction instead of discarding it, and (b) rerun each extractor with three independently written prompt templates per algorithm to assess prompt sensitivity. If the Sentence extractor's lead over Basic/Review shrinks to within run-to-run variation or reverses under either change, the headline ranking is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that SentenceFrameExtractor is the best-performing algorithm, as shown in Table 2. But the comparison does not hold prompt quality constant. The Methods say users iteratively develop prompt templates with the Prompt Editor, and the benchmark description points to a GitHub repository for details, but the paper itself does not provide the three extractor-specific prompt templates, the exemplar counts, or the iteration history. If the Sentence prompt was more carefully tuned or contained more informative few-shot exemplars, the F1 gap could be a property of the prompt, not the extractor. The paper's own limitation statement says 'Inconsistent elements in the JSON list are discarded.' This discard policy can bias the benchmark: if one extractor produces more malformed JSON elements, discarding them removes potential false positives and false negatives asymmetrically, inflating the F1 of the extractor with worse instruction-following. No parse-failure rates are reported, so we cannot tell whether the Sentence extractor's higher recall is genuine or an artifact of a higher discard rate. The absence of error bars or multiple runs further means the 0.05-0.15 F1 differences between extractors are not shown to be statistically reliable. Since the abstract and results both present the sentence-based algorithm as 'the best performance,' this is the load-bearing empirical support for the software's value proposition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes LLM-IE, a Python package for building LLM-based information extraction pipelines. It provides a uniform interface to different inference engines, three frame extractors (Basic, Review, Sentence), relation extractors, data management and visualization types, and a Prompt Editor agent intended to help users with schema definition and prompt design. The authors benchmark the extractors on three i2b2/n2c2 datasets using Llama-3.1-70B in an 8-shot setting and report that SentenceFrameExtractor achieves the highest F1 while consuming more GPU time. They also demonstrate an end-to-end system evaluation on a synthesized clinical note with a visualization.","tokens_in":6103,"tokens_out":4021,"duration_ms":37228,"significance":"If the results hold, LLM-IE would be a useful contribution as an integrated, modular toolkit that lowers the barrier to LLM-based clinical information extraction. The release on PyPI and GitHub, the coverage of multiple prompting algorithms, and the built-in Prompt Editor address a real practical gap. However, the paper's headline empirical claim is currently under-supported: the benchmark lacks error bars, the exact prompts are not reported, parse-failure rates are not given, and the key innovation (Prompt Editor) is never evaluated in isolation. These gaps must be closed before the software's value proposition is fully credible.","major_comments":[{"comment":"Table 2 reports that SentenceFrameExtractor outperforms the other extractors in F1 by large margins (e.g., 0.78 vs 0.55 on the 2012 EVENT task), but the comparison does not control for prompt quality. The prompt templates generated with the Prompt Editor and the exemplar counts are not included in the manuscript; readers are only referred to a GitHub repository. If the Sentence prompt was more carefully engineered or contained more informative few-shot examples, the measured advantage could be a property of the prompt rather than the algorithm. Please provide the exact prompt templates for all extractors, the exemplar counts, and a controlled comparison (for example, a shared prompt backbone or a systematic variation of exemplars) to support the claim that the Sentence algorithm itself is the best.","section":"Benchmarking and System Evaluation / Table 2"},{"comment":"The benchmark reports a single run per extractor with no variance estimate. Many differences are small (e.g., 0.555 vs 0.5755 F1 for Review vs Basic on the 2014 de-identification strict setting; 0.5505 vs 0.5567 on the relaxed setting), and without multiple seeds, bootstrap confidence intervals, or paired significance tests the ranking in Table 2 is not statistically grounded. At least three runs per condition with mean and standard deviation, or a paired test across documents, should be reported for the main comparisons.","section":"Benchmarking and System Evaluation / Table 2"},{"comment":"The paper states that 'Inconsistent elements in the JSON list are discarded,' but no parse-failure rates are given per extractor. This is a potential source of bias: if one extractor produces more malformed JSON outputs, discarding those elements removes false positives and false negatives asymmetrically, which can inflate its F1 and recall. Please report the number and proportion of discarded elements for each extractor, and consider a sensitivity analysis that treats unparsed elements as errors rather than excluding them.","section":"Discussion, limitation 3"},{"comment":"The abstract identifies the Prompt Editor as the key innovation, yet no experiment or user study evaluates it. There is no evidence that prompts created with the Prompt Editor are better than manually written prompts, or that the agent's suggestions improve downstream extraction performance. Please add an evaluation of the Prompt Editor (for example, comparing user-authored prompts with and without its assistance, or a qualitative/quantitative quality assessment of generated templates).","section":"Methods, System Design / Objective"},{"comment":"The text references 'Table SX' for implementation details, but no such table appears in the manuscript. This placeholder leaves the implementation details of the extractors, post-processing, and relation extraction incomplete. Please supply the table or remove the reference and provide the details in the main text or a proper supplement.","section":"Methods, System Design"}],"minor_comments":[{"comment":"The affiliation line contains a typo: 'Enteprise' should be 'Enterprise'.","section":"Title page / affiliations"},{"comment":"In the sentence 'We utilized the LLE-IE package', 'LLE-IE' should be 'LLM-IE'.","section":"System Evaluation section"},{"comment":"The column header 'GPU time (s)/ Note' is confusing; please clarify the units and what 'Note' refers to.","section":"Table 2"},{"comment":"The system prompt for the Prompt Editor is truncated with an ellipsis; either show the full prompt or indicate that it is abridged.","section":"Methods, Prompt Editor"},{"comment":"The template shown for the chat prompt includes placeholders such as {{prompt_guideline}}, but the text does not explain how these placeholders are populated; a brief explanation would improve readability.","section":"Methods, Prompt Editor"},{"comment":"The caption notes that only a few entity attributes are displayed for publication; the selection of displayed attributes should be described in the main text for transparency.","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears intended for an application/software-oriented track. The main concern is the fragility of the benchmark evidence: the missing prompt templates, the absent error bars, the unreported discard rates, and the missing Table SX all make the central empirical claim difficult to verify. The paper is not a rejectable contribution in principle, but the revision must address these load-bearing gaps. I would be willing to look at a revised version that provides the missing details and a more rigorous comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LLM-IE is a credible piece of software, and the paper's real contribution is the package itself: a modular, documented toolkit with a prompt-editing agent and support for several inference engines. The benchmark reruns published extractor algorithms and confirms the same broad ordering people have seen elsewhere, so the results have face validity. The i2b2/n2c2 numbers are a useful reference point for practitioners choosing among Basic, Review, and Sentence extractors. I also credit the authors for publishing code and data on GitHub, and for their honest limitation list.\n\nThe soft spots are real but mostly fixable. The benchmark claim that the Sentence extractor is best is the load-bearing result, yet the exact prompts, exemplar counts, and iteration history are not in the paper—they point to a GitHub repo. That makes the comparison vulnerable to the criticism that they tuned one prompt harder than the others. The Discussion admits \"Inconsistent elements in the JSON list are discarded,\" which means parse-failure rates could be changing the F1 scores asymmetrically. No error bars or multiple runs are reported, so the gaps between extractors are not shown to be reliable. Those three issues together mean the ranking in Table 2 is not yet secure. And the paper's own Table SX is missing, which breaks the otherwise clean presentation.\n\nThe Prompt Editor is the genuinely new piece, but it is never evaluated in isolation. Is it any better than handwriting prompts? We do not know. That is a missed opportunity, but not a fatal flaw. The \"no dedicated software available\" claim reads as overstated, though the package appears to be one of few integrated toolkits in this specific niche.\n\nWho is this for? Practitioners who want a ready-made LLM-IE pipeline and a baseline to cite. It is not a scientific breakthrough, and the empirical findings need revision. But it is a serious engineering paper with reproducible artifacts. I would send it to peer review—an experienced referee can demand the missing prompts, parse-failure rates, and error bars. It might be worth a reading group discussion on how to evaluate tool papers, but not for the benchmark numbers themselves.","headline":"A useful software package whose load-bearing benchmark ranking is not yet rigorous enough to trust, but which clearly deserves peer review rather than a desk rejection.","tokens_in":6577,"tokens_out":2631,"would_cite":false,"duration_ms":26156,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents LLM-IE, a Python package that turns LLM-based clinical information extraction into modular building blocks, and reports that its sentence-by-sentence extractor gets the best F1 scores on i2b2/n2c2 benchmarks.","keywords":["large language models","information extraction","named entity recognition","relation extraction","clinical NLP","prompt engineering","Python package","biomedical natural language processing"],"falsifier":"Re-run the three frame extractors on the same three challenges with several different sets of eight example prompts and at least one additional open-weight model, and count how often each extractor produces outputs that must be discarded. If sentence-by-sentence prompting no longer holds the top F1, or if the ranking changes across prompt sets and models, the reported order of extractors is not settled.","tokens_in":5649,"feed_emoji":"🩺","tokens_out":10595,"duration_ms":91473,"temperature":0.7,"pith_summary":"The paper's goal is to remove the engineering friction that keeps clinical NLP teams from using large language models for information extraction. It presents LLM-IE, a Python package whose building blocks cover the whole pipeline—task definition, prompt design, entity and attribute extraction, relation extraction, storage, and visualization—and whose central innovation is the Prompt Editor, an interactive LLM agent that turns a plain task description into a structured prompt template with schema and output format. The benchmark results are the paper's concrete evidence: on three i2b2/n2c2 clinical challenges, the sentence-based frame extractor achieved the highest F1 for named entity recognition and entity attributes, but at the cost of longer inference time. If the package works as described, it would let biomedical NLP practitioners build and inspect extraction pipelines without becoming prompt-engineering specialists.","feed_headline":"Sentence-by-sentence prompting wins in new LLM extraction toolkit","feed_subtitle":"Modular package wraps LLM engines, extractors, and a prompt-writing chat agent for clinical text mining.","key_machinery":"The load-bearing objects are frames and the Prompt Editor. A frame is the package's structured unit of extracted information, holding a frame ID, entity text, entity spans, and a set of attributes, produced when a FrameExtractor converts the LLM's JSON output through post-processing. The Prompt Editor is an LLM agent with access to stored prompt templates and writing guidelines; it generates prompt templates containing task description, schema definition, output format definition, and input placeholders, and users iterate with it until the template is final. The argument is carried by the extractors: BasicFrameExtractor prompts once, ReviewFrameExtractor prompts again to amend its own output, and SentenceFrameExtractor splits the document into sentences and prompts sentence by sentence, which the paper attributes to better recall and span detection.","core_discovery":"The central claim is that an integrated software layer for LLM-based information extraction is feasible and useful: one package can present different underlying LLM inference engines through a single interface, implement the prompting algorithms the field has published, and guide users through prompt and schema design with an agent. The reported discovery is that on the 2012 i2b2 temporal relations, 2014 i2b2 de-identification, and 2018 n2c2 adverse drug event benchmarks, the SentenceFrameExtractor—which prompts the model one sentence at a time—consistently beats the direct BasicFrameExtractor and the self-reviewing ReviewFrameExtractor in F1 for both entities and attributes, while the Review extractor improves recall over Basic on all NER tasks. The multi-class relation extractor reaches very high recall with lower precision. The package also demonstrates an end-to-end system that extracts drug, condition, and adverse drug event frames, links them with relations, and renders the result for human inspection.","pith_inferences":["The paper does not report whether Prompt Editor–generated prompts were held fixed across extractors or compared against hand-written prompts, so the agent's contribution to the benchmark results remains untested; a natural experiment is to run the same extractors with and without Prompt Editor guidance.","The benchmark is a single run in an 8-shot setting with one model and discards malformed JSON outputs, so the extractor ranking in Table 2 should be read as provisional; re-running across few-shot exemplar sets, temperatures, and smaller open-weight models would show whether the sentence extractor's lead is stable.","The frame data model, with explicit spans and overlap validation, points toward nested or overlapping entity extraction, a capability the paper does not claim or test.","A sentence-splitting extractor that also does a review pass is an obvious combination implied by the paper's two best mechanisms, but it is not implemented, so its possible gain is an untested prediction."],"forward_implications":["A user can start from a short task description and end with a working NER, attribute, or relation extraction pipeline by chatting with the Prompt Editor and then applying an extractor through a uniform engine interface.","For clinical entity and attribute extraction, sentence-by-sentence prompting should be the default algorithm when inference time is acceptable, because it produced the best F1 on every benchmark task reported.","A review pass is a cheap way to raise recall over one-shot prompting without switching to sentence splitting, since the Review extractor outperformed Basic in recall on all NER tasks.","Relation extraction can be kept cheap by pre-processing frame pairs with decision rules that skip LLM calls when a relation is impossible, as demonstrated with drug and dosage frames.","Because all prompt templates, inputs, and outputs are stored and visualizable, error analysis on complex extraction schemas is part of the pipeline rather than an afterthought."],"supporting_citations":[{"why":"Supplies the few-shot clinical extraction method that the BasicFrameExtractor implements.","marker":"[3]"},{"why":"Shows end-to-end structured information extraction from text, which motivates treating entity and attribute extraction as frame extraction.","marker":"[4]"},{"why":"The inference engine used to run the benchmarks, abstracted behind the Engines module's uniform interface.","marker":"[7]"},{"why":"Provide the self-reflection and self-check prompting strategies that the ReviewFrameExtractor turns into a second amendment pass.","marker":"[8,9]"},{"why":"Provide the sentence-level and self-improving entity extraction strategies that the SentenceFrameExtractor builds on.","marker":"[10,11,17]"},{"why":"Supply the pair-wise relation prompting and classification strategies behind the binary and multi-class relation extractors.","marker":"[12–14]"},{"why":"The 2012 i2b2 temporal relations challenge supplies the EVENT/TIMEX entity and attribute benchmark.","marker":"[18]"},{"why":"The 2014 i2b2 de-identification challenge supplies the strict and relaxed entity recognition benchmark.","marker":"[19]"},{"why":"The 2018 n2c2 adverse drug event challenge supplies the drug/ADE entity and relation benchmark.","marker":"[20]"},{"why":"The 70B open-weight LLM used for all benchmark and system evaluations.","marker":"[21]"}],"fun_headline_variants":["Sentence-wise prompting wins in new Python extraction toolkit","LLM-IE: One package for biomedical LLM extraction pipelines","Interactive agent in LLM-IE guides prompt and schema design","i2b2 tests: Sentence-based prompts boost LLM extraction F1","New LLM-IE package: Extract clinical entities sentence by sentence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that testing each extractor once on each challenge, with one model and one set of eight example prompts, and simply discarding outputs that do not follow the required format, is enough to rank the extractors fairly.","fun_headline_variants_meta":{"raw":{"variants":["Sentence-wise prompting wins in new Python extraction toolkit","LLM-IE: One package for biomedical LLM extraction pipelines","Interactive agent in LLM-IE guides prompt and schema design","i2b2 tests: Sentence-based prompts boost LLM extraction F1","New LLM-IE package: Extract clinical entities sentence by sentence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1559,"prompt_tokens":914,"completion_tokens":645,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":530,"tokens_out":645,"duration_ms":7035,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:08:52.539351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the three frame extractors on the same three challenges with several different sets of eight example prompts and at least one additional open-weight model, and count how often each extractor produces outputs that must be discarded. If sentence-by-sentence prompting no longer holds the top F1, or if the ranking changes across prompt sets and models, the reported order of extractors is not settled.","supporting_citations":[{"cited_title":"Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers","cited_arxiv_id":null,"evidence_quote":"Shows end-to-end structured information extraction from text, which motivates treating entity and attribute extraction as frame extraction."}],"review_version":1}