REVIEW 6 cited by
Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The capability of Large Language Models (LLMs) like ChatGPT to comprehend user intent and provide reasonable responses has made them extremely popular lately. In this paper, we focus on assessing the overall ability of ChatGPT using 7 fine-grained information extraction (IE) tasks. Specially, we present the systematically analysis by measuring ChatGPT's performance, explainability, calibration, and faithfulness, and resulting in 15 keys from either the ChatGPT or domain experts. Our findings reveal that ChatGPT's performance in Standard-IE setting is poor, but it surprisingly exhibits excellent performance in the OpenIE setting, as evidenced by human evaluation. In addition, our research indicates that ChatGPT provides high-quality and trustworthy explanations for its decisions. However, there is an issue of ChatGPT being overconfident in its predictions, which resulting in low calibration. Furthermore, ChatGPT demonstrates a high level of faithfulness to the original text in the majority of cases. We manually annotate and release the test sets of 7 fine-grained IE tasks contains 14 datasets to further promote the research. The datasets and code are available at https://github.com/pkuserc/ChatGPT_for_IE.
Forward citations
Cited by 6 Pith papers
-
ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts
A large-scale benchmark shows that leading multimodal language models still underperform expert humans at verifying climate claims from scientific charts.
-
LA-RL: Label-Aware Self-Reflection for Reinforcement Learning in Information Extraction
Label-aware diagnostic reflection plus two-stage outcome GRPO improves same-backbone IE F1 over SFT, with larger gains under relation-extraction domain shift.
-
MPL: Multiple Programming Languages with Large Language Models for Information Extraction
Using multiple programming languages as code-style prompts during fine-tuning improves LLM information extraction accuracy over single-language prompting.
-
Investigating Student Interaction Patterns with Large Language Model-Powered Course Assistants in Computer Science Courses
A deployed LLM course assistant served 589 students across three CS courses; logs show heavy evening use and homework questions, while only about 11% of responses included AI follow-ups that students mostly ignored.
-
Schema as Parameterized Tools for Universal Information Extraction
SPT turns predefined IE schemas into trainable token embeddings, enabling one small model to handle closed, open, and on-demand extraction through schema retrieval, generation, and infilling.
-
Improving TCM Question Answering through Tree-Organized Self-Reflective Retrieval with LLMs
A tree-organized, self-reflective retrieval framework over a TCM knowledge base lifts GPT-4 accuracy on a 600-question licensing-exam sample by 19.85 absolute percentage points.
Discussion (0). Sign in to comment.