REVIEW 4 cited by
DyKnow: Dynamically Verifying Time-Sensitive Factual Knowledge in LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
LLMs acquire knowledge from massive data snapshots collected at different timestamps. Their knowledge is then commonly evaluated using static benchmarks. However, factual knowledge is generally subject to time-sensitive changes, and static benchmarks cannot address those cases. We present an approach to dynamically evaluate the knowledge in LLMs and their time-sensitiveness against Wikidata, a publicly available up-to-date knowledge graph. We evaluate the time-sensitive knowledge in twenty-four private and open-source LLMs, as well as the effectiveness of four editing methods in updating the outdated facts. Our results show that 1) outdatedness is a critical problem across state-of-the-art LLMs; 2) LLMs output inconsistent answers when prompted with slight variations of the question prompt; and 3) the performance of the state-of-the-art knowledge editing algorithms is very limited, as they can not reduce the cases of outdatedness and output inconsistency.
Forward citations
Cited by 4 Pith papers
-
Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation
A turn-level faithfulness reward improves a Search-R1-style agent's Information-Think and Think-Answer faithfulness as judged by the same reward model used for training, while task accuracy is roughly unchanged.
-
An Agile Method for Implementing Retrieval Augmented Generation Tools in Industrial SMEs
EASI-RAG is a structured agile method for deploying RAG tools in industrial SMEs, validated by one case study where a no-experience team built a working assistant in three weeks.
-
Benchmarking and Rethinking Knowledge Editing for Large Language Models
Under autoregressive and sequential editing, parameter-based knowledge editing methods perform poorly, while the retrieval-based SCR baseline consistently outperforms them across datasets and models.
-
LLM-Assisted Question-Answering on Technical Documents Using Structured Data-Aware Retrieval Augmented Generation
A RAG pipeline with OCR, table and image to text conversion, and a RAFT-tuned reranker reports high QA scores, but its 50-question evaluation overlaps with its training manuals and its baseline comparison uses only 5 ...
Discussion (0). Continue with ORCID to comment.