Osiris: A Lightweight Open-Source Hallucination Detection System

Alex Shan; Christopher D. Manning; John Bauer

REVIEW 4 major objections 4 minor 13 references

Osiris: A Lightweight Open-Source Hallucination Detection System

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a 7B model fine-tuned on a perturbed multi-hop QA dataset detects RAG hallucinations with higher recall than GPT-4o on the RAGTruth benchmark.

desk verdict A reasonable extension of Lynx with a large recall claim over GPT-4o that is currently undermined by an asymmetric evaluation pipeline and inconsistent reporting. read the letter →

arxiv 2505.04844 v1 pith:IKILRD5D submitted 2025-05-07 cs.CL

Alex Shan , John Bauer , Christopher D. Manning This is my paper · ORCID

classification cs.CL

keywords hallucinationdetectionretrieval-augmentedgenerationmulti-hopquestionansweringdataperturbationsupervisedfine-tuningRAGTruthMuSiQueopen-sourceLLM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a lightweight, openly available 7B model can beat a much larger closed model at the practical task of telling whether a RAG-generated answer is supported by the retrieved context. The authors build a training set from the multi-hop MuSiQue question set: correct answers are verified with reasoning, and GPT-4o is prompted to replace them with plausible but unsupported answers that can even be drawn from the same context. After fine-tuning Qwen 2.5 7B on that data, they report 0.938 recall on the RAGTruth benchmark, compared with GPT-4o's 0.710, with competitive precision and F1. If this holds, production RAG systems could screen for hallucinations locally, cheaply, and in real time instead of paying for or waiting on a closed reviewer model.

What carries the argument

The central object is the perturbed MuSiQue dataset: roughly 39,876 question-answer pairs, split about evenly between verified answers and induced hallucinations, each carrying a reasoning explanation that cites the context and says why the answer is or is not supported. The perturbations are deliberately subtle, replacing the gold answer with another entity mentioned in the same evidence, so surface overlap alone cannot decide correctness. Fine-tuning on these contrastive pairs is what transfers multi-hop verification skill into the detector; the evaluation then runs on RAGTruth, a word-level hallucination corpus built from real RAG outputs, which measures recall and precision of flagged hallucinated spans.

What would settle it

Re-run the same RAGTruth evaluation twice: once with the JSON-repair post-processing applied to both Osiris-7B and GPT-4o, and once with it applied to neither. If Osiris's recall advantage disappears or reverses under symmetric scoring, the claimed edge is an artifact of asymmetric post-processing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that hallucination detection can be learned from a perturbed multi-hop QA dataset rather than from distilled single-hop examples. Osiris-7B, a Qwen2.5-7B model fine-tuned on the 39,876-example perturbed MuSiQue set, reaches recall 0.938 on RAGTruth while GPT-4o reaches 0.710, a gap the authors quote as 22.8% in the abstract and 23.8% in the conclusion; precision is 0.366 versus 0.446 and F1 is 0.527 versus 0.548. The paper argues that multi-hop structure matters: the model is forced to verify each reasoning step against the evidence, and the subtle perturbations, wrong answers that appear in the context but do not support the question, teach it to reject plausible but unsupported claims. The result is positioned as a practical detector, not a general fact-checker, with high recall so that human reviewers can focus on flagged spans.

Load-bearing premise

The load-bearing assumption is that the evaluation treats both systems alike: Osiris outputs are repaired with a GPT-4o-mini JSON prompt before scoring, and the paper never says the GPT-4o baseline gets the same repair, so the recall gap may partly reflect asymmetric post-processing.

Editorial extensions

If this is right

A locally hosted 7B detector can screen RAG outputs at scale, removing the need to route every generation through a closed API for review.
The recall-oriented design means nearly all hallucinations are flagged, allowing human reviewers to concentrate on flagged spans; the lower precision means some false positives will also land in that queue.
Because the detector is small and fast, it can operate in real time, before an answer is shown to a user, rather than as a post hoc audit.
The data recipe, perturb multi-hop QA with in-context distractors and add reasoning labels, can be transferred to other base models and domains, provided a reliable generator for the perturbations exists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported recall advantage could partly come from the JSON-repair post-processing applied to Osiris outputs but not, as far as the paper states, to the GPT-4o baseline; a symmetric evaluation is the direct test.
Because the same closed model that verifies correct answers also writes the hallucinated ones, the fine-tuned model may learn a house style of hallucination rather than a general capacity; checking against hallucinations produced by other generators would show generalization.
The 'open-source' label is partial, since perturbation generation and inference-time JSON repair both depend on closed models; a fully open pipeline would need a local source of perturbations and a local repair step.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper is a legitimate extension of the Lynx line of work: take a perturbation pipeline for inducing hallucinations, apply it to MuSiQue multi-hop QA, fine-tune a small Qwen model, and test on RAGTruth. The dataset construction is described in enough detail to reproduce, the code is released, and the fine-tuning recipe is concrete. That is real value. The improvement over the base Qwen model is also credible and shows the method does something.

The problem is the headline comparison. The claim that Osiris-7B beats GPT-4o by 22.8% (or 23.8%? the paper says both) in recall rests on an evaluation harness that is not shown to be symmetric. Section 2.2 says the fine-tuned models are used in conjunction with GPT-4o mini, with a JSON repair prompt from Appendix B.3, but it never states whether the GPT-4o baseline receives the same post-processing. Given the Limitations section openly admits that small models often struggle with JSON formatting, the repair step is not a no-op; it can convert free text, bullet lists, and malformed output into a uniform hallucination_list, which directly inflates recall. If the baseline was parsed more strictly, the 22.8-point gap could be largely an artifact of the harness.

Other soft spots: the abstract claims competitive precision and accuracy, but accuracy is never reported and precision drops from 0.446 to 0.366. The only baseline is GPT-4o; no other hallucination detector, open-source or closed, is compared. The paper also conflates absolute percentage points with relative improvement, and the 22.8% vs 23.8% inconsistency does not inspire confidence in the numbers. None of this disproves the method, but it means the central empirical claim is unverified without a clean, symmetric evaluation.

On the positive side, the RAGTruth labels are human-annotated, so the recall claim is not circular by construction. The citation pattern is fine, leaning heavily on preprints but appropriately crediting Lynx, MuSiQue, and RAGTruth. The writing is a bit rough but understandable.

This paper deserves a serious referee because the question it asks is important — can a small open model match or beat a frontier model on hallucination detection recall — and because the artifact is concrete. A referee should require a symmetric evaluation, explicit precision/accuracy numbers, at least one additional baseline, and a consistent reporting of the gain. With those revisions it could be a useful data point for the field.

I would bring it to a reading group as an example of an evaluation pitfall, not as a strong result, and I would not cite it in its current form.

Referee Report

4 major / 4 minor

Summary. The paper presents Osiris-7B, a Qwen2.5-7B model fine-tuned on a newly constructed perturbed MuSiQue dataset in which GPT-4o generates plausible but unsupported answers with reasoning labels. The authors evaluate the model on the human-annotated RAGTruth benchmark and report that the fine-tuned 7B model achieves recall 0.938 versus GPT-4o's 0.710, while also claiming competitive precision and accuracy and faster inference. The paper's contributions are an open-source fine-tuned detector, a data perturbation pipeline, and an evaluation against a closed-source model.

Significance. If the reported result holds, the paper would make a practically useful contribution: a small open-source model with higher recall than GPT-4o on a human-annotated hallucination detection benchmark, at lower inference cost, is of clear value to RAG deployment workflows. A notable strength is that the evaluation target, RAGTruth, is human-annotated and independent of the training labels, so the main recall claim is not circular by construction; the released code and dataset construction also help reproducibility. However, the central comparison currently rests on an underspecified and potentially asymmetric evaluation pipeline, and several headline claims in the abstract are not backed by the reported numbers.

major comments (4)

[Section 2.2, Appendix B.3, Table 3] The RAGTruth evaluation is described asymmetrically: the paper states that the fine-tuned models are used 'in conjunction with GPT-4o mini, employing a JSON prompt detailed in Appendix B.3 only when the output answers required correction,' but it never states whether the GPT-4o baseline receives the same JSON repair or parsing step. The Appendix B.3 prompt is not a no-op: it explicitly converts free text, bullet lists, numbered lists, and line-separated items into a uniform hallucination_list, and the Limitations section concedes that smaller fine-tuned models often fail to produce correct JSON. If the GPT-4o baseline was parsed with a different routine, the reported 0.938 vs 0.710 recall advantage could be an artifact of post-processing rather than of the model. I ask the authors to specify exactly which post-processing each arm received and to rerun both arms under identical post-processing.
[Section 2.2, Table 3] The paper never defines the prediction format or the matching rule used to compute recall, precision, and F1 on RAGTruth, which is described as a word-level hallucination corpus. It is not stated whether predictions are span-level exact matches, token-level overlaps, or sentence-level binary decisions, nor how the hallucination_list output is aligned with the human-annotated spans. Without this definition, the headline numbers in Table 3 are not reproducible or interpretable. Please provide the evaluation script or a precise statement of the matching criterion and any thresholds.
[Abstract, Table 3] The abstract claims 'competitive performance on precision and accuracy,' but Table 3 reports precision 0.366 for Osiris-7B versus 0.446 for GPT-4o (i.e., lower), and accuracy is never reported anywhere in the paper. The claim should either be removed or supported by a corresponding table entry and a definition of how accuracy is computed.
[Abstract, Section 3, Conclusion] The headline recall improvement over GPT-4o is stated inconsistently: the abstract says 22.8%, while Section 3 and the Conclusion say 23.8%. Neither number is the relative improvement from 0.710 to 0.938 (about 32.1%), and the text does not distinguish absolute percentage points (22.8 points) from relative improvement. The headline metric should be stated consistently and unambiguously.

minor comments (4)

[Section 2.1, Table 1] The verified example text is garbled: 'confirming the the that the country Tepuka is located in (Tuvalu) was Álvaro de Mendaña' should be rewritten, and there is a duplicated 'the' earlier in the same paragraph.
[Section 3, Inference Speed] The inference speed comparison is not apples-to-apples: Osiris-7B is measured with 4-bit quantization on an A100 80GB via vLLM, while GPT-4o's number is cited from a third-party source without the same hardware and serving conditions. Please state the conditions for both or clearly label the comparison as indicative only.
[Abstract, Table 3] The contribution bullet 'average increases of 32.98% in recall, 4.55% in precision, and 17.98% in F1' appears to report average percentage-point changes across four model sizes, yet the 7B model's precision actually decreases (0.402 to 0.366). Please state the aggregation method explicitly and note the per-size direction of change.
[Section 2.1, Figure 1] Figure 1 has no axis labels or legend; adding them would make the token-length distribution comparison interpretable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RAGTruth is an external human-annotated benchmark, and the MuSiQue-derived perturbed training set is constructed independently of it.

full rationale

The paper's central claim is that fine-tuning Qwen2.5-7B on a perturbed MuSiQue QA dataset yields higher recall than GPT-4o on RAGTruth. The evaluation target is an externally created, human-annotated hallucination corpus, not an output of the paper's pipeline. Training samples are generated from MuSiQue using GPT-4o verification and perturbation prompts (Appendices B.1-B.2), and none of the training labels are taken from RAGTruth; hence RAGTruth recall is not an input fitted value or a quantity reconstructed from the training set by construction. The use of GPT-4o-mini to repair JSON formatting in the fine-tuned model's outputs (Section 2.2, Appendix B.3) does create a possible evaluation-comparability concern: the paper does not state that the GPT-4o baseline was run through the same normalization, and the limitation that smaller models are 'often necessitating GPT-4o mini for post-processing corrections' makes the repair step potentially non-trivial. That concern is about an asymmetric or unclear evaluation harness, not about circularity: a format-repair step that extracts spans from the model's own free-text output is not equivalent to supplying the RAGTruth labels or to the training labels. No load-bearing self-citation, imported uniqueness theorem, or ansatz smuggled in by citation occurs; the only external frameworks cited (MuSiQue, Lynx, RAGTruth) are independent prior work. Under the rule that circularity must be shown by a quoted construction-level reduction, no circular step is identifiable, so the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its central claim rests on domain assumptions about LLM-as-judge labels and task transfer, none of which are independently verified within the paper.

assumptions (4)

domain assumption GPT-4o's verification and hallucination labels on MuSiQue are accurate enough to train a detector.
Dataset construction uses GPT-4o as the labelling oracle without human validation of the 39,876 samples; this enters in Sections 2.1 and 2.2.
domain assumption RAGTruth human annotations are the ground truth for hallucination detection.
Evaluation relies on RAGTruth as the sole benchmark; this is a standard assumption but is not validated beyond citing the benchmark in Section 2.2.
domain assumption Multi-hop QA training transfers to RAG hallucination detection in other domains.
The method assumes that MuSiQue-style multi-hop reasoning improves hallucination detection on RAGTruth, which contains different domains; this argument appears throughout Section 2.1.
domain assumption GPT-4o mini post-processing does not alter the relative comparison between models.
Evaluation uses GPT-4o mini to repair model outputs, but the paper does not state whether the GPT-4o baseline is repaired identically (Section 2.2, Appendix B.3).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Osiris: A Lightweight Open-Source Hallucination Detection System." pith.science (2026). https://pith.science/paper/IKILRD5D

@misc{pith2026250504844,
  author       = {Pith},
  title        = {Pith review of: Osiris: A Lightweight Open-Source Hallucination Detection System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IKILRD5D}},
  note         = {Machine review of arXiv:2505.04844}
}

read the original abstract

Retrieval-Augmented Generation (RAG) systems have gained widespread adoption by application builders because they leverage sources of truth to enable Large Language Models (LLMs) to generate more factually sound responses. However, hallucinations, instances of LLM responses that are unfaithful to the provided context, often prevent these systems from being deployed in production environments. Current hallucination detection methods typically involve human evaluation or the use of closed-source models to review RAG system outputs for hallucinations. Both human evaluators and closed-source models suffer from scaling issues due to their high costs and slow inference speeds. In this work, we introduce a perturbed multi-hop QA dataset with induced hallucinations. Via supervised fine-tuning on our dataset, we achieve better recall with a 7B model than GPT-4o on the RAGTruth hallucination detection benchmark and offer competitive performance on precision and accuracy, all while using a fraction of the parameters. Code is released at our repository.

Figures

Figures reproduced from arXiv: 2505.04844 by the authors.

**Figure 1.** Distribution of Token Context Lengths 2 Methodology To enhance hallucination detection performance, we built a data perturbation pipeline to construct fine-tuning data for developing Osiris-7B. For dataset construction, we began by selecting multihop QAs, as these require models to navigate through more diverse contexts. We then perturbed both positive and negative examples through our data pipeline to distill capa… view at source ↗

**Figure 2.** Q/A Dataset Perturbation Pipeline et al., 2024). By requiring stepwise integration of evidence, multi-hop frameworks enhance a model’s ability to verify factual claims, reducing instances where models confidently generate hallucinated responses that lack sufficient justification (Huang et al., 2025). Shortcut-Driven Reasoning A key advantage of using MuSiQue over other multi-hop QA datasets for training hallucinatio… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 8 canonical work pages

[1]

item1" "item2

Missing commas between array items: ["item1" "item2"] → ["item1", "item2"]

work page
[2]

Yang Deng, Yong Zhao, Moxin Li, See-Kiong Ng, and Tat-Seng Chua

Seemingly plausible distractors in multi-hop reasoning: Are large language models attentive read- ers? arXiv preprint arXiv:2409.05197. Yang Deng, Yong Zhao, Moxin Li, See-Kiong Ng, and Tat-Seng Chua. 2024. Gotcha! don’t trick me with unanswerable questions! self-aligning large language models for responding to unknown questions. arXiv preprint arXiv:2402...

arXiv 2024
[3]

list": [

Missing quotes: {list: [value]} → {"list": ["value"]}

work page
[4]

arXiv preprint arXiv:2410.11414

Redeep: Detecting hallucination in retrieval- augmented generation via mechanistic interpretabil- ity. arXiv preprint arXiv:2410.11414. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2021. Musique: Multi- hop questions via single-hop question composition. CoRR, abs/2108.00573. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten B...

arXiv 2021
[5]

QUESTION:

Hallucination is inevitable: An innate lim- itation of large language models. arXiv preprint arXiv:2401.11817. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 tech- nical report. arXiv preprint arXiv:2412.15115. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yan- h...

arXiv 2024
[6]

hallucination_list

Bullet points: • First item • Second item → {"hallucination_list": ["First item", "Second item"]}

work page
[7]

list": [{

Unclosed brackets: {"list": [{"item": "value"} → {"list": [{"item": "value"}]}

work page
[8]

First item \n Second item

Line-separated items: "First item \n Second item" → {"hallucination_list": ["First item", "Second item"]} If you see a plain text response with phrases like "I found these hallucinations:" or "Hallucinated content:", extract the listed items and format them as a proper JSON array in the hallucination_list. The JSON must follow this exact format: {"halluci...

work page

Show all 13 references

[9]

item1",

Trailing commas: ["item1", "item2",] → ["item1", "item2"]
[10]

Hallucinations: 1. First item 2. Second item

Unstructured lists: "Hallucinations: 1. First item 2. Second item" → {"hallucination_list": ["First item", "Second item"]}
[12]

1) First item 2) Second item

Numbered lists: "1) First item 2) Second item" → {"hallucination_list": ["First item", "Second item"]}
[2023]

arXiv preprint arXiv:2401.00396

Ragtruth: A hallucination corpus for develop- ing trustworthy retrieval-augmented language models. arXiv preprint arXiv:2401.00396. Anupam Purwar and Rahul Sundar. 2023. Keyword augmented retrieval: Novel framework for informa- tion retrieval integrated with speech interface. ...

2023 arXiv
[2024]

arXiv preprint arXiv:2409.05746

Llms will always hallucinate, and we need to live with this. arXiv preprint arXiv:2409.05746. Manish Bhattarai, Ryan Barron, Maksim Eren, Minh Vu, Vesselin Grantcharov, Ismael Boureima, Valentin Stanev, Cynthia Matuszek, Vladimir Valtchinov, Kim Rasmussen, et al. 2024. Heal: H...

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

item1" "item2

Missing commas between array items: ["item1" "item2"] → ["item1", "item2"]

work page

[2] [2]

Yang Deng, Yong Zhao, Moxin Li, See-Kiong Ng, and Tat-Seng Chua

Seemingly plausible distractors in multi-hop reasoning: Are large language models attentive read- ers? arXiv preprint arXiv:2409.05197. Yang Deng, Yong Zhao, Moxin Li, See-Kiong Ng, and Tat-Seng Chua. 2024. Gotcha! don’t trick me with unanswerable questions! self-aligning large language models for responding to unknown questions. arXiv preprint arXiv:2402...

arXiv 2024

[3] [3]

list": [

Missing quotes: {list: [value]} → {"list": ["value"]}

work page

[4] [4]

arXiv preprint arXiv:2410.11414

Redeep: Detecting hallucination in retrieval- augmented generation via mechanistic interpretabil- ity. arXiv preprint arXiv:2410.11414. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2021. Musique: Multi- hop questions via single-hop question composition. CoRR, abs/2108.00573. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten B...

arXiv 2021

[5] [5]

QUESTION:

Hallucination is inevitable: An innate lim- itation of large language models. arXiv preprint arXiv:2401.11817. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 tech- nical report. arXiv preprint arXiv:2412.15115. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yan- h...

arXiv 2024

[6] [6]

hallucination_list

Bullet points: • First item • Second item → {"hallucination_list": ["First item", "Second item"]}

work page

[7] [7]

list": [{

Unclosed brackets: {"list": [{"item": "value"} → {"list": [{"item": "value"}]}

work page

[8] [8]

First item \n Second item

Line-separated items: "First item \n Second item" → {"hallucination_list": ["First item", "Second item"]} If you see a plain text response with phrases like "I found these hallucinations:" or "Hallucinated content:", extract the listed items and format them as a proper JSON array in the hallucination_list. The JSON must follow this exact format: {"halluci...

work page