Dissecting Atomic Facts: Visual Analytics for Improving Fact Annotations in Language Model Evaluation

Daniel A. Keim; Frederik L. Dennig; Manuel Schmidt

REVIEW 3 major objections 5 minor 12 references

Dissecting Atomic Facts: Visual Analytics for Improving Fact Annotations in Language Model Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the definition of an atomic fact is underspecified, leading to high disagreement among both human and LLM annotators.

desk verdict A sound VA concept for a real problem, but the empirical motivation is unquantified and the prototype is untested; worth a serious referee, not a citation yet. read the letter →

arxiv 2509.01460 v1 pith:AKNA7ALW submitted 2025-09-01 cs.HC

Manuel Schmidt , Daniel A. Keim , Frederik L. Dennig This is my paper · ORCID

classification cs.HC

keywords visualanalyticsatomicfactstextannotationinter-annotatoragreementLLMevaluationfactextractionguidelinesknowledgegraphs

topics The Measurement Problem

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Factuality evaluation of large language models relies on decomposing model outputs into 'atomic facts' — minimal, self-contained units of meaning — and checking each against a source. The paper argues that no working definition of an atomic fact actually pins down where one fact ends and another begins. Its own annotation experiments, across 13 German administrative documents with two human annotators and several fine-tuned LLMs, show large variation in how many facts are extracted, how finely conditions and conjunctions are split, and whether context like ownership or conditions is included. The paper treats this disagreement not as noise but as evidence that annotation guidelines are underspecified, and proposes a visual analytics workflow to expose the ambiguities and iteratively refine the guidelines until humans and models converge on consistent extractions.

What carries the argument

The machinery is a four-stage visual analytics loop: aggregate text into a knowledge graph, highlight entities in the extracted facts, aggregate facts into small-multiple knowledge graphs, and project the fact semantics back onto the source text. Companion views — a semantic similarity heatmap computed from SBERT embeddings and Hungarian-assignment fact matching, a fact count histogram for granularity, and a branching-logic tree for conditionals — expose where annotators differ. The concept works by making disagreement inspectable so annotation guidelines can be revised and re-tested until humans and LLMs reach stable inter-annotator agreement.

What would settle it

Run a controlled experiment with two groups annotating the same set of documents: one group uses the proposed visual analytics tool with revision loops, the other uses static guidelines. If the tool group's inter-annotator agreement does not rise significantly above the control group after several revision cycles — or if agreement plateaus far below the threshold needed for benchmark reliability — the paper's central proposal fails, even though the observed disagreement itself would remain a valid empirical finding.

Watch

Extended reading notes

Core claim

The central claim is that the instability of atomic fact extraction is a measurement problem, not a model problem: annotators disagree because the rule for what counts as one fact is underconstrained. Empirical results reported for 13 German administrative documents show that human-human, human-LLM, and LLM-LLM inter-annotator agreement remained highly variable after multiple guideline iterations, with the largest divergence in granularity (how finely to split conjunctive and conditional structures) and referential dependency (whether contextual elements should be repeated for completeness). The paper proposes that a visual analytics system can surface these divergences through text-anchored

Load-bearing premise

The untested premise is that exposing disagreements through visualizations and guiding revision loops will actually make humans and LLMs converge on consistent atomic facts; the paper presents this as a goal, not a demonstrated result.

Editorial extensions

If this is right

Factuality benchmarks built on atomic facts inherit annotation instability; evaluation scores mean less when the ground truth disagrees with itself.
The proposed visual views (heatmap, histogram, knowledge-graph small multiples, branching trees) turn vague guideline disagreements into inspectable patterns, enabling systematic guideline revision.
Once agreement converges, the tool can switch to human-in-the-loop LLM-guided annotation, with majority-vote or interactive resolution pathways, increasing annotation efficiency.
Converged atomic facts provide a more stable input to factuality methods like FActScore, making LLM evaluation more interpretable and rigorous.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A measurable testable extension: use the same embedding-matching pipeline to compute inter-annotator agreement before and after each revision loop; if agreement does not increase across rounds, the visual analytics intervention is not the cause of convergence.
The disagreement signatures found here — granularity and referential dependency — likely generalize beyond German administrative text to any domain with long contexts and conditionals, so the convergence workflow is portable to English benchmarks with only guideline-adaptation cost.
The paper's own finding that inter-annotator agreement remains highly variable even after detailed guidelines suggests that some residual ambiguity may be irreducible; a realistic target may be bounded disagreement (e.g., explicit uncertainty flags) rather than full consensus.
The visual analytics approach could be repurposed as a calibration tool for LLM-as-judge annotation tasks beyond fact extraction, wherever inter-annotator agreement defines label quality.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Hi X,

Quick read on 2509.01460. It's a short concept paper from the Konstanz VA group proposing a visual analytics workflow to reduce disagreement in atomic-fact annotation for LLM factuality evaluation. The idea is reasonable and the problem is real: FActScore and claim-decomposition work have already shown that atomicity is underspecified. The paper's contribution is not the observation but the specific design: text-anchored fact highlighting, semantic similarity heatmaps, small-multiple knowledge graphs, and branching logic trees to expose guideline ambiguity and drive iterative convergence. That's a sensible translation of known issues into a VA research program. Credit where due: the authors are honest that the prototype is under development, they cite the relevant decomposition literature, and they don't oversell the system as validated.

The soft spots are where you'd expect. The motivating empirical claim—high IAA among humans and LLMs—is never quantified. No IAA values, thresholds, confidence intervals, or tests. The 13 documents are not described beyond domain; human-human agreement rests on 2 annotators and 3 documents. And the fact-matching threshold was empirically optimized on the same data used to report agreement, which makes any unstated number suspect. The stress-test note about circularity lands: the fitted threshold may inflate agreement, and since the abstract leans on "empirical results," that's a load-bearing weakness. However, the VA proposal itself doesn't depend on the precise IAA number, only on the plausible claim of meaningful variation, which the prior literature supports.

Given that this is explicitly a concept paper, I wouldn't demand a full evaluation before sending it out. But the empirical section, as written, is too thin for the abstract's claim. If this goes to review, the authors need to add real IAA numbers, a held-out threshold estimate, descriptive statistics, and ideally a small qualitative analysis of disagreement categories. The proposed convergence loop remains untested; that's acceptable for a position paper, but the reader should know that's what they're getting.

Bottom line: this deserves a serious referee—it raises an important problem and offers a plausible approach—but it needs heavy revision to make the empirical claim credible. I'd send it to a workshop or a VA venue with a proper revision request. Not something I'd cite in the next year until the data is concrete.

Best.

Referee Report

3 major / 5 minor

Summary. The paper argues that current definitions of 'atomic facts' used in LLM factuality evaluation are underspecified, and reports that annotators—both human and LLM-based—disagree substantially in fact extraction. To address this, the authors propose a visual analytics workflow that exposes annotation inconsistencies through views such as semantic similarity heatmaps, fact-count histograms, entity highlighting, knowledge-graph small multiples, and branching logic trees, with the goal of supporting iterative guideline refinement and convergence to higher inter-annotator agreement. The empirical basis is a study of 13 German administrative documents, three of which were independently annotated by two humans, and 800 documents annotated by several instruction-tuned LLMs. Fact-to-fact matching is performed with SBERT embeddings plus the Hungarian algorithm and a similarity threshold. The prototype of the VA system is stated to be under development, and no evaluation of the proposed workflow is reported.

Significance. The problem is timely and important: if atomic-fact annotations are as unstable as the authors claim, then factuality metrics such as FActScore rest on shaky ground. The paper identifies concrete ambiguity dimensions (granularity, referential dependency, conditionals/conjunctions) and offers a plausible visual design for exposing them. However, as submitted the manuscript does not establish its central empirical claim (no IAA values, confidence intervals, or statistical tests; a small and single-domain sample; a circularly tuned matching threshold), and the core proposal—that the VA workflow facilitates convergence—is not evaluated at all. The contribution is therefore a design concept with an unverified motivating observation, not a validated result. Credit is due for clearly framing disagreement as a signal of underspecified guidelines and for making the proposed system components concrete, but these strengths do not compensate for the missing evidence.

major comments (3)

[Section 2, Fig. 2] The paper's foundational claim that 'IAA for all human-human, human-LLM and LLM-LLM pairs remained highly variable' is not supported quantitatively. No IAA values, threshold values, confidence intervals, or statistical tests are reported; Fig. 2 is illustrative. Moreover, the fact-matching threshold was 'empirically optimized to minimize deviation from manual human-based matching' on what appears to be the same data later used to compute IAA. Without a held-out split or a pre-specified threshold, the reported IAA may be optimistically biased or even an artifact of the fitting procedure. This is load-bearing because the abstract's 'empirical results showing high disagreement' motivates the entire VA proposal.
[Section 2, document sample] The empirical design is too limited to support generalizations about human-model disagreement. Only 13 documents in a single domain (German administrative texts) are used, only three of these are independently annotated by two humans, and the 800 LLM-annotated documents are not paired with human references in the reported IAA. Consequently, the claim that human and LLM annotators 'highly disagree' is based on an extremely small human-human comparison and an unquantified human-LLM comparison. The authors should report per-pair agreement, document-level statistics, and a breakdown by disagreement type (granularity vs. referential dependency) to make the claim credible.
[Section 3, Fig. 1] The central proposal—that exposing disagreement through the four-stage loop and guided revision leads to convergence and higher IAA—is untested. The text states that the prototype is 'under development' and no user study, simulation, or before/after guideline-revision measurement is provided. There is also no comparison of the proposed VA tool against a text-only guideline condition. Since the abstract explicitly claims the approach 'facilitate[s] convergence through guided revision loops,' this is a load-bearing claim that needs empirical support or, at minimum, a pilot evaluation. As written, the paper is a concept note rather than a validated visual analytics contribution.

minor comments (5)

[Abstract] The phrase 'empirical results showing high disagreement' overstates what is demonstrated; the manuscript contains no reported IAA numbers. Consider softening to 'our preliminary annotations suggest substantial disagreement' or provide the missing quantitative evidence.
[Section 1] Typo: 'F ActScore' should be 'FActScore'. Also, 'Russel's logical atomism' should be 'Russell's' (cf. reference [6]).
[Section 2] The sentence 'The instruction fine-tuned LLMs...' is grammatically unclear; presumably 'instruction-tuned LLMs' is intended. Also, the annotation guidelines are not included or summarized in an appendix, which limits reproducibility of the human annotation process.
[Figure 2] The caption says the figure shows 'semantics (left) and granularity (right),' but the axes, color scale, and data sources are not described in the text. Clarify what exactly is plotted and what 'semantics' means in this context.
[After references] The manuscript includes an unlabeled slide deck (pages/sections such as 'Motivation', 'Fact Matching', 'Outlook') that duplicates and extends the main text. If this material is part of the submission, it should be integrated into the paper or explicitly identified as supplementary; otherwise it is confusing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claim of high annotator disagreement is an empirical observation, and the threshold calibration is a measurement choice, not a fitted prediction.

full rationale

The paper does not present a formal derivation or prediction chain that reduces to its inputs. Its central assertion—that atomicity definitions are underspecified and that human and LLM annotators disagree—is presented as an empirical finding from annotation experiments, not as the output of a fitted model. The only potentially circular element is the fact-matching pipeline: the similarity threshold is 'empirically optimized to minimize deviation from manual human-based matching' and then used to compute Jaccard-based IAA. However, this is a calibration step for a measurement instrument, not a parameter that directly determines the reported agreement level. The IAA is not a prediction from the threshold; it is an observed property of the annotations after applying a matching procedure. While the lack of a held-out split and the absence of reported IAA values are methodological weaknesses that affect the credibility of the empirical claim, they do not constitute definitional circularity. Moreover, the paper contains no load-bearing self-citations: all cited works are external benchmarks or foundational methods, and the proposed visual analytics system is explicitly described as a concept under development rather than a validated result. Thus, no circular step meets the standard of exhibiting a specific reduction by construction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new scientific entities. It depends on two untested premises: the diagnostic interpretation of disagreement and the efficacy of visual analytics for convergence. One free parameter, the similarity threshold, is fitted to human matching judgments and enters the reported IAA computation.

free parameters (1)

Similarity threshold for fact matching = Not reported
The threshold is 'empirically optimized to minimize deviation from manual human-based matching' (Section 2) and is used to discard low-quality matches before computing Jaccard IAA.

assumptions (3)

domain assumption Low inter-annotator agreement is primarily a signal of underspecified annotation guidelines.
The introduction and conclusion argue that disagreement is 'not noise, but a signal of underspecified annotation practices', which justifies the guideline-refinement approach.
ad hoc to paper Visualizing disagreements and guided revision loops will make annotation guidelines converge to high IAA.
This is the central untested premise of the proposed system, stated in the abstract and Section 3, and embodied in the Figure 1 revision loop.
domain assumption Atomic facts are a stable and meaningful unit for LLM factuality evaluation.
The paper motivates its approach by the reliance of FActScore, TruthfulQA, and FEVER on atomic facts, citing this prior work as the foundation without re-deriving it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dissecting Atomic Facts: Visual Analytics for Improving Fact Annotations in Language Model Evaluation." pith.science (2026). https://pith.science/paper/AKNA7ALW

@misc{pith2026250901460,
  author       = {Pith},
  title        = {Pith review of: Dissecting Atomic Facts: Visual Analytics for Improving Fact Annotations in Language Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKNA7ALW}},
  note         = {Machine review of arXiv:2509.01460}
}

read the original abstract

Factuality evaluation of large language model (LLM) outputs requires decomposing text into discrete "atomic" facts. However, existing definitions of atomicity are underspecified, with empirical results showing high disagreement among annotators, both human and model-based, due to unresolved ambiguity in fact decomposition. We present a visual analytics concept to expose and analyze annotation inconsistencies in fact extraction. By visualizing semantic alignment, granularity and referential dependencies, our approach aims to enable systematic inspection of extracted facts and facilitate convergence through guided revision loops, establishing a more stable foundation for factuality evaluation benchmarks and improving LLM evaluation.

Figures

Figures reproduced from arXiv: 2509.01460 by the authors.

**Figure 1.** Whose car is it? Our conceptual design illustrates a visual interface revealing inconsistencies in factual annotations across [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Initial analysis shows high disagreement between human [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

work page
[2]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

work page
[3]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

work page
[4]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

work page
[5]

Gunjal and G

A. Gunjal and G. Durrett. Molecular facts: Desiderata for decontextualization in LLM fact verification. In Find. Assoc. Comput. Linguist.: EMNLP 2024 , pp. 3751--3768. Association for Computational Linguistics, 2024. doi: 10 .1pt . . .4pt 18653 / / V1 / / 2024 .1pt . . .4pt FINDINGS - - EMNLP .1pt . . .4pt 215

work page 2024
[6]

S. Lin, J. Hilton, and O. Evans. Truthful QA : M easuring how models mimic human falsehoods. In Proc. 60th Annu. Meet. Assoc. Comput. Linguist. , pp. 3214--3252, 2022. doi: 10 .1pt . . .4pt 18653 / / V1 / / 2022 .1pt . . .4pt ACL - - LONG .1pt . . .4pt 229

work page 2022
[7]

S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generation. In Proc. 2023 Conf. Empir. Methods Nat. Lang. Process. , pp. 12076--12100, 2023. doi: 10 .1pt . . .4pt 18653 / / V1 / / 2023 .1pt . . .4pt EMNLP - - MAIN .1pt ...

work page 2023
[8]

Nenkova and R

A. Nenkova and R. J. Passonneau. Evaluating content selection in summarization: The pyramid method. In Hum. Lang. Technol. Conf. North Am. Chapter Assoc. Comput. Linguist. , pp. 145--152, 2004

work page 2004

Show all 12 references

[9]

Reimers and I

N. Reimers and I. Gurevych. Sentence- BERT : Sentence embeddings using S iamese BERT -networks. In Proc. 2019 Conf. Empir. Methods Nat. Lang. Process. 9th Int. Jt. Conf. Nat. Lang. Process. , pp. 3982--3992, 2019. doi: 10 .1pt . . .4pt 18653 / / v1 / / D19 - - 1410

2019
[10]

B. Russell. The Philosophy of Logical Atomism . Routledge, 2010

2010
[11]

Thorne, A

J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal. FEVER: a large-scale dataset for fact extraction and VER ification. In Proc. 2018 Conf. North Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol. , pp. 809--819, 2018. doi: 10 .1pt . . .4pt 18653 / / V1 / / N18...

2018
[12]

Wanner, S

M. Wanner, S. Ebner, Z. Jiang, M. Dredze, and B. V. Durme. A closer look at claim decomposition. In Proc. 13th Jt. Conf. Lexical Comput. Semant. , pp. 153--175, 2024. doi: 10 .1pt . . .4pt 18653 / / V1 / / 2024 .1pt . . .4pt STARSEM - - 1 .1pt . . .4pt 13

2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

work page

[2] [2]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

work page

[3] [3]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

work page

[4] [4]

write newline

" write newline "" before.all 'output.state := STRINGS d dc pc.str doi.urlstr INTEGERS nth.dash doi.string.pos doilen FUNCTION add.doi doi empty "\ doi 'd := "" 'doi.urlstr := #0 'nth.dash := #1 'doi.string.pos := d text.length 'doilen := nth.dash #3 < d doi.string.pos #1 substring 'dc := #1 doi.string.pos + 'doi.string.pos := dc "/" = #1 nth.dash + 'nth....

work page

[5] [5]

Gunjal and G

A. Gunjal and G. Durrett. Molecular facts: Desiderata for decontextualization in LLM fact verification. In Find. Assoc. Comput. Linguist.: EMNLP 2024 , pp. 3751--3768. Association for Computational Linguistics, 2024. doi: 10 .1pt . . .4pt 18653 / / V1 / / 2024 .1pt . . .4pt FINDINGS - - EMNLP .1pt . . .4pt 215

work page 2024

[6] [6]

S. Lin, J. Hilton, and O. Evans. Truthful QA : M easuring how models mimic human falsehoods. In Proc. 60th Annu. Meet. Assoc. Comput. Linguist. , pp. 3214--3252, 2022. doi: 10 .1pt . . .4pt 18653 / / V1 / / 2022 .1pt . . .4pt ACL - - LONG .1pt . . .4pt 229

work page 2022

[7] [7]

S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generation. In Proc. 2023 Conf. Empir. Methods Nat. Lang. Process. , pp. 12076--12100, 2023. doi: 10 .1pt . . .4pt 18653 / / V1 / / 2023 .1pt . . .4pt EMNLP - - MAIN .1pt ...

work page 2023

[8] [8]

Nenkova and R

A. Nenkova and R. J. Passonneau. Evaluating content selection in summarization: The pyramid method. In Hum. Lang. Technol. Conf. North Am. Chapter Assoc. Comput. Linguist. , pp. 145--152, 2004

work page 2004