Pith. sign in

REVIEW 4 major objections 5 minor 62 references

Spatially organizing source documents around a summary improves verification accuracy by 36% and cuts completion time by a third.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Placing source documents by how they align with summary sentences improves people's ability to verify multi-document AI summaries, outperforming a linear list.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A well-built VA system with a genuinely new layout, but the study's accuracy numbers may only show how well users read the system's own rankings, not how well they verify summaries against source truth. the 4 major comments →

arxiv 2607.28853 v1 pith:RDZPEFHT submitted 2026-07-30 cs.HC

Spatial Visual Analytics for Multi-Document Summary Verification

Jiahao Xu , Wei Liu , Yang Liu , Eric Krokos , Kirsten Whitley , Xuan Wang , Rebecca Faust , Chris North This is my paper
classification cs.HC
keywords summary verificationmulti-document summarizationvisual analyticsspatial layoutsprovenance visualizationLLM-as-a-judgeuser studydocument organization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that verifying an AI-generated summary against a collection of source documents becomes faster, more accurate, and less mentally demanding when the documents are arranged spatially on a canvas rather than presented as a linear list. The authors built SVS, a visual analytics system that offers two spatial layouts: one that organizes documents by their alignment with the summary's sentences, and one that arranges them by semantic similarity. In a comparative user study with 18 participants and the same provenance overlays in every condition, both spatial layouts raised overall accuracy by 36.0% over a linear baseline, and the summary-guided layout also cut completion time by 32.7%, lifted confidence by 11.9%, and lowered mental demand by 47.2%. The work matters because LLM-generated summaries are increasingly used for decision support, and verifying their grounding is currently a manual, slow process.

Core claim

SVS combines two coordinated views: a summary panel that decomposes each summary sentence into atomic claims and labels each claim as SUPPORTS, CONTRADICTS, or INSUFFICIENT EVIDENCE, and a document canvas that arranges source documents spatially. The SUMMARY-GUIDED layout places documents horizontally by their strongest similarity to any summary sentence (Rd = max|Md[i,j]|) and vertically by the weighted mean summary-sentence index of their evidence (µd). The SOURCE-GUIDED layout projects documents with UMAP and anchors summary sentences at weighted centroids of contributing documents. Holding provenance visualization constant, the user study found both spatial layouts outperformed a linear

What carries the argument

The load-bearing object is the document-card canvas with two alternative spatial mappings. SUMMARY-GUIDED encodes two signals per document: horizontal position from Rd (the maximum absolute sentence-level similarity between the document and any summary sentence, Eq. 2) and vertical position from µd (the weighted mean summary-sentence index of the document's strongest positive evidence, Eq. 4). SOURCE-GUIDED uses UMAP on whole-document embeddings and places each summary sentence at the normalized weighted centroid of contributing documents (Eq. 6). Both layouts are overlaid with identical provenance encodings—claim-level inference labels, evidence-highlighted sentences, and connector lines—so

Load-bearing premise

The study never describes an independent human annotation for the relevance labels used to score tasks T4–T7, so if those labels come from the same embedding and LLM pipeline that produces the layouts, the measured accuracy gains may partly be users reading the system's own ordering rather than verifying against external truth.

What would settle it

Rerun the study with ground-truth relevance labels provided by independent human raters (majority agreement across multiple annotators) for tasks T4–T7; if SUMMARY-GUIDED no longer beats LINEAR on accuracy, the reported gains are an artifact of layout–label alignment.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The 36.0% accuracy gain on both spatial layouts, driven mainly by topic-identification and coverage-estimation tasks, suggests spatial overview alone lets users judge relevance more accurately than linear lists.
  • SUMMARY-GUIDED's 32.7% time reduction and 47.2% mental-demand reduction make it a strong candidate for deployment in document-auditing workflows.
  • Consistency tasks (support/contradict/insufficient evidence) showed few significant layout effects, implying that claim-level provenance visualization, not spatial layout, is what supports those judgments.
  • Users preferred the layout that made document-to-summary relevance visible at a glance, indicating that spatial position can be a trust-building cue for automated verification outputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the study's relevance labels are derived from the same embeddings and LLM judge that create the layouts, part of the measured accuracy gain may be users aligning with the system's own ordering rather than verifying against independent truth; an independent human-annotation ground truth would separate these.
  • The SUMMARY-GUIDED idea — mapping evidence to the structure of the artifact being verified — could generalize beyond summaries to other structured outputs such as timelines, legal findings, or code review checklists.
  • The interaction between layout and ground-truth source suggests a testable design principle: for auditing tasks, spatial organization should encode the item under audit's structure, not merely the corpus's intrinsic semantic geometry.
  • The low sensitivity of consistency tasks to layout hints that future systems could decouple layout choice from provenance depth, letting users pick a layout per task type.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Summary Verification Space (SVS), a visual analytics system for verifying multi-document summaries. It contributes two spatial document layouts—SUMMARY-GUIDED, which positions documents by their relevance and alignment to summary sentences (Eqs. 2–4), and SOURCE-GUIDED, which projects documents by semantic similarity with summary-sentence anchors—plus a coordinated provenance visualization linking summary claims to evidence spans, inference labels, and document cards. The paper reports a usage scenario and a within-subject user study (N=18) comparing the two layouts against a LINEAR baseline, with provenance visualization held constant. The central claim is that both guided layouts improve verification accuracy by 36.0% over LINEAR, and that SUMMARY-GUIDED additionally reduces completion time by 32.7%, increases confidence by 11.9%, and lowers mental and physical demand by 47.2% and 42.3%. The quantitative analysis uses linear mixed-effects models with accuracy, time, confidence, and workload measures. The main weakness is that the study does not document an independent ground-truth protocol for scoring accuracy; the tasks and answer keys appear to coincide with SVS's own internally computed rankings and LLM inference labels, which threatens the validity of the headline accuracy claim.

Significance. If the central claims are valid, the paper makes a useful contribution to multi-document summarization verification: it shows that spatial organization plus provenance can improve relevance assessment, and it provides a carefully controlled comparison of layout strategies. The system design is thoughtful, and the authors provide an interactive demo, source code, and supplemental material, which supports reproducibility. The use of linear mixed-effects models with participant random intercepts is appropriate for the repeated-measures design, and holding provenance visualization constant across conditions is a sound way to isolate the effect of spatial layout. However, the significance of the empirical contribution is substantially reduced by the lack of any reported independent annotation or ground-truth procedure for the accuracy measure. Because several task answers are defined by the same quantities that generate the layouts and the LLM-judge labels, the reported accuracy gains may measure how easily users can read the system's own ordering rather than how well they verify summaries against source truth. The efficiency, confidence, and workload findings are less affected by this

major comments (4)
  1. [§7.1.2–7.1.3, §5.1, Eqs. (2)–(4)] The accuracy measure is not grounded in any independent answer key. Tasks T4–T7 ask users to identify the most/least relevant documents and sentences; the natural correct answers are the document with maximum Rd (Eq. 2) or minimum Rd, the top documents by wd(j), and the sentence index µd (Eq. 4). These are exactly the quantities used to build the SUMMARY-GUIDED layout. Tasks T8–T10 ask users to identify documents that 'contribute to its inference judgment label' or claims with SUPPORTS/CONTRADICTS/INSUFFICIENT_EVIDENCE labels; the only source of such labels is the LLM judge described in §5.1. No independent human annotation or external benchmark is reported for any of the ten tasks. If participant accuracy was scored against these system-derived labels, the 36.0% accuracy gain is at least partly circular: it measures whether participants can recover SVS's own ranking and labels, not whet
  2. [§7.1.5 and §9] The multiple-comparison burden is not addressed. The study reports ten tasks × three metrics, plus omnibus and pairwise tests, with no correction for multiplicity. Several isolated pairwise effects (e.g., T3 accuracy +27.78%, T5 accuracy +27.78%, T9 accuracy +16.67%) have raw p-values in the 0.02–0.05 range and could easily be false positives under 30+ comparisons. The omnibus effects are strong for the aggregate metrics, so the overall direction is likely robust, but the precision of the claimed 36.0% / 32.7% numbers and the task-level takeaways should be supported by corrected p-values or clearly labeled as exploratory.
  3. [§7.1.6 and §7.2] The accuracy metric is not defined. The report states mean accuracy scores of 6.33, 8.61, and 8.61 out of what scale, and the '36.0% increase' is computed as a relative improvement. It is unclear whether accuracy is the number of fully correct tasks out of ten, whether tasks with multiple correct answers (e.g., T6 'top two documents', T8 'select all documents') are scored as all-or-nothing or with partial credit, and whether ties or near-misses are treated as incorrect. This matters for interpreting the effect sizes. The authors should define the scoring rule for each task and describe how partial correctness, if any, was handled.
  4. [§7.1.5 and §7.2] The fixed dataset order is a secondary but real threat. The condition-order permutations balance each condition across datasets, which is good, but all participants saw Dataset 1, then Dataset 2, then Dataset 3. Any practice, fatigue, or dataset-difficulty trend is thus partially confounded with round, even if condition is balanced. The authors state that dataset progression did not require counterbalancing, but this assumption is not tested. At minimum, the analysis should include a round or order term, or justify its omission, and the paper should acknowledge this limitation.
minor comments (5)
  1. [Figure 3] Typography errors in the figure: 'Retreival' should be 'Retrieval', and 'INSUFFICENT EVIDENCE' should be 'INSUFFICIENT EVIDENCE'.
  2. [§5.1 and §4.2.1] The semantic and lexical thresholds for evidence retrieval and the UMAP parameters are never reported. Since these are free parameters that can affect both the layouts and the provenance labels, the authors should state the values used in the study, or at least refer to the supplemental code with specific configuration.
  3. [§7.1.2, Task T8] The wording 'select all documents that contribute to its inference judgment label' is ambiguous: does 'contribute' mean any retrieved evidence document, only documents with supporting/contradicting spans, or documents that the LLM judge explicitly cites? The scoring rule depends on this definition.
  4. [Figures 4 and 5] The error bars are described as 95% confidence intervals, but it is not stated whether they come from the fitted mixed-effects models or from raw scores. The caption should clarify the estimation and indicate whether the intervals are adjusted for participant random effects.
  5. [§7.1.4] The participant recruitment is described only as 'from a university mailing list.' Please report demographics, prior experience with summarization or visual analytics, and any screening criteria, as is standard for CHI/TVCG user studies.

Circularity Check

3 steps flagged

T4–T10 accuracy is keyed to SVS's own Eqs. (2)–(4) similarity rankings and §5.1 LLM-judge labels, with no independent annotation protocol reported; part of the reported 36% gain measures self-consistency rather than external summary verification.

specific steps
  1. self definitional [§7.1.2 Task T4/T5 vs §4.1.1 Eq. (2)]
    "Most relevant document to the summary: Identify the source document that is most relevant to the summary as a whole. ... Rd =max i,j |Md[i,j]|.(2) Documents are sorted in descending order of Rd and mapped to evenly spaced horizontal positions on the canvas."

    The study reports no independent relevance annotation for T4/T5. The natural answer key is argmax_d Rd (T4) and argmin_d Rd (T5), which are the same scores that place documents left-to-right in SUMMARY-GUIDED. Thus a user reading the leftmost card is, by construction, giving the 'correct' answer. As reported, the accuracy gain on these tasks reflects agreement with SVS's own ranking, not verification against external source truth.

  2. self definitional [§7.1.2 Task T6/T7 vs §4.1.2 Eqs. (3)–(4) and §4.2.2 Eq. (6)]
    "Most relevant documents to a summary sentence: For a given summary sentence, identify the top two documents that are most relevant to that sentence. ... wd(j) =max i max(Md[i,j],0).(3) ... p j = ∑ d αd(j)z d,(6)"

    Sentence-level relevance is defined by the same wd(j) used by both layouts: SUMMARY-GUIDED computes vertical positions from µd (Eq. 4) based on wd(j), and SOURCE-GUIDED places summary anchors at weighted averages of these same weights (Eq. 6). The correct top-two documents for T6 and the correct summary sentence for T7 are therefore determined by the same embedding similarities that generate the visualization. Without an external annotation protocol, T6/T7 accuracy is a read-back test of SVS's own similarity matrix.

  3. self definitional [§7.1.2 Task T8–T10 vs §5.1 Claim-level inference judgment]
    "Supporting documents for a summary sentence: For a summary sentence, select all documents that contribute to its inference judgment label. ... Finally, each atomic claim is passed to an LLM-based judge together with the retrieved evidence packets. For each claim, the model outputs one of three labels [26]:"

    The 'inference judgment label' referenced by T8 is generated by the system's own LLM judge; T9 and T10 ask for claims whose label is CONTRADICTS or INSUFFICIENT_EVIDENCE. The paper reports no human-created answer key for these tasks. Scoring accuracy against these labels measures whether users can recover the system's automated decisions, not whether those decisions are correct against the source documents. The Discussion calls these outputs 'inspectable and contestable cues,' but the study's dependent variable appears to treat them as ground truth.

full rationale

The paper's central accuracy claim—'Both guided layouts increased overall accuracy by 36.0%'—rests on tasks whose answer keys are never independently specified. For T4/T5, the most/least relevant document is naturally the argmax/argmin of Eq. (2), the same value that determines horizontal position in SUMMARY-GUIDED. For T6/T7, the same wd(j)/µd equations drive both the layout and the expected answer. For T8–T10, the correct answers are the system's own LLM-judge labels and retrieved document sets from §5.1. Since no external annotation procedure is reported anywhere in the manuscript, the accuracy component of the user study is at least partially circular: it measures how well users can read SVS's own similarity ranking and inference labels back, rather than whether summaries are verified against ground-truth source content. The efficiency, confidence, and workload findings are not circular—they compare conditions independent of the answer-key source—and T1–T3 could plausibly be scored from WCEP topic labels. The limitations section acknowledges that the pipeline 'partly depends on the underlying LLM,' but does not address the absence of independent answer keys for the accuracy metric. Because the reported accuracy gains are a central selling point and are partly self-referential, the circularity score is elevated; however, the paper still contains substantial non-circular contributions (layout designs, provenance visualization, time/workload effects), so the overall score is 6 rather than higher. No load-bearing self-citation chain was found.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on treating embedding similarity and LLM judgments as valid relevance/consistency signals, plus an unstated assumption that study accuracy labels are independent of those signals. Free parameters are the retrieval thresholds and UMAP settings, whose values are not reported.

free parameters (3)
  • semantic retrieval threshold
    Retains candidate evidence sentences in §5.1; affects which evidence links and layouts users see. Value not reported.
  • lexical retrieval threshold
    Second filter in evidence retrieval in §5.1; controls evidence recall versus precision. Value not reported.
  • UMAP parameters
    Controls the SOURCE-GUIDED document projection and sentence anchors in §4.2.1; exact settings not reported.
axioms (4)
  • domain assumption Cosine similarity of sentence embeddings is a valid proxy for summary–document relevance.
    Used to define Rd and µd in Eqs. (1)–(4) and to retrieve evidence in §5.1. If embeddings are poor, both layouts and the study's relevance tasks are built on a flawed signal.
  • ad hoc to paper LLM-based claim decomposition and inference labels are reliable enough to serve as provenance and as correctness ground truth.
    Pipeline in §5.1; tasks in §7.1.2. The paper acknowledges LLM dependence in §8 but does not externally validate the labels.
  • ad hoc to paper Task ground-truth labels are externally valid and comparable across the three fixed-order datasets.
    No annotation procedure is reported in §7.1.2–7.1.3, and dataset order is fixed, so difficulty and learning effects are assumed comparable without evidence.
  • domain assumption UMAP projection preserves meaningful semantic neighborhoods at the scale used.
    SOURCE-GUIDED layout in §4.2.1 and topic-cluster interpretation in §6.1 depend on this projection being faithful.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Spatial Visual Analytics for Multi-Document Summary Verification." pith.science (2026). https://pith.science/paper/RDZPEFHT

@misc{pith2026260728853,
  author       = {Pith},
  title        = {Pith review of: Spatial Visual Analytics for Multi-Document Summary Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RDZPEFHT}},
  note         = {Machine review of arXiv:2607.28853}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models increasingly generate summaries from collections of documents to support sensemaking and reporting, but verifying whether summary statements are grounded in source materials remains difficult. In multi-document summarization (MDS), evidence is distributed across many source documents and may be incomplete, conflicting, or missing. We present Summary Verification Space (SVS), a visual analytics system for verifying multi-document summaries through spatial document organization and coordinated provenance visualization. To support scalable verification, we investigate two alternative 2D canvas layouts: a SUMMARY-GUIDED layout that organizes documents by alignment with summary sentences, and a SOURCE-GUIDED layout that arranges documents by semantic similarity. Coordinated provenance visualization then makes relationships among summary content, source documents, and supporting evidence explicit, enabling users to trace support, contradiction, and missing evidence during verification. A task-driven usage scenario illustrates an auditing workflow in which users use the layouts to locate relevant documents and coverage gaps, then inspect linked claims and source evidence to make their own grounding judgments. In a comparative study with provenance held constant, both spatial layouts improved aggregate accuracy and reduced workload relative to a linear baseline, with the clearest gains on relevance tasks. The SUMMARY-GUIDED layout provides the strongest overall balance of accuracy, efficiency, confidence, and workload.

Figures

Figures reproduced from arXiv: 2607.28853 by Chris North, Eric Krokos, Jiahao Xu, Kirsten Whitley, Rebecca Faust, Wei Liu, Xuan Wang, Yang Liu.

Figure 1
Figure 1. Figure 1: Summary Verification Space (SVS). The summary panel (A) presents sentences (S1–S5), decomposed claims (C1–C2), [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: SOURCE-GUIDED layout. Documents form semantic clusters, while summary anchors show where evidence is concentrated. Brushed regions expose local topics, revealing that region A is largely uncovered by the summary. therefore proceeds in two stages: it first projects documents into a space, and then places summary sentences within this document space using semantic signals. 4.2.1 Document Organization: Semant… view at source ↗
Figure 3
Figure 3. Figure 3: Verification pipeline for provenance analysis. SVS transforms summary content into structured provenance through four stages: summary [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overall performance across the three document organiza [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Task-level performance across the ten study tasks (T1–T10) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Examples of commercial LLM interfaces that expose source materials for verification. (A) ChatGPT and (B) NotebookLM use inline citations [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The LINEAR condition used in the user study. The interface follows a familiar linear provenance layout, presenting the generated summary alongside source materials and highlighted evidence sentences. beddings with text-embedding-3-small. These embeddings sup￾port sentence-summary similarity computation, evidence retrieval, and the spatial layouts described in the main paper. The prototype uses gpt-5.1 as t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 12 canonical work pages · 2 internal anchors

  1. [1]

    A Brief Survey of Text Mining: Classification, Clustering and Extraction Techniques

    M. Allahyari, S. Pouriyeh, M. Assefi, S. Safaei, E. D. Trippe, J. B. Gutier- rez et al. A brief survey of text mining: Classification, clustering and extraction techniques, 2017. doi: 10.48550/arXiv.1707.02919 1

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1707.02919 2017

  2. [2]

    Andrews, A

    C. Andrews, A. Endert, and C. North. Space to think: large high-resolution displays for sensemaking. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’10, pp. 55–64. Association for Computing Machinery, New York, NY , USA, 2010. doi: 10.1145/ 1753326.1753336 2

    arXiv 2010

  3. [3]

    Andrews and C

    C. Andrews and C. North. The impact of physical navigation on spatial organization for sensemaking.IEEE Transactions on Visualization and Computer Graphics, 19(12):2207–2216, 2013. doi: 10.1109/TVCG.2013. 205 2

    doi:10.1109/tvcg.2013 2013

  4. [4]

    Arawjo, C

    I. Arawjo, C. Swoopes, P. Vaithilingam, M. Wattenberg, and E. L. Glass- man. ChainForge: A visual toolkit for prompt engineering and LLM hypothesis testing. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, art. no. 304, 18 pp. Association for Computing Machinery, New York, NY , USA, 2024. doi: 10.1145/ 3613904.3642016 3

    arXiv 2024

  5. [5]

    Bates, M

    D. Bates, M. Mächler, B. Bolker, and S. Walker. Fitting linear mixed- effects models using lme4.Journal of statistical software, 67:1–48, 2015. doi: 10.18637/jss.v067.i01 7

    doi:10.18637/jss.v067.i01 2015

  6. [6]

    N. Cao, J. Sun, Y .-R. Lin, D. Gotz, S. Liu, and H. Qu. FacetAtlas: Multifaceted visualization for rich text corpora.IEEE Transactions on Visualization and Computer Graphics, 16(6):1172–1181, 2010. doi: 10. 1109/TVCG.2010.154 2

    2010

  7. [7]

    G. Y .-Y . Chan, E. Bertini, L. G. Nonato, B. Barr, and C. T. Silva. Melody: Generating and visualizing machine learning model summary to under- stand data and classifiers together, 2020. doi: 10.48550/arXiv.2007.10614 2

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2007.10614 2020

  8. [8]

    S. Cho, M. Jang, J. Yeo, and D. Lee. RTSUM: Relation triple-based interpretable summarization with multi-level salience visualization. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 3: System Demonstrations), pp. 53–60. Association for Computationa...

    2024

  9. [9]

    Collins, S

    C. Collins, S. Carpendale, and G. Penn. DocuBurst: Visualizing document content using language structure.Computer Graphics Forum, 28(3):1039– 1046, 2009. doi: 10.1111/j.1467-8659.2009.01439.x 2

    arXiv 2009

  10. [10]

    DeYoung, S

    J. DeYoung, S. C. Martinez, I. J. Marshall, and B. C. Wallace. Do multi- document summarization models synthesize?Transactions of the As- sociation for Computational Linguistics, 12:1043–1062, 2024. doi: 10. 1162/tacl_a_00687 2

    2024

  11. [11]

    Dhaini, E

    M. Dhaini, E. Erdogan, S. Bakshi, and G. Kasneci. Explainability meets text summarization: A survey. InProceedings of the 17th International Natural Language Generation Conference, pp. 631–645. Association for Computational Linguistics, Tokyo, Japan, 2024. doi: 10.18653/v1/2024. inlg-main.49 1

    doi:10.18653/v1/2024 2024

  12. [12]

    Durmus, H

    E. Durmus, H. He, and M. Diab. FEQA: A question answering evalua- tion framework for faithfulness assessment in abstractive summarization. InProceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pp. 5055–5070. Association for Computational Linguistics, Online, 2020. doi: 10.18653/v1/2020.acl-main.454 2

    work page doi:10.18653/v1/2020.acl-main.454 2020

  13. [13]

    Endert, P

    A. Endert, P. Fiaux, and C. North. Semantic interaction for sensemaking: Inferring analytical reasoning for model steering.IEEE Transactions on Visualization and Computer Graphics, 18(12):2879–2888, 2012. doi: 10. 1109/TVCG.2012.260 2

    2012

  14. [14]

    Fabbri, I

    A. Fabbri, I. Li, T. She, S. Li, and D. Radev. Multi-news: A large- scale multi-document summarization dataset and abstractive hierarchical model. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074–1084. Association for Computational Linguistics, Florence, Italy, 2019. doi: 10.18653/v1/P19-1102 6

    doi:10.18653/v1/p19-1102 2019

  15. [15]

    A. R. Fabbri, W. Kry ´sci´nski, B. McCann, C. Xiong, R. Socher, and D. Radev. Summeval: Re-evaluating summarization evaluation.Transac- tions of the Association for Computational Linguistics, 9:391–409, 2021. doi: 10.1162/tacl_a_00373 2, 6

    doi:10.1162/tacl_a_00373 2021

  16. [16]

    Gabriel, A

    S. Gabriel, A. Celikyilmaz, R. Jha, Y . Choi, and J. Gao. GO FIGURE: A meta evaluation of factuality in summarization. InFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 478–

    2021

  17. [17]

    Y . Gao, W. Zhao, and S. Eger. SUPERT: Towards new frontiers in unsu- pervised evaluation metrics for multi-document summarization. InPro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1347–1354. Association for Computational Linguistics, Online, 2020. doi: 10.18653/v1/2020.acl-main.124 2

    work page doi:10.18653/v1/2020.acl-main.124 2020

  18. [18]

    Gholipour Ghalandari, C

    D. Gholipour Ghalandari, C. Hokamp, N. T. Pham, J. Glover, and G. Ifrim. A large-scale multi-document summarization dataset from the Wikipedia current events portal. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1302–1308. Association for Computational Linguistics, Online, 2020. doi: 10.18653/v1/2020. acl-m...

    doi:10.18653/v1/2020 2020

  19. [19]

    NotebookLM, 2026.https://notebooklm.google/

    Google. NotebookLM, 2026.https://notebooklm.google/. 6, 9

    2026

  20. [20]

    Gotz and M

    D. Gotz and M. X. Zhou. Characterizing users’ visual analytic activity for insight provenance.Information Visualization, 8(1):42–55, 2009. doi: 10. 1057/ivs.2008.31 2

    2009

  21. [21]

    M. A. Hearst. TileBars: visualization of term distribution information in full text information access. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’95, pp. 59–66. ACM Press/Addison-Wesley Publishing Co., USA, 1995. doi: 10.1145/223904. 223912 2

    doi:10.1145/223904 1995

  22. [22]

    Honnibal, I

    M. Honnibal, I. Montani, S. V . Landeghem, and A. Boyd. spaCy. Software,

  23. [23]

    Huang, X

    Y . Huang, X. Feng, X. Feng, and B. Qin. The factual inconsistency problem in abstractive text summarization: A survey, 2023. doi: 10.48550/ arXiv.2104.14839 1

    Pith/arXiv arXiv doi:10.48550/arxiv.2104.14839 2023

  24. [24]

    Isenberg and T

    P. Isenberg and T. Isenberg. Visualization on interactive surfaces: A research overview.i-com, 12(3):10–17, 2013. doi: 10.1524/icom.2013. 0020 2

    work page doi:10.1524/icom.2013 2013

  25. [25]

    Kryscinski, B

    W. Kryscinski, B. McCann, C. Xiong, and R. Socher. Evaluating the factual consistency of abstractive text summarization. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 9332–9346. Association for Computational Linguistics, Online, 2020. doi: 10.18653/v1/2020.emnlp-main.750 1

    doi:10.18653/v1/2020.emnlp-main.750 2020

  26. [26]

    Laban, T

    P. Laban, T. Schnabel, P. N. Bennett, and M. A. Hearst. Summac: Re- visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163– 177, 2022. doi: 10.1162/tacl_a_00453 2, 3, 5

    doi:10.1162/tacl_a_00453 2022

  27. [27]

    H. Li, Q. Dong, J. Chen, H. Su, Y . Zhou, Q. Ai et al. LLMs-as-Judges: A comprehensive survey on LLM-based evaluation methods, 2024. doi: 10. 48550/arXiv.2412.05579 2

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.05579 2024

  28. [28]

    C.-Y . Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74–81. Association for Computa- tional Linguistics, Barcelona, Spain, 2004. 2

    2004

  29. [29]

    S. Liu, Z. Li, T. Li, V . Srikumar, V . Pascucci, and P.-T. Bremer. NLIZE: A perturbation-driven visual interrogation tool for analyzing and interpreting natural language inference models.IEEE Transactions on Visualization and Computer Graphics, 25(1):651–660, 2019. doi: 10.1109/TVCG.2018. 2865230 2

    doi:10.1109/tvcg.2018 2019

  30. [30]

    W. Liu, C. North, and R. Faust. Visualizing spatial semantics of dimen- sionally reduced text embeddings, 2024. doi: 10.48550/arXiv.2409.03949 2

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.03949 2024

  31. [31]

    Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511–2522. Association for Computational Linguistics, Singapore,

    2023

  32. [32]

    C. C. Marshall and F. M. Shipman. Spatial hypertext: designing for change. Commun. ACM, 38(8):88–97, 1995. doi: 10.1145/208344.208350 2

    arXiv 1995

  33. [33]

    Maynez, S

    J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On faithfulness and factuality in abstractive summarization. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906–1919. Association for Computational Linguistics, Online, 2020. doi: 10.18653/v1/2020.acl-main.173 1

    doi:10.18653/v1/2020.acl-main.173 1906

  34. [34]

    McInnes, J

    L. McInnes, J. Healy, and J. Melville. UMAP: Uniform manifold approxi- mation and projection for dimension reduction, 2020. doi: 10.48550/arXiv .1802.03426 4

    Pith/arXiv arXiv doi:10.48550/arxiv.1802.03426 2020

  35. [35]

    B. Pan, Y . Fu, K. Wang, J. Lu, L. Pan, Z. Qian et al. VIS-Shepherd: Constructing critic for LLM-based data visualization generation, 2025. doi: 10.48550/arXiv.2506.13326 2 10 © 2026 IEEE. This is the author’s version of the article that has been published in IEEE Transactions on Visualization and Computer Graphics. The final version of this record is ava...

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.13326 2025

  36. [36]

    Pirolli and S

    P. Pirolli and S. Card. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of the 2005 International Conference on Intelligence Analysis, 6 pp. The MITRE Corporation, McLean, V A, USA, May 2005. 2

    2005

  37. [37]

    E. D. Ragan, A. Endert, J. Sanyal, and J. Chen. Characterizing prove- nance in visualization and data analysis: An organizational framework of provenance types and purposes.IEEE Transactions on Visualization and Computer Graphics, 22(1):31–40, 2016. doi: 10.1109/TVCG.2015. 2467551 2

    doi:10.1109/tvcg.2015 2016

  38. [38]

    Sarikaya, M

    A. Sarikaya, M. Gleicher, and D. A. Szafir. Design factors for summary visualization in visual analytics.Computer Graphics Forum, 37(3):145– 156, 2018. doi: 10.1111/cgf.13408 2

    work page doi:10.1111/cgf.13408 2018

  39. [39]

    F. M. Shipman III and C. C. Marshall. Formality considered harmful: Experiences, emerging themes, and directions on the use of formal repre- sentations in interactive systems.Computer Supported Cooperative Work (CSCW), 8(4):333–352, 1999. doi: 10.1023/A:1008716330212 2

    work page doi:10.1023/a:1008716330212 1999

  40. [40]

    Stasko, C

    J. Stasko, C. Görg, and Z. Liu. Jigsaw: Supporting investigative analysis through interactive visualization.Information Visualization, 7(2):118–132,

  41. [41]

    Strobelt, S

    H. Strobelt, S. Gehrmann, M. Behrisch, A. Perer, H. Pfister, and A. M. Rush. Seq2seq-vis: A visual debugging tool for sequence-to-sequence models.IEEE Transactions on Visualization and Computer Graphics, 25(1):353–363, 2019. doi: 10.1109/TVCG.2018.2865044 2

    arXiv 2019

  42. [42]

    S. Suh, B. Min, S. Palani, and H. Xia. Sensecape: Enabling multilevel exploration and sensemaking with large language models. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Tech- nology, UIST ’23, art. no. 1, 18 pp. Association for Computing Machinery, New York, NY , USA, 2023. doi: 10.1145/3586183.3606756 3

    arXiv 2023

  43. [43]

    S. Syed, T. Yousef, K. Al Khatib, S. Jänicke, and M. Potthast. Summary explorer: Visualizing the state of the art in text summarization. InProceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 185–194. Association for Com- putational Linguistics, Online and Punta Cana, Dominican Republic, 2021...

    work page doi:10.18653/v1/2021.emnlp-demo.22 2021

  44. [44]

    X. Tang, E. Krokos, C. Liu, K. Davidson, K. Whitley, N. Ramakrishnan et al. Steering LLM summarization with visual workspaces for sensemaking,

  45. [45]

    Tenney, J

    I. Tenney, J. Wexler, J. Bastings, T. Bolukbasi, A. Coenen, S. Gehrmann et al. The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pp. 107–118. Association for Computational Linguistics, Online,

    2020

  46. [46]

    van der Maaten and G

    L. van der Maaten and G. Hinton. Visualizing data using t-SNE.Journal of Machine Learning Research, 9(86):2579–2605, 2008. 4

    2008

  47. [47]

    J. Vig, W. Kryscinski, K. Goel, and N. Rajani. SummVis: Interactive visual analysis of models, data, and evaluation for text summarization. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 150–158. Associa- tio...

    doi:10.18653/v1/2021 2021

  48. [48]

    A. Wang, K. Cho, and M. Lewis. Asking and answering questions to evaluate the factual consistency of summaries. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5008–5020. Association for Computational Linguistics, Online, 2020. doi: 10.18653/v1/2020.acl-main.450 2

    doi:10.18653/v1/2020.acl-main.450 2020

  49. [49]

    C. Wang, A. Jain, D. Chen, and J. Gu. VizSeq: a visual analysis toolkit for text generation tasks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pp. 253–258. Association for Computational Linguistics, ...

    work page doi:10.18653/v1/d19-3043 2019

  50. [50]

    Z. Xie, Y . Guo, and K. Xu. AwesomeLit: Towards hypothesis generation with agent-supported literature research. In J. Byska, A. Ottley, and M. Waldner, eds.,EuroVis 2026 – Short Papers, 5 pp. The Eurographics Association, 2026. doi: 10.2312/evs.20261000 9

    work page doi:10.2312/evs.20261000 2026

  51. [51]

    doi: 10.18653/v1/2020.emnlp-demos.15 2

    work page doi:10.18653/v1/2020.emnlp-demos.15 2020

  52. [52]

    Zhang, Y

    H. Zhang, Y . Xu, and L. Perez-Beltrachini. Fine-grained natural language inference based faithfulness evaluation for diverse summarisation tasks. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1701–1722. Association for Computational Linguistics, St. Julian’s, Malta,

  53. [53]

    Zhang, P

    H. Zhang, P. S. Yu, and J. Zhang. A systematic survey of text summariza- tion: From statistical methods to large language models.ACM Comput. Surv., 57(11), art. no. 277, 41 pp., 2025. doi: 10.1145/3731445 1

    doi:10.1145/3731445 2025

  54. [54]

    Zhang, V

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi. BERTScore: Evaluating text generation with BERT, 2020. doi: 10.48550/arXiv.1904. 09675 2

    doi:10.48550/arxiv.1904 2020

  55. [55]

    Zhang, W

    Z. Zhang, W. Peng, X. Chen, L. Cao, and T. J.-J. Li. LADICA: A large shared display interface for generative AI cognitive assistance in co-located team collaboration. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, art. no. 147, 22 pp. Association for Computing Machinery, New York, NY , USA, 2025. doi: 10.1145/ 370...

    arXiv 2025

  56. [57]

    Y . Zha, Y . Yang, R. Li, and Z. Hu. AlignScore: Evaluating factual consis- tency with a unified alignment function. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11328–11348. Association for Computational Linguistics, Toronto, Canada, 2023. doi: 10.18653/v1/2023.acl-long.634 2

    doi:10.18653/v1/2023.acl-long.634 2023

  57. [59]

    doi: 10.18653/v1/2024.eacl-long.102 3

    doi:10.18653/v1/2024.eacl-long.102 2024

  58. [487]

    Association for Computational Linguistics, Online, 2021. doi: 10. 18653/v1/2021.findings-acl.42 1

    2021

  59. [2008]

    doi: 10.1057/palgrave.ivs.9500180 2

    work page doi:10.1057/palgrave.ivs.9500180

  60. [2020]

    doi: 10.5281/zenodo.1212303 4

    doi:10.5281/zenodo.1212303

  61. [2023]

    doi: 10.18653/v1/2023.emnlp-main.153 2

    doi:10.18653/v1/2023.emnlp-main.153 2023

  62. [2024]

    doi: 10.48550/arXiv.2409.17289 2

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.17289

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.