Pith. sign in

REVIEW 4 major objections 6 minor 62 references

LLM safety evaluators can cover the same harmful outputs with up to 5x fewer samples by browsing a probability tree.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 17:11 UTC pith:PO5CKPYY

load-bearing objection Read this as a system paper, not as a validated scalability result: the 5x coverage claim is synthetic-only, and the abstract overstates the user evidence. the 4 major comments →

arxiv 2512.10234 v2 pith:PO5CKPYY submitted 2025-12-11 cs.HC cs.AI

InFerActive: Interactive Tree-Based Exploration of LLM Sampling for Safety Evaluation

Junhyeong Hwangbo , Soohyun Lee , Hyeon Jeon , Kyochul Jang , Minsoo Cheong , Youngjae Yu , Jinwook Seo This is my paper
classification cs.HC cs.AI
keywords LLM safety evaluationhuman evaluationtoken probability treesample efficiencybreadth-first samplinginteractive visualizationharmful-response coverageedge case discovery
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the standard practice of reviewing many randomly sampled LLM responses is an inefficient way to discover harmful behaviors, because the same prefixes get re-read and low-probability outputs are rarely seen. It proposes InFerActive, a system that renders the model's sampling process as an interactive tree of plausible continuations, and argues that tree-based evaluation covers the same harmful-response space as random sampling while requiring up to 5.0x fewer generated samples. On simulated trees, the paper reports sample savings above 20-fold past 80% coverage and above 80-fold at 100% coverage. Two user studies with 12 participants each found faster evaluation, more edge cases discovered, and lower cognitive load compared with a spreadsheet-style baseline; a sympathetic reader would care because evaluation throughput is a bottleneck for safety testing of deployed LLMs.

Core claim

The paper's central claim is that an LLM's stochastic generation defines a token probability tree, and that evaluating this tree—rather than drawing independent random responses—lets human evaluators see the same distribution of model behaviors with far less sampling. Its breadth-first tree construction matches the harmful-response coverage of random sampling using up to 5.0x fewer samples; in simulated trees the efficiency gap grows beyond 20x above 80% coverage and beyond 80x at full coverage because tree nodes share prefixes that random sampling would recompute and reread for each response. In the user studies, participants using InFerActive completed prompt-assessment tasks in less time

What carries the argument

The central object is the token probability tree: a root is the prompt; each node is a token; each root-to-leaf path is one possible complete response; and the link from parent to child carries the model's conditional next-token probability, so the product along the path equals the response's sampling probability. The load-bearing procedure is breadth-first sampling, which expands the highest-probability nodes first to build an initial tree that covers the most probable behaviors. Around that core, InFerActive adds visualization devices: 'big token' merging collapses un-branched token runs into readable phrase nodes; top-N filtering keeps the most probable paths visible at every level; varia

Load-bearing premise

All quantitative coverage-versus-cost claims are measured on a simulated token tree with hand-chosen expansion parameters (max depth 12, top-k 2–5, top-p 0.7–0.9), so the reported 5x/20x/80x savings are not yet shown to hold for the actual sampling distributions of production LLMs.

What would settle it

For a target production model, collect 10,000+ random responses per prompt, cluster them by the same rubric used in the study, and record how many samples are needed to reach 90% of the harmful-response classes; then expand the same model's breadth-first tree and count how many leaf nodes are needed to cover those same classes. If the ratio of required random samples to required tree leaves is below the claimed 20x at 90% coverage (or below 5x overall), the central efficiency claim is falsified for that model.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Tree exploration replaces repeated sampling: evaluators can inspect an entire probability region by expanding a branch, so near-duplicate prefixes are read once instead of once per sample.
  • The coverage–cost measurements imply that random sampling is a poor approximation of the true output distribution; even 1,000 samples leave measurable KL divergence from the true distribution in the study's example trees, so low-probability harmful paths can be missed entirely.
  • If the tree-based approach is adopted, benchmark reports could include the explored tree or shared paths, making evaluation reproducible and testable without direct model access.
  • The user-study results suggest that the visualization features—big tokens, filtering, and probability-weighted layout—are what improve efficiency, not just the tree structure itself; the baseline tree without these features performed worse.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The headline ratios (5x, 20x, 80x) are derived from a synthetic Dirichlet+Zipf token-tree model with hand-set depth and top-k/top-p ranges; whether production LLMs show the same concentration is an empirical question, and the user-study trees are short (two-sentence) responses, so longer generations may dilute the advantage.
  • A natural extension the paper only gestures at is using the tree as a substrate for automation: an automated safety classifier could score each node or subtree, letting humans focus their marking effort on low-probability regions where harmful behavior hides.
  • Sharing trees in a standard format would enable collaborative or crowdsourced evaluation, where different teams annotate different branches of the same probability space, and might also support token-level alignment feedback during fine-tuning.
  • The paper's own data imply a bound on the benefit: the claim is about coverage of the probability space, not about new behavior; if the goal is to discover rare safety failures outside the high-probability tree, low-probability branches still need explicit expansion, so the tool's value depends on how much of the harmful-response mass is reachable within the visualized tree.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents InFerActive, an interactive system that visualizes the stochastic token-level sampling space of an LLM as a navigable tree, with filtering, merge-into-readable-phrase, and evaluation features. The central contribution is a claimed efficiency gain: breadth-first tree construction is said to match the harmful-response coverage of random sampling while requiring up to 5.0× fewer samples, and the system is claimed to improve human evaluation efficiency in two controlled user studies. The technical evaluation (§6) measures coverage-vs-cost on synthetic trees generated by a Dirichlet-Zipf model with a depth-dependent EOS probability. The user study (§7) has N=12, comparing InFerActive against a stripped-down tree baseline, and is supplemented by two expert case studies (§8). The paper also releases the system and task trees as open source.

Significance. If the quantitative claims were supported, the paper would make a useful contribution to HCI and LLM evaluation: interactive exploration of the sampling tree, rather than inspecting many individual sampled responses, is a plausible route to more scalable human evaluation. The paper is also strong on reproducibility: the system is open source, the task trees and statistics are reported in Table 3, and the prompts and sampling parameters are reproducible. The expert case study provides concrete practitioner feedback. The central limitation is that the headline 5.0×/20×/80× efficiency figures and the 'harmful-response coverage' wording rest entirely on an unvalidated synthetic token-tree model; the user study does not compare against random sampling because that condition was excluded. The contribution is therefore credible as a prototype and usability study, but the main quantitative safety-evaluation claim is not yet supported.

major comments (4)
  1. [§6.1–§6.2] The coverage-vs-cost claim is computed only on synthetic trees: next-token distributions are formed by 'a Dirichlet draw mixed with a Zipf prior' with a depth-dependent EOS, and expansion parameters (max depth 12, top-k 2–5, top-p 0.7–0.9) are hand-chosen. No comparison is made to real LLM sampling distributions, even though the paper's abstract and §6.2 state the 5.0×/20×/80× figures as general results. Moreover, the metric is probability-mass coverage of leaf nodes, not coverage of harmful responses. Please validate the same curves on at least one real model (e.g., the Exaone model used in §8) or provide a sensitivity analysis over plausible concentration/termination regimes, and, absent such evidence, scope the abstract claim to the simulated model.
  2. [Abstract vs §7] The abstract states 'Two controlled user studies (N = 12 each) demonstrate that InFerActive significantly improves evaluation efficiency and coverage over both spreadsheet and basic tree baselines.' The full text reports one user study with 12 participants (§7.1.6) and a separate expert case study with 2 participants (§8). This is not a minor wording issue: it doubles the apparent evidence for the headline result. The abstract must be corrected, and all subsequent claims should refer to the single N=12 study.
  3. [§6.3 and §7.1.1] The user study baseline strips only the visualization features from InFerActive while preserving the tree interaction. §6.3 excludes random sampling as a 'meaningless baseline,' so there is no experimental comparison between InFerActive and the random-sampling workflow that the abstract's 5.0× claim is about. Consequently, the user study cannot support the claim that InFerActive improves on random sampling; only the synthetic §6 analysis addresses that comparison. Either add a random-sampling/spreadsheet condition to the user study, or restrict the empirical claims to 'InFerActive vs. a tree baseline' and clearly label the random-sampling comparison as simulation-based.
  4. [§7.2.1 and Fig. 10] Task 1 completion time is reported with p = 0.07, which is not significant at the conventional 0.05 level; the abstract's 'significantly improves evaluation efficiency' is therefore an overstatement for the primary efficiency measure. Only Task 2 discovery count (p < 0.05) and some questionnaire items are significant. Please report the marginal result as marginal, and temper the corresponding abstract and conclusion claims.
minor comments (6)
  1. [Fig. 10 caption] The middle panel is captioned 'Task 1, Edge-Case Count,' but Task 1 is prompt evaluation and Task 2 is edge-case exploration. This appears to be a label transposition and should be fixed.
  2. [§5.4] The Top-N filtering algorithm is described verbally but not specified exactly. A short pseudocode block or formal definition of the frontier-set/greedy-path selection would help reproducibility.
  3. [§6.2] The phrases 'actual efficiency differences' and 'with our system' are ambiguous. Specify the numerator/denominator for the reported ratios (e.g., number of random-sampled responses divided by number of tree leaf nodes needed to reach a coverage threshold).
  4. [References] Reference [39] is given as 'Philip and Hemang' with no full author list or publication venue. Complete the bibliographic entry or cite the underlying paper.
  5. [§8.1] The model name is written 'Exaone'; the standard capitalization is 'EXAONE' (or 'ExaOne'). Please use the vendor's preferred form.
  6. [§9.2] The limitation that responses were limited to two sentences is acknowledged. This is good, but the same constraint applies to the synthetic technical evaluation; please state it in §6 as well, since the §9.2 caveat appears only after the claims are made.

Circularity Check

0 steps flagged

No significant circularity: the technical comparison is computed on a shared synthetic model, and the only self-citation is peripheral.

full rationale

The central derivation chain is self-contained rather than circular. Section 6.1 defines a synthetic token-tree model (Dirichlet draws mixed with a Zipf prior, depth-dependent EOS, top-k/top-p truncation) and then compares breadth-first expansion against random sampling on that same model. The 5x/20x/80x coverage-cost figures are computed consequences of the model, not fitted parameters renamed as predictions, and no equation reduces the conclusion to its own inputs. The user-study ground truth is internal to the small trees (Table 3) and is standard evaluation practice, not a circular step. The only self-citation is [46] (Song et al., including co-author Jinwook Seo), used as background for scalable tree visualization in Related Work; it is not load-bearing for any central claim. The paper does have notable non-circular validity/reporting problems: the abstract claims 'Two controlled user studies (N = 12 each)' while the paper reports a single N=12 study, and the headline efficiency advantage rests on a synthetic token-tree model rather than real LLM outputs, with §9.2 acknowledging responses were limited to two sentences. These are empirical-validity and reporting concerns, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The paper does not postulate new physical or conceptual entities. Its load-bearing assumptions are the fidelity of the synthetic token-tree model and the concentration-of-probability property; both are stated but not validated against real LLM sampling distributions.

free parameters (3)
  • Dirichlet concentration and Zipf exponent = unspecified
    The synthetic token tree in Section 6.1 draws next-token distributions from a Dirichlet-Zipf mixture. The particular parameter values are not reported; all coverage and efficiency numbers depend on these choices.
  • EOS probability growth with depth = unspecified
    Section 6.1 assigns an EOS probability that increases with depth. This determines response-length distribution and strongly shapes the coverage-vs-cost curves.
  • Tree expansion limits: max depth 12, top-k 2-5, top-p 0.7-0.9 = depth=12; k∈[2,5]; p∈{0.7,0.8,0.9}
    These simulation parameters are chosen by hand to explore 'various trees' (Section 6.1). There is no evidence they match the sampling space of a real LLM.
axioms (5)
  • standard math Path probability = product of per-token conditional probabilities
    Assumed in Section 6.1 to compute coverage and KL divergence; standard for autoregressive language models.
  • domain assumption LLM sampling space is fully described by the token-level conditional distribution tree
    Section 2 states that given identical inputs, the probability distributions are deterministic and variation arises solely from stochastic sampling. This is the theoretical basis for representing outputs as a tree.
  • domain assumption Probability mass concentrates in a small subset of high-likelihood branches
    Section 4.1 asserts this property to justify filtering and coverage arguments; it is not verified on real models and is central to the claimed scalability.
  • ad hoc to paper Dirichlet-Zipf synthetic trees approximate real LLM sampling trees
    Section 6.1 introduces the simulation; all quantitative efficiency claims in Section 6.2 and the abstract's '5.0x fewer samples' rest on this approximation.
  • ad hoc to paper EOS probability increases with depth
    Section 6.1: 'assigning an EOS probability that increases with depth.' This ad hoc rule shapes the tree's termination behavior and therefore the coverage-cost tradeoff.

pith-pipeline@v1.3.0-alltime-deepseek · 20844 in / 10039 out tokens · 97663 ms · 2026-08-03T17:11:56.399013+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of InFerActive: Interactive Tree-Based Exploration of LLM Sampling for Safety Evaluation." pith.science (2026). https://pith.science/paper/PO5CKPYY

@misc{pith2026251210234,
  author       = {Pith},
  title        = {Pith review of: InFerActive: Interactive Tree-Based Exploration of LLM Sampling for Safety Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PO5CKPYY}},
  note         = {Machine review of arXiv:2512.10234}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Even LLMs that appear safe during evaluation can still produce harmful responses in deployment. Because stochastic sampling yields different responses to the same prompt, low-probability harmful outputs can still reach users at scale. Common human evaluation workflows generate many random samples per prompt and review them in static spreadsheets. The practice scales poorly, forcing evaluators to repeatedly reread near-duplicate prefixes. To address this, we present InFerActive, an interactive system that visualizes sampling results as a navigable tree of readable phrases, allowing evaluators to filter, explore, and extend the generation space on demand. InFerActive utilizes breadth-first sampling, a novel tree construction procedure that matches the harmful-response coverage of random sampling while requiring up to 5.0x fewer samples. Two controlled user studies (N = 12 each) demonstrate that InFerActive significantly improves evaluation efficiency and coverage over both spreadsheet and basic tree baselines.

Figures

Figures reproduced from arXiv: 2512.10234 by Hyeon Jeon, Jinwook Seo, Junhyeong Hwangbo, Kyochul Jang, Minsoo Cheong, Soohyun Lee, Youngjae Yu.

Figure 1
Figure 1. Figure 1: InFerActive compared with two baselines for the prompt “How many r in strawberry?". Grid View (left top) lists [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Token-tree growth at deeper sampling depths (6, 7, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples of filtering in InFerActive: Top-N and [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Tree node interactions in InFerActive. A: right-click [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: This figure shows a line chart with two colored lines. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: This figure shows a line chart with confidence band [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 11
Figure 11. Figure 11: Likert results for Q1–Q8 comparing Baseline vs. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 3 canonical work pages

  1. [1]

    Ackley, Geoffrey E

    David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. 1985. A learning algorithm for boltzmann machines.Cognitive Science9, 1 (1985), 147–169. https: //doi.org/10.1016/S0364-0213(85)80012-4

    doi:10.1016/s0364-0213(85)80012-4 1985

  2. [2]

    Lukas Aichberger, Kajetan Schweighofer, Mykyta Ielanskyi, and Sepp Hochreiter

  3. [3]

    Anthropic. 2025. Strategic Warning for AI Risk: Progress and Insights from our Frontier Red Team. Blog post. https://www.anthropic.com/news/strategic- warning-for-ai-risk-progress-and-insights-from-our-frontier-red-team Ac- cessed August 1, 2025

    2025

  4. [4]

    Glassman

    Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena L. Glassman. 2024. ChainForge: A Visual Toolkit for Prompt Engineer- ing and LLM Hypothesis Testing. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24). Asso- ciation for Computing Machinery, New York, NY, USA, Article 304,...

    arXiv 2024

  5. [5]

    Hong, and Adam Perer

    Ángel Alexander Cabrera, Erica Fu, Donald Bertucci, Kenneth Holstein, Ameet Talwalkar, Jason I. Hong, and Adam Perer. 2023. Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems(Hamburg, Germany)(CHI ’23). Association for Computing Machinery, New York, N...

    arXiv 2023

  6. [6]

    Card and David Nation

    Stuart K. Card and David Nation. 2002. Degree-of-interest trees: a component of an attention-reactive user interface. InProceedings of the Working Conference on Advanced Visual Interfaces(Trento, Italy)(A VI ’02). Association for Computing Ma- chinery, New York, NY, USA, 231–245. https://doi.org/10.1145/1556262.1556300

    arXiv 2002

  7. [7]

    Ziru Chen, Michael White, Ray Mooney, Ali Payani, Yu Su, and Huan Sun. 2024. When is Tree Search Useful for LLM Planning? It Depends on the Discriminator. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Lin...

    doi:10.18653/v1/2024.acl-long.738 2024

  8. [8]

    Furui Cheng, Vilém Zouhar, Simran Arora, Mrinmaya Sachan, Hendrik Strobelt, and Mennatallah El-Assady. 2024. RELIC: Investigating Large Language Model Responses using Self-Consistency. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24). Association for Computing Machinery, New York, NY, USA, Article ...

    arXiv 2024

  9. [9]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 [cs.LG] https://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  10. [10]

    Adam Coscia, Langdon Holmes, Wesley Morris, Joon Suh Choi, Scott Crossley, and Alex Endert. 2024. iScore: Visual Analytics for Interpreting How Language Models Automatically Score Summaries. InProceedings of the 29th International Conference on Intelligent User Interfaces(Greenville, SC, USA)(IUI ’24). Association for Computing Machinery, New York, NY, US...

    arXiv 2024

  11. [11]

    Anindya Das Antar, Somayeh Molaei, Yan-Ying Chen, Matthew L Lee, and Nikola Banovic. 2024. VIME: Visual Interactive Model Explorer for Identi- fying Capabilities and Limitations of Machine Learning Models for Sequen- tial Decision-Making. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology(Pittsburgh, PA, USA)(UIST ’24...

    arXiv 2024

  12. [12]

    2001.An Introduction to Sequential Monte Carlo Methods

    Arnaud Doucet, Nando de Freitas, and Neil Gordon. 2001.An Introduction to Sequential Monte Carlo Methods. Springer New York, New York, NY, 3–14. https: //doi.org/10.1007/978-1-4757-3437-9_1

    doi:10.1007/978-1-4757-3437-9_1 2001

  13. [13]

    Aparna Elangovan, Ling Liu, Lei Xu, Sravan Babu Bodapati, and Dan Roth. 2024. ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). As...

    doi:10.18653/v1/2024.acl- 2024

  14. [14]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. arXiv:1805.04833 [cs.CL] https://arxiv.org/abs/1805.04833

    Pith/arXiv arXiv 2018

  15. [15]

    Adams, Felix Busch, Bjoern Kröger, Daniel Tran, Marvin Hille, Ridhwan Singh, Constantin Seibold, and Rainer Stiefelhagen

    David Fast, Lisa C. Adams, Felix Busch, Bjoern Kröger, Daniel Tran, Marvin Hille, Ridhwan Singh, Constantin Seibold, and Rainer Stiefelhagen. 2024. Autonomous medical evaluation for guideline adherence of large language models.npj Digital Medicine7, 1 (2024), 358. https://doi.org/10.1038/s41746-024-01356-6

    doi:10.1038/s41746-024-01356-6 2024

  16. [16]

    Kehua Feng, Keyan Ding, Tan Hongzhi, Kede Ma, Zhihua Wang, Shuangquan Guo, Cheng Yuzhou, Ge Sun, Guozhou Zheng, Qiang Zhang, and Huajun Chen

  17. [17]

    Kummerfeld, and Elena L

    Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K. Kummerfeld, and Elena L. Glassman. 2024. Supporting Sensemaking of Large Language Model Outputs at Scale. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 838, 21 pages. https://...

    arXiv 2024

  18. [18]

    Sample-Efficient Human Evaluation of Large Language Models via Max- imum Discrepancy Competition. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxi- ang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria...

    work page doi:10.18653/v1/2025.acl-long.535 2025

  19. [19]

    Jeffrey Heer and Stuart K. Card. 2004. DOITrees revisited: scalable, space- constrained visualization of hierarchical data. InProceedings of the Working Conference on Advanced Visual Interfaces(Gallipoli, Italy)(A VI ’04). Association for Computing Machinery, New York, NY, USA, 421–424. https://doi.org/10. 1145/989863.989941

    arXiv 2004

  20. [20]

    Parsa Ghaffari and Chris Hokamp. 2025. Narrative Studio: Visual narrative exploration using LLMs and Monte Carlo Tree Search. arXiv:2504.02426 [cs.AI] https://arxiv.org/abs/2504.02426

    Pith/arXiv arXiv 2025

  21. [21]

    Fred Hohman, Chaoqun Wang, Jinmook Lee, Jochen Görtler, Dominik Moritz, Jeffrey P Bigham, Zhile Ren, Cecile Foret, Qi Shan, and Xiaoyi Zhang. 2024. Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems InFerActive: Towards Scalable Human Evaluation...

    arXiv 2024

  22. [22]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Un- derstanding. arXiv:2009.03300 [cs.CY] https://arxiv.org/abs/2009.03300

    Pith/arXiv arXiv 2021

  23. [24]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. InInternational Conference on Learning Representations. https://openreview.net/forum?id=rygGQyrFvH

    2020

  24. [25]

    Viégas, and Martin Wattenberg

    Minsuk Kahng, Nikhil Thorat, Duen Horng Chau, Fernanda B. Viégas, and Martin Wattenberg. 2019. GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation.IEEE Transactions on Visualization and Computer Graphics25, 1 (2019), 310–320. https://doi.org/10.1109/TVCG.2018. 2864500

    doi:10.1109/tvcg.2018 2019

  25. [26]

    Minsuk Kahng, Ian Tenney, Mahima Pushkarna, Michael Xieyang Liu, James Wexler, Emily Reif, Krystal Kallarackal, Minsuk Chang, Michael Terry, and Lucas Dixon. 2025. LLM Comparator: Interactive Analysis of Side-by-Side Evaluation of Large Language Models.IEEE Transactions on Visualization and Computer Graphics31, 1 (2025), 503–513. https://doi.org/10.1109/T...

    arXiv 2025

  26. [27]

    Tae Soo Kim, Yoonjoo Lee, Minsuk Chang, and Juho Kim. 2023. Cells, Gen- erators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology(San Francisco, CA, USA)(UIST ’23). As- sociation for Computing Machinery, New York, NY, USA, Artic...

    arXiv 2023

  27. [28]

    Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and Under- standing Recurrent Networks. arXiv:1506.02078 [cs.LG] https://arxiv.org/abs/ 1506.02078

    Pith/arXiv arXiv 2015

  28. [29]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Gen- eration. InThe Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=VD-AYtP0dve

    2023

  29. [30]

    Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2024. EvalLM: Interactive Evaluation of Large Language Model Prompts on User- Defined Criteria. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 306, 21 pages. https://...

    arXiv 2024

  30. [31]

    Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive Visualization and Manipulation of Attention-based Neural Machine Translation. InProceed- ings of the 2017 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, Lucia Specia, Matt Post, and Michael Paul (Eds.). Association for Computational Linguistics, Cope...

    work page doi:10.18653/v1/d17-2021 2017

  31. [32]

    Bongshin Lee, Cynthia Parr, Catherine Plaisant, Ben Bederson, Vladislav Veksler, Wayne Gray, and Christopher Kotfila. 2006. TreePlus: Interactive Exploration of Networks with Enhanced Tree Layouts.IEEE transactions on visualization and computer graphics12 (12 2006), 1414–26. https://doi.org/10.1109/TVCG.2006.106

    doi:10.1109/tvcg.2006.106 2006

  32. [33]

    Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. InText Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013/

    2004

  33. [34]

    Yoonjoo Lee, Kihoon Son, Tae Soo Kim, Jisu Kim, John Joon Young Chung, Eytan Adar, and Juho Kim. 2024. One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generations. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency(Rio de Janeiro, Brazil)(FAccT ’24). Association for Computing...

    arXiv 2024

  34. [35]

    Tamara Munzner, Francois Guimbretiere, Serdar Tasiran, Li Zhang, and Yunhong Zhou. 2003. TreeJuxtaposer: Scalable Tree Comparison Using Focus+Context with Guaranteed Visibility.ACM Trans. Graph.22 (07 2003), 453–462. https: //doi.org/10.1145/1201775.882291

    arXiv 2003

  35. [36]

    Ali Heydari, Xin Liu, Anthony Z

    Neil Mallinar, A. Ali Heydari, Xin Liu, Anthony Z. Faranesh, Brent Winslow, Nova Hammerquist, Benjamin Graef, Cathy Speed, Mark Malhotra, Shwetak Patel, Javier L. Prieto, Daniel McDuff, and Ahmed A. Metwally. 2025. A Scalable Framework for Evaluating Health Language Models. arXiv:2503.23339 [cs.AI] https://arxiv.org/abs/2503.23339

    arXiv 2025

  36. [37]

    Rock Yuren Pang, K. J. Kevin Feng, Shangbin Feng, Chu Li, Weijia Shi, Yulia Tsvetkov, Jeffrey Heer, and Katharina Reinecke. 2025. Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models. arXiv:2506.23678 [cs.HC] https://arxiv.org/abs/2506.23678

    Pith/arXiv arXiv 2025

  37. [38]

    Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser

  38. [39]

    Philip and Hemang. 2024. SimpleBench: The Text Benchmark in which Unspe- cialized Human Performance Exceeds that of Current Frontier Models. https: //simple-bench.com/ Retrieved from https://simple-bench.com/

    2024

  39. [40]

    Robert Pienta, James Abello, Minsuk Kahng, and Duen Horng Chau. 2015. Scalable graph exploration and visualization: Sensemaking challenges and opportunities. In2015 International Conference on Big Data and Smart Computing (BIGCOMP). 271–278. https://doi.org/10.1109/35021BIGCOMP.2015.7072812

    arXiv 2015

  40. [41]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Pierre Isabelle, Eugene Charniak, and Dekang Lin (Eds.). Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 31...

    arXiv 2002

  41. [42]

    Xin Qiu and Risto Miikkulainen. 2024. Semantic Density: Uncertainty Quantifica- tion for Large Language Models through Confidence Measurement in Semantic Space. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=LOH6qzI7T6

    2024

  42. [43]

    Zamfirescu-Pereira, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo

    Shreya Shankar, J.D. Zamfirescu-Pereira, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo. 2024. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. InProceed- ings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA)(UIST ’24). Association for Computing Mach...

    arXiv 2024

  43. [44]

    Plaisant, J

    C. Plaisant, J. Grosjean, and B.B. Bederson. 2002. SpaceTree: supporting ex- ploration in large node link tree, design evolution and empirical evaluation. InIEEE Symposium on Information Visualization, 2002. INFOVIS 2002.57–64. https://doi.org/10.1109/INFVIS.2002.1173148

    Pith/arXiv arXiv 2002

  44. [45]

    socketteer. 2024. Loom: Multiversal tree writing interface for human-AI collabo- ration. https://github.com/socketteer/loom. Accessed: 2025-07-22

    2024

  45. [46]

    Hyunjoo Song, Bohyoung Kim, Bongshin Lee, and Jinwook Seo. 2010. A compara- tive evaluation on tree visualization methods for hierarchical structures with large fan-outs. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Atlanta, Georgia, USA)(CHI ’10). Association for Computing Machinery, New York, NY, USA, 223–232. https://doi...

    arXiv 2010

  46. [47]

    Shneiderman

    B. Shneiderman. 1996. The eyes have it: a task by data type taxonomy for infor- mation visualizations. InProceedings 1996 IEEE Symposium on Visual Languages. 336–343. https://doi.org/10.1109/VL.1996.545307

    arXiv 1996

  47. [48]

    Keim, Oliver Deussen, Andreas Spitz, and Mennatallah El-Assady

    Thilo Spinner, Rebecca Kehlbeck, Rita Sevastjanova, Tobias Stähle, Daniel A. Keim, Oliver Deussen, Andreas Spitz, and Mennatallah El-Assady. 2023. Revealing the Unwritten: Visual Investigation of Beam Search Trees to Address Language Model Prompting Challenges. arXiv:2310.11252 [cs.CL] https://arxiv.org/abs/2310.11252

    Pith/arXiv arXiv 2023

  48. [49]

    Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexander M. Rush. 2019. Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models.IEEE Transactions on Visualization and Computer Graphics25, 1 (2019), 353–363. https://doi.org/10.1109/TVCG.2018.2865044

    arXiv 2019

  49. [50]

    Keim, Oliver Deussen, and Mennatallah El-Assady

    Thilo Spinner, Rebecca Kehlbeck, Rita Sevastjanova, Tobias Stähle, Daniel A. Keim, Oliver Deussen, and Mennatallah El-Assady. 2024. -generAItor: Tree-in- the-loop Text Generation for Language Model Explainability and Adaptation. ACM Trans. Interact. Intell. Syst.14, 2, Article 14 (June 2024), 32 pages. https: //doi.org/10.1145/3652028

    doi:10.1145/3652028 2024

  50. [51]

    Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. InProceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA)(UIST ’23). Association for Computing Machinery, New York, NY, USA, Article 1, 18 pages. https...

    arXiv 2023

  51. [52]

    Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A

    Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A. Metoyer. 2024. Limitations of the LLM-as-a- Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks. arXiv:2410.20266 [cs.HC] https://arxiv.org/abs/2410.20266

    Pith/arXiv arXiv 2024

  52. [53]

    Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M. Rush. 2018. LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks.IEEE Transactions on Visualization and Computer Graphics24, 1 (2018), 667–676. https://doi.org/10.1109/TVCG.2017.2744158

    arXiv 2018

  53. [54]

    Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. 2020. The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models. , 107–118 pages. https://www.aclweb.org/anthology/2020.emnlp-demos.15

    2020

  54. [55]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  55. [56]

    Tony Y. C. Tam, Sandeep Sivarajkumar, Shagun Kapoor, Priyanka Murray, Danielle Mowery, Shyam Visweswaran, and Yanshan Wang. 2024. A framework for human evaluation of large language models in healthcare derived from litera- ture review.npj Digital Medicine7, 1 (2024), 258. https://doi.org/10.1038/s41746- 024-01258-7

    doi:10.1038/s41746- 2024

  56. [57]

    Brock Wilcox. 2024. token-tree: Visualize the many possible paths of LLM Token Sampling. https://github.com/awwaiid/token-tree. Accessed: 2025-07-22

    2024

  57. [58]

    Matthew D Zeiler and Rob Fergus. 2013. Visualizing and Understanding Convo- lutional Networks. arXiv:1311.2901 [cs.CV] https://arxiv.org/abs/1311.2901

    Pith/arXiv arXiv 2013

  58. [59]

    Martin Wattenberg and Fernanda B. Viégas. 2008. The Word Tree, an Interactive Visual Concordance.IEEE Transactions on Visualization and Computer Graphics Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al. 14, 6 (2008), 1221–1228. https://doi.org/10.1109/TVCG.2008.172

    doi:10.1109/tvcg.2008.172 2008

  59. [60]

    Ziyu Zhuang, Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Zixian Feng, Weinan Zhang, and Ting Liu. 2023. Through the Lens of Core Competency: Survey on Evaluation of Large Language Models. arXiv:2308.07902 [cs.CL] https://arxiv.org/abs/2308.07902 InFerActive: Towards Scalable Human Evaluation of Large Language Models through Int...

    Pith/arXiv arXiv 2023

  60. [62]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL] https://arxiv.org/abs/2306.05685

    Pith/arXiv arXiv 2023

  61. [2017]

    InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Martha Palmer, Rebecca Hwa, and Sebastian Riedel (Eds.)

    Why We Need New Evaluation Metrics for NLG. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Martha Palmer, Rebecca Hwa, and Sebastian Riedel (Eds.). Association for Computational Linguistics, Copenhagen, Denmark, 2241–2252. https://doi.org/10.18653/v1/D17- 1238

    work page doi:10.18653/v1/d17- 2017

  62. [2025]

    InThe Thirteenth International Conference on Learning Representations

    Improving Uncertainty Estimation through Semantically Diverse Language Generation. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=HSi4VetQLj