Pith. sign in

REVIEW 4 major objections 5 minor 73 references

RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that pre-materialized retrieval indexes and process checkpoints cut RAG parameter-change latency from minutes or hours to under a second, and that expert developers debug retrieval before generation.

desk verdict A genuinely useful RAG debugging tool paper with a solid qualitative finding, but the headline sub-second latency claim rests on a precomputed grid whose coverage is not demonstrated. read the letter →

arxiv 2504.13587 v1 pith:U6635Z6C submitted 2025-04-18 cs.HC cs.AI

Quentin Romero Lauro , Shreya Shankar , Sepanta Zeighami , Aditya Parameswaran This is my paper · ORCID
classification cs.HCcs.AI
keywords retrieval-augmentedgenerationRAGdebuggingdevelopertoolsinteractivewhat-ifanalysispre-materializedindexesprocesscheckpointingretriever-firstworkflowsLLMpipelines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the slow part of RAG development is not reasoning about failures but waiting on the pipeline: changing retrieval parameters such as chunk size normally forces documents to be re-indexed, which can take minutes or hours. It claims that a tool called raggy removes this delay by pre-materializing retrieval indexes over a grid of chunk sizes and overlaps, and by checkpointing the Python process at each pipeline primitive so a developer can edit a step and resume from there in under a second. To show why this matters, the paper reports a qualitative study of 12 engineers who built a hospital question-answering system; it found that they consistently inspected retrieved chunks before generation components, and that on average 71.3% of their parameter changes would have required slow re-indexing in a conventional workflow. The paper also derives design implications: RAG tools should foreground retrieval inspection, support coordinated changes across components, and help developers build a small golden-answer test set during debugging.

What carries the argument

Two mechanisms carry the argument. The first is the pre-materialized index grid: a one-time initialization builds vector indexes across chunk sizes from 100 to 2000 characters and overlaps from 0 to 400 characters, using cosine similarity, TF-IDF, max marginal relevance, and RAPTOR, all stored in a Chroma database; any interaction inside the grid selects an existing index, so what would normally be a document re-indexing job becomes a lookup. The second is process-state preservation via forking: before each Retriever or LLM primitive returns, raggy forks the Python process, pauses the child as a checkpoint, and later lets an edited checkpoint resume as the new main process while terminating stale branches to prevent memory leaks. Together these turn edit-and-see cycles that previously took half a day into dropdown-level experiments with immediate feedback.

What would settle it

Run the same debugging task on a corpus whose documents are not well served by chunk sizes under 2000 characters, and record how many useful parameter changes fall outside the pre-computed grid; if those changes require re-indexing delays of minutes or hours, the central latency claim fails for exactly the configurations that matter. A direct timing check would also settle it: changing a retriever parameter inside the grid must complete in under a second on a large corpus, rather than only on the study's 220-PDF collection.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that interactive RAG debugging is achievable without leaving the Python-based workflow developers already use: by pre-computing hundreds of retrieval indexes (chunk sizes 100–2000 characters, overlaps 0–400, four retrieval methods) and by forking the executing Python process at every primitive, raggy lets developers change chunk size, overlap, retrieval method, k, prompts, and even manually override retrieved chunks, and observe the effect locally or through the rest of the pipeline in under a second. The user study then claims that this latency removal changes debugging behavior: all 12 participants examined retrieved chunks before touching LLM outputs, treating retrieval quality as the gating question; when chunks were inadequate, the dominant fix was increasing chunk size, often followed by prompt adjustments to handle larger context. The paper further claims that saving answers as golden references lets developers accumulate a small evaluation set while debugging, while acknowledging that participants wanted more systematic evaluation across multiple configurations at once.

Load-bearing premise

The central claim depends on developers wanting retrieval configurations inside the pre-indexed grid of chunk sizes 100–2000 characters and overlaps 0–400; any configuration outside that grid still requires the slow re-indexing that raggy promises to eliminate.

Editorial extensions

If this is right

  • Retrieval parameter tuning—chunk size, chunk overlap, retrieval method, and k—becomes an interactive operation for configurations inside the pre-indexed grid, so developers can explore retrieval choices they currently avoid because of cost.
  • Because checkpoints let a user edit a prompt, a retrieved-chunk set, or an LLM output and resume downstream, developers can isolate whether an error originated in retrieval, generation, or the prompt without rerunning the whole pipeline.
  • The observed retriever-first workflow implies that RAG tools should surface retrieval internals—chunk scores, selected versus unselected chunks, document provenance—as a first-class debugging surface rather than burying them under prompt engineering.
  • Saving correct answers during debugging gives teams a golden set that grows with use, enabling a lightweight evaluation loop even when no labeled data exists upfront.
  • For in-grid changes, the 71.3% of parameter changes that would have required re-indexing become near-instant, lowering the cost of trial and error in pipeline development.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested implication is that the fixed grid is the real bottleneck: if a corpus's optimal chunking falls outside 100–2000 characters or 0–400 overlap, raggy's speed advantage disappears for exactly the configuration that matters; a natural extension is adaptive, on-demand index expansion triggered by out-of-grid requests.
  • The retriever-first behavior may be partly an artifact of the tool's design: because retrieval gets the most detailed interactive cell, raggy may pull attention toward retrieval even from developers who would otherwise start with prompts. A comparison with a prompt-first interface would separate the workflow claim from the interface effect.
  • The checkpoint-by-forking mechanism generalizes beyond RAG: any multi-step pipeline with expensive, stateful steps—data queries, tool calls, agent messages—could expose the same resume-from-any-step debugging pattern, with process memory as the main scaling constraint.
  • The paper's one-time indexing cost of over an hour suggests a testable design: lazy, on-demand index construction keyed to the developer's actual next parameter choice, rather than a fixed grid.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents raggy, a Python library and interactive web-based debugging interface for retrieval-augmented generation (RAG) pipelines. raggy precomputes multiple document chunking configurations and corresponding retrieval indexes, and uses Python process forking to checkpoint pipeline state at each primitive component, enabling developers to modify parameters and re-run individual steps without reprocessing the corpus. The authors report a formative interview study with 6 practitioners, from which they derive three design goals, and a think-aloud user study with 12 engineers who used raggy to build a hospital question-answering system over 220 PDFs. The main findings are that participants valued rapid iteration, consistently inspected and debugged retrieval before LLM components, engaged in iterative foraging and sensemaking, and needed to coordinate changes across components. The paper contributes the raggy design and implementation, qualitative insights into expert debugging patterns, and design implications for future RAG tools.

Significance. If the latency claims are substantiated, raggy addresses a clearly documented pain point in RAG development, and the qualitative findings (retriever-first debugging, coordinated component changes) are a useful empirical addition to the emerging literature on LLM application development. The study procedures, interview protocols, and analysis approach are described in sufficient detail to defend the qualitative claims, and the paper includes honest limitations. However, the paper's headline quantitative claim—that parameter changes are reduced from minutes or hours to under a second—is not backed by latency measurements and rests on a finite precomputed grid whose coverage is not established. The 71.3% re-indexing statistic also lacks a precise measurement definition. These gaps are load-bearing because the low-latency capability motivates the entire tool design.

major comments (4)
  1. [§4.2.1] The paper's central latency claim ('reducing what would typically be minutes (or hours) of processing to under a second') is asserted but never measured. No experiment, timing logs, or benchmark of the precomputed-index lookup is reported, nor is the number of chunking configurations or the exact step sizes for chunk size and overlap given ('increasing intervals' and 'a mix' are not reproducible specifications). Please add a latency benchmark for representative parameter changes (in-grid and, if applicable, out-of-grid) and specify the grid construction precisely.
  2. [§4.2.1] The sub-second benefit applies only to configurations inside the precomputed grid (chunk sizes 100–2000 characters, overlaps 0–400 characters). The paper asserts this grid 'covers a large portion of the parameter space developers typically explore' but provides no distribution of participant-chosen parameter values relative to the grid, no corpus-general evidence for this range, and no fallback for out-of-grid choices such as chunk_size=3000 or overlap=500. Consequently, a developer whose optimal configuration lies outside the grid experiences the same slow re-indexing that raggy claims to eliminate. Please report whether and which participant changes fell inside the grid, and discuss or implement an on-demand incremental indexer.
  3. [§6.1.1 and Figure 7] The 71.3% statistic is undefined as stated. It is unclear which parameter types are counted as 'require re-indexing' (e.g., does changing retrieval method from cosine to TF-IDF count? does changing k count?), whether the denominator is the total number of parameter changes pooled across participants or a per-participant average, and whether the percentage reflects only changes within raggy's precomputed grid. Please provide a precise definition and a per-participant table.
  4. [§6.2 and §5] The study lacks any comparison condition: all 12 participants used raggy, so the observed 'retriever-first' pattern may be an artifact of the interface's presentation (which surfaces retrieved chunks immediately) rather than a general debugging strategy. The paper frames this as a consistent developer pattern and derives design implications from it. Please either temper the generalizability claims or add a discussion of how the interface could bias this behavior, ideally with a comparison to a conventional code-only debugging workflow.
minor comments (5)
  1. [§4.1 (LLM Generator)] The text refers to 'Figure 4.2' and 'Figure 5.3' where Figure 4 and Figure 5 are meant; these cross-references are broken and should be corrected.
  2. [§6.2.1] Participant range notation is inconsistent (e.g., 'P1.P6' appears instead of 'P1–P6'); please standardize the delimiter used throughout the paper.
  3. [§6.2.2 and §6.4] There are minor grammatical errors, such as 'how they interpreted the same results different' (should be 'differently') and 'Table 2 illustrates' where the subject is plural ('tactics illustrate'). A light copyedit would improve readability.
  4. [Figure 7 caption] The caption does not define the unit of 'parameter changes' or state whether the values are counts or percentages; please clarify in the caption.
  5. [Reference [15]] Reference [15] is a Kaggle discussion; for a scientific claim about chunk-size grid selection, a more established citation or a description of the authors' own heuristic would be preferable.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the latency and user-study claims are empirically grounded and not derived from fitted inputs.

full rationale

This is an HCI systems paper with no mathematical derivation chain; its central claims are engineering latency behavior and qualitative user-study findings. The latency claim in Section 4.2.1 is explicitly conditional: raggy precomputes indexes for chunk sizes 100-2000 characters and overlaps 0-400 characters, and the sub-second behavior applies when a requested configuration matches one of those precomputed indexes. The finite grid is an acknowledged design boundary, not a hidden fit; the paper does not derive 'under a second' from data it later reports as a prediction. The 71.3% re-indexing statistic in Section 6.1.1 is an observational summary of participant actions, not a fitted parameter renamed as an outcome, and it is not used to validate grid coverage. The retriever-first finding is an empirical pattern from 12 think-aloud sessions; while raggy's interface may influence behavior, the paper reports it as a tool-mediated observation and does not define 'retriever-first' in terms of the tool's own layout. The only self-citations are non-load-bearing: reference [54] supports data drift and lifecycle claims in Sections 6.3.1 and 6.1.3, and references [62, 66] support experiment-tracking context; none are invoked as uniqueness theorems or to forbid alternatives. The limitations the skeptic identifies (out-of-grid configurations requiring full re-indexing, no corpus-general coverage evidence, no latency benchmark for grid misses) are genuine correctness and coverage limitations and are partly acknowledged in the paper, but they are not circularity: the claims are not equivalent to their inputs by construction. Therefore no circular step is exhibited, and the circularity score is minimal.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric free parameters or invented physical entities are present. The central claims rest on the design and sampling assumptions listed as axioms.

assumptions (4)
  • domain assumption Adjusting chunk size, overlap, retrieval method, and k are the parameters with the greatest impact on RAG output quality.
    Motivates the raggy design; asserted in the introduction and in the formative interviews (Section 3), not from a quantitative sensitivity analysis.
  • domain assumption The pre-computed index grid (chunk sizes 100-2000 chars, overlaps 0-400 chars) covers the parameter configurations developers actually need during RAG debugging.
    Section 4.2.1 states the grid covers a large portion of the parameter space developers typically explore but provides no evidence that the ranges are sufficient across corpora.
  • domain assumption Process forking at primitive boundaries preserves enough Python program state to rerun pipelines reliably from checkpoints.
    Section 4.2.2 describes the checkpoint design but reports no tests or error cases; relies on CPython fork semantics being safe in the supported environments.
  • domain assumption Findings from 12 engineers on a single hospital QA task generalize to RAG developers in other domains.
    The paper acknowledges in Section 7 that one-hour sessions do not reflect full production complexity and that participants varied in IR/ML expertise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines." pith.science (2026). https://pith.science/paper/U6635Z6C

@misc{pith2026250413587,
  author       = {Pith},
  title        = {Pith review of: RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U6635Z6C}},
  note         = {Machine review of arXiv:2504.13587}
}
read the original abstract

Retrieval-augmented generation (RAG) pipelines have become the de-facto approach for building AI assistants with access to external, domain-specific knowledge. Given a user query, RAG pipelines typically first retrieve (R) relevant information from external sources, before invoking a Large Language Model (LLM), augmented (A) with this information, to generate (G) responses. Modern RAG pipelines frequently chain multiple retrieval and generation components, in any order. However, developing effective RAG pipelines is challenging because retrieval and generation components are intertwined, making it hard to identify which component(s) cause errors in the eventual output. The parameters with the greatest impact on output quality often require hours of pre-processing after each change, creating prohibitively slow feedback cycles. To address these challenges, we present RAGGY, a developer tool that combines a Python library of composable RAG primitives with an interactive interface for real-time debugging. We contribute the design and implementation of RAGGY, insights into expert debugging patterns through a qualitative study with 12 engineers, and design implications for future RAG tools that better align with developers' natural workflows.

Figures

Figures reproduced from arXiv: 2504.13587 by the authors.

Figure 1
Figure 1. An example of how a user might use raggy, our tool for debugging RAG pipelines. The developer debugs by first increasing chunk size to retrieve more information, then by adding a new LLM step to rewrite the query to address ambiguity. raggy enables real-time modifications with instant feedback, and facilitates visualization of how changes impact the pipeline. Abstract Retrieval-augmented generation (RAG) pipelines h… view at source ↗
Figure 2
Figure 2. Python implementation of a RAG pipeline (left) and the corresponding debugging interface generated by [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. In raggy’s Retriever cell, users can (A) adjust hyper￾parameters (e.g., retrieval method, chunk size), (B) manually select or deselect chunks, and (C) visualize the distribution of selected vs. unselected chunks. low-latency “what-if” analysis by pre-materializing retrieval in￾dexes and checkpoints that users may want to restart pipelines from (Section 4.2). 4.1 Debugging Interface As shown in [PITH_FULL_IMAGE:figu… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: In raggy’s LLM cell, users can (1) experiment with prompting strategies and (2) directly modify the LLM’s out￾put to test downstream effects. This example shows query decomposition, where a prompt is broken into sub-questions, each triggering its own retrieval. to use …
Figure 6
Figure 6. Figure 6: The iterative debugging workflow supported by [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Darker bars indicate changes that would have re [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 36 canonical work pages

  1. [1]

    LangChain AI. 2024. LangGraph: Overview. https://langchain-ai.github.io/ langgraph/#overview Accessed: 2024-09-02

    work page 2024

  2. [2]

    Rama Akkiraju, Anbang Xu, Deepak Bora, Tan Yu, Lu An, Vishal Seth, Aaditya Shukla, Pritam Gundecha, Hridhay Mehta, Ashwin Jha, et al . 2024. FACTS About Building Retrieval Augmented Generation-based Chatbots. arXiv preprint arXiv:2407.07858 (2024)

    arXiv 2024

  3. [3]

    Shm Garanganao Almeda, JD Zamfirescu-Pereira, Kyu Won Kim, Pradeep Mani Rathnam, and Bjoern Hartmann. 2024. Prompting for discovery: Flex- ible sense-making for ai art-making with dreamsheets. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems . 1–17

    2024

  4. [4]

    Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 291–300

    2019

  5. [5]

    Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI magazine 35, 4 (2014), 105–120

    2014

  6. [6]

    Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023. ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing. arXiv:2309.09128 [cs.HC]

    work page Pith review arXiv 2023

  7. [7]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations

    2023

  8. [8]

    Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Ni- hal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al

    work page

Show all 73 references
  1. [9]

    Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024. Seven failure points when engineering a retrieval augmented generation system. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering ...

    2024

  2. [10]

    Alex Bäuerle, Ángel Alexander Cabrera, Fred Hohman, Megan Maher, David Koski, Xavier Suau, Titus Barik, and Dominik Moritz. 2022. Symphony: Compos- ing interactive interfaces for machine learning. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . 1–14

    2022

  3. [11]

    Andrei-Laurentiu Bornea, Fadhel Ayed, Antonio De Domenico, Nicola Pi- ovesan, and Ali Maatouk. 2024. Telco-RAG: Navigating the challenges of retrieval-augmented language models for telecommunications. arXiv preprint arXiv:2404.15939 (2024)

    2024 arXiv

  4. [12]

    Andrew Bragdon, Robert Zeleznik, Steven P Reiss, Suman Karumuri, William Cheung, Joshua Kaplan, Christopher Coleman, Ferdi Adeputra, and Joseph J LaViola Jr. 2010. Code bubbles: a working set-based interface for code under- standing and maintenance. In Proceedings of the SIGCH...

    2010

  5. [13]

    Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. 335–336

    1998

  6. [14]

    Jaegul Choo and Shixia Liu. 2018. Visual analytics for explainable deep learning. IEEE computer graphics and applications 38, 4 (2018), 84–92

    2018

  7. [15]

    Kaggle Community. 2024. Guide to Chunk Size and Overlap. https://www. kaggle.com/discussions/general/503436 Accessed: 2025-04-04

    2024

  8. [16]

    Victor Dibia, Jingya Chen, Gagan Bansal, Suff Syed, Adam Fourney, Erkang Zhu, Chi Wang, and Saleema Amershi. 2024. Autogen studio: A no-code devel- oper tool for building and debugging multi-agent systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural L...

    2024

  9. [17]

    Will Epperson, Gagan Bansal, Victor Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, and Saleema Amershi. 2025. Interactive Debugging and Steering of Multi- Agent AI Systems. arXiv preprint arXiv:2503.02068 (2025)

    2025 arXiv

  10. [18]

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ra- gas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217 (2023)

    2023 arXiv

  11. [19]

    Katy Ilonka Gero, Tao Long, and Lydia B Chilton. 2023. Social dynamics of AI support in creative writing. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–15

    2023

  12. [20]

    Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K Kummerfeld, and Elena L Glassman. 2024. Supporting Sensemaking of Large Language Model Outputs at Scale. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–21

    2024

  13. [21]

    Philip J Guo, Sean Kandel, Joseph M Hellerstein, and Jeffrey Heer. 2011. Proac- tive wrangling: Mixed-initiative end-user programming of data transformation scripts. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 65–74

    2011

  14. [22]

    Marti A Hearst. 2009. Search user interfaces. Cambridge university press

    2009

  15. [23]

    Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE transactions on visualization and computer graphics 25, 8 (2018), 2674–2693

    2018

  16. [24]

    Fred Hohman, Chaoqun Wang, Jinmook Lee, Jochen Görtler, Dominik Moritz, Jeffrey P Bigham, Zhile Ren, Cecile Foret, Qi Shan, and Xiaoyi Zhang. 2024. Talaria: Interactively optimizing machine learning models for efficient inference. In Proceedings of the CHI Conference on Human ...

    2024

  17. [25]

    Jennifer Hsia, Afreen Shaikh, Zhiruo Wang, and Graham Neubig. 2024. RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems. arXiv preprint arXiv:2403.09040 (2024)

    2024 arXiv

  18. [26]

    Mina Huh, Yi-Hao Peng, and Amy Pavel. 2023. GenAssist: Making image gen- eration accessible. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–17

    2023

  19. [27]

    Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. 2024. Long-context llms meet rag: Overcoming challenges for long inputs in rag. In The Thirteenth International Conference on Learning Representations

    2024

  20. [28]

    K2view. 2024. GenAI Adoption 2024: The Challenge with Enterprise Data. https: //www.k2view.com/genai-adoption-survey/ Accessed: 2025-03-11

    2024

  21. [29]

    Minsuk Kahng, Pierre Y Andrews, Aditya Kalro, and Duen Horng Chau. 2017. Activis: Visual exploration of industry-scale deep neural network models. IEEE transactions on visualization and computer graphics 24, 1 (2017), 88–97

    2017

  22. [30]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering.. In EMNLP (1). 6769–6781

    2020

  23. [31]

    Amy J Ko, Thomas D LaToza, and Margaret M Burnett. 2015. A practical guide to controlled experiments of software engineering tools with human participants. Empirical Software Engineering 20 (2015), 110–141

    2015

  24. [32]

    Amy J Ko and Brad A Myers. 2004. Designing the whyline: a debugging inter- face for asking questions about program behavior. In Proceedings of the SIGCHI conference on Human factors in computing systems . 151–158

    2004

  25. [33]

    Amy J Ko and Brad A Myers. 2008. Debugging reinvented: asking and answering why and why not questions about program behavior. In Proceedings of the 30th international conference on Software engineering . 301–310

    2008

  26. [34]

    Amy J Ko, Brad A Myers, Michael J Coblenz, and Htet Htet Aung. 2006. An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Transactions on software engineering 32, 12 (2006), 971–987

    2006

  27. [35]

    Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods in human-computer interaction . Morgan Kaufmann

    2017

  28. [36]

    Doris Jung-Lin Lee. 2020. Towards an Integrated Solution for Intelligent Visual Data Discovery. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems . 1–7

    2020

  29. [37]

    Hyunji Lee, Sohee Yang, Hanseok Oh, and Minjoon Seo. 2022. Generative Multi- hop Retrieval. In 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

    2022

  30. [38]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing...

    2020

  31. [39]

    Tom Lieber. 2013. Theseus: understanding asynchronous code. InCHI’13 Extended Abstracts on Human Factors in Computing Systems . 2731–2736

    2013

  32. [40]

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 5303–5315

    2023

  33. [41]

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 1 (2020)

    2020 arXiv

  34. [42]

    Christopher D Manning. 2009. An introduction to information retrieval

    2009

  35. [43]

    Kelong Mao, Zheng Liu, Hongjin Qian, Fengran Mo, Chenlong Deng, and Zhicheng Dou. 2024. RAG-Studio: Towards In-Domain Adaptation of Retrieval Augmented Generation Through Self-Alignment. In Findings of the Association for Computational Linguistics: EMNLP 2024 , Yaser Al-Onaiza...

    2024 doi

  36. [44]

    Murphy, Robert J

    Gail C. Murphy, Robert J. Walker, and ELA Banlassad. 2002. Evaluating emerging software development technologies: Lessons learned from assessing aspect- oriented programming. IEEE Transactions on software engineering 25, 4 (2002), 438–455

    2002

  37. [45]

    Stephen Oney, Brad Myers, and Joel Brandt. 2014. Interstate: a language and environment for expressing interface behavior. In Proceedings of the 27th annual RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines ACM symposium on User interface ...

    2014

  38. [46]

    Jorge Piazentin Ono, Sonia Castelo, Roque Lopez, Enrico Bertini, Juliana Freire, and Claudio Silva. 2020. Pipelineprofiler: A visual analytics tool for the explo- ration of automl pipelines. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 390–400

    2020

  39. [47]

    Chris Parnin, Gustavo Soares, Rahul Pandita, Sumit Gulwani, Jessica Rich, and Austin Z Henley. 2023. Building Your Own Product Copilot: Challenges, Oppor- tunities, and Needs. arXiv preprint arXiv:2312.14231 (2023)

    2023 arXiv

  40. [48]

    Kayur Patel, Naomi Bancroft, Steven M Drucker, James Fogarty, Amy J Ko, and James Landay. 2010. Gestalt: integrated support for implementation and analysis in machine learning. In Proceedings of the 23nd annual ACM symposium on User interface software and technology . 37–46

    2010

  41. [49]

    Michael Quinn Patton. 2002. Qualitative research & evaluation methods . sage

    2002

  42. [50]

    Peter Pirolli and Stuart Card. 1999. Information foraging. Psychological review 106, 4 (1999), 643

    1999

  43. [51]

    Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning , Vol. 242. Citeseer, 29–48

    2003

  44. [52]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN...

    2019

  45. [53]

    Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. 2024. Raptor: Recursive abstractive processing for tree-organized retrieval. arXiv preprint arXiv:2401.18059 (2024)

    2024 arXiv

  46. [54]

    We Have No Idea How Models will Behave in Pro- duction until Production

    Shreya Shankar, Rolando Garcia, Joseph M Hellerstein, and Aditya G Parameswaran. 2024. " We Have No Idea How Models will Behave in Pro- duction until Production": How Engineers Operationalize Machine Learning. Proceedings of the ACM on Human-Computer Interaction 8, CSCW1 (2024), 1–34

    2024

  47. [55]

    Shangeetha Sivasothy, Scott Barnett, Stefanus Kurniawan, Zafaryab Rasool, and Rajesh Vasa. 2024. RAGProbe: An Automated Approach for Evaluating RAG Applications. arXiv preprint arXiv:2409.19019 (2024)

    2024 arXiv

  48. [56]

    Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M Rush. 2022. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE trans- actions on visualization and computer graph...

    2022

  49. [57]

    Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: En- abling multilevel exploration and sensemaking with large language models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–18

    2023

  50. [58]

    Ben Swanson, Kory Mathewson, Ben Pietrzak, Sherol Chen, and Monica Di- nalescu. 2021. Story centaur: Large language model few shot learning as a cre- ative writing tool. InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisti...

    2021

  51. [59]

    Steven J Taylor, Robert Bogdan, and Marjorie L DeVault. 2015. Introduction to qualitative research methods: A guidebook and resource . John Wiley & Sons

    2015

  52. [60]

    Chroma Core Team. 2025. Chroma: AI-native embedding database. https: //github.com/chroma-core/chroma Accessed: 2025-03-13

    2025

  53. [61]

    Manasi Vartak, Harihar Subramanyam, Wei-En Lee, Srinidhi Viswanathan, Saadiyah Husnoo, Samuel Madden, and Matei Zaharia. 2016. ModelDB: a system for machine learning model management. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics. 1–3

    2016

  54. [62]

    Dakuo Wang, Justin D Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019. Human- AI collaboration in data science: Exploring data scientists’ perceptions of au- tomated AI. Proceedings of the ACM on human-co...

    2019

  55. [63]

    James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fer- nanda Viégas, and Jimbo Wilson. 2019. The what-if tool: Interactive probing of machine learning models. IEEE transactions on visualization and computer graphics 26, 1 (2019), 56–65

    2019

  56. [64]

    Kevin Wu, Eric Wu, and James Zou. 2024. How faithful are RAG models? Quantifying the tug-of-war between RAG and LLMs’ internal prior. arXiv preprint arXiv:2404.10198 (2024)

    2024 arXiv

  57. [65]

    Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. Promptchainer: Chaining large language model prompts through visual programming. InCHI Conference on Human Factors in Computing Systems Extended Abstracts . 1–10

    2022

  58. [66]

    Doris Xin, Eva Yiwei Wu, Doris Jung-Lin Lee, Niloufar Salehi, and Aditya Parameswaran. 2021. Whither automl? understanding the role of automation in machine learning workflows. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1–16

    2021

  59. [67]

    Rui Yang, Michael Fu, Chakkrit Tantithamthavorn, Chetan Arora, Lisa Van- denhurk, and Joey Chua. 2025. RAGVA: Engineering Retrieval Augmented Generation-based Virtual Assistants in Practice. arXiv preprint arXiv:2502.14930 (2025)

    2025 arXiv

  60. [68]

    Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. 2018. Accelerating the machine learning lifecycle with MLflow. IEEE Data Eng. Bull. 41, 4 (2018), 39–45

    2018

  61. [69]

    JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang

  62. [70]

    Yijia Zhang, Qingyu Chen, Zhihao Yang, Hongfei Lin, and Zhiyong Lu. 2019. BioWordVec, improving biomedical word embeddings with subword information and MeSH. Scientific data 6, 1 (2019), 52

    2019

  63. [71]

    The purpose of this interview is to explore ways to improve interfaces for building RAG pipelines. Your responses in this interview will help us design better tools

    Yuqi Zhou, Sunhao Dai, Zhanshuo Cao, Xiao Zhang, and Jun Xu. 2024. Length- Induced Embedding Collapse in Transformer-based Models. arXiv preprint arXiv:2410.24200 (2024). Romero Lauro and Shankar et al. A Formative Interview Study Protocol This appendix contains the semi-struc...

    2024 arXiv

  64. [2022]

    arXiv preprint arXiv:2202.01279 (2022)

    Promptsource: An integrated development environment and repository for natural language prompts. arXiv preprint arXiv:2202.01279 (2022)

    2022 arXiv

  65. [2023]

    In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

    Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.