pith. sign in

arxiv: 2605.14612 · v1 · pith:FED2GNOFnew · submitted 2026-05-14 · 💻 cs.SE · cs.AI

In-IDE Toolkit for Developers of AI-Based Features

Yaroslav Sokolov , Yury Khudyakov , Lenar Sharipov , Andrei Gasparian , Parth Tiwary , Artem Trofimov This is my paper

Pith reviewed 2026-06-30 20:32 UTC · model grok-4.3

classification 💻 cs.SE cs.AI
keywords AI toolkitIDE integrationLLM observabilityAI evaluationtrace capturedebugging workflowsoftware engineeringPyCharm plugin
0
0 comments X

The pith

An IDE plugin integrates AI tracing and evaluation into the standard run and debug workflow.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces an AI Toolkit plugin for JetBrains IDEs that places tracing and evaluation inside the familiar Run/Debug cycle. Practitioners in a mixed-methods study consistently reported needs for repeatable evaluations, traces visible at execution time, and minimal setup or switching. The plugin responds with automatic trace capture on run, hierarchical views of those traces, one-click dataset addition, and evaluations that behave like unit tests with interchangeable metrics. Early telemetry from the PyCharm release shows developers adopt the features when prompted at run time and continue using them with low churn.

Core claim

The AI Toolkit enables IDE-native workflows for AI features through run-triggered trace capture, immediate hierarchical inspection, one-click addition of traces to datasets, and unit-test-like evaluations using pluggable metrics. This integration responds to the three needs identified in the mixed-methods study and shows initial adoption signals in the first release.

What carries the argument

The AI Agents Debugger and AI Evaluation components that trigger trace capture on run and support immediate inspection plus metric-based evaluations inside the IDE.

If this is right

  • Trace capture occurs automatically during normal runs without extra configuration.
  • Traces become available for immediate hierarchical review right after execution.
  • Any captured trace can be added to a dataset in one click for reuse in evaluations.
  • Evaluations execute in the same manner as unit tests and accept interchangeable metrics.
  • Lower setup cost leads to higher rates of trace capture and continued usage.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same pattern of embedding observability could apply in other IDEs and languages beyond the initial JetBrains release.
  • Teams without dedicated ML staff might maintain consistent evaluation habits if the tools stay inside daily coding flows.
  • Expanding metric and framework support would further reduce custom scripting for different AI agent setups.
  • Early integration of these practices could shift how product requirements for AI features are validated during development.

Load-bearing premise

That the three needs from the practitioner study are the primary barriers and that addressing them via IDE integration will produce sustained adoption of disciplined AI practices.

What would settle it

Telemetry showing that users who capture traces still abandon the workflow or continue relying on separate tools after initial use.

Figures

Figures reproduced from arXiv: 2605.14612 by Andrei Gasparian, Artem Trofimov, Lenar Sharipov, Parth Tiwary, Yaroslav Sokolov, Yury Khudyakov.

Figure 1
Figure 1. Figure 1: AI Toolkit observe–evaluate flow. scores, and – when using LLM-as-a-judge [11] – the optional ex￾planation. Aggregated statistics such as average score, token usage, and wall-clock time are also summarized. We persist the complete trace per data point so that when an output is undesired, the developer can look at the internal calls and prompts to understand what happened ( [PITH_FULL_IMAGE:figures/full_fi… view at source ↗
read the original abstract

AI-enabled features built on LLMs and agentic workflows are difficult to test, debug, and reproduce, especially for product-focused software engineers without a machine learning background. We present the AI Toolkit plugin for JetBrains IDEs, which brings tracing and evaluation directly into the Run/Debug loop. A mixed methods study with practitioners presents three consistent needs: (1) make evaluation regular and repeatable, (2) expose traces at the moment of execution, and (3) minimize setup and context switching. Guided by these needs, the AI Toolkit introduces an IDE-native workflow: run-triggered trace capture; immediate, hierarchical inspection; one-click "Add to Dataset" from traces; and unit-test-like evaluations with pluggable metrics. The first release in PyCharm shows promising early signals - strong conversion when promoted at Run, sustained usage among those who capture traces, and low churn - suggesting that IDE-native observability lowers activation energy and helps developers adopt disciplined practices. We detail the design and implementation of the AI Agents Debugger and AI Evaluation, report initial adoption telemetry, and outline next steps to broaden framework coverage and scale evaluations. Together, these results indicate that integrating AI observability and evaluation into everyday IDE workflows can make modern AI development accessible to non-ML specialists while preserving software-engineering practices.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript presents the AI Toolkit plugin for JetBrains IDEs that integrates tracing, debugging, and evaluation of AI-based features directly into the Run/Debug workflow. It reports results from a mixed-methods practitioner study that identified three needs—making evaluation regular and repeatable, exposing traces at execution, and minimizing setup—and describes how the plugin addresses them via run-triggered trace capture, hierarchical inspection, one-click dataset addition, and pluggable unit-test-like evaluations. Initial telemetry from the PyCharm release is presented as showing strong conversion at promotion, sustained usage conditional on trace capture, and low churn, supporting the conclusion that IDE-native observability can make disciplined AI practices accessible to non-ML software engineers.

Significance. If the central claims hold, the work would be significant for cs.SE by demonstrating a practical mechanism to embed AI observability and evaluation into standard IDE practices without disrupting existing workflows. The practitioner-grounded design and emphasis on preserving software-engineering conventions (e.g., unit-test-like evaluations) are strengths. The mixed-methods study and telemetry reporting provide a starting point for tool evaluation in this space.

major comments (2)
  1. [Abstract] Abstract: The telemetry is summarized only as 'strong conversion when promoted at Run, sustained usage among those who capture traces, and low churn' with no quantitative values, participant demographics (including ML experience), pre/post measures, control conditions, or evidence that captured traces or added datasets were used in downstream evaluation; this leaves the inference that the toolkit produces adoption of disciplined practices by non-ML specialists unsupported.
  2. [Mixed methods study] Mixed methods study description: The three needs are presented as 'consistent' findings that directly motivate the design, yet no details on study protocol, sample, recruitment, or how primacy of these barriers was established are supplied; without this the claim that IDE integration of these features will lead to sustained disciplined behavior rests on an unverified assumption.
minor comments (1)
  1. [Abstract] Abstract: The term 'promising early signals' is imprecise; replacing it with a direct statement of the observed telemetry metrics would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and note where we will revise the manuscript to improve clarity and support for our claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The telemetry is summarized only as 'strong conversion when promoted at Run, sustained usage among those who capture traces, and low churn' with no quantitative values, participant demographics (including ML experience), pre/post measures, control conditions, or evidence that captured traces or added datasets were used in downstream evaluation; this leaves the inference that the toolkit produces adoption of disciplined practices by non-ML specialists unsupported.

    Authors: We agree the abstract provides only a high-level summary. The full manuscript reports specific telemetry metrics (e.g., conversion percentages and usage durations) from the PyCharm release. We will revise the abstract to include key quantitative values. Participant demographics and controlled pre/post measures are not available from the real-world telemetry deployment due to privacy constraints and the observational nature of the data; we will add explicit language noting these limitations and that downstream evaluation use is inferred from sustained trace-capture patterns rather than directly measured. revision: yes

  2. Referee: [Mixed methods study] Mixed methods study description: The three needs are presented as 'consistent' findings that directly motivate the design, yet no details on study protocol, sample, recruitment, or how primacy of these barriers was established are supplied; without this the claim that IDE integration of these features will lead to sustained disciplined behavior rests on an unverified assumption.

    Authors: We acknowledge that the current manuscript does not supply sufficient detail on the mixed-methods protocol, sample, recruitment, or how the three needs were prioritized. We will expand the study description section in the revision to include these elements (protocol overview, participant count and recruitment method, and analysis approach establishing primacy of the barriers). This will strengthen the grounding for the design decisions and the link to adoption claims. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical claims rest on independent study and telemetry without reduction to inputs

full rationale

The paper presents a practitioner mixed-methods study identifying three needs, describes an IDE plugin built to address them, and reports usage telemetry as early signals. No equations, fitted parameters, predictions, or self-citation chains appear. The central claim that IDE integration lowers activation energy for disciplined AI practices is supported by the study data and telemetry signals rather than being equivalent to its inputs by construction. This is the normal case of a self-contained tool paper whose derivation does not reduce to self-definition or fitted renaming.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no information on free parameters, axioms, or invented entities; the ledger is therefore empty.

pith-pipeline@v0.9.1-grok · 5774 in / 1202 out tokens · 32903 ms · 2026-06-30T20:32:08.728715+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 4 canonical work pages · 2 internal anchors

  1. [1]

    Sangeetha Annam, Merry Saxena, Ujjwal Kaushik, and Shikha Mittal. 2025. LangChain: Simplifying Development with Language Models.Textual Intelligence: Large Language Models and Their Real-World Applications(2025), 287–304

    2025

  2. [2]

    Arize AI, Inc. 2025. Arize AI. https://arize.com. AI & Agent Engineering Platform: development, observability, and evaluation. Accessed: 2025-10-27

    2025

  3. [3]

    Lukas Biewald. 2020. Experiment Tracking with Weights and Biases. https: //www.wandb.com/. Software available from https://wandb.com. Accessed: 2025-10-27

    2020

  4. [4]

    Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, et al

  5. [5]

    InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency

    Visibility into AI agents. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. 958–973

    2024

  6. [6]

    Infrastructure for AI Agents

    Alan Chan, Kevin Wei, Sihao Huang, Nitarshan Rajkumar, Elija Perrier, Seth Lazar, Gillian K. Hadfield, and Markus Anderljung. 2025. Infrastructure for ai agents. https://arxiv.org/abs/2501.10114. arXiv:2501.10114

    work page arXiv 2025

  7. [7]

    Harrison Chase. 2022. LangChain. https://github.com/langchain-ai/langchain. Open-source Python framework for developing applications powered by language models. Accessed: 2025-10-27

    2022

  8. [8]

    Samuel Colvin, Eric Jolibois, Hasan Ramezani, Adrian Garcia Badaracco, Terrence Dorsey, David Montague, Serge Matveenko, Marcelo Trylesinski, Sydney Runkle, David Hewitt, Alex Hall, and Victorien Plot. 2025. Pydantic Validation. https: //docs.pydantic.dev/latest/. Version v2.13.0a0+dev. Widely used data validation library for Python 3.9+ with strong typin...

    2025

  9. [9]

    Liming Dong, Qinghua Lu, and Liming Zhu. 2024. AgentOps: Enabling Observ- ability of LLM Agents. https://arxiv.org/abs/2411.05285. arXiv:2411.05285

    work page arXiv 2024

  10. [10]

    Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik

  11. [11]

    Empowering biomedical discovery with AI agents.Cell187, 22 (2024), 6125–6151

    2024

  12. [12]

    Gartner, Inc. 2025. Gartner Identifies the Top Strategic Trends in Software Engineering for 2025 and Beyond. https://www.gartner.com/en/newsroom/press- releases/2025-07-01-gartner-identifies-the-top-strategic-trends-in-software- engineering-for-2025-and-beyond. Press release. Contains the prediction that by 2027, at least 55% of software engineering teams...

    2025

  13. [13]

    Jiawei Gu, Jianqing Zhao, Chenghao Liu, Wenxin Du, Zihuiwen Fang, Yao Wang, Wenqiang Lei, and Wayne Xin Zhao. 2024. A Survey on LLM-as-a-Judge. https: //arxiv.org/abs/2411.15594. arXiv:2411.15594. Survey of LLMs as judges for generative evaluation, with techniques to improve reliability and consistency. Accessed: 2025-10-27

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    Jeffrey Ip and Kritin Vongthongsri. 2025. deepeval. https://github.com/confident- ai/deepeval. Version 3.6.8. License: Apache-2.0. Accessed: 2025-10-27

    2025

  15. [15]

    JetBrains. 2024. Koog by JetBrains. https://www.jetbrains.com/koog/. Open- source framework for building fault-tolerant, scalable, production-ready AI agents with a Kotlin DSL; graph-based workflows; built-in tracing/monitoring; integrates with Weave and Langfuse. Accessed: 2025-10-27

    2024

  16. [16]

    JetBrains. 2025. PyCharm: The Only Python IDE You Need. https://www.jetbrains. com/pycharm/. Professional Python IDE with AI-enhanced experience; sup- ports Python, databases, Jupyter, Git, Conda, and frameworks incl. PyTorch, TensorFlow, Django, Flask, FastAPI. Accessed: 2025-10-27

    2025

  17. [17]

    Hyoungwook Jin, Seonghee Lee, Hyungyu Shin, and Juho Kim. 2024. Teach ai how to code: Using large language models as teachable agents for program- ming education. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–28

    2024

  18. [18]

    LangChain Inc. 2024. LangGraph. https://github.com/langchain-ai/langgraph. Low-level orchestration framework and runtime for building long-running, state- ful agents with durable execution, HITL, memory, and production deployment. Accessed: 2025-10-27

    2024

  19. [19]

    LangChain Inc. 2024. LangSmith. https://smith.langchain.com. Tooling for developing, debugging, evaluating, and deploying LLM applications; integrates with LangChain and LangGraph. Accessed: 2025-10-27

    2024

  20. [20]

    LangChain Inc. 2025. LangChain Callback System. https://python.langchain. com/docs/concepts/callbacks. Callback system for hooking into LLM app stages; handlers receive start/end events, tokens, errors; pass callbacks per-request or at construction. Accessed: 2025-10-27

    2025

  21. [21]

    LangChain Inc. 2025. LangChain Runnable. https://python.langchain.com/ api_reference/core/runnables/langchain_core.runnables.base.Runnable.html. Runnable = unit of work that can be invoked, batched, streamed, transformed, and composed; LCEL primitives: RunnableSequence, RunnableParallel. Accessed: 2025-10-27

    2025

  22. [22]

    Microsoft. 2025. AI Toolkit for Visual Studio Code. https://github.com/microsoft/ vscode-ai-toolkit. Accessed: 2025-10-27

    2025

  23. [23]

    Microsoft. 2025. Visual Studio Code. https://code.visualstudio.com/. Accessed: 2025-10-27

    2025

  24. [24]

    Yu, et al

    Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S. Yu, et al. 2025. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. InProceedings of the 31st ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining V. 2. 6140–6150

    2025

  25. [25]

    OpenAI. 2025. OpenAI Python SDK. https://github.com/openai/openai-python. Official Python SDK for the OpenAI API with typed sync/async clients; supports the Responses and Chat Completions APIs, vision, and more. Accessed: 2025-10- 27

    2025

  26. [26]

    Pydantic Team. 2025. Pydantic AI. https://github.com/pydantic/pydantic-ai/. Framework for structured, type-safe, debuggable AI applications with agentic workflows and function calling. Accessed: 2025-10-27

    2025

  27. [27]

    Clemens Rawert, Marc Klingen, and Maximilian Deichmann. 2023. Langfuse — Open-Source LLM Engineering Platform. https://langfuse.com/. Software available from https://langfuse.com. Accessed: 2025-10-27

    2023

  28. [28]

    Stack Overflow. 2025. Stack Overflow Developer Survey 2025 — AI agent observability & security. https://survey.stackoverflow.co/2025/ai#ai-agent- observability-and-security. Accessed: 2025-10-27

    2025

  29. [29]

    Stack Overflow. 2025. Stack Overflow Developer Survey 2025 — Technology: Dev IDEs (Professionals that use AI). https://survey.stackoverflow.co/2025/ technology#most-popular-technologies-dev-envs-prof-ai. Accessed: 2025-10-27

    2025

  30. [30]

    Guangzhi Sun, Xiao Zhan, and Jose Such. 2024. Building better ai agents: A provocation on the utilisation of persona in llm-based conversational agents. In Proceedings of the 6th ACM Conference on Conversational User Interfaces. 1–6

    2024

  31. [31]

    SuperAnnotate AI, Inc. 2025. SuperAnnotate. https://www.superannotate.com. Build efficient human data and evaluation pipelines to ship better agentic, multi- modal, and frontier AI faster. Accessed: 2025-10-27

    2025

  32. [32]

    Artem Trofimov, Mikhail Kostyukov, Sergei Ugdyzhekov, Natalia Ponomareva, Igor Naumov, and Maksim Melekhovets. 2024. JetTrain: IDE-Native Machine Learning Experiments. InProceedings of the 1st ACM/IEEE Workshop on Integrated Development Environments. 59–61

    2024

  33. [33]

    Weights & Biases. 2024. Weave: Track, Test and Improve LLM Applications. https: //wandb.ai/site/products/weave. Track LLM calls in production, test prompts, and run evaluations for performance and safety. Accessed: 2025-10-27

    2024

  34. [34]

    Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2025. Survey on evaluation of LLM-based agents. https://arxiv.org/abs/2503.16416. arXiv:2503.16416

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, et al. 2025. MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains. InFindings of the Association for Computational Linguistics: NAACL 2025. 4737–4765

    2025

  36. [36]

    Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, Fen Xie, and Corey Zumar

    Matei A. Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, Fen Xie, and Corey Zumar. 2018. Accelerating the Machine Learning Lifecycle with MLflow.IEEE Data Engineering Bulletin41 (2018), 39–45. MLflow is an open-source platform for managing the ML lifecycle: track...

    2018

  37. [37]

    Yusheng Zheng, Yanpeng Hu, Tong Yu, and Andi Quinn. 2025. AgentSight: System-Level Observability for AI Agents Using eBPF. InProceedings of the 4th Workshop on Practical Adoption Challenges of ML for Systems. 110–115

    2025

  38. [38]

    Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoor- thi, Yuandong Tian, et al

    Mingchen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoor- thi, Yuandong Tian, et al. 2025. Agent-as-a-Judge: Evaluate Agents with Agents. InForty-second International Conference on Machine Learning

    2025