Pith. sign in

REVIEW 3 major objections 5 minor 62 references

Mined workflow graphs double the boundaries a black-box agent test can reach

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 23:47 UTC pith:W3IGHSTK

load-bearing objection Solid empirical contribution to black-box testing of LLM agents. The ablation design is genuinely clean, but two intermediate LLM steps lack quantitative validation, which is where the skepticism should focus. the 3 major comments →

arxiv 2607.06873 v1 pith:W3IGHSTK submitted 2026-07-08 cs.SE

Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents

Liting Lin , Boxi Yu , Yuzhong Zhang , Lionel Briand , David-Paul Niland , Emir Mu\~noz This is my paper
classification cs.SE
keywords agentevalconversationalboundariesagentagentsgraphtestingblack-box
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the most dangerous failures of conversational AI agents—canceling a booking before the user confirms, acting for an unverified caller, accepting a value that should be refused—sit at stateful boundaries hidden behind multi-turn conversational prerequisites. A tester who sends single prompts cannot reach these boundaries because the agent only exposes them after a sequence of turns has navigated to the right state. The paper presents AgentEval, a framework that solves this by first interacting with the agent as a black box to mine a conversational workflow graph: a directly-follows graph whose nodes are abstracted agent activities and whose edges carry the user actions observed between them. Once the graph is built, AgentEval does not prompt blindly for tests. Instead it deterministically enumerates every structural location in the graph—every node, edge, entry point, and terminal state—as a potential boundary target, asks an LLM to rank each target by how likely stressing it would expose a guard or prerequisite, and then generates a perturbation test for the highest-ranked targets. Each test inherits the conversational route to its target location, so the runner walks the agent through the prerequisite turns before applying the perturbation, reaching state-dependent behavior that a single prompt cannot. The central empirical claim is that this structural enumeration is what drives the gain: on an airline agent, graph-guided generation covers 23 distinct boundaries versus 12 with a prompt-only baseline, at a lower duplicate rate (0.26 vs 0.56) and near-zero false-alarm rate. Ablation shows that merely feeding the graph as prompt context does not help—only using it to enumerate structural targets does.

Core claim

The paper's central discovery is that the gain in boundary-test coverage comes not from having a graph but from using its structure to enumerate targets. When the same mined graph is passed to an LLM only as textual context, the generator produces 9 distinct boundaries—no better than the 12 from a no-graph baseline, because a free-form prompt still clusters on a few salient flows and repeats them. Only when every node, edge, start, and end of the graph becomes an explicit, deterministic test target does coverage jump to 23 distinct boundaries. The graph's structural locations force the generator to spread tests across the entire observed workflow rather than re-prompting the same few high-sy

What carries the argument

The conversational workflow graph is a directly-follows graph built from black-box session traces. An LLM first abstracts each conversation turn into a user action and an agent activity label, merging surface variants of the same activity into one node. Then a deterministic frequency-counting algorithm (Algorithm 1) adds an edge between any two activities observed in direct succession, recording the user actions that drove each transition. The resulting graph's structural locations—nodes, edges, entry points, terminal states—are then enumerated as boundary targets. Each target inherits the observed route (the sequence of user actions from session start to that location), and each generated边界

Load-bearing premise

The entire graph construction depends on an LLM reliably assigning the same activity label to semantically identical agent replies across different conversations. If the LLM labels 'What is your reservation code?' and 'Could you give me the booking reference?' as different activities, the graph fragments into spurious nodes and missing edges, and the structural target enumeration that drives the boundary-testing gain operates on an unreliable model. The paper states that theL

What would settle it

If the LLM-driven event abstraction step is noisy—assigning different labels to the same activity or the same label to different activities—the directly-follows graph would fragment or collapse, producing spurious nodes and missing edges. The structural target enumeration would then operate on an unreliable model, and the 23-vs-12 boundary gain could be an artifact of graph topology rather than genuine boundary discovery. A direct falsification would show that abstraction consistency is low (e.g., re-labeling the same traces produces different graphs) and that the boundary gain disappears when

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If structural enumeration of a mined behavioral model is the key driver of boundary-test diversity, then any conversational agent with a recurring interaction structure—routing, slot-filling, validation, confirmation gates—can be tested this way without source code access, making regression testing of deployed agents feasible as a CI/CD artifact.
  • The separation between the tester (black-box, graph-driven) and the auditor (white-box, source-code-reading) establishes a protocol for validating black-box test quality against ground truth, which other agent-testing frameworks could adopt.
  • The finding that graph-as-context fails but graph-as-structure succeeds suggests a broader principle: LLM test generators benefit from deterministic structural scaffolding that constrains where they look, rather than from being given more information in their prompt.
  • The identity-mismatch problem—where the generator pairs an authenticated caller with another customer's reservation, making the boundary unreachable—points to a need for state-aware test grounding that goes beyond copying values from observed traces.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents AGENTEVAL, a black-box testing framework for conversational LLM agents. The framework operates in two phases: Discovery and Execution. In the Discovery phase, AGENTEVAL interacts with the agent via its chat interface to collect session traces. It abstracts these traces into agent activities and user actions, then constructs a 'conversational workflow graph' using the directly-follows graph technique from process mining. This graph is used to enumerate structural locations (nodes, edges, starts, ends) as boundary targets. For each target, an LLM generates a boundary test that replays the conversational path to that location before applying a perturbation (e.g., skipping a prerequisite). In the Execution phase, an LLM-driven runner executes each test against a fresh session, and a separate LLM judge evaluates the outcome based solely on the visible conversation. The authors evaluate AGENTEVAL on four τ³-bench domains using a privileged, white-box LLM auditor (with access to the source code) as the ground-truth oracle. The results show that graph-guided boundary generation yields 23–38 distinct boundaries per agent, outperforming a prompt-only baseline (23 vs. 12 distinct boundaries on the airline domain) at lower duplicate and false-alarm rates.

Significance. The paper addresses a practically important problem: testing stateful workflow failures in black-box conversational LLM agents, where critical boundaries (e.g., confirmation gates) are hidden behind multi-turn prerequisites. The adaptation of process mining (directly-follows graphs) to abstract unstructured conversation turns into a behavioral model for test generation is a novel and well-motivated contribution. The benchmark design, which uses a privileged white-box auditor to score a black-box tester, is a significant methodological strength. The central ablation (Table V) is well-designed: by comparing 'Graph-context' (graph as prompt text) against 'Ours' (structural target enumeration), both using the identical mined graph, the paper cleanly isolates structural target selection as the cause of the boundary-testing gain. The framework produces reusable regression test plans, adding practical value for CI/CD pipelines.

major comments (3)
  1. The headline metric—distinct-boundary count—is derived from LLM-based clustering, which is unvalidated and potentially biased by the independent variable. Section VI-B states the auditor extracts a 'boundary card' and clusters cards describing the same boundary. However, because AGENTEVAL generates tests by enumerating structural locations (Table I), each test carries a distinct graph-location label. When the LLM auditor clusters tests by boundary similarity, it may be biased toward treating tests from different structural locations as distinct boundaries even when they test semantically equivalent guards. This would systematically inflate the 'Ours' distinct count relative to the 'Prompt-only' baseline (which lacks structural labels). The paper acknowledges 'stricter independent pairwise judgments are left to future work' (Section VI-E), but this threat is load-bearing for the central R
  2. The LLM-driven event abstraction step (Section IV-A) lacks quantitative validation, yet the entire graph construction depends on it. The paper states 'The model produces labels that fit the trace and assigns the same label to all surface variants' but provides no measure of abstraction consistency (e.g., inter-run agreement, stability under rephrasing). If the LLM fragments semantically identical activities into separate nodes, the directly-follows graph will contain spurious nodes and missing edges. This directly affects the structural target enumeration that drives the boundary-testing gain: more nodes yield more targets, potentially inflating the distinct-boundary count for 'Ours' without a corresponding increase in actual boundary coverage. A quantitative check of abstraction stability is needed to ensure the mined graph is a reliable foundation for test generation.
  3. The evaluation relies on a single run per configuration (Section VI-E, Internal threats). The paper notes that 'small differences... should not be overread,' but the boundary-count gap (23 vs. 12) is the central claim. Given the inherent randomness of LLM-driven generation and judging, a single run provides no confidence intervals or variance estimates. It is possible that the 23 vs. 12 gap is partly an artifact of a single favorable run. To solidify the claim, the ablation should be repeated across multiple seeds, reporting mean and standard deviation for the distinct-boundary count and duplicate rate.
minor comments (5)
  1. Section VI-D, RQ2: The lower boundary validity on airline and retail (0.78 and 0.79) is attributed to the generator pairing the authenticated caller with an order belonging to a different customer. This is a significant limitation that effectively wastes ~20% of the boundary test budget. The paper should discuss whether this is a fundamental limitation of the LLM generator or a prompt engineering issue that could be mitigated.
  2. Table II: The auditor model selection is based on agreement with a single expert human reference on 26 airline scenarios. It would strengthen the benchmark to validate the auditor on a larger, more diverse set of scenarios across all four domains, not just airline.
  3. Section VI-D, RQ4: The judge reliability study on value/accuracy faults (Table VII) shows that a doubled stored reservation price is caught in only 1 of 10 relevant tests. The paper explains this is because the agent recomputes the correct total. This suggests the judge is comparing the displayed value rather than the executed outcome, which may limit its ability to detect certain classes of faults. This limitation should be discussed more prominently.
  4. Figure 1: The graph excerpt is helpful, but the full graph (46 activities, 54 transitions) is not shown. Providing access to the full graph, perhaps in a supplementary artifact, would aid reproducibility and allow readers to assess the graph's quality.
  5. Section III-A: The warm-up schedule parameters (m_cap=1, m_hp=2, m_cc=1) are fixed. The paper should briefly discuss whether the system is sensitive to these choices, as they determine the initial exploration coverage before the LLM planner takes over.

Circularity Check

0 steps flagged

No significant circularity: the central claim is grounded in an ablation design that isolates structural target selection, with an independent auditor validated against expert reference.

full rationale

The paper's central claim—that graph-guided structural target enumeration yields more distinct boundaries than prompt-only generation—is supported by an ablation design (RQ3, Table V) where Graph-context and Ours share identical mined graphs, discovery traces, and budgets, differing only in whether the graph is used as prompt text or for structural target enumeration. This isolates the cause without circular dependency. The auditor uses a different model family (GLM-5.2 vs. MiMo-v2.5-pro) from the tester, and its coverage audit is validated against an expert human reference (Jaccard 0.94, Table II). The directly-follows graph construction (Algorithm 1) is a standard process-mining algorithm applied to LLM-abstracted traces; it does not define its outputs in terms of its inputs in a circular manner. The boundary-potential scoring (Section V-B) uses an LLM to rank targets but this is a filtering step, not a derivation claiming first-principles prediction. The distinct-boundary count relies on LLM-proposed clustering, which the paper itself flags as a validity threat ('stricter independent pairwise judgments are left to future work,' Section VI-E), but this is a measurement-validation concern, not circularity—the count is not defined in terms of the structural enumeration that produced the targets. No self-citation chain is load-bearing for the central claim. The paper is largely self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The free parameters are mostly fixed budgets chosen globally rather than fitted to data, which is reasonable for a testing framework. The ad-hoc-to-paper axiom on LLM abstraction consistency is the most concerning: it is load-bearing (the graph depends on it) and unvalidated. The domain assumptions (resettable agent, recurring patterns) are standard for the testing setting but limit generality.

free parameters (6)
  • R_disc (discovery rounds) = 12
    Fixed to balance coverage against cost; not tuned per agent but chosen globally (Section VI-C).
  • B_turn (turn budget per session) = 6
    Fixed globally; determines trace depth and graph richness.
  • m_cap, m_hp, m_cc (warm-up sessions) = 1, 2, 1
    Predefined warm-up schedule before free LLM planner choice; hand-set.
  • N_f (functional test target) = 50
    Cap on functional tests; fixed budget.
  • N_b (boundary test target) = 50
    Cap on boundary tests; fixed budget. The paper notes full graphs usually yield more targets than this budget allows.
  • Boundary potential threshold = not stated
    LLM scores each target in [0,1]; top N_b kept. The threshold is implicitly the rank cutoff, but the scoring itself is an uncalibrated LLM judgment.
axioms (5)
  • domain assumption The agent can be reset to a fresh session (reset(A) operation).
    Stated in Section II-E as a standard requirement for automated testing environments. Load-bearing: all replay-based testing depends on this.
  • domain assumption The agent exhibits recurring interaction patterns (routing, slot filling, validation, confirmation gates) observable in bounded traces.
    Stated in Section II-E. If false (e.g., highly open-ended agent), the mined graph remains thin and yields fewer boundary targets than a trace-only baseline.
  • ad hoc to paper LLM-based event abstraction produces consistent agent-activity labels across surface variants.
    Section IV-A states the model 'assigns the same label to all surface variants' but provides no quantitative consistency measure. The entire graph construction depends on this.
  • domain assumption The LLM judge can reliably distinguish real faults from correct behavior using only visible conversation turns.
    Section III-B and RQ4. Validated partially: 22/27 process/guard faults caught (0.81), 4/4 value/accuracy mutants killed. But the judge's false-alarm rate on unmodified controls is the control, and residual auditor error in run-audit labels is acknowledged as a threat (Section VI-E).
  • domain assumption The directly-follows graph is an adequate behavioral model for the purpose of enumerating boundary test targets.
    Section II-D justifies the choice by its deterministic construction and flowchart-like readability. The paper acknowledges it 'can imply paths never observed as full traces' (Section VII), which could generate unreachable targets.
invented entities (2)
  • Conversational workflow graph independent evidence
    purpose: A directly-follows graph over LLM-abstracted agent activities, serving as the structural model for boundary test target enumeration.
    The graph is constructed deterministically (Algorithm 1) from observed traces and its effectiveness is validated empirically through ablation (Table V). It is not postulated; it is mined and tested.
  • Boundary potential score no independent evidence
    purpose: An LLM-generated score in [0,1] estimating how likely stressing a graph location is to expose a guard, limit, prerequisite, or recovery behavior worth testing.
    The score is an uncalibrated LLM judgment with no validation against a ground-truth boundary-potential measure. It functions as a ranking heuristic; its accuracy is not independently verified.

pith-pipeline@v1.1.0-glm · 22165 in / 3164 out tokens · 283382 ms · 2026-07-09T23:47:47.265342+00:00 · methodology

0 comments
read the original abstract

Conversational LLM agents can cause real-world harm when their internal workflows fail, such as completing a transaction without confirmation. Testing these state-dependent failures is difficult because critical boundaries, such as identity checks and confirmation gates, are hidden behind multi-turn conversational prerequisites, rendering them inaccessible to standard tests. We present AgentEval, a black-box testing framework that discovers and stresses these stateful boundaries. AgentEval interacts with an agent to mine a \emph{conversational workflow graph}, a model of its behavior. Instead of prompting blindly, AgentEval uses this graph's structure to enumerate specific guards and prerequisites as test targets, replaying the conversational path to a boundary before applying a perturbation. AgentEval then executes each test, determining whether it passes or fails using only the conversation turns. We benchmark AgentEval against a privileged, white-box auditor with access to the agent's underlying source code, which AgentEval never sees. On four $\tau^3$-bench agents, AgentEval successfully generates tests covering $23$--$38$ distinct boundaries per agent; ablation studies attribute the gain to the graph's structure: $23$ distinct boundaries versus $12$ with a prompt-only baseline, at lower duplicate and false-alarm rates.

Figures

Figures reproduced from arXiv: 2607.06873 by Boxi Yu, David-Paul Niland, Emir Mu\~noz, Lionel Briand, Liting Lin, Yuzhong Zhang.

Figure 1
Figure 1. Figure 1: Excerpt of a conversational workflow graph A [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A workflow-graph fragment AGENTEVAL induces from the τ 3 -bench airline agent (real activity labels, abbreviated). Solid edges are observed transitions labeled with the user action or condition that caused them; the shaded (amber) node is a confirmation gate. The two dashed edges (premature confirm? and cancel anyway?) are boundary perturbations that tests will attempt, not observed transitions; deny_and_e… view at source ↗
Figure 3
Figure 3. Figure 3: AGENTEVAL overview, in two phases that share the black-box agent. Discovery explores the agent over Rdisc rounds and collects traces. From those traces AGENTEVAL synthesizes functional tests directly and induces the workflow graph; the graph then drives boundary-test synthesis. Execution drives each test turn by turn with the agent and makes a judgment from the resulting session trace. The LLM generates la… view at source ↗
Figure 4
Figure 4. Figure 4: Event abstraction maps each turn to a user action (from the user utterance) and an agent activity (from the agent reply). Differently worded replies that express the same activity (surface variants) receive the same agent￾activity label and merge into one node. from those same traces (Section V-B). B. Execution Execution runs each test in the plan as an isolated session: an LLM driven runner resets the age… view at source ↗
Figure 5
Figure 5. Figure 5: Two tests AGENTEVAL generated for the τ 3 -bench airline agent (abbreviated from the run artifacts), one of each kind. Both use the test schema of Section II: a goal, user-turn hints, observable pass and fail criteria, and a turn budget. The boundary test also specifies the graph location it targets and the type of boundary. tests synthesized directly from them, and boundary tests guided by the workflow gr… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 62 canonical work pages · 5 internal anchors

  1. [1]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023

    work page 2023

  2. [2]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

  3. [3]

    $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ-bench: A benchmark for tool-agent-user interaction in real-world domains,”arXiv preprint arXiv:2406.12045, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    Preventing repeated real world AI failures by cataloging incidents: The AI incident database,

    S. McGregor, “Preventing repeated real world AI failures by cataloging incidents: The AI incident database,” inProceedings of the AAAI Conference on Artificial Intelligence (IAAI track), vol. 35, no. 17, 2021, pp. 15 458–15 463

    work page 2021

  5. [5]

    RealHarm: A collection of real-world language model application failures,

    P. Le Jeune, J. Liu, L. Rossi, and M. Dora, “RealHarm: A collection of real-world language model application failures,” inProceedings of the First Workshop on Large Language Model Security (LLMSEC), 2025, pp. 87–100

    work page 2025

  6. [6]

    $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment

    V . Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan, “τ 2-bench: Evaluating conversational agents in a dual-control environment,”arXiv preprint arXiv:2506.07982, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    WebArena: A realistic web environment for building autonomous agents,

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Fried, U. Alon, and G. Neubig, “WebArena: A realistic web environment for building autonomous agents,” inInternational Conference on Learning Representations (ICLR), 2024

    work page 2024

  8. [8]

    GAIA: A benchmark for general AI assistants,

    G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y . LeCun, and T. Scialom, “GAIA: A benchmark for general AI assistants,” inInternational Con- ference on Learning Representations (ICLR), 2024

    work page 2024

  9. [9]

    AppWorld: A controllable world of apps and people for benchmarking interactive coding agents,

    H. Trivedi, T. Khot, M. Hartmann, R. Manku, V . Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian, “AppWorld: A controllable world of apps and people for benchmarking interactive coding agents,” in Annual Meeting of the Association for Computational Linguistics (ACL), 2024, pp. 16 022–16 076

    work page 2024

  10. [10]

    SWE-bench: Can language models resolve real-world GitHub issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” inInternational Conference on Learning Representa- tions (ICLR), 2024

    work page 2024

  11. [11]

    G. J. Myers, C. Sandler, and T. Badgett,The Art of Software Testing, 3rd ed. Wiley, 2011

    work page 2011

  12. [12]

    SpecOps: A fully automated AI agent testing framework in real-world GUI environments,

    S. Y . Ahmed, S. Feng, C. Bae, C. Barrus, and X. Zhang, “SpecOps: A fully automated AI agent testing framework in real-world GUI environments,” inProceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE), 2026

    work page 2026

  13. [13]

    STELLAR: A search-based testing framework for large language model applications,

    L. Sorokin, I. Vasilev, K. E. Friedl, and A. Stocco, “STELLAR: A search-based testing framework for large language model applications,” arXiv preprint arXiv:2601.00497, 2026

    work page arXiv 2026

  14. [14]

    W. M. P. van der Aalst,Process Mining: Data Science in Action, 2nd ed. Springer, 2016

    work page 2016

  15. [15]

    A practitioner’s guide to process mining: Limitations of the directly-follows graph,

    ——, “A practitioner’s guide to process mining: Limitations of the directly-follows graph,” inInternational Conference on Enterprise In- formation Systems (CENTERIS), Procedia Computer Science vol. 164, 2019, pp. 321–328

    work page 2019

  16. [16]

    τ 3-bench: From text-only to multimodal, knowledge- aware agent evaluation,

    Sierra Research, “τ 3-bench: From text-only to multimodal, knowledge- aware agent evaluation,” https://github.com/sierra-research/tau2-bench, 2026, accessed: 2026-06-30

    work page 2026

  17. [17]

    Event abstraction for process mining using supervised learning techniques,

    N. Tax, N. Sidorova, R. Haakma, and W. M. P. van der Aalst, “Event abstraction for process mining using supervised learning techniques,” in Intelligent Systems Conference (IntelliSys), 2016, pp. 251–269

    work page 2016

  18. [18]

    Event abstraction in process mining: Literature review and taxonomy,

    S. J. van Zelst, F. Mannhardt, M. de Leoni, and A. Koschmider, “Event abstraction in process mining: Literature review and taxonomy,” Granular Computing, vol. 6, no. 3, pp. 719–736, 2021

    work page 2021

  19. [19]

    Boundary value exploration for software analysis,

    F. Dobslaw, F. G. de Oliveira Neto, and R. Feldt, “Boundary value exploration for software analysis,” inIEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 2020, pp. 346–353

    work page 2020

  20. [20]

    Automated robustness testing of off-the-shelf software components,

    N. P. Kropp, P. J. Koopman, and D. P. Siewiorek, “Automated robustness testing of off-the-shelf software components,” inInternational Sympo- sium on Fault-Tolerant Computing (FTCS), 1998, pp. 230–239

    work page 1998

  21. [21]

    τ- knowledge: Evaluating conversational agents over unstructured knowl- edge,

    Q. Shi, A. Zytek, P. Razavi, K. Narasimhan, and V . Barres, “τ- knowledge: Evaluating conversational agents over unstructured knowl- edge,”arXiv preprint arXiv:2603.04370, 2026

    work page arXiv 2026

  22. [22]

    ToolLLM: Facilitating large language models to master 16000+ real-world APIs,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun, “ToolLLM: Facilitating large language models to master 16000+ real-world APIs,” inInternational Conference on Learning Representations (ICLR), 2024

    work page 2024

  23. [23]

    Towards self-evolving benchmarks: Synthesizing agent trajectories via test-time exploration under validate- by-reproduce paradigm,

    D. Guo, T. Zhou, D. Liu, C. Qian, Q. Ren, S. Shao, Z. Fan, Y . R. Fung, K. Wang, L. Zhang, and J. Shao, “Towards self-evolving benchmarks: Synthesizing agent trajectories via test-time exploration under validate- by-reproduce paradigm,”arXiv preprint arXiv:2510.00415, 2025

    work page arXiv 2025

  24. [24]

    TestAgent: Automatic benchmarking and exploratory interaction for evaluating LLMs in vertical domains,

    W. Wang, Z. Ma, X. Wang, Y . Zhang, P. Liu, and M. Chen, “TestAgent: Automatic benchmarking and exploratory interaction for evaluating LLMs in vertical domains,”arXiv preprint arXiv:2410.11507, 2024

    work page arXiv 2024

  25. [25]

    Graph2Eval: Automatic multimodal task generation for agents via knowledge graphs,

    Y . Chen, X. Hu, Y . Liu, Z. Wang, Z. Liao, L. Chen, F. Wei, Y . Qian, B. Zheng, K. Yin, and S. Zhang, “Graph2Eval: Automatic multimodal task generation for agents via knowledge graphs,”arXiv preprint arXiv:2510.00507, 2025

    work page arXiv 2025

  26. [26]

    Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems

    J. Geng, H. Chen, D. Arumugam, and T. L. Griffiths, “Are large language models reliable AI scientists? assessing reverse-engineering of black-box systems,”arXiv preprint arXiv:2505.17968, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [27]

    Mining specifications,

    G. Ammons, R. Bod ´ık, and J. R. Larus, “Mining specifications,” in ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), 2002, pp. 4–16

    work page 2002

  28. [28]

    Discovering models of software processes from event-based data,

    J. E. Cook and A. L. Wolf, “Discovering models of software processes from event-based data,”ACM Transactions on Software Engineering and Methodology, vol. 7, no. 3, pp. 215–249, 1998

    work page 1998

  29. [29]

    Automatic generation of software behavioral models,

    D. Lorenzoli, L. Mariani, and M. Pezz `e, “Automatic generation of software behavioral models,” inInternational Conference on Software Engineering (ICSE), 2008, pp. 501–510

    work page 2008

  30. [30]

    Inferring models of concurrent systems from logs of their behavior with CSight,

    I. Beschastnikh, Y . Brun, M. D. Ernst, and A. Krishnamurthy, “Inferring models of concurrent systems from logs of their behavior with CSight,” inInternational Conference on Software Engineering (ICSE), 2014, pp. 468–479

    work page 2014

  31. [31]

    Workflow mining: Discovering process models from event logs,

    W. M. P. van der Aalst, T. Weijters, and L. Maruster, “Workflow mining: Discovering process models from event logs,”IEEE Transactions on Knowledge and Data Engineering, vol. 16, no. 9, pp. 1128–1142, 2004

    work page 2004

  32. [32]

    Discov- ering block-structured process models from event logs—a constructive approach,

    S. J. J. Leemans, D. Fahland, and W. M. P. van der Aalst, “Discov- ering block-structured process models from event logs—a constructive approach,” inApplication and Theory of Petri Nets and Concurrency, LNCS 7927, 2013, pp. 311–329

    work page 2013

  33. [33]

    Applying graph reduction techniques for identifying structural conflicts in process models,

    W. Sadiq and M. E. Orlowska, “Applying graph reduction techniques for identifying structural conflicts in process models,” inInternational Conference on Advanced Information Systems Engineering (CAiSE), LNCS 1626, 1999, pp. 195–209

    work page 1999

  34. [34]

    Learning regular sets from queries and counterexamples,

    D. Angluin, “Learning regular sets from queries and counterexamples,” Information and Computation, vol. 75, no. 2, pp. 87–106, 1987

    work page 1987

  35. [35]

    Unsupervised dialog structure learning,

    W. Shi, T. Zhao, and Z. Yu, “Unsupervised dialog structure learning,” inConference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2019, pp. 1797–1807

    work page 2019

  36. [36]

    Dialog2Flow: Pre-training soft-contrastive action-driven sentence embeddings for automatic dia- log flow extraction,

    S. Burdisso, S. Madikeri, and P. Motlicek, “Dialog2Flow: Pre-training soft-contrastive action-driven sentence embeddings for automatic dia- log flow extraction,” inConference on Empirical Methods in Natural Language Processing (EMNLP), 2024, pp. 5421–5440

    work page 2024

  37. [37]

    Agenda-based user simulation for bootstrapping a POMDP dialogue system,

    J. Schatzmann, B. Thomson, K. Weilhammer, H. Ye, and S. Young, “Agenda-based user simulation for bootstrapping a POMDP dialogue system,” inHuman Language Technologies: Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Short Papers, 2007, pp. 149–152

    work page 2007

  38. [38]

    ConvLab-2: An open-source toolkit for building, evaluating, and diagnosing dialogue systems,

    Q. Zhu, Z. Zhang, Y . Fang, X. Li, R. Takanobu, J. Li, B. Peng, J. Gao, X. Zhu, and M. Huang, “ConvLab-2: An open-source toolkit for building, evaluating, and diagnosing dialogue systems,” inAnnual Meeting of the Association for Computational Linguistics (ACL), System Demonstrations, 2020, pp. 142–149

    work page 2020

  39. [39]

    A taxonomy of model- based testing approaches,

    M. Utting, A. Pretschner, and B. Legeard, “A taxonomy of model- based testing approaches,”Software Testing, Verification and Reliability, vol. 22, no. 5, pp. 297–312, 2012

    work page 2012

  40. [40]

    Principles and methods of testing finite state machines—a survey,

    D. Lee and M. Yannakakis, “Principles and methods of testing finite state machines—a survey,”Proceedings of the IEEE, vol. 84, no. 8, pp. 1090–1123, 1996

    work page 1996

  41. [41]

    Testing software design modeled by finite-state machines,

    T. S. Chow, “Testing software design modeled by finite-state machines,” IEEE Transactions on Software Engineering, vol. SE-4, no. 3, pp. 178– 187, 1978

    work page 1978

  42. [42]

    RESTler: Stateful REST API fuzzing,

    V . Atlidakis, P. Godefroid, and M. Polishchuk, “RESTler: Stateful REST API fuzzing,” inInternational Conference on Software Engineering (ICSE), 2019, pp. 748–758

    work page 2019

  43. [43]

    RESTful API automated test case generation with Evo- Master,

    A. Arcuri, “RESTful API automated test case generation with Evo- Master,”ACM Transactions on Software Engineering and Methodology, vol. 28, no. 1, pp. 3:1–3:37, 2019

    work page 2019

  44. [44]

    Morest: Model-based RESTful API testing with execution feedback,

    Y . Liu, Y . Li, G. Deng, Y . Liu, R. Wan, R. Wu, D. Ji, S. Xu, and M. Bao, “Morest: Model-based RESTful API testing with execution feedback,” inInternational Conference on Software Engineering (ICSE), 2022, pp. 1406–1417

    work page 2022

  45. [45]

    KAT: Dependency-aware automated API testing with large language models,

    T. Le, T. Tran, D. Cao, V . Le, T. N. Nguyen, and V . Nguyen, “KAT: Dependency-aware automated API testing with large language models,” inIEEE Conference on Software Testing, Verification and Validation (ICST), 2024

    work page 2024

  46. [46]

    Testing RESTful APIs: A survey,

    A. Golmohammadi, M. Zhang, and A. Arcuri, “Testing RESTful APIs: A survey,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 1, pp. 27:1–27:41, 2023

    work page 2023

  47. [47]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,”IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2024

    work page 2024

  48. [48]

    CoverUp: Effective high coverage test generation for Python,

    J. Altmayer Pizzorno and E. D. Berger, “CoverUp: Effective high coverage test generation for Python,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, 2025

    work page 2025

  49. [49]

    Evaluating and improving ChatGPT for unit test generation,

    Z. Yuan, M. Liu, S. Ding, K. Wang, Y . Chen, X. Peng, and Y . Lou, “Evaluating and improving ChatGPT for unit test generation,” inACM International Conference on the Foundations of Software Engineering (FSE), 2024, pp. 1703–1726

    work page 2024

  50. [50]

    CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models,” inInternational Conference on Software Engineering (ICSE), 2023, pp. 919–931

    work page 2023

  51. [51]

    TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration

    S. Gu, Q. Zhang, K. Li, C. Fang, F. Tian, L. Zhu, J. Zhou, and Z. Chen, “TestART: Improving LLM-based unit testing via co-evolution of auto- mated generation and repair iteration,”arXiv preprint arXiv:2408.03095, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  52. [52]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,”IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015

    work page 2015

  53. [53]

    Pseudo-oracles for non-testable programs,

    M. D. Davis and E. J. Weyuker, “Pseudo-oracles for non-testable programs,” inProceedings of the ACM ’81 Conference, 1981, pp. 254– 257

    work page 1981

  54. [54]

    On testing non-testable programs,

    E. J. Weyuker, “On testing non-testable programs,”The Computer Journal, vol. 25, no. 4, pp. 465–470, 1982

    work page 1982

  55. [55]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” inAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023

    work page 2023

  56. [56]

    Large language models cannot self-correct reasoning yet,

    J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou, “Large language models cannot self-correct reasoning yet,” in International Conference on Learning Representations (ICLR), 2024

    work page 2024

  57. [57]

    On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

    K. Stechly, K. Valmeekam, and S. Kambhampati, “On the self- verification limitations of large language models on reasoning and planning tasks,”arXiv preprint arXiv:2402.08115, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  58. [58]

    Self-Refine: Iterative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self-Refine: Iterative refinement with self-feedback,” inAdvances in Neural Infor- mation Processing Systems (NeurIPS), 2023

    work page 2023

  59. [59]

    CRITIC: Large language models can self-correct with tool-interactive critiquing,

    Z. Gou, Z. Shao, Y . Gong, Y . Shen, Y . Yang, N. Duan, and W. Chen, “CRITIC: Large language models can self-correct with tool-interactive critiquing,” inInternational Conference on Learning Representations (ICLR), 2024

    work page 2024

  60. [60]

    Metamorphic testing: A review of challenges and opportunities,

    T. Y . Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities,” ACM Computing Surveys, vol. 51, no. 1, pp. 4:1–4:27, 2018

    work page 2018

  61. [61]

    Not what you’ve signed up for: Compromising real-world LLM- integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world LLM- integrated applications with indirect prompt injection,” inACM Work- shop on Artificial Intelligence and Security (AISec), 2023, pp. 79–90

    work page 2023

  62. [62]

    Ignore previous prompt: Attack techniques for language models,

    F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” inNeurIPS ML Safety Workshop, 2022

    work page 2022