Pith. sign in

REVIEW 4 major objections 8 minor 15 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Tool-Calling Agents Pick the Same Tools but Vary in Arguments

2026-07-04 21:27 UTC pith:EFC3SR46

load-bearing objection Novel structural/parametric consistency distinction for tool-calling agents; correctness validation needs semantic grounding the 4 major comments →

arxiv 2605.28840 v1 pith:EFC3SR46 submitted 2026-04-23 cs.CL cs.AIcs.SE

How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines

Abel Yagubyan This is my paper
classification cs.CL cs.AIcs.SE
keywords LLM agentsbehavioral consistencytool callingreproducibilityagent reliabilitystructural consistencyparametric variance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether LLM tool-calling agents behave the same way when run repeatedly on the same task. The author finds a pattern called 'structural consistency, parametric variance': agents reliably select the same tools in the same order across repeated runs (mean Tool Sequence Similarity = 0.87), but vary substantially in the arguments they provide (mean Argument Consistency = 0.69). This gap is large (Cohen's d = 0.75, p < 10^-13) and holds across all six models tested. The paper then shows that only structural consistency predicts task success: high-TSS runs achieve 90.2% correctness versus 61.2% for low-TSS runs (d = 0.81, p < 0.001), while argument-level variance is uncorrelated with correctness (r = 0.12, n.s.). The author argues that TSS can serve as a lightweight, label-free reliability proxy for production agent deployments, and that task specification clarity matters more for consistency than model choice.

Core claim

The paper introduces a formal distinction between two behavioral layers in LLM tool-calling agents: the structural layer (which tools are called, in what order) and the argument layer (what parameter values are passed). It shows these layers decouple sharply. Agents are highly consistent in tool selection but inconsistent in argument instantiation, and only the structural layer matters for task success. The author attributes this to training dynamics: fine-tuning reinforces correct procedural schemas (which tool to use) more strongly than argument instantiation (what values to pass), because procedure selection carries a cleaner training signal. The paper also finds that 60% of behavioral分歧

What carries the argument

Tool Sequence Similarity (TSS), a normalized Levenshtein distance over tool-name sequences across repeated runs of the same task. TSS ranges from 0 to 1, where 1 means all runs produced identical tool sequences.

Load-bearing premise

The paper's correctness rubric uses regex-matched argument patterns and required tool coverage to judge whether a trace succeeded. If this rubric systematically rewards traces that match expected patterns regardless of whether they are semantically correct, then the finding that TSS predicts correctness could partly reflect the rubric's own structural bias rather than a true reliability signal.

What would settle it

If a subsequent study with human-graded correctness (rather than regex-based rubric) found that argument-level variance does predict failure — for instance, because semantically wrong arguments cause silent failures that the regex rubric misses — the core claim that AC is benign would collapse.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agent tests should assert on structured tool-call sequences, not natural language outputs, which match less than 5% of the time even when tool sequences are identical.
  • Monitoring only the first 1-2 tool calls against a reference trace captures 60% of behavioral divergence, enabling cheap online consistency checks.
  • Engineering effort spent clarifying ambiguous task specifications yields more consistency improvement than switching to a more capable model.
  • TSS can be computed without ground-truth correctness labels, making it deployable as a real-time reliability signal in production agent systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If TSS predicts correctness because structural divergence concentrates failures, then agents with low TSS on a task type could be automatically routed to retry or human review before side effects execute, creating a pre-commitment safety layer.
  • The structural/parametric split suggests that argument-level diversity could be deliberately exploited (e.g., sampling multiple argument variants for the same tool sequence) to improve robustness without risking structural failures — a form of variance that helps rather than hurts.
  • The finding that divergence concentrates in early steps implies that error propagation through multi-step pipelines is primarily topological: a wrong first tool call cascades through all subsequent steps, which would make early-step verification disproportionately valuable in longer pipelines than the 3-5 step tasks tested here.
  • If the procedural-schema interpretation is correct, then fine-tuning specifically on argument-level consistency (not just procedure selection) might close the TSS-AC gap, but the paper's results suggest this would not improve task success — raising the question of whether argument consistency is worth optimizing at all.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This manuscript presents an empirical study of behavioral consistency in multi-step tool-calling LLM agents across 19 tasks, 6 models, and 1,140 traces. The authors introduce two formal metrics—Tool Sequence Similarity (TSS), based on Levenshtein distance over tool-name sequences, and Argument Consistency (AC), based on Jaccard similarity over exact key-value argument pairs. The central finding is a 'structural consistency, parametric variance' pattern: agents reliably select the same tools in the same order (mean TSS = 0.87) but vary in arguments (mean AC = 0.69). The paper further reports that TSS predicts task correctness (d = 0.81) while AC does not (r = 0.12, n.s.), and that task ambiguity is a stronger driver of inconsistency than model choice. Code, traces, and benchmark definitions are released.

Significance. The paper addresses a practically important and underexplored question: behavioral reproducibility of tool-calling agents under identical inputs. The structural/parametric distinction is a useful conceptual contribution that goes beyond prior ReAct-agent consistency work (Mehta et al., 2026) by decomposing behavior into tool-selection and argument-instantiation layers. The release of raw traces, code, and benchmark definitions is a genuine strength. The finding that TSS predicts correctness without requiring labels, if valid, has actionable implications for agent monitoring. However, the significance of the correctness-prediction claim is tightly coupled to the validity of the correctness rubric and the semantic sensitivity of the AC metric, both of which require scrutiny.

major comments (4)
  1. §4.5 and §5.5: The correctness rubric is regex-based (required tool coverage, argument pattern matching, output completeness) and is the load-bearing instrument for the paper's most actionable claim—that TSS predicts correctness while AC does not. The rubric is not semantically validated: no human annotation, no LLM-as-judge cross-check, and no inter-annotator agreement is reported. The paper acknowledges this in §6.5 ('Our rubric is not a full semantic evaluation') but the rubric validity concern is more than a limitation footnote—it directly affects Finding 4 and the practical guidelines in §6.3. At minimum, a human-validated subset (even 100–200 traces) should be scored and the rubric's agreement with human judgment reported. Without this, the d = 0.81 result could be an artifact of a rubric that rewards structural completeness by construction (since 'required tool coverage' is itself
  2. Definition 4 (AC) and §5.5: The AC metric uses exact key-value Jaccard similarity, so semantically equivalent arguments (e.g., {name: 'Alice'} vs {name: 'alice'}, or {date: '2026-03-02'} vs {date: 'March 2, 2026'}) contribute zero overlap. This has two consequences for the central claim. First, it may artificially deflate AC, inflating the TSS > AC gap (d = 0.75). Second, and more critically, it may make the 'argument variance is benign' finding (r = 0.12, n.s.) partly tautological: if AC measures surface-form rather than semantic consistency, it will naturally fail to predict a correctness rubric that evaluates semantic outcomes. The paper acknowledges in §6.1 that variations are 'largely semantically equivalent (e.g., Alice vs alice in a case-insensitive lookup)' but frames this as a discovery rather than a metric limitation. A semantically normalized AC variant (even a simple case-ins
  3. §5.5: The claim that TSS predicts correctness while AC does not is based on separate bivariate correlations (TSS-correctness r = 0.32; AC-correctness r = 0.12). No multivariate regression of correctness on both TSS and AC is reported. Given that AC is partially confounded with TSS (as noted in Definition 4: when traces call different tools at step i, AC = 0 for that step), a regression with both predictors would help disentangle whether AC has no independent effect or whether the null result is driven by the metric's syntactic sensitivity. This is directly relevant to the practical recommendation in §6.3, guideline 4 ('Use TSS as an online reliability signal' while ignoring argument-level consistency).
  4. §4.3: Temperature is fixed at 1.0 for all experiments. The paper states this is 'a deliberate choice: we measure consistency as deployed.' However, T = 1.0 is the maximum default temperature for most providers and represents the high-variance end of the deployment spectrum. Many production deployments use T = 0 or low temperature precisely for reproducibility. Without a T = 0 baseline, it is unclear whether the structural/parametric pattern and the TSS-correctness relationship hold at lower temperatures, which limits the generalizability of the practical guidelines. The paper lists this as a limitation (§6.5) but it is load-bearing for the deployment recommendations.
minor comments (8)
  1. Table 1: The 'Output Match' column for o1 shows '—' but the footnote only explains exclusion from aggregate statistics. Clarify whether output match was not computed or not applicable for o1.
  2. Figure 2 caption: 'Llama 3.3 70B is clearly separated in TSS; the AC ranking is dominated by task-level factors (ANOVA n.s.).' The ANOVA result is reported in §5.6, not referenced in the figure. Consider adding the F-statistic or p-value to the caption for self-containedness.
  3. §5.3: '60% of first-divergence events occur within the first two pipeline steps (mean divergence point = 2.2).' The text also states '72% in steps 1–2' in Figure 4. These two percentages (60% vs 72%) appear inconsistent. Please reconcile.
  4. Table 2 caption states 'fine-grained category comparisons are exploratory' but the body text (§5.2) reports specific d-values and p-values for the ambiguous-vs-structured contrast. Clarify which comparisons are confirmatory (pre-registered hypotheses) vs exploratory.
  5. §4.6: The paper acknowledges that a linear mixed-effects model would be more appropriate but uses simple ANOVA. Consider reporting at least one LME result to confirm that the key findings (model differences in TSS, ambiguity effect on AC) survive.
  6. Appendix D (Table 5): The correctness criteria for ambig-001 state 'Any invocation accepted; multiple valid strategies are treated as correct.' This is a very lenient standard that may inflate correctness for ambiguous tasks. Consider discussing how this affects the ambiguity-correctness relationship.
  7. The reference to 'Mehta et al. (2026)' appears to be a future-dated arXiv preprint (2602.11619). Verify the citation details.
  8. §3.2, Definition 4: The notation |S| for the normalization constant is introduced but S is defined as indexing 'all step-aligned pairs.' Clarify whether pairs where only one trace reaches step i are excluded, and how traces of different lengths are handled.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises four major concerns: (1) the correctness rubric lacks semantic validation, (2) the AC metric's exact-match design may artificially deflate AC and create a tautological null result, (3) no multivariate regression disentangling TSS and AC is reported, and (4) the fixed T=1.0 limits generalizability of deployment recommendations. We agree that all four points identify genuine weaknesses. We can fully address points 1–3 with additional analysis and experiments within the revision timeline. Point 4 (temperature ablation) requires new experiments that we can partially address but cannot fully complete to the standard the referee rightly requests. We outline our planned revisions below.

read point-by-point responses

  1. Referee: §4.5 and §5.5: The correctness rubric is regex-based and not semantically validated. No human annotation, no LLM-as-judge cross-check, no inter-annotator agreement. The d=0.81 result could be an artifact of a rubric that rewards structural completeness by construction.

    Authors: The referee is correct that the correctness rubric is the load-bearing instrument for Finding 4, and that without semantic validation, the d=0.81 result could partly reflect the rubric's structural orientation rather than genuine correctness prediction. We accept this criticism and will address it concretely in the revision. Specifically, we will: (1) sample 200 traces stratified across TSS tertiles, task categories, and models; (2) have two annotators independently score correctness using a semantic rubric that evaluates whether the agent accomplished the user's actual goal, not whether it matched regex patterns; (3) report inter-annotator agreement (Cohen's kappa) and the correlation between human scores and the automated rubric. If the rubric-human agreement is high (which we expect given that the rubric does check argument validity and output completeness, not just tool coverage), this substantially mitigates the concern. If agreement is moderate or low, we will report the human-validated d-value alongside the rubric-based one and adjust our claims accordingly. We acknowledge that the rubric's 'required tool coverage' component does share conceptual overlap with TSS, and we will explicitly discuss this potential circularity in the revised paper. We will also add an LLM-as-judge cross-check on the same 200 traces as a secondary validation. We agree this is more than a limitation footnote—it affects the paper's central actionable claim—and we will elevate the discussion accordingly. revision: yes

  2. Referee: Definition 4 (AC) uses exact key-value Jaccard similarity, so semantically equivalent arguments contribute zero overlap. This may artificially deflate AC, inflate the TSS>AC gap, and make the 'argument variance is benign' finding partly tautological.

    Authors: This is a fair and important point. We agree that the exact-match AC metric conflates surface-form variation with genuine semantic divergence, and that this framing has implications for both the TSS>AC gap and the AC-correctness null result. In the revision, we will: (1) implement a semantically normalized AC variant (AC_sem) that applies case-insensitive matching, date-format normalization (parsing to ISO 8601), and basic string normalization (stripping whitespace, punctuation); (2) report AC_sem alongside the original AC for all key results; (3) re-run the TSS>AC comparison and the AC-correctness correlation with AC_sem. If AC_sem is substantially higher than AC and the AC_sem-correctness correlation becomes significant, we will revise the 'argument variance is benign' claim to be more nuanced—e.g., 'surface-form argument variance does not predict failure, but this may partly reflect the metric's semantic insensitivity.' We will also reframe the §6.1 discussion to acknowledge that the semantic equivalence of argument variants is both a finding about agent behavior and a limitation of the metric, rather than presenting it purely as a discovery. We note that even with normalization, there will remain cases where arguments are genuinely semantically different (e.g., searching 'budget Q1' vs. 'expenses March'), so we do not expect AC_sem to fully converge to TSS. But the referee's point that the current metric may be too coarse is well-taken. revision: yes

  3. Referee: §5.5: No multivariate regression of correctness on both TSS and AC is reported. Given that AC is partially confounded with TSS, a regression would help disentangle whether AC has no independent effect.

    Authors: The referee is correct that the bivariate correlations alone are insufficient to disentangle the independent effects of TSS and AC on correctness, especially given the structural confound noted in Definition 4 (AC=0 when tools differ at a step). We will add a multivariate logistic regression (or linear regression on the binary correctness outcome) with TSS and AC as joint predictors, and report standardized coefficients, p-values, and model fit statistics. We will also run a version with AC_sem (from the normalized metric above) as an additional predictor. This will allow us to determine whether AC has no independent predictive value beyond what TSS captures, or whether the null result was an artifact of the metric's syntactic sensitivity and the confound. We will revise the practical guideline in §6.3 (guideline 4) to be conditional on these regression results—if AC_sem shows an independent effect, we will recommend monitoring argument consistency in addition to TSS. If it does not, we will state this more cautiously than the current guideline, which we agree is stronger than the evidence warrants. revision: yes

  4. Referee: §4.3: Temperature is fixed at 1.0 for all experiments. Without a T=0 baseline, it is unclear whether the structural/parametric pattern and the TSS-correctness relationship hold at lower temperatures, limiting generalizability of deployment recommendations.

    Authors: We agree that the absence of a temperature ablation is a significant limitation for the deployment recommendations, and we acknowledge that T=1.0 represents the high-variance end of the deployment spectrum. The referee is right that many production systems use T=0 or low temperature for reproducibility, and our guidelines should speak to those settings. In the revision, we will: (1) run a subset of the benchmark (at least 8 of 19 tasks, covering all five categories) at T=0 and T=0.5 for 2–3 representative models (e.g., GPT-4.1-mini, Claude Sonnet 4, Llama 3.3 70B) with 10 runs each; (2) report TSS, AC, and correctness at each temperature; (3) discuss whether the structural/parametric pattern and the TSS-correctness relationship hold, weaken, or disappear at lower temperatures. We expect TSS to increase at T=0 (potentially ceiling effects) and the TSS-correctness relationship to weaken if structural variance is compressed, but this is an empirical question. We will be transparent that a full temperature ablation across all 19 tasks and 6 models is beyond the scope of this revision, and we will frame the partial ablation as preliminary evidence rather than a definitive answer. We will also temper the deployment guidelines in §6.3 to explicitly note the temperature regime under which they were validated and caution against extrapolating to T=0 without the ablation data. We note that some providers (e.g., o1) do not expose temperature as a parameter, which complicates a fully systematic ablation, but this does not excuse the absence of the T=0 baseline for the models where it is available. revision: partial

Circularity Check

0 steps flagged

No significant circularity. Metrics are independently defined from first principles; the correctness rubric is specified independently of the consistency metrics; no prediction reduces to a fitted input by construction.

full rationale

The paper defines TSS (Definition 3) via Levenshtein distance over tool-name sequences and AC (Definition 4) via Jaccard similarity over exact key-value pairs. The correctness rubric (Section 4.5) is independently specified with three components (tool coverage, argument validity, output completeness) using regex patterns. No metric is defined in terms of the outcome it predicts. The central finding—that TSS predicts correctness (d=0.81) while AC does not (r=0.12)—is an empirical observation, not a definitional consequence: a consistently wrong agent would have high TSS and low correctness, which would break the relationship, so the finding is not forced by construction. The paper has no load-bearing self-citations; Mehta et al. (2026) is cited for context and comparison, not as a premise for any derivation. The skeptic's concern about AC's exact string matching inflating the TSS>AC gap and making the 'argument variance is benign' finding partly tautological is a validity threat (the correctness rubric uses case-insensitive regex while AC uses exact matching, creating an asymmetry), but it is not circularity: AC could still predict correctness if argument variation were semantically meaningful enough to cause rubric failures, and the paper acknowledges this limitation in §6.1 and §6.5. Similarly, the partial overlap between TSS (tool sequence consistency) and the correctness rubric's 'required tool coverage' criterion is a confound, not circularity—an agent that consistently selects wrong tools would have high TSS but low correctness, so the positive TSS-correctness relationship is informative rather than tautological. Score 1 reflects the minor metric-design interaction that the paper itself flags as a limitation, but the derivation chain is self-contained.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities (particles, forces, dimensions). Free parameters are fixed experimental conditions (temperature, N, max iterations), not fitted constants. The key axiom is the correctness rubric, which is ad hoc to the paper and unvalidated by human judgment.

free parameters (3)
  • Temperature = 1.0
    Set to 1.0 (provider default) for all experiments. Not fitted to data but a fixed experimental condition. Acknowledged as a limitation.
  • N (runs per cell) = 10
    Number of repeated invocations per task-model pair. Chosen by design, not fitted.
  • Max iterations = 10
    Maximum tool-calling loop iterations per run. Fixed design parameter.
axioms (4)
  • domain assumption LLM tool-calling behavior is stochastic at temperature > 0, and repeated identical invocations yield different traces.
    Invoked in Section 3.1 and Section 4.3 to justify measuring consistency across repeated runs. Standard assumption for LLM sampling.
  • ad hoc to paper The correctness rubric (required tool coverage, argument validity, output completeness) is a valid proxy for true task success.
    Invoked in Section 4.5 and used in Section 5.5 to establish that TSS predicts correctness. This is the weakest link; the rubric is regex-based and not semantically validated.
  • domain assumption Deterministic simulated tools isolate LLM variance from environmental non-determinism.
    Invoked in Section 4.2. Reasonable by construction, but may overestimate real-world consistency (acknowledged in Section 6.5).
  • ad hoc to paper Tool-name sequences are the primary structural unit of agent behavior.
    Invoked in Section 3.1 (Definition 1) and motivates the TSS metric. The choice to separate structural (tool names) from parametric (arguments) layers is a modeling decision that drives all findings.

pith-pipeline@v1.1.0-glm · 14291 in / 1922 out tokens · 592282 ms · 2026-07-04T21:27:19.433670+00:00 · methodology

0 comments
read the original abstract

Large language model (LLM) agents with tool-calling capabilities are increasingly deployed in production systems, yet a fundamental reliability question remains under-explored: does the same agent behave the same way twice? We present a systematic empirical study of behavioral consistency in multi-step tool-calling agents, measuring whether agents select the same tools, in the same order, with the same arguments, across repeated identical invocations. Unlike prior work on consistency in ReAct-style agents(search-only, free-text actions), we study the richer setting of structured tool-calling interfaces with typed parameters and consequential side effects.

Figures

Figures reproduced from arXiv: 2605.28840 by Abel Yagubyan.

Figure 1
Figure 1. Figure 1: The “structural consistency, parametric variance” pattern. Two independent runs of the same agent [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Model comparison across TSS (left) and AC (right) with 95 % CIs. Llama 3.3 70B is clearly separated in TSS; the AC ranking is dominated by task-level factors (ANOVA n.s.). 5.2 Ambiguity Is the Dominant Driver of Inconsistency Task category strongly affects consistency ( [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ambiguous tasks show the largest consistency drop; difficulty has a modest additional effect [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of first divergence points. 60 % of divergence originates in steps 1–2, enabling lightweight [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Left: Correctness rate climbs monotonically across TSS tertiles (61 %→77 %→90 %), broken out by category. Right: AC shows no relationship with correctness (r = 0.12, p = 0.31, n.s.), confirming that argument-level variance is benign. This enables a concrete production workflow: (a) run the agent on each task type k times to establish a reference tool sequence; (b) in production, compare each run’s first tw… view at source ↗
Figure 6
Figure 6. Figure 6: TSS across models and task categories. Llama 3.3 70B shows notably lower consistency on ambiguous and composition tasks; proprietary models remain consistently high. 6.5 Limitations • Sample size. N = 10 runs per cell yields split-half reliability r = 0.66. Aggregate results are well-powered (1,140 traces), but per-cell CIs are wide. Future work should use N ≥ 30. • Single temperature. We measure at T = 1.… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 3 internal anchors

  1. [1]

    AI Agents That Matter

    Kapoor, S. and Narayanan, A. AI Agents That Matter . arXiv preprint arXiv:2407.01502, 2024

    work page Pith review arXiv 2024

  2. [2]

    Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles

    Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. In Advances in Neural Information Processing Systems, 2017

    work page 2017

  3. [3]

    API-Bank : A Comprehensive Benchmark for Tool-Augmented LLMs

    Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., and Li, Y. API-Bank : A Comprehensive Benchmark for Tool-Augmented LLMs . In Proceedings of EMNLP, 2023

    work page 2023

  4. [4]

    AgentBench : Evaluating LLMs as Agents

    Liu, X., Yu, H., Zhang, H., et al. AgentBench : Evaluating LLMs as Agents. In Proceedings of ICLR, 2024

    work page 2024

  5. [5]

    Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

    Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. In Proceedings of ACL, 2022

    work page 2022

  6. [6]

    When Agents Disagree With Themselves: Measuring Behavioral Consistency in LLM -Based Agents

    Mehta, A., Ramesh, A., and Singla, A. When Agents Disagree With Themselves: Measuring Behavioral Consistency in LLM -Based Agents. arXiv preprint arXiv:2602.11619, 2026

    work page arXiv 2026

  7. [7]

    Gorilla: Large Language Model Connected with Massive APIs

    Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla : Large Language Model Connected with Massive APIs . arXiv preprint arXiv:2305.15334, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    True Few-Shot Learning with Language Models

    Perez, E., Kiela, D., and Cho, K. True Few-Shot Learning with Language Models. In Advances in Neural Information Processing Systems, 2022

    work page 2022

  9. [9]

    ToolLLM : Facilitating Large Language Models to Master 16000+ Real-World APIs

    Qin, Y., Liang, S., Ye, Y., et al. ToolLLM : Facilitating Large Language Models to Master 16000+ Real-World APIs . In Proceedings of ICLR, 2024

    work page 2024

  10. [10]

    and Guven, E

    Renze, M. and Guven, E. Self-Reflection in LLM Agents: Effects on Problem-Solving Performance. arXiv preprint arXiv:2405.06682, 2024

    work page arXiv 2024

  11. [11]

    Toolformer : Language Models Can Teach Themselves to Use Tools

    Schick, T., Dwivedi-Yu, J., Dess \`i , R., et al. Toolformer : Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems, 2023

    work page 2023

  12. [12]

    Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting

    Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design. arXiv preprint arXiv:2310.11324, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Wang, X., Wei, J., Schuurmans, D., et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In Proceedings of ICLR, 2023

    work page 2023

  14. [14]

    Ethical and social risks of harm from Language Models

    Weidinger, L., Mellor, J., Rauh, M., et al. Ethical and Social Risks of Harm from Language Models. arXiv preprint arXiv:2112.04359, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  15. [15]

    ReAct : Synergizing Reasoning and Acting in Language Models

    Yao, S., Zhao, J., Yu, D., et al. ReAct : Synergizing Reasoning and Acting in Language Models. In Proceedings of ICLR, 2023

    work page 2023