Pith. Sign in

REVIEW

Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding

PaperPDF from arXiv

data files

ReviewThe objections and verdict, as JSON Paper breakdownThe authors' claims, conditions and measurements, as JSON Signed receiptProof this review has not been altered
X Bluesky LinkedIn Reddit Hacker News
Daniel Maninger, Leon Chemnitz, Jannis Brugger, Tushar Lamba, Amir Molzam Sharifloo, Mira Mezini

Software Engineering · Submitted 7 Jul 2026 · 59 references

Pith's verdict: CD contribution is solid and immediately deployable; RAG+CD appendix tables are duplicated, undermining RQ3 verifiability.

T0 · AI review

How this review was made: An AI referee (glm-5.2) read the paper on July 8, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

  1. T0 AI review An AI read the paper and answered a fixed list of questions.
  2. T1 Audit Automatic checks that the references and files the paper relies on exist and match.
  3. T2 Compute The paper's code or calculations re-run with the same results.
  4. T3 Formal statement The claims written out in a formal math language a computer can read.
  5. T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 3 major objections · 6 minor · Ask about this paper · Read the full review →
Challenge this review Comment (0) Read on arXiv ↗
Cite ▾
arXiv ID2607.05936 (version 1)
Pith IDTDVAACU5, Pith's permanent record of this paper and its review
CitationAmir Molzam Sharifloo, Daniel Maninger, Jannis Brugger, Leon Chemnitz, Mira Mezini, Tushar Lamba. "Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding." arXiv:2607.05936 (2026). https://pith.science/paper/TDVAACU5
Cite this reviewBibTeX for Pith's review of this paper
Claim paper

Why Pith says this

Highlight any text to comment on it.

Referee report

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

Constrained decoding zeroes hallucinated API calls

The reading

This paper tackles the problem of LLMs generating incorrect web API invocation code—code that calls endpoints that don't exist, uses wrong HTTP methods, or passes arguments not defined in the API specification. The authors evaluate two mitigation techniques: retrieval-augmented generation (RAG), which retrieves relevant endpoint documentation from OpenAPI specifications and injects it into the model's prompt, and constrained decoding (CD), which automatically translates OpenAPI specifications into regular-expression constraints that block the model from generating any URL, HTTP method, or argument not permitted by the specification. The central mechanism is the constraint generator: a system that parses an OpenAPI specification, handles interdependencies between methods, URLs, and parameter locations (path, query, header, body), and produces hierarchical regex-based generation rules enforced token-by-token during decoding. The key finding is that CD eliminates specification violations entirely—zero hallucinated endpoints or arguments across all 24 evaluated models, both datasets, and both experimental setups—while also substantially improving overall correctness, with average gains of 143% to 209% on the synthetic dataset. RAG, by contrast, reduces hallucinations but has ambivalent effects on correctness: it helps when the model must select the endpoint itself (full completion) but can hurt when the endpoint is already provided, because models tend to over-include optional parameters they see in the retrieved documentation. The combination of RAG and CD achieves the highest peak correctness for some models but is less consistent than CD alone.

What carries the argument

The key machinery is the automatic constraint translator: a system that takes an OpenAPI specification and produces a set of hierarchical generation rules, each consisting of a regex-encoded start condition, stop condition, and body. The rules are triggered sequentially—first matching the HTTP method, then narrowing the URL space to endpoints supporting that method, then narrowing parameters to those defined for the identified endpoint, and finally constraining parameter values by data type. Capture groups and conditional patterns prevent duplicate parameters; negative lookahead assertions prevent closing a parameter list before all required parameters are present. The constraints are model-

Extended reading notes

Core claim

The paper's central discovery is that OpenAPI specifications can be automatically translated into regex-based decoding constraints that make it structurally impossible for an LLM to generate a web API invocation violating the specification—no hallucinated URLs, methods, or arguments—while simultaneously improving overall correctness across models. This works because the constraints are derived directly from the machine-readable specification rather than learned or approximated, turning a soft knowledge problem (does the model remember the API?) into a hard structural guarantee (can the model generate tokens outside the allowed space?). The paper also discovers that RAG and CD interact non-tr

Load-bearing premise

The evaluation measures correctness by comparing generated request configurations against a single ground-truth configuration per task, which may not capture the full space of valid solutions—particularly for the real-world dataset, where some tasks require no parameters beyond path variables, meaning a model can score correct simply by generating a closing parenthesis.

Editorial extensions

If this is right

  • Any externally documented interface with a machine-readable specification—database schemas, protocol buffers, GraphQL schemas—could in principle use the same specification-to-constraint translation to guarantee specification-compliant code generation.
  • The finding that RAG causes parameter over-inclusion suggests that retrieval-augmented code generation systems broadly may suffer from a 'completeness bias' where models treat retrieved documentation as a checklist rather than a reference, a problem that constrained decoding can partially mitigate by blocking unnecessary parameters.
  • The zero-hallucination guarantee from CD could shift the burden of API correctness verification from post-generation testing to generation-time enforcement, changing how developers interact with LLM-based code assistants for integration tasks.
  • The model-specific variability in RAG's effectiveness implies that production deployments of retrieval-augmented code generation need per-model validation, whereas CD can be applied more uniformly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The constraint approach could be extended beyond Axios/JavaScript to other HTTP client libraries and languages, since the OpenAPI specification is language-agnostic—only the syntax-level regexes would need adaptation.
  • The observation that CD slightly increases non-executable code rates suggests a tension between structural constraint and generation fluency; investigating whether more permissive constraint formulations (e.g., allowing multiple valid syntax patterns) could reduce this penalty would be a natural next step.
  • If OpenAPI specifications could be automatically version-detected from task descriptions, the requirement for users to manually specify the API could be removed, enabling fully automated specification-to-constraint pipelines in IDE-integrated code assistants.
  • The finding that smaller constrained models can match or exceed larger unconstrained models (e.g., Code Llama 70B with CD outperforming GPT-4o without CD on argument completion) suggests that constraint enforcement may be a more cost-effective path to correctness than scaling model size.
Original abstract

Integration of web APIs is a cornerstone of modern software systems, yet writing correct web API invocation code remains challenging due to complex and evolving API specifications. Although LLMs are increasingly used for code generation, previous work has empirically shown that their ability to generate correct web API integrations is limited. At the same time, mitigation techniques and their effectiveness for this setting remain insufficiently understood. In this paper, we propose and systematically evaluate retrieval-augmented generation (RAG) and constrained decoding (CD) as two complementary approaches to improving LLM-generated web API invocation code. For RAG, we design a retriever that processes OpenAPI specifications and retrieves compact endpoint representations to inject into model prompts. For CD, we introduce an automatic translation from OpenAPI specifications to regex-based constraints enforced during generation. We evaluate both approaches on WAPIIBench's existing synthetic dataset and on a new real-world dataset derived from GitHub repositories. Our results show that RAG reduces hallucinations and improves correctness when generating full API invocations but reduces it when the endpoint is already provided as it encourages the generation of unnecessary parameters. In contrast, CD reliably prevents illegal URLs, HTTP methods, and arguments and substantially improves overall correctness for both starter codes.

Figures
Figure 1. Benchmark design for evaluating the capabilities of LLMs in generating web API invocation code. (0) Based on an API
Figure 2. Retrieval-augmented generation framework for generating web API invocations. Instead of passing the prompt directly to
Figure 3. Constrained decoding framework for generating web API invocations. In regular autoregressive generation, a decoder takes the
Figure 4. Comparison of a task and starter code from the synthetic dataset [
Figure 5. Comparison of correct implementations between generation settings on the synthetic dataset 6.2 Vanilla Generation The results for vanilla generation on the synthetic dataset are consistent with our previous findings [25] 13. We will focus our analysis here on a few key aspects and refer the reader to our prior work for a broader discussion. The overall benchmark performance for the full completion setup is low, at an average correctness rate among open-source models of 13% and Code Llama (70B) as the best performing model at 30%. GPT-4o achieves 58%. All Qwen2.5-Coder and Llama 3.1…
Figure 6. Comparison of hallucinated endpoints between generation settings on the synthetic dataset they are unable to continue the given starter code in a syntactically valid way. The remaining models hallucinate on average 31% of all endpoints. On the real-world dataset, we observe a higher baseline performance—47% correctness on average among open-source models producing executable implementations. We attribute this to the tasks in this dataset being drawn from real-world repositories, and thus having a high likelihood of representing common use cases of wide-spread APIs that are also…
Figure 7. Comparison of correct implementations between generation settings on the real-world dataset On both datasets, models achieve generally higher correctness rates for argument completion than for full completion— 26% and 55% on average, respectively. On the synthetic dataset, Qwen2.5-Coder (14B) scores highest at 47%. Interestingly, GPT-4o-mini performs better than GPT-4o (63% vs. 59%). An important caveat for the real-world dataset is that several tasks included do not require parameters in locations other than the path, i.e., there are no further arguments to complete and all the…
Figure 8. Comparison of hallucinated endpoints between generation settings on the real-world dataset In contrast, for full completion, it is free to select any endpoint it is familiar with—even if it is the wrong choice, it is less likely to be a hallucination. Summary: Vanilla generation yields low overall correctness, particularly for full completion (13% average across open-source models on the synthetic dataset), with hallucinated endpoints affecting nearly a 1/3 of all generations. Model size is only a weak predictor of performance, and several recent models fail to produce executable…

Figures reproduced from arXiv:2607.05936 by the authors.

References

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.