Pith. sign in

REVIEW 5 major objections 5 minor 25 references

When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Careful supervised fine-tuning, not complex reinforcement learning, fixes multilingual API-call language errors.

desk verdict Useful failure-mode definition and a plausible SFT-vs-RL story, but the main comparison is unverifiable because the same GRPO/RM-3 configuration has different numbers in Tables 4, 5, 6, and 7. read the letter →

arxiv 2608.11715 v1 pith:LDUSE6FL submitted 2026-08-12 cs.CL cs.AI

Siddharth Chauhan , Thomas Butler , Abhishek Singhania , Pankaj Porwal , Honey Gupta This is my paper · ORCID
classification cs.CLcs.AI
keywords argumentlanguagemismatchmultilingualtoolusefunctioncallingbenchmarksupervisedfine-tuningreinforcementlearningGRPOcross-lingualtransferpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multilingual API calling has a failure mode the paper names Argument Language Mismatch (ALM): the model picks the right tool and arguments but writes argument values in the wrong language, e.g., a Spanish currency amount as "dollars" rather than "dólares". The paper's aim is to test whether eliminating ALM requires sophisticated reinforcement learning or whether ordinary supervised fine-tuning suffices. On a parallel five-language extension of the BFC function-calling benchmark, it finds that SFT alone substantially raises both argument-language consistency and end-to-end function call accuracy, and that with validation-based checkpoint selection SFT matches or beats PPO and GRPO on the learnability split. The paper concludes that most multilingual API-grounding performance comes from careful supervised training and data construction, with RL contributing targeted gains on unseen APIs, cross-lingual transfer, and reasoning preservation. This matters because teams building multilingual agents need to know where to spend post-training effort.

What carries the argument

The paper's central named object is Argument Language Mismatch (ALM), a failure where the tool and argument names are correct but argument values are generated in a language inconsistent with the user's utterance. The machinery that carries the argument is a five-step hierarchical evaluation chain—Tool Invocation Detection, Tool Selection Accuracy, Argument Completion Accuracy, Argument Language Consistency, and Function Call Match—whose strict ordering FCM ≤ ALC ≤ ACA ≤ TSA ≤ TID lets the authors separate language errors from structural errors. On the training side, the load-bearing mechanism is RM-3, an argument-factorized reward that scores each argument value's language on a graded scale (exact match, minor variation, mismatch) and combines those scores with structural gates, giving the policy per-argument credit; combined with GRPO's group-relative advantage normalization, this reward design produces the paper's strongest RL results while remaining stable under token-level reward weighting. The benchmark itself is a translated, parallel version of a human-annotated function-calling dataset built with a rule-based protocol that translates user-derived values while preserving canonical identifiers and API names.

What would settle it

Take a random sample of the translated benchmark's 832 ALM-relevant turns, replace the automated judge's language-consistency scores with human annotations, and compare best-checkpoint SFT against GRPO; if the SFT advantage (e.g., 67.4 vs 55.3 FCM on Split-1) shrinks or reverses under human labels, the paper's central claim is an artifact of judge or translation bias.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that Argument Language Mismatch is a generative conditioning failure, not a structural or intent failure, and that this kind of failure is largely learnable by imitation. The paper defines a strict metric hierarchy TID ≥ TSA ≥ ACA ≥ ALC ≥ FCM, isolating language consistency from tool selection and argument completion, and shows that SFT on 832 ALM-relevant turns from a translated function-calling benchmark lifts ALC and FCM far above the base model on both a high-API-overlap learnability split and a low-overlap generalization split. At best validation checkpoints on Split-1, SFT reaches 79.1 ALC and 67.4 FCM, exceeding GRPO (74.0 and 55.3) on both, and exceeding SFT+GRPO (79.3 and 61.3) on FCM. RL's remaining value appears in generalization: GRPO improves cross-lingual ALC consistently across Italian, Dutch, and French, whereas SFT's gains are uneven and include negative transfer on Dutch, and GRPO preserves English reasoning on MGSM while the best-checkpoint SFT model drops from 70.8 to 62.2. The paper concludes that even the strongest argument-factorized reward design, RM-3, improves RL only incrementally over a strong SFT baseline, and that careful supervised training and honest model selection should be the default for structured multilingual generation.

Load-bearing premise

The whole ranking rests on the assumption that translation and an automated language judge reproduce how real multilingual API calls fail; if translation artifacts or judge bias drive the ALC scores, the SFT-over-RL conclusion may not transfer to natural use.

Editorial extensions

If this is right

  • Multilingual agent teams should treat a strong, validation-selected SFT model as the default and reserve RL for generalization or multi-objective needs, because best-checkpoint SFT already beats GRPO on end-to-end FCM in the learnability split.
  • Language consistency should be scored separately from tool selection and argument completion; the hierarchy TID→TSA→ACA→ALC→FCM turns ALM from a hidden failure into a measurable diagnostic.
  • If RL is used, argument-factorized rewards (RM-3) are the main driver of gains; sparse or stepwise rewards leave language consistency under-optimized, and token-level upweighting works under GRPO but destabilizes PPO.
  • RL's honest advantage is generalization: GRPO transfers the 'match argument language to user locale' rule to unseen APIs and languages more consistently than SFT, and preserves general reasoning in English where the best-checkpoint SFT model loses 8.6 MGSM points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paper's account is right, the SFT-vs-RL ranking should extend to other surface-level conditioning problems in structured generation, such as localized date, number, and unit formats in code or SQL; a quick test would be applying the same metric hierarchy to a code-generation task with locale-dependent literals.
  • The benchmark's translation protocol explicitly enforces utterance-value coherence, so natural multilingual tool-use data with code-switching or mixed-language inputs may behave differently; collecting such logs and re-running the comparison would show whether SFT's dominance survives realistic language mixing.
  • Since the paper's ALM-aware prompt intervention already recovers part of the ALC gap (+7.5 points) without any training, a constrained-decoding or language-identifier-conditioned variant of SFT may close much of the remaining RL margin, which the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper formalizes Argument Language Mismatch (ALM), a failure mode in multilingual API calling where a model selects the correct tool but generates argument values in the wrong language. It constructs a multilingual extension of the Berkeley Function Calling benchmark and compares supervised fine-tuning (SFT) with PPO and GRPO under sparse, stepwise, and argument-factorized reward models. The central claim is that SFT is a strong baseline, achieving performance comparable to and sometimes exceeding RL, while RL provides only incremental gains concentrated in generalization and multi-objective trade-offs.

Significance. If validated, the paper would offer a useful formalization of ALM, a reproducible translated benchmark, and a cautionary result about the importance of strong SFT baselines in structured multilingual generation. The systematic comparison of reward granularity and the cross-lingual evaluation are valuable contributions. However, the significance is currently tempered by unresolved internal-reporting inconsistencies and by the absence of calibration or error bars for the primary metric, so the headline SFT-versus-RL comparison cannot be fully verified as written.

major comments (5)
  1. [Tables 4–7, Section 5.5–5.7] The same configuration (GRPO, RM-3 reward, Qwen2.5-14B, best validation checkpoint, Split-1) is reported as 74.0/55.3 ALC/FCM in Table 4 and Table 5, as 74.04/55.32 in Table 7, but as 81.2/66.9 in Table 6. Since the central claim in Section 5.3 that SFT (79.1/67.4) exceeds GRPO depends on the 74.0/55.3 numbers, the headline conclusion is not verifiable from the paper as written. Please reconcile these values and explicitly label the split and checkpoint-selection protocol for every table.
  2. [Section 6 and Appendix F] Section 6 states that 'SFT yields limited improvements in language consistency, whereas GRPO with argument-level rewards learns a transferable rule,' and Appendix F.7 concludes that 'outcome-driven optimization via argument-factorized reinforcement learning is necessary to achieve robust multilingual grounding.' These statements directly contradict the abstract, Section 5.2, and Section 7, which argue that SFT is a strong baseline and RL gains are incremental. Please resolve this contradiction or clearly distinguish the settings to which each statement applies.
  3. [Section 5.8, Table 8] The text says GRPO 'demonstrates stronger generalization than SFT' on cross-lingual transfer, but Table 8 shows SFT achieving a higher average ALC (57.88 vs. 57.72) and larger absolute improvements on Italian and French, with GRPO only better on Dutch. The claim as stated is not supported by the displayed numbers.
  4. [Section 2.3, Section 4.2, Appendix J] The primary evaluation metric ALC is computed via an LLM judge described in Appendix J, but the paper reports no calibration against human labels, no inter-annotator agreement, and no confidence intervals or standard deviations across training seeds. Since ALC is the basis for the main SFT-versus-RL comparison and for all reward models, please provide judge accuracy, human agreement, and error bars for the main tables.
  5. [Section 8.4, Appendix G] The paper acknowledges that the translated benchmark 'may introduce artifacts or reduce variability compared to naturally occurring multilingual data.' The translation prompt in Appendix G explicitly enforces utterance-value coherence and localizes argument values, which may make the task unusually learnable by SFT. Please add an analysis on natural (untranslated) multilingual API failures or explicitly scope the conclusions to translated benchmarks.
minor comments (5)
  1. [Section 3.2, Appendix A.3] Section 3.2 says the benchmark covers 'five languages' but lists only Spanish, French, Italian, and Dutch, while Appendix A.3 says 'five additional languages' including Hindi. Please correct the inconsistency.
  2. [Table 2] Table 2 contains stray 'High' and 'Low' labels that do not correspond to any column header; please remove them.
  3. [Equations (2) and (4)] The notation for ALCcont is inconsistent: Eq. (2) uses 1/(K*I) without defining K and I, while Eq. (4) uses 1/K. Please define the indexing or simplify the notation.
  4. [Tables 6 and 7] Tables 6 and 7 do not state which split they report; since Section 5.1 says results use Split-1 unless otherwise noted, please label each table explicitly.
  5. [Abstract and Introduction] The phrase 'visiting post-training strategies' in the Introduction appears to be a typo for 'revisiting'; please fix.

Circularity Check

1 steps flagged · score 1.0 of 10

No load-bearing circularity; one minor definitional entanglement where the RM-3 reward equals the reported ALC metric by construction, while the central SFT-versus-RL comparison remains an independent empirical result.

  1. self definitional [Section 4.2.3 (Eq. 4); Section 4.2.2 (ALC threshold); evaluation judge in Appendix J]
    "RM-3 combines discrete structural gates with continuous argument-level rewards: ... ALCcont(Y) otherwise. ... The graded argument-level score is defined as: ALCcont(Y) = 1 K X i,k S(vi,k). ... A binary language-consistency score is obtained by thresholding this value: ALC= 1 if ALCcont≥1.8 (i.e., at least 90% of the maximum score 2.0), and ALC= 0 otherwise."

    By construction, the RM-3 training reward for structurally correct outputs is exactly the continuous judge score ALCcont(Y) (Eq. 4), and the primary reported metric ALC is defined as that same judge score thresholded at 1.8 (Section 4.2.2), using the same LLM judge prompt in Appendix J. Under RM-3, the reward is, by definition, the graded precursor of the reported ALC metric, so claims that RM-3 yields the highest ALC and that GRPO with RM-3 improves ALC partially restate the training objective rather than an outcome measured independently of it.

full rationale

The paper contains no self-citations, so patterns 3, 4, and 5 do not apply: every cited basis (BFC benchmark, GRPO, PPO, RLHF, tool-use datasets) is external, and no load-bearing argument reduces to prior work of the present authors. The benchmark is an external dataset (BFC) that is translated rather than generated to fit the hypothesis, and the central claim is an empirical ranking between SFT and RL methods; checkpoints are selected on validation FCM, not on the reported test metrics, so the comparison is not a fitted parameter renamed as a prediction. Cross-lingual transfer is a genuine held-out evaluation (trained on Spanish, tested on Italian, Dutch, and French), so those results are not forced by construction. The one definitional entanglement, flagged as a step above, is that the RM-3 reward equals the continuous ALC metric built from the same LLM judge; per the reviewing rule, the paper's own limitation statement in Section 8.4 ('our study focuses on a translated benchmark, which may introduce artifacts or reduce variability compared to naturally occurring multilingual data') is weighed here: it is a construct-validity risk, not a circularity. Similarly, the unreconciled GRPO/RM-3 numbers across Tables 4, 5, 6, and 7 (74.0/55.3 vs. 81.2/66.9 vs. 74.04/55.32) and the absence of human calibration for the LLM judge are correctness and verifiability concerns, not cases of a derivation equating its output to its input by construction. The derivation chain is therefore substantially self-contained, and only a minor, openly stated metric-reward identity keeps the circularity score above zero.

Assumptions & free parameters 7 free parameters · 4 assumptions · 1 invented entities

The central claim relies on the assumption that language consistency with the user is the right correctness requirement, that the LLM judge measures it reliably, and that the LLM-translated BFC corpus preserves realistic multilingual behavior. The paper also filters to ALM-relevant turns, and several reward and sampling hyperparameters are hand-chosen. No physical entities are introduced; ALM is a definitional construct rather than an independently evidenced phenomenon.

free parameters (7)
  • RM-2 ALC binary threshold = 1.8
    Section 4.2.2 defines binary ALC as ALC_cont >= 1.8; this hand-chosen threshold changes the reward signal and the measured ALC.
  • RM-2 language scores = 2.0 / 1.5 / 1.0
    Hand-assigned judge scores in Section 4.2.2; the scale determines the density of the hierarchical reward.
  • RM-3 language scores = 2.5 / 2.0 / 1.0
    Hand-assigned judge scores in Section 4.2.3 and Eq. (3); these values are central to the reported RM-3 gains.
  • Token reward weight beta = 1.5 and 3
    Section 4.3; beta=3 improves GRPO but destabilizes PPO, so this hyperparameter drives a headline comparison.
  • GRPO sampling configuration = K=8, T=0.6, top-p=0.95
    Appendix K fixes group size and sampling; these choices affect group-relative advantages and exploration.
  • ALM-turn selection criteria = 832 turns, 16.35%
    Section 3.3 selects turns with translated arguments or baseline ALM; this filtering defines the benchmark and all reported metrics.
  • API overlap split thresholds = 17% and 6%
    Appendix A.6 manually sets train/test API overlap; the learnability versus generalization distinction depends on these thresholds.
assumptions (4)
  • domain assumption Language consistency with the user is a correctness requirement for API arguments unless the schema says otherwise.
    Stated in Section 2.1 as the definition of correct grounding; if this is wrong, optimizing ALC targets an artificial constraint.
  • domain assumption An LLM judge reliably assigns language scores to argument values.
    ALC and rewards RM-2/RM-3 depend on judge outputs in Eq. (2), Eq. (4), and Appendix J; no calibration, agreement, or sensitivity analysis is reported.
  • domain assumption The LLM-based translation protocol preserves the distribution and difficulty of natural multilingual API calls.
    Appendix G uses a translation prompt that enforces utterance-value coherence; the authors concede in Section 8.4 that a translated benchmark may introduce artifacts.
  • ad hoc to paper The selected ALM-relevant turns are representative for measuring ALM improvements.
    Section 3.3 restricts training and evaluation to 16.35% of the dataset selected partly by baseline ALM behavior, so results may not represent the full benchmark.
invented entities (1)
  • Argument Language Mismatch and the ALC metric
    purpose: Names and measures a failure mode in multilingual API calling; ALC is both the evaluation metric and the reward target.
    No independent handle outside the paper: operational validity rests on the language-consistency assumption and the uncalibrated LLM judge; standard benchmarks do not measure it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use." pith.science (2026). https://pith.science/paper/LDUSE6FL

@misc{pith2026260811715,
  author       = {Pith},
  title        = {Pith review of: When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LDUSE6FL}},
  note         = {Machine review of arXiv:2608.11715}
}
read the original abstract

The reliability of Large Language Models (LLMs) for API calling degrades in multilingual settings. A common failure occurs when a model selects the correct tool but generates argument values in an inconsistent language, which we term Argument Language Mismatch (ALM). Although semantically correct, such outputs are operationally invalid and not captured by standard API-calling metrics. We revisit post-training strategies for mitigating ALM and find that, in our benchmark, supervised fine-tuning (SFT) provides a strong baseline, substantially improving argument language consistency and end-to-end function call accuracy. Under consistent model selection, SFT achieves performance comparable to, and sometimes exceeding more complex reinforcement learning (RL) approaches. We further examine whether RL with structured, argument-aware rewards offers additional benefits. While methods such as Group Relative Policy Optimization (GRPO) can improve language consistency and better preserve general reasoning ability, these gains are incremental and most pronounced in generalization and multi-objective trade-offs. Overall, our results suggest that much of the performance in multilingual API grounding can be achieved through careful supervised training, with RL providing targeted rather than fundamental improvements.

Figures

Figures reproduced from arXiv: 2608.11715 by the authors.

Figure 1
Figure 1. Overview of our language-consistent API grounding framework. Given a multilingual user request and available tool specifications, the model generates candidate API calls. A structured reward evaluates both structural correctness and whether argument values match the language of the user input. Reinforcement learning with argument-aware rewards encourages the model to produce language-consistent arguments, improving … view at source ↗
Figure 2
Figure 2. Reward granularity increases from RM-1 to RM-3. RM-1 provides sparse, response-level feedback; RM-2 introduces hierarchy-aware step rewards; RM-3 factorizes reward across argument values, enabling fine-grained credit assignment for mitigating Argument Language Mismatch (ALM). consistency is considered successful, the continuous score is used within RM-2 to distinguish between complete language failure (ALCcont ≤ 1.0… view at source ↗
Figure 3
Figure 3. visualizes the distribution of function calls per turn for BFC and APIGen. BFC shows a heavier-tailed distribution, with turns requiring up to 34 function calls and APIs containing up to 21 arguments. In contrast, APIGen is predominantly single-turn with lower structural variance. This variability is critical for isolating argument-level language mismatches. 0 1 2 3 4 5 6+ 0 1,000 2,000 Function Calls per Turn # Tur… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 23 canonical work pages

  1. [1]

    Structural Richness APIGen provides the largest supervision signal (60,000 samples), but exhibits strictly single-turn structure with Turns/Dialog= 1.00

    Scale vs. Structural Richness APIGen provides the largest supervision signal (60,000 samples), but exhibits strictly single-turn structure with Turns/Dialog= 1.00. In contrast, BFC contains fewer total samples but demonstrates substantially higher structural variance: Turns/DialogBFC = 1.47 (max = 7). This difference increases contextual dependency and cr...

    work page

  2. [2]

    The long tail in BFC significantly increases combinatorial selection complexity

    Tool Selection Difficulty BFC exposes the model to an average of Tools/TurnBFC = 3.99 (max = 37), compared to 1.00 in API-Bank and 4.85 (max 11) in ToolAlpaca. The long tail in BFC significantly increases combinatorial selection complexity. The probability of correct API selection under uniform choice decreases inversely with candidate set size, making BF...

    work page

  3. [3]

    This is the highest among human-annotated benchmarks

    Argument-Level Complexity Argument entropy is measured via the average number of parameters per API: Args/APIBFC = 2.37 (max = 21). This is the highest among human-annotated benchmarks. Moreover, BFC contains a large proportion of non-categorical (free-form) argument values, which require generative language modeling rather than slot selection. Since ALM ...

    work page

  4. [4]

    Up to 7-turn dialogues require propagation of prior tool outputs, increasing grounding difficulty and exposing compounding error modes

    Dialogue Structure and Context Propagation Unlike APIGen and ToolAlpaca, BFC contains multi-turn interactions with dependency chains across turns. Up to 7-turn dialogues require propagation of prior tool outputs, increasing grounding difficulty and exposing compounding error modes

    work page

  5. [5]

    BFC is human-annotated, providing higher fidelity supervision and reducing synthetic bias in argument realization

    Annotation Fidelity Datasets such as APIGen and Glaive FC are fully synthetic, introducing potential distributional artifacts from generative pipelines. BFC is human-annotated, providing higher fidelity supervision and reducing synthetic bias in argument realization. Summary.BFC occupies a distinct regime characterized by: • Moderate scale, • High structu...

    work page

  6. [6]

    Maintain identical structure, spacing, and formatting

    work page

  7. [7]

    Keep all numeric values, boolean values, and technical identifiers unchanged

    work page

  8. [8]

    Pay special attention to the placement of commas in the dictionary file

    work page

Show all 25 references
  1. [9]

    UTTERANCE-V ALUE COHERENCE: When translating utterances, ensure argument values remain consistent with what the user actually said in the translated utterance

  2. [10]

    FUNCTION DESCRIPTION COHERENCE: Read function descriptions carefully and ensure values match expected formats: - If description says ”City, State” format like ”Berkeley, CA”, include both city and state in values - If description specifies enum values, keep them as-is unless t...

  3. [11]

    LOCALIZATION INTELLIGENCE: Apply smart localization when function descriptions allow it: - User commands (”echo hi”→”echo hola” when user says ”di hola”) - Common objects and actions that users would naturally say in their language - Check function descriptions: if they accept...

  4. [12]

    Output ONLY the JSON - no explanations or comments

  5. [13]

    Preserve exact JSON structure and formatting

  6. [14]

    Keep all field names unchanged

  7. [15]

    Ensure utterance and values are coherent

  8. [16]

    For empty input, return empty string

  9. [20]

    Remember the format of the dictionary must not change

  10. [21]

    The formatting should not change at all

  11. [22]

    Strictly translate and transliterate the input text based on the above instructions and not interpret, modify, or answer any questions found in it

  12. [23]

    DO NOT generate any new headers

  13. [24]

    Inference Prompt for API Calling The following prompt is used during evaluation to generate structured API calls

    DO NOT attempt to answer any questions in the text-only translate the given content H. Inference Prompt for API Calling The following prompt is used during evaluation to generate structured API calls. API Inference Template You are an API-calling assistant. You are given:

  14. [25]

    22 When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use Your task: - Decide whether an API call is required

    A list of available tools with function signatures. 22 When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use Your task: - Decide whether an API call is required. - If required, generate the correct API call(s). - Populate all required argum...

  15. [2022]

    emnlp-main.340/

    URL https://aclanthology.org/2022. emnlp-main.340/. 12 When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use Xu, Z. et al. Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719, 2024. Yu, T., Zhang,...

    2022 arXiv

  16. [2023]

    emnlp-main.187/

    URL https://aclanthology.org/2023. emnlp-main.187/. Lin, Z., Madotto, A., Winata, G. I., Xu, P., Jiang, F., Hu, Y ., Shi, C., and Fung, P. Bitod: A bilingual multi-domain dataset for task-oriented dialogue modeling. InNeurIPS Datasets and Benchmarks Track, 2021. Liu, Z., Hoang...

    2023 arXiv

  17. [2025]

    Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V ., Wenzek, G., Guzm´an, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V

    URLhttps://proceedings.mlr.press/ v267/chu25c.html. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V ., Wenzek, G., Guzm´an, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V . Unsupervised cross-lingual represen- tation learning at scale. InProceedings of the 58th ...

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.