{"id":"7457dce4-8c50-4712-af8c-d598e433c84e","arxiv_id":"2608.11715","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"On a new multilingual function-calling benchmark, supervised fine-tuning largely resolves argument language mismatch, and reinforcement learning adds targeted but incremental gains.","lead":"The paper defines a failure mode where LLMs select the correct API but fill argument values in the wrong language, and compares post-training methods on a new multilingual function-calling benchmark. It finds supervised fine-tuning is a strong baseline, with reinforcement learning adding targeted but incremental gains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unreconciled numbers for the same GRPO/RM-3 configuration in Tables 4, 5, 6, and 7 make the central SFT-versus-RL comparison unverifiable as reported.","rationale":"The reader identified inconsistencies in Tables 5–7 in their rationale and therefore assigned CONDITIONAL, but their stated weakest assumption was the validity of the translated benchmark and LLM judge. I agree that benchmark artifacts matter, but the more immediately load-bearing issue is internal: the same configuration is assigned different primary results in tables that feed the headline claim. The central empirical conclusion cannot be evaluated until the canonical numbers are established. If Table 6's GRPO values are correct, the strongest version of the SFT-beats-RL claim is not supported; if Tables 5 and 7 are correct, Table 6 needs correction. In either case the paper as submitted does not provide a single verified set of results for its key comparison. I would therefore withhold a final verdict pending reconciliation and release of the training/evaluation artifacts.","tokens_in":19771,"tokens_out":5259,"duration_ms":51393,"concrete_test":"Re-run or re-report the Split-1 best-checkpoint GRPO/RM-3 result under one canonical protocol, and use that value in Table 4. Concretely: take the Table 6 protocol (best checkpoint, RM-3, Qwen2.5-14B, Split 1, no token-weighting) and regenerate GRPO's ALC/FCM; if the result is 81.2/66.9, recompute Table 4 and re-derive the SFT-versus-RL conclusion; if the result is 74.0/55.3, audit Table 6 for an unstated difference. This requires releasing the benchmark, training configuration, and checkpoint-selection code.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim — that SFT is comparable to, and sometimes better than, RL for multilingual API grounding — rests on the best-checkpoint Split-1 comparison in Table 4, where SFT reaches 79.1 ALC / 67.4 FCM and GRPO with RM-3 reaches 74.0 ALC / 55.3 FCM, making SFT look clearly superior. However, the same GRPO/RM-3 configuration is reported as 81.2 ALC / 66.9 FCM in Table 6 (Section 5.6) and as 74.04 ALC / 55.32 FCM in Table 7 (Section 5.7), while Table 5 (Section 5.5) reports 74.0 ALC / 55.3 FCM. These are supposed to be the same method, reward model, split, and validation-based checkpoint selection. If Table 6's numbers are correct, the claimed SFT advantage over GRPO on Split-1 mostly disappears on FCM and reverses on ALC, so the 'sometimes exceeding' claim in the abstract and conclusion is unsupported. If Tables 5 and 7 are correct, Table 6 is inconsistent. Either way, the headline conclusion depends on a numerical comparison that is not stable across the paper's own tables. This is an internal reporting inconsistency in the evidence that directly carries the central claim, not a disagreement with external consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes Argument Language Mismatch (ALM), a failure mode in multilingual API calling where a model selects the correct tool but generates argument values in the wrong language. It constructs a multilingual extension of the Berkeley Function Calling benchmark and compares supervised fine-tuning (SFT) with PPO and GRPO under sparse, stepwise, and argument-factorized reward models. The central claim is that SFT is a strong baseline, achieving performance comparable to and sometimes exceeding RL, while RL provides only incremental gains concentrated in generalization and multi-objective trade-offs.","tokens_in":20102,"tokens_out":5282,"duration_ms":51222,"significance":"If validated, the paper would offer a useful formalization of ALM, a reproducible translated benchmark, and a cautionary result about the importance of strong SFT baselines in structured multilingual generation. The systematic comparison of reward granularity and the cross-lingual evaluation are valuable contributions. However, the significance is currently tempered by unresolved internal-reporting inconsistencies and by the absence of calibration or error bars for the primary metric, so the headline SFT-versus-RL comparison cannot be fully verified as written.","major_comments":[{"comment":"The same configuration (GRPO, RM-3 reward, Qwen2.5-14B, best validation checkpoint, Split-1) is reported as 74.0/55.3 ALC/FCM in Table 4 and Table 5, as 74.04/55.32 in Table 7, but as 81.2/66.9 in Table 6. Since the central claim in Section 5.3 that SFT (79.1/67.4) exceeds GRPO depends on the 74.0/55.3 numbers, the headline conclusion is not verifiable from the paper as written. Please reconcile these values and explicitly label the split and checkpoint-selection protocol for every table.","section":"Tables 4–7, Section 5.5–5.7"},{"comment":"Section 6 states that 'SFT yields limited improvements in language consistency, whereas GRPO with argument-level rewards learns a transferable rule,' and Appendix F.7 concludes that 'outcome-driven optimization via argument-factorized reinforcement learning is necessary to achieve robust multilingual grounding.' These statements directly contradict the abstract, Section 5.2, and Section 7, which argue that SFT is a strong baseline and RL gains are incremental. Please resolve this contradiction or clearly distinguish the settings to which each statement applies.","section":"Section 6 and Appendix F"},{"comment":"The text says GRPO 'demonstrates stronger generalization than SFT' on cross-lingual transfer, but Table 8 shows SFT achieving a higher average ALC (57.88 vs. 57.72) and larger absolute improvements on Italian and French, with GRPO only better on Dutch. The claim as stated is not supported by the displayed numbers.","section":"Section 5.8, Table 8"},{"comment":"The primary evaluation metric ALC is computed via an LLM judge described in Appendix J, but the paper reports no calibration against human labels, no inter-annotator agreement, and no confidence intervals or standard deviations across training seeds. Since ALC is the basis for the main SFT-versus-RL comparison and for all reward models, please provide judge accuracy, human agreement, and error bars for the main tables.","section":"Section 2.3, Section 4.2, Appendix J"},{"comment":"The paper acknowledges that the translated benchmark 'may introduce artifacts or reduce variability compared to naturally occurring multilingual data.' The translation prompt in Appendix G explicitly enforces utterance-value coherence and localizes argument values, which may make the task unusually learnable by SFT. Please add an analysis on natural (untranslated) multilingual API failures or explicitly scope the conclusions to translated benchmarks.","section":"Section 8.4, Appendix G"}],"minor_comments":[{"comment":"Section 3.2 says the benchmark covers 'five languages' but lists only Spanish, French, Italian, and Dutch, while Appendix A.3 says 'five additional languages' including Hindi. Please correct the inconsistency.","section":"Section 3.2, Appendix A.3"},{"comment":"Table 2 contains stray 'High' and 'Low' labels that do not correspond to any column header; please remove them.","section":"Table 2"},{"comment":"The notation for ALCcont is inconsistent: Eq. (2) uses 1/(K*I) without defining K and I, while Eq. (4) uses 1/K. Please define the indexing or simplify the notation.","section":"Equations (2) and (4)"},{"comment":"Tables 6 and 7 do not state which split they report; since Section 5.1 says results use Split-1 unless otherwise noted, please label each table explicitly.","section":"Tables 6 and 7"},{"comment":"The phrase 'visiting post-training strategies' in the Introduction appears to be a typo for 'revisiting'; please fix.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a useful benchmark and a plausible central claim, but the internal inconsistencies in the quantitative tables and in the narrative relative to Section 6 and Appendix F currently prevent verification of the main comparison. These issues are fixable with a careful consistency pass, additional labeling of experimental settings, and reporting of metric calibration and error bars. I would recommend inviting a revision rather than rejecting outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this paper because it defines a genuinely useful failure mode — Argument Language Mismatch — and asks the right question about whether RL earns its complexity on top of SFT for multilingual API calling. That is worth someone's time. But the central empirical claim is not currently supported by the paper's own tables.\n\nThe problem is specific and load-bearing. The identical GRPO/RM-3/14B configuration on Split-1 is reported as 74.0 ALC / 55.3 FCM in Tables 4 and 5, 81.2 / 66.9 in Table 6, and 74.04 / 55.32 in Table 7. All are described as best validation checkpoint. Table 4 is the table that carries the abstract's claim that SFT \"sometimes exceeds\" RL. If Table 6 is right, GRPO beats SFT on ALC and nearly ties on FCM, so that claim collapses. If Tables 4/5/7 are right, Table 6 is wrong. Either way, the headline comparison is unstable across the paper's own reporting. This is not a minor typo; it is the evidence for the main conclusion.\n\nThere is real value here beyond the numbers. The ALM definition is clear, and the hierarchical metric chain (TID→TSA→ACA→ALC→FCM) is a sensible way to isolate language-consistency failures. The decision to translate BFC rather than invent a synthetic benchmark is reasonable, and the reward-granularity progression from sparse RM-1 to factorized RM-3 is a well-posed controlled comparison. The conclusion that strong SFT baselines need to be the default is also the right instinct, even if this paper doesn't yet prove it.\n\nSecondary issues are mostly what you would expect: no error bars, an uncalibrated LLM judge, and the authors' own caveat in Section 8.4 that translated data may introduce artifacts. Those would be survivable if the tables agreed. They don't.\n\nWho is this for? People working on multilingual tool-use and post-training baselines will get useful framing and a clear target for improvement. It deserves a serious referee, but the authors need to reconcile the numbers, release the benchmark and judge prompts, and ideally show human agreement on the judge before the SFT-versus-RL comparison is credible. I would accept it for peer review, expecting heavy revision.","headline":"Useful failure-mode definition and a plausible SFT-vs-RL story, but the main comparison is unverifiable because the same GRPO/RM-3 configuration has different numbers in Tables 4, 5, 6, and 7.","tokens_in":20664,"tokens_out":2298,"would_cite":false,"duration_ms":21982,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Careful supervised fine-tuning, not complex reinforcement learning, fixes multilingual API-call language errors.","keywords":["argument language mismatch","multilingual tool use","function calling benchmark","supervised fine-tuning","reinforcement learning","GRPO","cross-lingual transfer","post-training"],"falsifier":"Take a random sample of the translated benchmark's 832 ALM-relevant turns, replace the automated judge's language-consistency scores with human annotations, and compare best-checkpoint SFT against GRPO; if the SFT advantage (e.g., 67.4 vs 55.3 FCM on Split-1) shrinks or reverses under human labels, the paper's central claim is an artifact of judge or translation bias.","tokens_in":19543,"feed_emoji":"🌐","tokens_out":10750,"duration_ms":96255,"temperature":0.7,"pith_summary":"Multilingual API calling has a failure mode the paper names Argument Language Mismatch (ALM): the model picks the right tool and arguments but writes argument values in the wrong language, e.g., a Spanish currency amount as \"dollars\" rather than \"dólares\". The paper's aim is to test whether eliminating ALM requires sophisticated reinforcement learning or whether ordinary supervised fine-tuning suffices. On a parallel five-language extension of the BFC function-calling benchmark, it finds that SFT alone substantially raises both argument-language consistency and end-to-end function call accuracy, and that with validation-based checkpoint selection SFT matches or beats PPO and GRPO on the learnability split. The paper concludes that most multilingual API-grounding performance comes from careful supervised training and data construction, with RL contributing targeted gains on unseen APIs, cross-lingual transfer, and reasoning preservation. This matters because teams building multilingual agents need to know where to spend post-training effort.","feed_headline":"Supervised fine-tuning beats RL on multilingual API calls","feed_subtitle":"A new failure mode—argument language mismatch—is mostly solved by careful SFT, so RL earns only incremental gains.","key_machinery":"The paper's central named object is Argument Language Mismatch (ALM), a failure where the tool and argument names are correct but argument values are generated in a language inconsistent with the user's utterance. The machinery that carries the argument is a five-step hierarchical evaluation chain—Tool Invocation Detection, Tool Selection Accuracy, Argument Completion Accuracy, Argument Language Consistency, and Function Call Match—whose strict ordering FCM ≤ ALC ≤ ACA ≤ TSA ≤ TID lets the authors separate language errors from structural errors. On the training side, the load-bearing mechanism is RM-3, an argument-factorized reward that scores each argument value's language on a graded scale (exact match, minor variation, mismatch) and combines those scores with structural gates, giving the policy per-argument credit; combined with GRPO's group-relative advantage normalization, this reward design produces the paper's strongest RL results while remaining stable under token-level reward weighting. The benchmark itself is a translated, parallel version of a human-annotated function-calling dataset built with a rule-based protocol that translates user-derived values while preserving canonical identifiers and API names.","core_discovery":"On the paper's own terms, the discovery is that Argument Language Mismatch is a generative conditioning failure, not a structural or intent failure, and that this kind of failure is largely learnable by imitation. The paper defines a strict metric hierarchy TID ≥ TSA ≥ ACA ≥ ALC ≥ FCM, isolating language consistency from tool selection and argument completion, and shows that SFT on 832 ALM-relevant turns from a translated function-calling benchmark lifts ALC and FCM far above the base model on both a high-API-overlap learnability split and a low-overlap generalization split. At best validation checkpoints on Split-1, SFT reaches 79.1 ALC and 67.4 FCM, exceeding GRPO (74.0 and 55.3) on both, and exceeding SFT+GRPO (79.3 and 61.3) on FCM. RL's remaining value appears in generalization: GRPO improves cross-lingual ALC consistently across Italian, Dutch, and French, whereas SFT's gains are uneven and include negative transfer on Dutch, and GRPO preserves English reasoning on MGSM while the best-checkpoint SFT model drops from 70.8 to 62.2. The paper concludes that even the strongest argument-factorized reward design, RM-3, improves RL only incrementally over a strong SFT baseline, and that careful supervised training and honest model selection should be the default for structured multilingual generation.","pith_inferences":["If the paper's account is right, the SFT-vs-RL ranking should extend to other surface-level conditioning problems in structured generation, such as localized date, number, and unit formats in code or SQL; a quick test would be applying the same metric hierarchy to a code-generation task with locale-dependent literals.","The benchmark's translation protocol explicitly enforces utterance-value coherence, so natural multilingual tool-use data with code-switching or mixed-language inputs may behave differently; collecting such logs and re-running the comparison would show whether SFT's dominance survives realistic language mixing.","Since the paper's ALM-aware prompt intervention already recovers part of the ALC gap (+7.5 points) without any training, a constrained-decoding or language-identifier-conditioned variant of SFT may close much of the remaining RL margin, which the paper does not test."],"forward_implications":["Multilingual agent teams should treat a strong, validation-selected SFT model as the default and reserve RL for generalization or multi-objective needs, because best-checkpoint SFT already beats GRPO on end-to-end FCM in the learnability split.","Language consistency should be scored separately from tool selection and argument completion; the hierarchy TID→TSA→ACA→ALC→FCM turns ALM from a hidden failure into a measurable diagnostic.","If RL is used, argument-factorized rewards (RM-3) are the main driver of gains; sparse or stepwise rewards leave language consistency under-optimized, and token-level upweighting works under GRPO but destabilizes PPO.","RL's honest advantage is generalization: GRPO transfers the 'match argument language to user locale' rule to unseen APIs and languages more consistently than SFT, and preserves general reasoning in English where the best-checkpoint SFT model loses 8.6 MGSM points."],"supporting_citations":[{"why":"Supplies the BFC function-calling dataset that this paper translates into five languages and uses for all training and evaluation.","marker":"Patil et al., 2024"},{"why":"Defines PPO, one of the two reinforcement-learning algorithms compared against SFT under identical reward formulations.","marker":"Schulman et al., 2017"},{"why":"Defines GRPO, the group-relative policy-optimization algorithm that yields the paper's stronger RL results and cross-lingual transfer.","marker":"Shao et al., 2024"},{"why":"Advances the 'SFT memorizes, RL generalizes' thesis that this paper qualifies by showing SFT can match RL under consistent model selection.","marker":"Chu et al., 2025"}],"fun_headline_variants":["SFT beats RL for multilingual API calls","Language mismatch in API calls: SFT wins","RL only incremental after SFT for tool use","Multilingual API: SFT is the real fix"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ranking rests on the assumption that translation and an automated language judge reproduce how real multilingual API calls fail; if translation artifacts or judge bias drive the ALC scores, the SFT-over-RL conclusion may not transfer to natural use.","fun_headline_variants_meta":{"raw":{"variants":["SFT beats RL for multilingual API calls","Language mismatch in API calls: SFT wins","RL only incremental after SFT for tool use","Multilingual API: SFT is the real fix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1717,"prompt_tokens":1044,"completion_tokens":673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":614}},"tokens_in":660,"tokens_out":673,"duration_ms":6329,"temperature":1.0,"reasoning_tokens":614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:30:27.919081+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the translated benchmark's 832 ALM-relevant turns, replace the automated judge's language-consistency scores with human annotations, and compare best-checkpoint SFT against GRPO; if the SFT advantage (e.g., 67.4 vs 55.3 FCM on Split-1) shrinks or reverses under human labels, the paper's central claim is an artifact of judge or translation bias.","supporting_citations":[],"review_version":1}