{"id":"200f216e-c071-4d24-a1c2-008c63045846","arxiv_id":"2606.06835","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"RL learns a cost-aware policy for invoking translation on LLMs that improves task reward at reduced cost across 22 languages in three resource tiers and five domains.","lead":"The paper trains a single RL policy on a post-trained LLM to decide when to translate non-English inputs into the model's strong language, using reward signals instead of hand-crafted rules. Smart generalists might read it to see a practical way to close multilingual performance gaps without new pretraining data while controlling translation costs.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Reward signal validity hinges on unverified fidelity of answer-preserving translation pipeline","rationale":"The reader's weakest_assumption directly identifies the same load-bearing dependency on pipeline fidelity and non-gaming reward. Because the numerical claims and transfer results are derived from this unverified signal, the UNVERDICTED status is appropriate; the concern does not require a different verdict category.","tokens_in":1886,"tokens_out":334,"duration_ms":13175,"concrete_test":"Sample 200 examples from the XLow tier (and 100 from each synthetic language); have two independent annotators judge whether the gold label remains valid after translation. If >4% of cases are invalid, recompute the RL reward table using only the verified subset and re-evaluate the gated policy's reported lifts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The RL objective optimizes a policy whose reward is computed on task performance after optional translation. The paper states that data is built via an 'answer-preserving translation pipeline' and that the gated policy yields +23.5 / +17.5 reward lift on Low/XLow tiers. For this to support the claim, two conditions must hold: (1) post-translation gold labels remain correct (no semantic drift, no answer change), and (2) the resulting reward truly measures native comprehension rather than pipeline artifacts. Neither is shown to be validated at scale for the 22 languages or the 2 synthetic languages; Low/XLow tiers are precisely where MT quality is weakest. If either condition fails, the reported Pareto front and zero-shot transfer results rest on corrupted supervision.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes learning a single RL policy (via confidence-gated GSPO) that decides when to invoke translation as a tool for LLMs on multilingual tasks. Using an answer-preserving translation pipeline to generate training data across 22 languages in High/Low/XLow tiers plus 5 domains, the gated policy is shown to improve task reward over a no-translation baseline (+4.6 High, +23.5 Low, +17.5 XLow) while using only 63% of the cost of an always-translate policy, achieving Pareto optimality over 87% of the cost-sensitivity range; it also transfers zero-shot to 9 held-out languages and improves on two synthetic languages.","tokens_in":2032,"tokens_out":675,"duration_ms":13358,"significance":"If the reward signal is valid, the work demonstrates that RL can induce language- and domain-adaptive tool-use introspection without language-specific heuristics or external routers, offering a scalable route to close performance gaps for low-resource languages. The explicit cost-reward trade-off analysis and zero-shot transfer results would be a concrete advance over prior rule-based or router-based approaches.","major_comments":[{"comment":"§3 (Data Construction) and §4 (Reward Definition): The central numerical claims (+23.5 / +17.5 reward lift on Low/XLow) rest on the assumption that the answer-preserving translation pipeline produces inputs whose gold labels remain valid. No quantitative validation (human consistency checks, label-flip rate, or semantic-drift metrics) is reported for the 22 languages or the two synthetic languages, especially in tiers where MT quality is weakest. This directly affects whether the RL objective measures native comprehension or pipeline artifacts.","section":"§3, §4"},{"comment":"§5.2 (Pareto Optimality and Cost-Sensitivity): The claim that the gated policy is Pareto-optimal across 87% of the cost-sensitivity range and preserves full reward at 63% cost is load-bearing for the cost-aware contribution. The manuscript does not specify how the cost-sensitivity parameter is swept, how the unconstrained baseline's translation rate is measured, or whether the 63% figure is averaged across domains or languages.","section":"§5.2"},{"comment":"§4.3 (Zero-Shot Transfer): The zero-shot transfer result to 9 held-out languages is presented as evidence of generalization, yet the training/test language split, domain overlap, and whether the held-out languages were seen during the answer-preserving pipeline construction are not detailed; this is required to assess whether the policy truly generalizes or exploits residual pipeline artifacts.","section":"§4.3"}],"minor_comments":[{"comment":"The abstract states concrete numerical gains but the methods section should include explicit data-split tables and the exact number of examples per tier/domain to allow reproduction.","section":"Abstract, §3"},{"comment":"Notation for the gated policy (GSPO) and the confidence threshold should be introduced with an equation in §4 rather than only in prose.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive feedback. We address each major comment below with clarifications and planned revisions to improve the manuscript.","responses":[{"response":"We agree that quantitative validation would strengthen the claims regarding pipeline validity. The answer-preserving pipeline is constructed to retain original gold labels after input translation, with the design intended to minimize semantic drift. However, we acknowledge that no explicit metrics such as label consistency rates or human checks are currently reported. In the revised version, we will add a subsection in §3 reporting back-translation consistency rates across a sample of languages in each tier and a limited human evaluation of label validity on 100 examples per tier. This directly addresses the concern about potential artifacts.","revision_made":"partial","referee_comment":"[§3, §4] §3 (Data Construction) and §4 (Reward Definition): The central numerical claims (+23.5 / +17.5 reward lift on Low/XLow) rest on the assumption that the answer-preserving translation pipeline produces inputs whose gold labels remain valid. No quantitative validation (human consistency checks, label-flip rate, or semantic-drift metrics) is reported for the 22 languages or the two synthetic languages, especially in tiers where MT quality is weakest. This directly affects whether the RL objective measures native comprehension or pipeline artifacts."},{"response":"We thank the referee for highlighting the need for additional methodological detail. The cost-sensitivity parameter is swept linearly from 0 to 1.0 in steps of 0.05; the unconstrained baseline corresponds to λ=0 (no cost penalty). The 63% cost figure represents the average translation invocation rate of the gated policy (across all 22 languages and 5 domains) at the λ value where its reward matches that of the always-translate policy. We will expand §5.2 with an explicit description of the sweep procedure, the definition of the unconstrained baseline, and confirmation that the 63% value is an aggregate average. This will make the Pareto analysis fully reproducible.","revision_made":"yes","referee_comment":"[§5.2] §5.2 (Pareto Optimality and Cost-Sensitivity): The claim that the gated policy is Pareto-optimal across 87% of the cost-sensitivity range and preserves full reward at 63% cost is load-bearing for the cost-aware contribution. The manuscript does not specify how the cost-sensitivity parameter is swept, how the unconstrained baseline's translation rate is measured, or whether the 63% figure is averaged across domains or languages."},{"response":"The 22 languages constitute the training set for RL; the 9 held-out languages are drawn from the same resource tiers but excluded from policy optimization. All languages (including held-out) receive the answer-preserving pipeline during data construction, but the policy never trains on held-out examples. All five domains are shared between train and held-out sets. We will revise §4.3 to include an explicit table of the train/held-out language split, confirm the pipeline usage, and state that held-out languages appear only at evaluation time. This clarifies the zero-shot nature of the transfer results.","revision_made":"yes","referee_comment":"[§4.3] §4.3 (Zero-Shot Transfer): The zero-shot transfer result to 9 held-out languages is presented as evidence of generalization, yet the training/test language split, domain overlap, and whether the held-out languages were seen during the answer-preserving pipeline construction are not detailed; this is required to assess whether the policy truly generalizes or exploits residual pipeline artifacts."}],"tokens_in":1678,"tokens_out":770,"duration_ms":21857,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core move is training a policy on Qwen3-4B via reward alone to decide translation use across 22 languages in three resource tiers. They add confidence-gated GSPO to trade off cost and keep the policy from defaulting to always-translate. The reported lifts (+4.6 High, +23.5 Low, +17.5 XLow) and the Pareto result at 63% cost while holding reward look like the main empirical points, plus zero-shot transfer to nine held-out languages and a synthetic-language test.\n\nThe new piece is the reward-only policy plus the gating mechanism; prior work used language IDs or external routers. That replacement is clean on paper and the domain coverage plus per-language analysis during training give it some practical flavor.\n\nThe soft spot is the answer-preserving translation pipeline that supplies the training data and the reward. The stress-test note is right to flag this: if post-translation labels drift or the MT quality in Low/XLow tiers corrupts the signal, the policy is learning on noisy supervision and the transfer claims weaken. The abstract gives no scale validation for the 22 languages or the two synthetic ones, and that assumption carries the central numbers. If the full text has explicit checks or human verification, it changes the picture; otherwise the soundness stays limited.\n\nThis is for people working on multilingual tool use or cost-aware RL for LLMs. A reader who wants concrete numbers on adaptive translation would get something usable from the method and the cost curves. It deserves a serious referee because the setup is specific enough to test and the idea is straightforward, even if the pipeline validation needs to be the first thing reviewers check.","headline":"The paper learns a single RL policy for when to translate inputs instead of using rules or routers, with a confidence gate for cost control, but the gains rest on an unverified answer-preserving pipeline.","tokens_in":2553,"tokens_out":421,"would_cite":false,"duration_ms":14367,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A reinforcement-learned policy decides when to translate inputs so LLMs solve tasks in low-resource languages at reduced cost.","keywords":["reinforcement learning","tool use","machine translation","multilingual LLMs","cost-aware policy","introspection","zero-shot transfer"],"falsifier":"Measure whether the policy still achieves high reward on low-resource language inputs when translation is artificially disabled; if reward remains high, the claim that the policy has learned genuine introspection fails.","tokens_in":2801,"feed_emoji":"🌐","tokens_out":740,"duration_ms":20867,"temperature":0.7,"pith_summary":"The paper establishes that reward alone can train a single policy to assess whether the model understands an input and to invoke translation only when it cannot. This matters because always translating wastes compute on familiar languages while letting the model decide fails due to overconfidence on unfamiliar ones. Using an answer-preserving translation pipeline to generate training data, the authors apply continued RL to Qwen3-4B across 22 languages in three resource tiers and five domains with a new confidence-gated variant of policy optimization. The resulting gated policy delivers large reward gains on low-resource languages, matches the performance of always translating at 63 percent of the cost, and generalizes zero-shot to held-out languages.","feed_headline":"RL policy translates only when needed, matching reward at 63% cost","feed_subtitle":"Gated training on 22 languages across three resource tiers yields Pareto-optimal decisions that generalize zero-shot.","key_machinery":"The confidence-gated GSPO, which modifies policy optimization to condition tool invocation on the model's estimated comprehension for cost-sensitive decisions.","core_discovery":"The central claim is that a confidence-gated policy trained via reinforcement learning on translation decisions learns language- and domain-adaptive introspection from reward alone, raising task reward by 4.6 points on high-resource, 23.5 on low-resource, and 17.5 on extra-low-resource languages while using only 63 percent of the translation cost of an unconstrained policy and remaining Pareto-optimal across 87 percent of the cost-sensitivity range; the same policy improves reward by 18.7 on synthetic unseen languages and transfers zero-shot to nine held-out languages.","pith_inferences":["The same reward-driven approach could be applied to deciding use of other external tools such as code execution or retrieval.","Patterns in how translation decisions emerge over training may reveal general mechanisms by which models develop self-assessment of capability.","Extending the pipeline to real user queries rather than benchmark tasks would test whether the learned policy remains stable outside controlled domains."],"forward_implications":["The gated policy improves reward over the no-translation baseline by +4.6 on High, +23.5 on Low, and +17.5 on XLow resource languages.","It preserves the full reward of an always-translate policy while incurring only 63 percent of the cost.","The policy is Pareto-optimal across 87 percent of the cost-sensitivity range.","It transfers zero-shot to nine held-out languages and raises reward by +18.7 on two synthetic languages that the base model cannot comprehend.","Tool-use behavior emerges during training in patterns that vary by language resource tier and domain."],"fun_headline_variants":["RL gate learns translation needs from reward across 22 languages","Gated RL preserves reward at 63% translation cost","Policy improves XLow languages by 17.5 via cost aware RL","Zero shot transfer to 9 languages with gated translation policy"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The answer-preserving translation pipeline keeps task labels valid after translation and the reward signal used for RL accurately reflects native comprehension without the model learning to game the translation decision.","fun_headline_variants_meta":{"raw":{"variants":["RL gate learns translation needs from reward across 22 languages","Gated RL preserves reward at 63% translation cost","Policy improves XLow languages by 17.5 via cost aware RL","Zero shot transfer to 9 languages with gated translation policy"]},"model":"grok-4.3","cost_usd":0.009061,"raw_usage":{"total_tokens":4146,"prompt_tokens":828,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":90612000,"prompt_tokens_details":{"text_tokens":828,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3250,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":828,"tokens_out":68,"duration_ms":20362,"temperature":1.0,"reasoning_tokens":3250,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T22:23:34.816965+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Measure whether the policy still achieves high reward on low-resource language inputs when translation is artificially disabled; if reward remains high, the claim that the policy has learned genuine introspection fails.","supporting_citations":[],"review_version":1}