Pith. sign in

REVIEW 3 major objections 4 minor 40 references

Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A tool-using agent given the same task in another language keeps only 71–73 percent of its action policy, measured against its own reproducibility.

desk verdict A careful measurement study whose central 71–73% retention headline is chance-inclusive and overstated as written, but the underlying methodology and qualitative findings are solid and deserve serious review. read the letter →

arxiv 2608.11110 v2 pith:IYD5IO6P submitted 2026-08-11 cs.CL

Sourabrata Mukherjee , Kalika Bali , Sunayana Sitaram This is my paper · ORCID
classification cs.CL
keywords cross-lingualpolicyretentiontool-usingagentsaction-traceevaluationEnglishpivotmeasurementconfoundsmultilingualtraceextractionparsefailuremodelconvergenceundergreedydecoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multilingual evaluation has mostly compared final answers and thrown away the intermediate steps, but for a tool-using agent the steps are the product: they set cost and latency, determine failure modes, and are the only part of behaviour that audits can target. This paper tries to establish that cross-lingual action-policy divergence is a real, structural property of current frontier agents, not sampling noise. It builds a measurement that removes five confounds—missing baseline, trace length, empty traces, reproducibility ceiling, and a chance floor—and under greedy decoding four very different frontier models retain 71–73% of their own action-policy self-consistency when the language changes, with model identity explaining only 5.7% of the variance. The paper also argues that the cause is an English pivot that is causally load-bearing and that models will not abandon when instructed, and that a trace-extraction regex alone can manufacture a 26-fold accuracy drop.

What carries the argument

The load-bearing object is the ceiling-corrected estimand $\tilde I=I_{\mathrm{cross}}/I_{\mathrm{within}}$, the share of a model's own reproducibility that survives a change of language. It only means anything because every cell is generated twice with only the serving seed changing, so decoding noise enters both sides identically and language is the only difference; on top of that, comparisons are length-matched in both directions, pairs with empty traces are dropped, and the chance floor is measured by permuting task-to-trace assignments rather than assumed. The second mechanism is the English pivot: Translate is the most-used tool, reasoning text is about 99% ASCII even on Devanagari input, and interventions that remove or mandate the translation tool move cross-lingual agreement in a dose-response pattern that was pre-registered. The machinery works by converting raw trace similarity, which is dominated by artifacts, into a quantity that can be compared across models and benchmarks.

What would settle it

Run the same replicated, length-matched protocol on a scaffold that actually executes tool calls and feeds observations back; if the cross-lingual gap falls well below the reported ~0.20 or the four frontier models spread beyond the 2.6-point band, the structural-divergence claim is an artifact of the symbolic setting. Alternatively, a high-adherence frontier model whose normalized greedy retention sits clearly outside [0.708, 0.733] would bound the claimed regularity.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that language change imposes a nearly constant loss on the action policies of capable tool-using agents. Every task–language cell is generated twice under identical decoding so that same-language agreement $I_{\mathrm{within}}$ and cross-language agreement $I_{\mathrm{cross}}$ are measured the same way, and normalized policy retention $\tilde{I}=I_{\mathrm{cross}}/I_{\mathrm{within}}$ divides out each model's own reproducibility. Under greedy decoding the four frontier models land in a band of 2.6 percentage points, each keeping 71–73% of its policy, and across all 24 cells model identity explains only 5.7% of the variance in $\tilde{I}$; the divergence is positive in every cell, survives greedy decoding, and stays flat as temperature rises while self-consistency falls. Removal of the English pivot lowers agreement in proportion to how much a model uses it, a pre-registered head-room prediction holds across all four models, and instructing models to reason in the task language is refused at above 99% in the models tested. Below roughly 10B parameters the regularity breaks down, and the apparent ordering among small models is mostly an artifact of a measured chance floor.

Load-bearing premise

The decisive assumption is that a parsed, never-executed tool call stands in for real tool use: the experiment measures the induced policy, not grounded execution, so if real tool outputs change the state, the 71–73% regularity and the English-pivot mechanism may not transfer.

Editorial extensions

If this is right

  • Answer-level parity in multilingual evaluation is compatible with large behavioural divergence, so cost, failure modes, and auditability must be assessed from the action trace, not the final answer.
  • Raw cross-lingual trace similarity is not interpretable on its own: any of the five confounds can flip a conclusion, so future studies should report length-matched, empty-excluded, ceiling-normalized, and chance-corrected values.
  • Deployed multilingual agents will pay different costs and fail in different places per language; the English pivot means non-English routes carry an extra translation step that English regression tests never exercise.
  • Model rankings built from uncorrected cross-lingual gaps are rankings of determinism rather than of multilingual ability; the paper finds the rank correlation between the T=0.5 and T=0 gaps is -0.80.
  • Trace-extraction parsing should be reported with every headline number; a single regex suppressed one model's measured accuracy by 26 times while its accuracy on readable outputs barely moved.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the measured regularity reflects language-conditioned serial behaviour rather than tool use specifically, then the same ceiling-corrected protocol could detect comparable cross-lingual divergence in code generation or multi-step math, where intermediate states play the role of tool calls.
  • The near-total refusal to reason in the task's language suggests the English pivot is not a surface preference; a testable extension is whether fine-tuning or decoding-level constraints that block English tokens can reduce it, and whether reducing it closes the retention gap.
  • The paper's chance-floor values imply that published cross-lingual comparisons with short traces may be mostly noise; a conservative rule of thumb would treat any raw trace-similarity score near 0.56 on a five-tool alphabet as uninformative until corrected.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper measures whether multilingual tool-using agents execute the same action policy across languages, using a fixed symbolic scaffold and 2.38M rollouts across 8 models, 6 parallel benchmarks, and 41 languages. The authors identify five confounds in naive trace-similarity measurement and introduce a protocol that generates every cell twice, excludes empty traces, length-matches comparisons in both directions, measures a chance floor by permutation, and normalizes cross-language agreement by same-language reproducibility. The main results are that cross-lingual policy divergence survives greedy decoding, that four frontier models retain 71–73% of their self-consistency under language change, and that this retention is driven by an English pivot that resists instruction to abandon it. The paper also documents a trace-extraction regex artifact that manufactured a 26× multilingual accuracy failure in one model.

Significance. If the central claims survive scrutiny, this is a significant methodological contribution to multilingual agent evaluation. The authors are unusually careful: every cell is replicated, empty traces are excluded, length is matched in both directions, the chance floor is measured by permutation rather than assumed, routing and truncation are verified from prompt logs, provenance checks are exact, and one prediction was written into the analysis script before compute was spent. The paper also ships code, prompts, benchmarks, and all 2.38M per-rollout traces, making the results auditable. The cross-lingual divergence result, the English-pivot mechanism, and the measurement-artifact case study are all likely to influence benchmark design. The main reservation is that the headline 71–73% retention figure is chance-inclusive; chance-corrected retention is 15–18%, and the paper's abstract and conclusion do not make this clear.

major comments (3)
  1. [§4, Table 3, Appendix Q] The headline claim that the four frontier models 'each retain 71–73% of its own action policy when the language changes' is stated on the chance-inclusive scale Ĩ=I_cross/I_within (Appendix B), but Appendix Q reports a permutation chance floor c≈0.56 and a chance-corrected retention κ=(I_cross−c)/(I_within−c) of 15–18%. Because a model with zero above-chance cross-lingual retention would already score Ĩ≈0.65, the 71–73% band is close to the floor and the level claim is materially overstated as written. The abstract and conclusion should either headline the chance-corrected value or clearly label the 71–73% figure as chance-inclusive and quote the corrected value whenever the retention level is discussed.
  2. [§4, Table 3, Appendix Q] The variance decomposition in Table 3, which reports that model identity explains only 5.7% of the variance in Ĩ, is computed on chance-inclusive cells. Appendix Q shows that chance correction lowers the level roughly fivefold and widens the relative spread of the band from 4.1% to 18.8%, but it does not report the η² decomposition for the corrected estimand κ. The claim that model identity explains only 5.7% of the variance is therefore not established on the corrected scale; the corrected decomposition should be reported, or the claim should be explicitly restricted to the chance-inclusive estimand.
  3. [Abstract, §1, Appendix Q] The paper repeatedly states that 'every correction makes the effect larger,' but the chance-floor correction in Appendix Q lowers the retention ratio roughly fivefold, from 71–73% to 15–18%. This statement is accurate only for the corrections in Table 1 applied to the gap Δ, not for the chance-floor correction applied to Ĩ. The wording should be made precise so that readers do not infer that chance correction also strengthens the headline level.
minor comments (4)
  1. [§4, paragraph after Table 3] The sentence 'Correcting for the measured chance floor lowers the level roughly fivefold and preserves the absolute band' is important enough to appear in the abstract, since it directly qualifies the paper's central numerical claim.
  2. [Appendix Q] The definition of κ should be written in the notation of Appendix B, and the aggregation rule used for the 15–18% figure should be stated in the main text, so that readers can reproduce the corrected band.
  3. [§5, Table 4, Appendix N] The term 'pre-registered' is used for predictions written into an analysis script; if there is no external timestamp or registry, the paper should describe this as a commitment made in the analysis script before compute, to avoid ambiguity about the strength of the guarantee.
  4. [Figure 2, Table 2 captions] The captions should state explicitly that the reported Ĩ values are chance-inclusive and that chance-corrected values are given in Appendix Q, since readers may otherwise interpret 71–73% as chance-corrected retention.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the retention ratio, chance floor, and pre-registered pivot test are all measured quantities rather than fitted targets, and the paper's self-citations are contextual only.

full rationale

I walked the paper's derivation chain and found no step where a claimed prediction is equivalent by construction to its inputs, nor any load-bearing self-citation. The central estimand is I~ = I_cross / I_within (Appendix B, Eq. 1), where both quantities are directly measured cross-seed trace similarities. The 71–73% band is a ratio of these measured values, not a parameter fitted to the target. The chance floor is measured by permutation, not assumed: Appendix Q states 'we therefore measure c rather than assume it, by permuting the task→ trace assignment within a language arm and within length bin and recomputing S.' The paper also discloses that chance-corrected retention is 15–18% rather than 71–73% chance-inclusive, and says 'We report both, and say which the claim rests on' (Section 4); this is an overstatement/scale caveat, not a circular reduction. The English-pivot head-room prediction was 'written into the analysis script before the compute was spent' (Section 5), with two further models chosen on their measured baseline pivot rate; the predicted ordering is a genuine out-of-sample test, not a fit. Self-citations to MEGA (Ahuja et al., 2023, with two co-authors overlapping) appear only as background for cross-lingual evaluation and are not load-bearing. No uniqueness theorem, ansatz, or renamed known result is used to force the conclusions. The paper is transparent about limitations (Appendix S: tools never executed, n=4 frontier band, independent-vendor gap, failed think-native manipulation), and these limitations are correctness risks rather than evidence of circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. Its load-bearing inputs are the verified parallel benchmarks, the similarity metric, the attenuation model for the chance floor, and the scaffold itself. The chance floor and eligibility threshold are data-derived or hand-set quantities that shape the corrected and pooled results, but the raw 71-73% claim depends mainly on directly measured trace similarities.

free parameters (2)
  • Chance floor c = 0.561 pooled; per-cell range 0.266-0.753 at T=0; up to 0.947 on short-trace cells
    Measured by permutation of task-to-trace assignment within 66 cells and 200 permutations (Appendix Q). Used in the attenuation correction kappa=(S-c)/(1-c) that yields the corrected 15-18% retention figures. It is empirically estimated rather than hand-set, but it materially changes the corrected headline.
  • Adherence eligibility threshold = 20% parse-failure rate
    The paper recommends reporting any model above ~20% parse failure as unranked and applies this reasoning to GPT-OSS-120B and Aya-Expanse-8B (Figure 8c, Appendix L/R). This hand-chosen threshold influences which models enter the frontier and boundary analyses, though the four frontier models all have very low parse-failure rates.
assumptions (5)
  • domain assumption Verified parallel alignment keys guarantee that translated items are the same task across languages.
    Used in §2 and Appendix F; five screened corpora fail this test and Belebele needed re-keying, so the guarantee is checked but still an assumption about translation equivalence.
  • domain assumption Generating each cell twice with only the serving seed varied makes language identity the only systematic difference between Iwithin and Icross.
    This is the identification argument in Appendices B and C. It assumes that no other uncontrolled factor varies systematically across languages or replicates.
  • domain assumption Normalized matching-block similarity S over a five-tool alphabet is an adequate measure of action-policy retention.
    Appendix C defines S; Appendix S notes that alternative trace-similarity families are untested, so the metric choice is a load-bearing modeling assumption.
  • domain assumption The classical attenuation model S_obs = c + (1-c) S_true applies with a common chance floor c for same-language and cross-language pairs.
    Appendix Q uses this model to produce chance-corrected retention of 15-18%. No cell has Icross <= c, but the additive form and shared floor are assumed rather than directly validated.
  • domain assumption A fixed ReAct-style scaffold with a single regex extractor captures the induced tool-use policy equivalently across languages and models.
    §2 and Appendix L. Parse-failure rates are measured and GPT-OSS is excluded, but the extraction regex remains part of the measurement and can asymmetrically affect languages.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents." pith.science (2026). https://pith.science/paper/IYD5IO6P

@misc{pith2026260811110,
  author       = {Pith},
  title        = {Pith review of: Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IYD5IO6P}},
  note         = {Machine review of arXiv:2608.11110}
}
read the original abstract

When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions. Yet those actions are the product: they fix cost and latency, decide how the system fails, and are the only auditable part of its behaviour. We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages (2.38M rollouts). The naive measurement fails: five confounds sit between raw trace similarity and any defensible claim, each able to flip a conclusion. Short traces score higher, empty traces score perfectly, unrelated traces agree by chance over half the time, the gap is capped by each model's reproducibility, and a model asked the same question twice in one language answers differently, leaving no baseline. We remove all five, and every correction makes the effect larger. Divergence proves structural, not sampling noise: it survives greedy decoding in every cell and stays flat as temperature rises, even as models grow less self-consistent. Normalised by their own reproducibility, four very different frontier models converge under greedy decoding, each keeping 71-73% of its action policy across languages, with model identity explaining only 5.7% of the variance. Below roughly 10B parameters it breaks down, and the ordering among smaller models is largely an artifact of a chance floor we measure by permutation rather than assume. Agents route non-English tasks through English; this pivot is causally load-bearing, confirmed by a pre-registered prediction across four models, and models will not abandon it when told to. Finally, a single trace-extraction regex, not the model, manufactured a multilingual failure: two worked examples raise one model's measured accuracy twenty-sixfold while its accuracy on readable outputs barely moves.

Figures

Figures reproduced from arXiv: 2608.11110 by the authors.

Figure 1
Figure 1. Semantically identical tasks in 41 languages pass through one fixed symbolic tool [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The central result. (a) Greedy decoding raises [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 2
Figure 2. The central result. (a) Greedy decoding raises [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (9 more)
Figure 3
Figure 3. Figure 3: The temperature ladder separates language from sampling. (a) Cross-lingual [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 4
Figure 4. Figure 4: Each rule is one (model, benchmark) cell, running from its sampling-inclusive gap [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The regime, its boundary, and why the boundary is a measurement result. (a) [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: English pivoting, tested causally and pre-registered across four models. (a) Length [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Measurement pathology and its repair. (a) GPT-OSS-120B’s empty-trace rate per [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Trace length, from correlation to cause. (a) The correlational picture across all 30 [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 8
Figure 8. Figure 8: Measurement pathology and its repair. (a) GPT-OSS-120B’s empty-trace rate under [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Correctness moves with language, but invariance does not predict it. (a) Within [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]
Figure 9
Figure 9. Figure 9: Trace length, from correlation to cause. (a) The correlational picture across all 30 [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 6 canonical work pages

  1. [1]

    MEGA: Multilingual evaluation of generative AI

    Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. MEGA: Multilingual evaluation of generative AI. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4232–4267,

    2023

  2. [2]

    language pairs generated by one task are not independent; resampling pairs would understate the intervals substantially. The same resample indices are used forIwithin and Icross within a draw, so ˜I and ∆ inherit the correlation between numerator and denominator rather than treating them as independent. Under C3 a pair is dropped whenevereithertrace is em...

    work page 2026

  3. [8]

    at most 2

    were run at T=0 on two models with task, language, seed, decoding and token budget held fixed, for 72 cells and 271,200 rollouts. Three levels rather than two matters: a monotone move across three points is hard to attribute to an uncontrolled covariate, and it also lets a non-monotone response beseenrather than averaged away. The manipulation took. Mean ...

    work page 2026

  4. [10]

    Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk

    doi: 10.1145/3600006.3613165. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA: Evaluating cross-lingual extractive question answering. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7315–7330,

    arXiv

  5. [11]

    doi: 10.18653/ v1/2020.acl-main.653. 11 Published as a conference paper at COLM 2026 Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R´e, et al. Hol...

    work page 2020

  6. [12]

    arXiv:2211.09110. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. InInternational Conference on Lear...

    arXiv

  7. [13]

    Gr´egoire Mialon, Cl´ementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom

    arXiv:2308.03688. Gr´egoire Mialon, Cl´ementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general AI assistants. InInternational Conference on Learning Representations,

    arXiv

  8. [14]

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M

    arXiv:2311.12983. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xian- gru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through...

    arXiv

Show all 40 references
  1. [15]

    NLLB Team, Marta R

    doi: 10.18653/v1/2023.acl-long.891. NLLB Team, Marta R. Costa-juss `a, James Cross, Onur C ¸elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic...

    2023 arXiv

  2. [17]

    TALM: Tool augmented language models.arXiv preprint arXiv:2205.12255,

    Aaron Parisi, Yao Zhao, and Noah Fiedel. TALM: Tool augmented language models.arXiv preprint arXiv:2205.12255,

    arXiv

  3. [18]

    XCOPA: A multilingual dataset for causal commonsense reasoning

    Edoardo Maria Ponti, Goran Glavaˇs, Olga Majewska, Qianchu Liu, Ivan Vuli´c, and Anna Korhonen. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 2362–2376,

    2020

  4. [19]

    Martin L

    doi: 10.18653/v1/2020.emnlp-main.185. Martin L. Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. John Wiley & Sons, New York,

    2020 doi

  5. [21]

    doi: 10.18653/ v1/2023.emnlp-main.658. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: ...

    2023

  6. [22]

    arXiv:2307.16789. Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyan- shu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunch...

    arXiv

  7. [23]

    12 Published as a conference paper at COLM 2026 Timo Schick, Jane Dwivedi-Yu, Roberto Dess `ı, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom

    doi: 10.1162/tacl a 00452. 12 Published as a conference paper at COLM 2026 Timo Schick, Jane Dwivedi-Yu, Roberto Dess `ı, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InAdv...

    2026 doi

  8. [24]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr

    arXiv:2302.04761. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design, or: How i learned to start worrying about prompt formatting. InInternational Conference on Learning Representations,

    arXiv

  9. [25]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei

    arXiv:2310.11324. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. InInternational Conference o...

    arXiv

  10. [26]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V

    arXiv:2210.03057. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InInternational Conference on Learning Representations,

    arXiv

  11. [27]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H

    arXiv:2203.11171. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, volume 35, pp. 2...

    arXiv

  12. [28]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, et al

    doi: 10.18653/v1/2024.acl-long.820. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    2024 arXiv

  13. [29]

    Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich

    arXiv:2210.03629. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multi- lingual neural machine translation and zero-shot translation. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1628–1639,

    arXiv

  14. [30]

    Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing

    doi: 10.18653/v1/2020.acl-main.148. Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. How do large language models handle multilingualism? InAdvances in Neural Information Processing Systems, volume 37,

    2020 doi

  15. [31]

    24 cells

    arXiv:2402.18815. 13 Published as a conference paper at COLM 2026 Appendix Appendix Guide The appendices are grouped by what they are for. Method and estimator come first, then the data and the checks on it, then the full results behind each figure in the main text, then the b...

    2026 arXiv

  16. [32]

    and ask how far apart its members are. BecauseT is fixed and language-independent, traces from two languages are sequences over the same alphabet and can be compared exactly, without a translation or entailment model in the measurement path. The estimator has to be cross-seed ...

    1993

  17. [34]

    and learned tool use (Schick et al., 2023; Parisi et al., 2022; Qin et al.,

    2023

  18. [35]

    Agentic benchmarks evaluate task completion (Liu et al., 2024; Mialon et al.,

    established the interleaved reason–act format we adopt: the model alternates free-form deliberation with a symbolic call drawn from a declared tool inventory. Agentic benchmarks evaluate task completion (Liu et al., 2024; Mialon et al.,

    2024

  19. [36]

    Two consequences follow for our purposes

    but are overwhelmingly English and scoreoutcomes, using the action sequence only as a means to that score. Two consequences follow for our purposes. First, a model that reaches the right answer by a different route is indistinguishable from one that reaches it by the intended ...

    2020

  20. [37]

    compare final predictions, and the underlying datasets we build on (Conneau et al., 2018; Ponti et al., 2020; Artetxe et al., 2020; Bandarkar et al.,

    2018

  21. [38]

    are translations or careful adaptations of a common item pool. Their alignment guarantees are exactly what makes a matched cross-lingual comparison possible, but they are guarantees aboutitems, not about pipelines: Appendix F documents five multilingual corpora whose per-langu...

    2022

  22. [39]

    hard benchmark

    examine intermediatetext. Text is the natural object when the question is whether a model reasons at all, but it is a poor object for a cross-lingual comparison: two rationales in different languages cannot be scored for agreement without a translation or entailment model, whi...

    2023

  23. [40]

    non-English

    and OPUS-100 (Zhang et al., 2020), which are En–X bitext with no X–Y corre- spondence; XL-Sum (Hasan et al., 2021), whose articles are collected independently per language; MLQA (Lewis et al., 2020), which is parallel only through a shared question id and whose 7 languages are...

    2020

  24. [1993]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    arXiv

  25. [1994]

    Cross-lingual consistency of factual knowledge in multilingual language models

    Jirui Qi, Raquel Fern ´andez, and Arianna Bisazza. Cross-lingual consistency of factual knowledge in multilingual language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10650–10666,

    2023

  26. [2018]

    doi: 10.18653/v1/D18-1269. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Flor...

    arXiv

  27. [2020]

    Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa

    doi: 10.18653/v1/2020.acl-main.421. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The belebele benchmark: a parallel reading comprehension dataset in 122 l...

    2020 doi

  28. [2021]

    doi: 10.18653/v1/2021.findings-acl

    2021 doi

  29. [2022]

    gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925,

    OpenAI. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925,

    arXiv

  30. [2023]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama

    doi: 10.18653/v1/2023.emnlp-main.258. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4623–4637,

    2023 doi

  31. [2024]

    Jonathan H

    doi: 10.18653/v1/2024.acl-long.44. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.Transactions of the A...

    2024 doi

  32. [2025]

    Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong- Bin Kang, M

    Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong- Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. XL-Sum: Large-scale multilingual abstractive summarization for 44 languages. InFindings of the Association for Computational Linguistics: AC...

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.