REVIEW 3 major objections 3 minor
Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper establishes that after removing five measurement confounds, frontier tool-using agents retain 71–73% of their action policy when the task language changes, and model identity explains only 5.7% of the variance across cells.
desk verdict Strong empirical study, but the 'model-independent' claim rests on an invalid variance decomposition; the four-model agreement stands, the universal constant does not. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the ceiling-corrected estimand $\tilde{I}=I_{\mathrm{cross}}/I_{\mathrm{within}}$, computed from matched replicates: every (model, benchmark, language) cell is generated twice under identical decoding and token budget, varying only the serving seed, so that same-language agreement $I_{\mathrm{within}}$ and cross-language agreement $I_{\mathrm{cross}}$ are both cross-seed and differ only in language. The protocol removes the five confounds by strict empty-trace exclusion, two-directional length matching, a permutation-measured chance floor, and the division by $I_{\mathrm{within}}$ itself, with task-level bootstrap intervals. This machinery is what converts a raw gap into a quantity that can be compared across models.
What would settle it
Run the same matched-replicate protocol at greedy decoding on a new frontier model that is not one of the four (for instance, a model from a fourth vendor that emits parseable traces), and compute $\tilde{I}$ pooled over the six benchmarks; if the value falls outside the [0.708, 0.733] band, the model-independence claim is refuted. Alternatively, replace the matching-block similarity with an edit distance under a tool-cost matrix and check whether the four models still land within 2.6 points of each other; the paper's own Appendix S notes this is untested.
Extended reading notes
Core claim
The paper's central discovery is that cross-lingual policy retention—defined as $\tilde{I}=I_{\mathrm{cross}}/I_{\mathrm{within}}$, the share of a model's own same-language reproducibility that survives a change of language—is nearly model-independent at the frontier. Under greedy decoding, four models spanning dense and mixture-of-experts architectures, 24B–235B parameters, and different training recipes each land between 71% and 73% retention, with the four-model band only 2.6 percentage points wide and model identity explaining 5.7% of the variance across 24 cells. The paper further shows that this result emerges only after correcting five confounds, each of which otherwise biases or reverses conclusions: a missing same-language baseline, trace-length sensitivity, empty-trace inflation, the reproducibility ceiling, and a chance floor measured by permutation at $c\approx 0.56$. It locates the boundary of the regularity below roughly 10B parameters, where the raw ordering of models is largely a chance-floor artifact, and it identifies a causal mechanism: agents pivot non-English tasks through English, a behaviour that is load-bearing for cross-lingual agreement and that resists direct instruction to abandon it.
Load-bearing premise
The model-independence claim rests on treating the 24 (model, benchmark) cells as independent in a variance decomposition even though they are nested within four models and use a single fixed trace-similarity metric; if those models are not representative, or a different similarity metric would spread the ratios, the 71–73% convergence could be an artifact of selection and metric rather than a property of frontier policies.
Editorial extensions
If this is right
- Final-answer accuracy is not behavioural agreement: two languages can agree on every answer while the action route differs in cost, failure mode, and auditability.
- Uncorrected cross-lingual invariance gaps rank models by determinism rather than by language robustness; the observed rank correlation between the sampling-inclusive and greedy gaps is $\rho=-0.80$.
- The English pivot is causally load-bearing: removing it lowers length-matched cross-lingual agreement in proportion to usage, and instructed abandonment fails at over 99% non-compliance.
- Self-consistency voting is a variance reducer, not a retention improver: on the ceiling-corrected estimand it costs 1.6–1.9 points with disjoint intervals.
- Trace-extraction parsing must be reported alongside every headline number, since a single regex changed one model's measured accuracy twenty-sixfold.
Reading between the lines
- If the 71–73% band is a genuine frontier property, a natural testable extension is to run the same protocol on newly released frontier models; the prediction is that they land inside the band, and a miss would bound the regime.
- The chance-floor correction implies that the above-chance retention is roughly 15–18% rather than 71–73%, so downstream citations of the headline figure should state which scale they mean; this is a direct arithmetic consequence of the paper's own correction.
- The causal length dose-response suggests benchmark designers can move retention by changing trace length, so cross-benchmark comparisons are not meaningful without length control; this is our editorial inference, not stated as a recommendation by the paper.
- The English-pivot result connects to latent-English interpretability work, but extends it: prompting is unlikely to remove the pivot, so deployment in non-English settings should budget for translation-related costs and failure modes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a measurement protocol for cross-lingual action-policy retention in tool-using agents. It defines same-language and cross-language trace agreement from paired replicates, corrects five confounds, and normalizes by self-consistency to obtain tilde-I = I_cross / I_within. Across 8 models, 6 parallel benchmarks, 41 languages and 2.38M rollouts, the study reports that under greedy decoding four frontier models retain 71-73% of their action policy across languages, with model identity explaining 5.7% of the per-cell variance. It further locates a breakdown below roughly 10B parameters, identifies an English pivot as the causal mechanism, and documents a trace-extraction regex artifact that can manufacture multilingual failure.
Significance. This is an unusually careful empirical study. The matched-replicate design, length matching in both directions, measured permutation chance floor, design checks, and release of code and traces are all strengths. The GPT-OSS regex case is a particularly convincing control and the causal length manipulation is valuable. If the model-independence claim survives statistical scrutiny, the 71-73% convergence is an important finding for multilingual agent evaluation. The main weakness is that the headline inference from 24 cells nested in four models to a 'model-independent quantity' is not supported by the current variance decomposition, and the chance-inclusive retention level is presented without sufficient emphasis on the chance-corrected 15-18% figure. These issues are fixable with reanalysis and more careful framing.
major comments (3)
- [§4 and Table 3; Appendix S] The claim that 'the data already answer' the n=4 objection is not supported by the reported variance decomposition. The 24 cells in Table 3 are nested within four models, so a one-way eta-squared computed on cells treats non-independent observations as independent; the effective sample size for any statement about generalization across models is four, not twenty-four. The task-level bootstrap intervals in Table 10 and Appendix I quantify within-model precision only, not model-level uncertainty, and Appendix S in fact concedes that the intervals are task-level. Please redo the decomposition with a cluster bootstrap resampling whole models, a mixed-effects model with random intercepts for model and benchmark, or a leave-one-model-out sensitivity analysis, and report the resulting uncertainty or range for the 5.7% figure and for the 71-73% band. The residual (67.4%) is not small and includes benchmark and model-by-benchmark variation, so it does not support the reading that per-cell variation is negligible. Without such an estimate, the abstract's 'model-independent quantity' is an overstatement; the data support 'four frontier models agree closely under the chosen metric and greedy decoding.'
- [Abstract and Appendix Q] The paper states that all five confounds are removed, but the headline 71-73% retention is not chance-corrected. Appendix Q shows that the chance-corrected retention is 15-18%, roughly one-fifth of the headline value, because unrelated traces already agree at c about 0.56. Since the chance floor biases the ratio toward 1 and differentially across models, the chance-inclusive 71-73% should not be presented without this qualifier. Please either report the chance-corrected level as the primary retention figure while noting that the band is preserved, or explicitly state in the abstract and Section 4 that 71-73% is chance-inclusive and that the above-chance retention is 15-18%.
- [§4 and Appendix S] The model-independence claim is also conditional on the choice of the trace-similarity metric S. Appendix S acknowledges that no alternative metric was tested, but S is not a neutral measurement: C2 shows the metric is strongly length-sensitive, and a set-based or edit-distance family could spread the four models' ratios beyond the 2.6-point band. Because the abstract promotes 'close to a model-independent quantity,' a sensitivity analysis with at least one alternative S, or a clear restriction of the claim to matching-block similarity, is needed before the stronger form of the claim is made.
minor comments (3)
- [§1 and §6] Section 1 says 'Every correction we apply makes it larger, never smaller,' but Section 6 reports that self-consistency voting and the scale extension remove results and lower tilde-I; please align the wording, for example by saying 'every confound correction' instead of 'every correction.'
- [Appendix N and §5] The term 'pre-registered' is used to mean predictions written into the analysis script before compute was spent; this is not an external preregistration. Please say so explicitly and describe how the script version and timestamps were fixed, so readers do not infer a formal registry.
- [Appendix S] The statement that the per-cell decomposition is 'the stronger form of the evidence' should be reconciled with the acknowledged n=4 limitation; the main text should not resolve the n=4 objection by invoking the same decomposition without reporting its model-level uncertainty.
Circularity Check
No significant circularity: the central retention ratio is measured directly from traces, and supporting quantities are either measured or pre-registered rather than fitted.
full rationale
The paper's central claim is an empirical measurement, not a derivation. The estimand tilde-I = Icross/Iwithin is defined directly from observed executed traces (Appendix B, Eq. 1), with both numerator and denominator measured under matched replicate seeds; no fitted parameter enters the 71-73% retention figure. The chance floor c is measured by a permutation null over task-to-trace assignments (Appendix Q), not assumed, and the pre-registered head-room prediction was written into the analysis script before the additional compute was spent (Appendix N), so it is an independent test rather than a retrofitted explanation. The self-citations that exist (e.g., MEGA, Ahuja et al. 2023, which includes two of the authors) are background benchmark references and are not load-bearing for any claim. The skeptical concern about treating 24 (model, benchmark) cells as independent units in the eta-squared decomposition is a statistical-generalization limitation, not circularity: the decomposition is a descriptive summary of measured cells, and Appendix S explicitly acknowledges n=4, the model-level generalization limit, and the single-metric choice. Because no equation in the paper is equivalent to its own input by construction, and no fitted constant is renamed as a prediction, the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption The fixed scaffold and shared five-tool alphabet induce a model's action policy Fθ(z,l)=(A(l)(z), y(l)(z)) that is comparable across languages because the trace alphabet is language-independent.
- domain assumption Same-language cross-seed agreement Iwithin measures the model's own stochasticity, and language identity is the only difference in Icross because both are cross-seed.
- domain assumption The matching-block similarity S is an adequate measure of trace agreement; conclusions about retention are assumed invariant to the choice of S.
- standard math The permutation null (task->trace reassignment within language arm) estimates the chance floor c without perturbing other properties.
Cite this review
Pith. "Pith review of Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents." pith.science (2026). https://pith.science/paper/IYD5IO6P
@misc{pith2026260811110,
author = {Pith},
title = {Pith review of: Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/IYD5IO6P}},
note = {Machine review of arXiv:2608.11110}
}
read the original abstract
When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions. Yet those actions are the product: they fix cost and latency, decide how the system fails, and are the only auditable part of its behaviour. We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages (2.38M rollouts). The naive measurement fails: five confounds sit between raw trace similarity and any defensible claim, each able to flip a conclusion. Short traces score higher, empty traces score perfectly, unrelated traces agree by chance over half the time, the gap is capped by each model's reproducibility, and a model asked the same question twice in one language answers differently, leaving no baseline. We remove all five, and every correction makes the effect larger. Divergence proves structural, not sampling noise: it survives greedy decoding in every cell and stays flat as temperature rises, even as models grow less self-consistent. Normalised by their own reproducibility, four very different frontier models converge under greedy decoding, each keeping 71-73% of its action policy across languages, with model identity explaining only 5.7% of the variance. Below roughly 10B parameters it breaks down, and the ordering among smaller models is largely an artifact of a chance floor we measure by permutation rather than assume. Agents route non-English tasks through English; this pivot is causally load-bearing, confirmed by a pre-registered prediction across four models, and models will not abandon it when told to. Finally, a single trace-extraction regex, not the model, manufactured a multilingual failure: two worked examples raise one model's measured accuracy twenty-sixfold while its accuracy on readable outputs barely moves.
Figures
Figures from the paper (6 more)
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.