Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Analysis of Numerical Localisation in LLM Translations

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Embedding formatting rules in the prompt improves LLM localisation accuracy.

desk verdict Useful, transparent first study of numerical localisation with small open LLMs, but the headline significance claim does not survive its own Table 3 as written. read the letter →

arxiv 2608.05232 v1 pith:ZUHFZC2N submitted 2026-08-05 cs.CL cs.LG

Patrizia Kaye This is my paper · ORCID
classification cs.CLcs.LG
keywords numericallocalisationtranslationlargelanguagemodelsin-contextlearningchain-of-thoughtpost-editingEnglish-Germandateandtimeformatting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether small, commodity-hardware large language models can reliably localise dates, times, and numbers when translating between English and German, and how that accuracy can be improved. It reports that adding explicit localisation principles to the prompt, such as digit and decimal separators, date conventions, and the 24-hour clock, gives higher mean accuracy than direct translation, chain-of-thought, or post-editing on all five tested 2-7B instruction-tuned models. This matters because a mislocalised number such as 123.456 can change value by three orders of magnitude, and because the result contradicts the earlier finding that post-editing is the strongest strategy for numerical translation.

What carries the argument

The central mechanism is the in-context-learning prompt: a system prompt that states the localisation principles for each language, namely that German uses a comma as decimal marker and a period as thousands separator, a period in numerical dates, and the 24-hour clock, while English uses a period as decimal marker, a comma or apostrophe as thousands separator, a slash in numerical dates, and a 12- or 24-hour clock. The comparison is carried by a measurement pipeline that extracts values with regular expressions, parses dates and times with dateparser and numbers with babel, and awards one point for the correct value and one for the correct format.

What would settle it

Re-run the same 700 sentence pairs with an independent human-annotated gold standard for value and format, then recompute mean accuracy per strategy; if direct translation or post-editing reaches or exceeds in-context learning, the paper's central claim is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that in-context learning with localisation rules is the most accurate of four strategies for English-German localisation of times, dates, and numbers in the tested models (Qwen2-7B, Qwen3-4B, TowerInstruct-7B, and the two SalamandraTA models). Mean accuracy across all model/type/direction combinations was 53.3% with an unspecified locale and 53.8% with an explicit locale, versus 50.3% and 49.8% for direct translation, 40.2% and 40.3% for chain-of-thought, and 26.9% and 28.4% for post-editing; a Wilcoxon test at alpha = 8.33e-3 supports the difference between in-context learning and the other strategies. The paper also reports that explicitly naming the target locale had only a small and mixed effect, that the best single model was the 4B-parameter Qwen3, and that post-editing collapsed for the Salamandra models, which produced unrelated outputs.

Load-bearing premise

The ranking of strategies depends on the assumption that the hand-written regular expressions find every date, time, and number that matters and that the dateparser and babel libraries judge value and format correctly.

Editorial extensions

If this is right

  • Simply listing formatting rules in the prompt is a cheap and reproducible way to raise localisation accuracy on models that fit in 8GB of VRAM, without extra inference passes.
  • Post-editing with regex extraction and a parser library, which was the winning strategy for numerical translation in the prior study, is the weakest strategy for localisation on these models.
  • Chain-of-thought can degrade localisation accuracy below direct translation, so asking the model to explain its process is not a safe default for format-sensitive output.
  • Stating the target locale explicitly is not enough by itself; the model's implicit formatting behaviour matters more than the declared locale.
  • Model size does not determine localisation skill: the 4B-parameter model beat the 7B-parameter models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the in-context-learning advantage suggests a design rule for format-restricted generation: when the target conventions are enumerable and few, stating them in the prompt may beat reasoning or correction pipelines; this is testable on other language pairs such as English-French or English-Spanish.
  • Because correctness was judged only on regex-extracted values and accepted both US and UK date formats, the absolute percentages are pipeline-dependent; the paper's ranking should be re-tested with human-annotated ground truth before being used as a benchmark.
  • If the mechanism is really the model applying stated rules, then fine-tuning a small model on localisation pairs may compress most of the in-context-learning gain, which would be a natural follow-up experiment.
  • The Salamandra failure mode in post-editing suggests that instruction-tuned translation models can respond to extraction prompts with hallucinated example-like output; checking whether this is specific to the prompt template would separate pipeline failure from model failure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates numerical localisation (dates, times, and numbers with locale-specific formatting) in English–German translation by five instruction-tuned LLMs in the 2–7B parameter range. It introduces four strategies: direct translation, in-context learning (ICL) with explicit localisation principles, chain-of-thought, and post-editing, tested under both unspecified and explicitly specified locales. Mean accuracy across all models and data types is reported, with ICL showing the highest mean accuracy, and Wilcoxon pairwise tests are used to claim statistical significance. The authors provide a reproducible pipeline with public code and data, and they contrast their ICL finding with Tang et al.'s earlier result that post-editing was best for numerical translation.

Significance. If the claims hold, the paper is a useful empirical extension of numerical translation to numerical localisation, with practical relevance for translation pipelines on commodity hardware. The main strengths are the publicly available scripts and data, the transparent description of the prompt templates and regular expressions, and the use of a Bonferroni-corrected pairwise comparison framework rather than a single uncorrected test. The central claim, however, is currently not established as stated: the headline ICL-vs-direct comparison for the unspecified locale exceeds the paper's own significance threshold, and the measurement pipeline is defined entirely through hand-written regexes and parser choices whose sensitivity is not assessed. These issues do not invalidate the study's direction, but they require revision before the abstract's 'statistically significant improvement' claim can be accepted.

major comments (4)
  1. [§4, Tables 3 and 4] The headline statistical claim is not supported as written. For the unspecified-locale condition, the Direct-vs-ICL Wilcoxon p-value in Table 3 is 1.06e-2, which exceeds the stated Bonferroni threshold α=0.05/6=8.33e-3. The paper does not state whether the tests are one- or two-sided; Table 4's Direct-vs-ICL p-value of 4.03e-3 passes either way, but the abstract claims improvement 'compared to direct translation or the alternative strategies' without restricting to the explicit-locale condition. Please state the test direction and the number of paired observations (and what each observation is, e.g., model × direction × data-type cells), then re-evaluate the abstract and conclusions under the stated threshold; with two-sided tests, the unspecified-locale Direct-vs-ICL comparison is not significant at α=8.33e-3.
  2. [§3.5 and Appendix A.1] The evaluation is only defined on regex-extracted values, and the paper does not quantify the reliability of that extraction. The hand-written regular expressions determine which sentences enter the sample and which value/format pairs are compared, a dateparser bug is patched for German times, and both US and UK date formats are accepted as valid even in the explicit-locale condition. A different regex set or parser choice could change the relative standing of the strategies, which is load-bearing for the central claim. Please provide a manual audit of regex recall/precision on the sampled sentences, or a sensitivity analysis showing that the ranking of strategies is unchanged under alternative parsers or stricter locale validation.
  3. [§3.4 and §3.5] The post-editing strategy is scored on a subset of samples, which makes the comparison unequal. The manuscript states that 'Where the direct translation failed to produce values, these were excluded from the post-editing results so as to avoid bias,' while all other strategies are scored on the full 700 inputs. Because post-editing uses the direct translation as its base, excluding unparseable outputs removes exactly the cases in which the base translation already failed. Please report the subset size and score post-editing on the full set as well, or demonstrate explicitly that the exclusion does not affect the reported strategy ranking.
  4. [§3, 'Prompts were iteratively adjusted'] The prompt-development process is a potential source of advantage for ICL. Section 3 says prompts were 'iteratively adjusted during initial development' to improve output and clarify formatting, but the ICL prompt is the one that explicitly states the localisation rules used by the evaluation, whereas CoT and PE receive no equivalent prompt optimisation. Please report how many prompt iterations were tried, on which development data, and whether the final prompts were selected before or after inspecting the test outputs. If the ICL prompt was tuned on the evaluation data, the claim that ICL is the most accurate strategy would be substantially weakened.
minor comments (5)
  1. [Tables 5 and 6] Several table cells appear to have missing delimiters, for example '81.464.1', '92.679.1', and '75.271.8'; these should be clearly separated into distinct entries so the per-model values are legible.
  2. [§5, explicit-locale discussion] The statement that explicitly specifying the locale had a 'small but significant impact' is not supported by any statistical test in the paper; the 66/120 improvement versus 52/120 loss split should be analysed with an appropriate paired test, or the wording should be softened to 'numerically small'.
  3. [Appendix B.2] The ICL prompt contains typographical errors: 'a a comma' and 'a a period' should read 'a comma' and 'a period'.
  4. [§3.5] The random seed used for the 100-sample selection was not recorded. Since the samples themselves are provided in the supplementary material this is not blocking, but reporting the seed would make the sampling step fully reproducible without relying on the supplementary data.
  5. [§2.4 and §3.1] The dataset descriptions are duplicated across Sections 2.4 and 3.1; consider consolidating them to avoid redundancy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ICL-versus-baseline comparison is an empirical intervention with fixed a priori rules, and no fitted parameter or self-citation chain is load-bearing.

full rationale

This paper is an empirical benchmark, not a derivation. The in-context-learning prompt states localisation principles that also define the evaluation criteria, but those principles are fixed a priori from DIN 5008 / ISO 8601 conventions and were not fitted to the measured outcomes. The study measures whether models follow explicitly provided formatting rules, which is a legitimate experimental intervention rather than a circular derivation. The post-editing strategy uses the dateparser library both as a translation component and as part of the evaluation checker; this creates a measurement-consistency concern, but it is not a circular reduction because the parser is an external, fixed tool and the paper does not claim to derive its results from the parser's behaviour. No parameters were fitted to the outcome data, no uniqueness theorem is imported from the authors' prior work, and no load-bearing self-citation appears in the argument. The statistical-significance issue concerning Table 3 is a correctness or reporting matter, not a circularity defect. After walking the claimed derivation chain, I find no step where a prediction is equivalent to its input by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the extraction and validation pipeline (regexes plus dateparser/babel) producing accurate ground truth, and on the parallel corpus sides being treated as the authoritative localised reference. No outcome-dependent free parameters are fitted; the manual choices (prompts, regexes, Bonferroni threshold) are methodological rather than fitted.

assumptions (4)
  • domain assumption The parallel-corpus target sentence is the authoritative ground truth for the localised value and format.
    Used in Section 3.5; the model's output is compared against the corpus reference, but the paper accepts both US and UK date formats as valid due to ambiguity, weakening the ground truth for dates.
  • domain assumption The regular expressions in Appendix A.1 correctly identify every relevant date, time, and number in the source sentences.
    Used in Section 3.5; lines that do not match are excluded, and some data types were dropped due to false positives or insufficient matches, so the test sample is defined by these patterns.
  • domain assumption The dateparser and babel Python libraries correctly parse and validate values and formats for en and de locales.
    Used in Section 3.5; the paper itself patches a dateparser bug for German times (Appendix A.1), indicating the tooling can mis-handle valid formats.
  • domain assumption Temperature=0 greedy decoding at vLLM 0.17.0 yields a stable, representative model output for each prompt.
    Used in Section 3.3; each sample is processed once, so run-to-run variance is not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analysis of Numerical Localisation in LLM Translations." pith.science (2026). https://pith.science/paper/ZUHFZC2N

@misc{pith2026260805232,
  author       = {Pith},
  title        = {Pith review of: Analysis of Numerical Localisation in LLM Translations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZUHFZC2N}},
  note         = {Machine review of arXiv:2608.05232}
}
read the original abstract

The work of Tang et. al. (2025) on numerical translation is extended by analysing the capability of five large language models (LLMs) for the localisation of times, numbers, and dates instead of translation. Models were selected that could be loaded onto and run on commodity hardware and a baseline quality for each mode is computed, then three different strategies to improve on that accuracy were tested. In contrast to Tang et. al., it was discovered that on the tested LLMs, embedding the localisation principles into the prompt context provided a statistically significant improvement in accuracy compared to direct translation or the alternative strategies.

Figures

Figures reproduced from arXiv: 2608.05232 by the authors.

Figure 1
Figure 1. Accuracy for each model and strategy with unspecified locale [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Accuracy for each model and strategy with explicit locale [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages

  1. [1]

    Investigating Numerical Translation with Large Language Models

    Tang, W.; Yu, J.; Li, Y.; Zhao, Y.; Zhang, W.; Feng, W.; Zhang, M.; Yang, H. Investigating Numerical Translation with Large Language Models. In Proceedings of the ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5. https://doi.org/10.1109/ICASSP4 9660.2025.10887726

    work page arXiv 2025

  2. [2]

    Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet

    Kocmi, T.; Avramidis, E.; Bawden, R.; Bojar, O.; Dvorkovich, A.; Federmann, C.; Fishel, M.; Freitag, M.; Gowda, T.; Grundkiewicz, R.; et al. Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet. In Proceedings of the Proceedings of the Ninth Conference on Machine Translation, 2024, pp. 1–46

    work page 2024

  3. [3]

    Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets

    Kocmi, T.; Artemova, E.; Avramidis, E.; Bawden, R.; Bojar, O.; Dranch, K.; Dvorkovich, A.; Dukanov, S.; Fishel, M.; Freitag, M.; et al. Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets. In Proceedings of the Proceedings of the Tenth Conference on Machine Translation; Haddow, B.; Kocmi, T.; Koehn, P ....

    doi:10.18653/v1/2025.wmt-1.22 2025

  4. [4]

    Attention is All You Need

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention is All You Need. 2017

    work page 2017

  5. [5]

    Improving Language Understanding by Generative Pre-Training

    Radford, A.; Narasimhan, K.; Salimans, T.; Sutskever, I. Improving Language Understanding by Generative Pre-Training

    work page

  6. [6]

    Achieving Professional Translation in the Military Field Through Fine-Tuning LLM

    Tao, Y.; Yang, R.; Meng, F.; Liu, Y.; Zhuo, W. Achieving Professional Translation in the Military Field Through Fine-Tuning LLM. In Proceedings of the 2025 IEEE 6th International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT). IEEE, 2025, pp. 2010–2016. https://doi.org/10.1109/ AINIT65432.2025.11035793

    work page arXiv 2025

  7. [7]

    BLEURT: Learning Robust Metrics for Text Generation

    Sellam, T.; Das, D.; Parikh, A. BLEURT: Learning Robust Metrics for Text Generation. Association for Computational Linguistics, 2020, pp. 7881–7892. https://doi.org/10.18653/v1/2020.acl-main.704

    doi:10.18653/v1/2020.acl-main.704 2020

  8. [8]

    Findings of the 2020 Conference on Machine Translation (WMT20)

    Barrault, L.; Biesialska, M.; Bojar, O.; Costa-jussà, M.R.; Federmann, C.; Graham, Y.; Grundkiewicz, R.; Haddow, B.; Huck, M.; Joanis, E.; et al. Findings of the 2020 Conference on Machine Translation (WMT20). In Proceedings of the Proceedings of the Fifth Conference on Machine Translation, Online, 11 2020; pp. 1–55

    work page 2020

Show all 19 references
  1. [9]

    Tilde MODEL - Multilingual Open Data for EU Languages

    Rozis, R.; Skadin, š, R. Tilde MODEL - Multilingual Open Data for EU Languages. In Proceedings of the Proceedings of the 21st Nordic Conference on Computational Linguistics, Gothenburg, Sweden, 5 2017; pp. 263–265

    2017

  2. [10]

    Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here but Not Quite There Yet

    Kocmi, T.; Avramidis, E.; Bawden, R.; Bojar, O.; Dvorkovich, A.; Federmann, C.; Fishel, M.; Freitag, M.; Gowda, T.; Grundkiewicz, R.; et al. Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here but Not Quite There Yet. In Proceedings of the Proceedings...

    2023 doi

  3. [11]

    Efficient Memory Management for Large Language Model Serving with PagedAttention

    Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C.H.; Gonzalez, J.E.; Zhang, H.; Stoica, I. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Princip...

    2023

  4. [12]

    Many-to-English Machine Translation Tools, Data, and Pretrained Models

    Gowda, T.; Zhang, Z.; Mattmann, C.; May, J. Many-to-English Machine Translation Tools, Data, and Pretrained Models. In Proceedings of the Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu...

    2021 doi

  5. [13]

    Available online: https://www.dinmedia.de/de/ themenseiten/din-5008 (accessed on 2026-06-17)

    DIN 5008 – der Standard für Geschäftskommunikation. Available online: https://www.dinmedia.de/de/ themenseiten/din-5008 (accessed on 2026-06-17)

    2026

  6. [14]

    Standard, International Organization for Standardization, Genva, Switzerland, 2019

    ISO 8601-1:2019 Date and time — Representations for information interchange. Standard, International Organization for Standardization, Genva, Switzerland, 2019

    2019

  7. [15]

    Technical report, Unicode Consortium, San Francisco, USA, 2026

    Common Locale Data Repository. Technical report, Unicode Consortium, San Francisco, USA, 2026

    2026

  8. [16]

    Qwen2 Technical Report.arXiv preprint arXiv:2407.106712024

    Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; et al. Qwen2 Technical Report.arXiv preprint arXiv:2407.106712024

    arXiv

  9. [17]

    Qwen3 Technical Report, 2025, [arXiv:cs.CL/2505.09388]

    Team, Q. Qwen3 Technical Report, 2025, [arXiv:cs.CL/2505.09388]

    2025 arXiv

  10. [18]

    Tower: An Open Multilingual Large Language Model for Translation-Related Tasks, 2024, [arXiv:cs.CL/2402.17733]

    Alves, D.M.; Pombal, J.; Guerreiro, N.M.; Martins, P .H.; Alves, J.; Farajian, A.; Peters, B.; Rei, R.; Fernandes, P .; Agrawal, S.; et al. Tower: An Open Multilingual Large Language Model for Translation-Related Tasks, 2024, [arXiv:cs.CL/2402.17733]

    2024 arXiv

  11. [19]

    Salamandra Technical Report, 2025, [arXiv:cs.CL/2502.08489]

    Gonzalez-Agirre, A.; Pàmies, M.; Llop, J.; Baucells, I.; Dalt, S.D.; Tamayo, D.; Saiz, J.J.; Espuña, F.; Prats, J.; Aula-Blasco, J.; et al. Salamandra Technical Report, 2025, [arXiv:cs.CL/2502.08489]

    2025 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.