REVIEW 4 major objections 5 minor 24 references
Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Adaptive few-shot prompting that retrieves demonstrations with the deployed LLM's own hybrid embeddings and reranks multiple candidates consistently improves LLM-based Chinese-English and English-Chinese translation over zero-shot, fixed…
desk verdict The AFSP recipe is plausible and the reranker idea is clever, but the reported gains rest on a possibly leaked Diplomatic split and test-set-tuned hyperparameters. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is a train-free hybrid demonstration retrieval module built on the deployed LLM's embedding layer. For a query source sentence, it computes dense embeddings by max-pooling the LLM's token embeddings, sparse weights via a linear projection with ReLU, and multi-vector embeddings via a projection of the full embedding sequence; the final relevance score is $s_{\text{rank}} = \alpha_1 s_{\text{dense}} + \alpha_2 s_{\text{sparse}} + \alpha_3 s_{\text{multi}}$ with weights $0.4$, $0.4$, $0.2$, and the top-$k$ parallel pairs are inserted into a fixed prompt template. The second component is a reranker: a BERT-based small language model that scores candidate translations, trained with mean squared error on self-generated labels $s'_i = 1 - 0.2|b_i|$, where $b_i$ is the number of applied degeneration operations (parallel text, back translation, source insertion, spelling mistakes, repeated translation, synonym replacement). The reranker selects the best of $n$ sampled outputs from the translation LLM, and the paper sets $k=3$ demonstrations in the main experiments.
What would settle it
Remove from the Diplomatic demonstration corpus any source sentence with high n-gram overlap with a test sentence (for example, BLEU above 0.8 against a test source or its reference), rerun the AFSP versus kNN-few-shot comparison, and check whether the BLEU gap collapses; if it does, the claimed advantage is mostly leakage.
Extended reading notes
Core claim
The paper's central claim is that translation demonstrations for in-context learning should be chosen per input rather than fixed, and that the deployed LLM itself supplies the best representation for that choice. It argues that a hybrid retriever combining dense, sparse, and multi-vector embeddings derived from the translation model's own embedding layer selects more semantically relevant example pairs than third-party embedding models, and that sampling multiple candidate translations and reranking them with a small BERT-based model—trained in a self-supervised way by degrading reference translations—yields additional gains. Across ChatGLM3-6B, InternLM2-7B, Llama3-8B, and ChatGPT-3.5, on a newly collected diplomatic corpus and on the UN Chinese-English corpus, the approach reports higher BLEU, METEOR, ROUGE, CHRF, and COMET-Kiwi scores than zero-shot, fixed few-shot, and kNN few-shot prompting.
Load-bearing premise
The demonstration corpus and the test set are drawn from the same 5,528-sentence Diplomatic corpus with no overlap or near-duplicate analysis, so the reported gains could partly reflect the model copying reference translations from retrieved near-duplicates rather than better prompting.
Editorial extensions
If this is right
- Inference-time demonstration retrieval and reranking can raise LLM translation quality without fine-tuning the translation model.
- The deployed LLM's own embedding layer is sufficient for high-quality demonstration retrieval, outperforming third-party embedding models in the reported experiments.
- A reranker trained purely on the parallel corpus via text-perturbation negative sampling adds further gains on top of retrieval.
- The gains appear across open-source and closed-source LLMs, suggesting the method transfers across model families.
Reading between the lines
- Editorial inference: Because the demonstration pool and test set come from the same 5,528-sentence corpus without reported deduplication, part of the reported gain may come from retrieving near-duplicate reference translations rather than from the prompting strategy itself.
- Editorial inference: The method's design assumes the demonstration corpus is domain-matched to the test input; with a mismatched pool, retrieval quality and hence translation quality would likely drop, separating the value of domain overlap from the value of the retrieval algorithm.
- Editorial inference: The self-supervised reranker could be applied to other text-generation tasks, such as summarization or grammatical error correction, where reference-style texts can be automatically degraded to create training labels.
- Editorial inference: For the closed-source model, the paper uses ChatGLM3-6B embeddings as a proxy; whether the benefit persists when the proxy model is from a different family than the translation model is untested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Adaptive Few-shot Prompting (AFSP), an inference-time framework for machine translation with large language models. AFSP first retrieves top-k translation demonstrations using a hybrid relevance score built from dense, sparse, and multi-vector embeddings computed with the deployed LLM's own embedding layer, then samples multiple candidate translations from the few-shot-prompted LLM and reranks them with a BERT-based reranker trained in a self-supervised manner on perturbed references. The method is evaluated on a newly constructed Diplomatic Chinese-English parallel corpus and a Chinese-English subset of the UN Open Corpus, across four LLMs, and is reported to outperform zero-shot, fixed few-shot, and kNN-based few-shot prompting on BLEU, METEOR, ROUGE, CHRF, and COMET-Kiwi.
Significance. If the reported gains hold up, AFSP would be a practically useful, fine-tuning-free way to improve LLM translation, and the design choice of grounding demonstration retrieval in the deployed LLM's own embeddings is interesting and worth testing. The self-supervised reranker idea is also attractive because it avoids human annotation, and the new Diplomatic corpus could be a useful resource. However, the current evaluation contains several load-bearing gaps: possible test/demonstration leakage in the Diplomatic split, hyperparameters tuned on the test set, no variance or significance testing on 500-sentence test sets, and a reranker training signal with no demonstrated correlation to translation quality. These issues prevent confidence in the central claim.
major comments (4)
- [Experiment Settings] For the Diplomatic corpus, the 500 test sentences are randomly drawn from the same 5,528-sentence corpus whose remaining sentences form the demonstration pool, and no deduplication or overlap analysis is reported. This is a press-conference Q&A corpus with heavy formulaic repetition; because retrieval scores by semantic similarity to the test source, the top-k demonstrations may include near-copies of the test sentence together with its reference translation, allowing the model to copy or lightly revise the reference. That could explain a substantial share of the reported 4-9 BLEU gains over fixed few-shot prompting. Please report exact and near-duplicate rates between the test set and the demonstration pool, and either re-run the main comparison on a deduplicated split or evaluate on a separate held-out corpus.
- [Ablation on Weights of Hybrid Demonstration Retrieval; Implementation Details] The hyperparameters alpha1, alpha2, alpha3 (set to 0.4, 0.4, 0.2) and the demonstration count k=3 are selected via ablations in Table 7 and Table 8 conducted on the same 500-sentence test sets used for the main results in Tables 3 and 4. Consequently, the main comparison is in-sample for these hyperparameters, and the statement that 0.4/0.4/0.2 achieves the best performance is a claim about the test set, not a validated configuration. Please either fix these hyperparameters before looking at the test set, use a separate validation split, or report the sensitivity of the main findings to the choice of alpha and k.
- [Main Results (Tables 3 and 4)] All results are point estimates on 500-sentence test sets, with no bootstrap confidence intervals, paired significance tests, or variance reporting. Several improvements over KNN Few-shot are small relative to the expected sampling variance at this test size (e.g., Table 3, ChatGLM3-6B English-to-Chinese COMET-Kiwi 89.12 vs 88.33, and various ROUGE scores differing by less than 1 point). The claim that AFSP 'always' produces higher-quality translations is therefore not supported for every metric. Please add confidence intervals and significance tests over multiple test samples or bootstrap resamples.
- [Result Re-ranking, Eq. (2)] The reranker is trained to predict the invented score 1 - 0.2*|b|, where b is a combination of degeneration operations. This score is not shown to correlate with human translation quality, BLEU, or COMET, so it is unclear what property of the candidates the reranker actually optimizes. Without validation of the synthetic supervision signal, the contribution of the reranking component to translation quality is not established. Please validate the reranker's ranking against human judgments or established metrics, and report how often the reranker selects the candidate with the highest BLEU/COMET among the sampled outputs.
minor comments (5)
- [Tables 3-5] Several typos appear in the tables: 'BLUE-4' should be 'BLEU-4' throughout Tables 3 and 4; Table 5's header 'ASFP' should be 'AFSP'; and in Table 3, the ChatGPT-3.5 English-to-Chinese row for CHRF contains a stray '0' before the value 58.67.
- [Hybrid Demonstration Retrieval] In the description of multi-vector retrieval, the formula for the multi-vector score is labeled 'ssparse = ...' but should be 'smulti = ...' to match the notation in Eq. (1).
- [Human Evaluation] The human evaluation (Table 5) is based on only 5 randomly selected examples per translation direction per dataset. This is a very small sample and the confidence intervals would be extremely wide; please state this limitation explicitly in the text and, ideally, expand the evaluation or report it as a pilot study.
- [Experiment Settings] The paper does not state whether any filtering or cleaning was applied to the crawled Diplomatic corpus beyond aligning the parallel sentences; please clarify how the 5,528 pairs were extracted and whether duplicates within the corpus were removed before the random split.
- [General] The claim that AFSP 'mitigates semantic bias caused by probabilistic sampling' would benefit from an example or analysis showing that the reranker's selected output is indeed more semantically consistent than a randomly chosen candidate; a qualitative case study would help the reader understand the mechanism beyond the aggregate metrics.
Circularity Check
Minor in-sample selection of hybrid-retrieval weights on the Diplomatic test set; the central AFSP claim still has independent support from the UN benchmark.
-
fitted input called prediction
[Implementation Details; Ablation on Weights of Hybrid Demonstration Retrieval (Table 7)]
"The α1, α2, and α3 are set to 0.4, 0.4, 0.2 for the computation of the final relevance score srank in hybrid demonstration retrieval. ... The results in Table 7 show that α1, α2, and α3 are set to 0.4, 0.4, and 0.2 can achieve the best performance on most of the metrics."
The optimal hybrid-retrieval weights are chosen by running the full evaluation on the Diplomatic test set and picking the best configuration in Table 7. These same weights and the same 500-sentence test set produce the headline Diplomatic results in Table 3, so the reported 'effectiveness and superiority' on Diplomatic is partly an in-sample model-selection result, not an out-of-sample prediction. The circularity is mild: even the worst searched α configuration (0.2/0.4/0.4) beats the few-shot baseline by about 4 BLEU in Chinese-to-English, and the UN benchmark, where the weights are transferred without tuning, independently shows large AFSP gains. The number of demonstrations k=3 is likewise validated on the same test set (Table 8), though it is shared by the few-shot baselines.
full rationale
The only concrete circular step is the selection of the hybrid-retrieval mixing weights (and the demonstration count) on the same Diplomatic test set that is later used for the headline comparison; this is an in-sample selection, but it is not load-bearing because the AFSP advantage over few-shot baselines persists across all searched weights and because the UN experiments use the same weights without tuning and show comparable or larger gains. The self-supervised reranker labels (s′ = 1 − 0.2·|b|) are an invented training target, but the reranker's contribution is validated externally by the AFSP vs. AFSP-w/o-rerank comparisons (Tables 3 and 4) and by human evaluation, so it is not a circular reduction. Finally, the absence of deduplication between the random 500-sentence Diplomatic test split and the remaining demonstration pool is a genuine data-contamination risk for the numbers, but contamination is not a circularity step under the definitions used here; it is a correctness and robustness concern that would need overlap analysis to resolve.
Assumptions & free parameters
free parameters (4)
- Hybrid retrieval weights alpha1, alpha2, alpha3 =
0.4, 0.4, 0.2
- Penalty coefficient in negative sampling score =
0.2
- Number of demonstrations k =
3
- Number of sampled candidates =
30 for open LLMs, 5 for ChatGPT-3.5
assumptions (3)
- domain assumption The embedding layer of the deployed LLM provides a better semantic representation for retrieval than dedicated embedding models.
- ad hoc to paper The synthetic quality scores from perturbation operations (Eq. 2) are a valid training signal for ranking real translation outputs.
- domain assumption Retrieved demonstrations from the same corpus as the test sentences do not leak the answers.
Cite this review
Pith. "Pith review of Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models." pith.science (2026). https://pith.science/paper/SM5IB5H5
@misc{pith2026250101679,
author = {Pith},
title = {Pith review of: Adaptive Few-shot Prompting for Machine Translation with Pre-trained Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/SM5IB5H5}},
note = {Machine review of arXiv:2501.01679}
}
read the original abstract
Recently, Large language models (LLMs) with in-context learning have demonstrated remarkable potential in handling neural machine translation. However, existing evidence shows that LLMs are prompt-sensitive and it is sub-optimal to apply the fixed prompt to any input for downstream machine translation tasks. To address this issue, we propose an adaptive few-shot prompting (AFSP) framework to automatically select suitable translation demonstrations for various source input sentences to further elicit the translation capability of an LLM for better machine translation. First, we build a translation demonstration retrieval module based on LLM's embedding to retrieve top-k semantic-similar translation demonstrations from aligned parallel translation corpus. Rather than using other embedding models for semantic demonstration retrieval, we build a hybrid demonstration retrieval module based on the embedding layer of the deployed LLM to build better input representation for retrieving more semantic-related translation demonstrations. Then, to ensure better semantic consistency between source inputs and target outputs, we force the deployed LLM itself to generate multiple output candidates in the target language with the help of translation demonstrations and rerank these candidates. Besides, to better evaluate the effectiveness of our AFSP framework on the latest language and extend the research boundary of neural machine translation, we construct a high-quality diplomatic Chinese-English parallel dataset that consists of 5,528 parallel Chinese-English sentences. Finally, extensive experiments on the proposed diplomatic Chinese-English parallel dataset and the United Nations Parallel Corpus (Chinese-English part) show the effectiveness and superiority of our proposed AFSP.
Figures
Reference graph
Works this paper leans on
-
[4]
arXiv preprint arXiv:2402.03216
Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216. Costa-juss`a, M. R.; Cross, J.; C ¸ elebi, O.; Elbayad, M.; Heafield, K.; Heffernan, K.; Kalbassi, E.; Lam, J.; Licht, D.; Maillard, J.; et al
-
[7]
arXiv preprint arXiv:2407.21783
The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. Fan, A.; Bhosale, S.; Schwenk, H.; Ma, Z.; El-Kishky, A.; Goyal, S.; Baines, M.; Celebi, O.; Wenzek, G.; Chaudhary, V .; et al
-
[9]
The unreasonable effectiveness of few-shot learning for machine translation. ArXiv, abs/2302.01398. Guo, S.; Zhang, S.; Ma, Z.; Zhang, M.; and Feng, Y
-
[10]
SiLLM: Large Language Models for Simultaneous Machine Translation. ArXiv, abs/2402.13036. Hendy, A.; Abdelrehim, M. G.; Sharaf, A.; Raunak, V .; Gabr, M.; Matsushita, H.; Kim, Y . J.; Afify, M.; and Awadalla, H. H
-
[11]
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. ArXiv, abs/2302.09210. Jiang, Z.; and Zhang, Z. 2024a. Can ChatGPT Rival Neural Machine Translation? A Comparative Study. arXiv preprint arXiv:2401.05176. Jiang, Z.; and Zhang, Z. 2024b. Convergences and Diver- gences between Automatic Assessment and Human Evalu- ation: Insights f...
-
[12]
Is ChatGPT A Good Translator? A Preliminary Study. ArXiv, abs/2301.08745. Kirk, R.; Mediratta, I.; Nalmpantis, C.; Luketina, J.; Ham- bro, E.; Grefenstette, E.; and Raileanu, R
-
[13]
arXiv preprint arXiv:2310.06452
Understand- ing the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452. Lin, C.-Y
-
[14]
Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language. ArXiv, abs/2404.04809. Nori, H.; Lee, Y . T.; Zhang, S.; Carignan, D.; Edgar, R.; Fusi, N.; King, N.; Larson, J.; Li, Y .; Liu, W.; Luo, R.; McKin- ney, S. M.; Ness, R. O.; Poon, H.; Qin, T.; Usuyama, N.; White, C.; and Horvitz, E
Show all 24 references
-
[15]
arXiv:2311.16452
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine. arXiv:2311.16452. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al
-
[16]
In Koehn, P.; Barrault, L.; Bojar, O.; Bougares, F.; Chatterjee, R.; Costa-juss `a, M
CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. In Koehn, P.; Barrault, L.; Bojar, O.; Bougares, F.; Chatterjee, R.; Costa-juss `a, M. R.; Fed- ermann, C.; Fishel, M.; Fraser, A.; Freitag, M.; Graham, Y .; Grundkiewicz, R.; Guzman, P.; Haddow, B....
2022
-
[18]
arXiv preprint arXiv:2307.09288
Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Vilar, D.; Freitag, M.; Cherry, C.; Luo, J.; Ratnakar, V .; and Foster, G. F
-
[19]
ArXiv, abs/2211.09102
Prompting PaLM for Translation: Assess- ing Strategies and Performance. ArXiv, abs/2211.09102. Wang, L.; Lyu, C.; Ji, T.; Zhang, Z.; Yu, D.; Shi, S.; and Tu, Z
-
[20]
ArXiv, abs/2405.11804
(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Col- laboration for Translating Ultra-Long Literary Texts. ArXiv, abs/2405.11804. Yao, B.; Chen, G.; Zou, R.; Lu, Y .; Li, J.; Zhang, S.; Liu, S.; Hendler, J.; and Wang, D
-
[21]
arXiv preprint arXiv:2311.09782
More Samples or More Prompt Inputs? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineering. arXiv preprint arXiv:2311.09782. Yuan, F.; Lu, Y .; Zhu, W.; Kong, L.; Li, L.; Qiao, Y .; and Xu, J
-
[22]
In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Findings of the Association for Computational Linguistics: ACL 2023 , 11518–11533
Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Findings of the Association for Computational Linguistics: ACL 2023 , 11518–11533. Toronto, Canada: Association for Computa- tional Lingui...
2023
-
[23]
arXiv preprint arXiv:2210.02414
Glm- 130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. Zhang, B.; Haddow, B.; and Birch, A. 2023a. Prompting large language model for machine translation: A case study. In International Conference on Machine Learning , 41092– 41110. PMLR. Zhang, B.; H...
-
[24]
In Findings of the Association for Computational Linguistics: NAACL 2024, 2765–2781
Multilingual Machine Transla- tion with Large Language Models: Empirical Results and Analysis. In Findings of the Association for Computational Linguistics: NAACL 2024, 2765–2781
2024
-
[2014]
CoRR, abs/1409.0473
Neural Ma- chine Translation by Jointly Learning to Align and Trans- late. CoRR, abs/1409.0473. Bahdanau, D.; Cho, K. H.; and Bengio, Y
-
[2015]
Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y .; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al
Neural Ma- chine Translation of Rare Words with Subword Units.ArXiv, abs/1508.07909. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y .; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al
-
[2019]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Burstein, J.; Doran, C.; and Solorio, T., eds., Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies...
2019
-
[2021]
Journal of Machine Learning Research , 22(107): 1–48
Beyond english-centric multilingual ma- chine translation. Journal of Machine Learning Research , 22(107): 1–48. Feng, Z.; Chen, R.; Zhang, Y .; Meng, Z.; and Liu, Z. 2024a. Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level. arXiv prep...
-
[2022]
arXiv preprint arXiv:2207.04672
No language left behind: Scal- ing human-centered machine translation. arXiv preprint arXiv:2207.04672. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K
-
[2023]
arXiv preprint arXiv:2310.14735
Un- leashing the potential of prompt engineering in Large Lan- guage Models: a comprehensive review. arXiv preprint arXiv:2310.14735. Chen, J.; Xiao, S.; Zhang, P.; Luo, K.; Lian, D.; and Liu, Z
-
[2024]
ArXiv, abs/2404.11018
Many-Shot In-Context Learning. ArXiv, abs/2404.11018. Bahdanau, D.; Cho, K.; and Bengio, Y
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.