{"id":"0ec94f90-fbfc-4dd7-9850-cf09b5fc636e","arxiv_id":"2501.01679","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adaptive selection of translation examples through a hybrid LLM-embedding retriever and a self-supervised reranker improves few-shot machine translation across several LLMs.","lead":"A new system picks the best example translations to show a language model before it translates, then picks the best of several candidate translations. This improves Chinese-English translation on a new diplomatic dataset and on UN documents, compared with fixed prompts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unchecked overlap between the Diplomatic demonstration pool and test set could make the reported BLEU gains an artifact of reference copying; deduplication must be reported.","rationale":"Identified the single most load-bearing concern: potential leakage between demonstration and test sets. It targets the central claim directly. The reader flagged the same weakness; I agree. This is more load-bearing than the absence of significance tests or code release because if near-duplicates are common, the headline numbers do not measure what they claim. The concrete test is feasible (the corpus is publicly crawlable) and would settle the question. I also considered test-set-based hyperparameter tuning and the small human evaluation; while real, they are secondary. The verdict should remain CONDITIONAL: the idea is plausible and the flaws are addressable, but the superiority claim is not established until the overlap check is performed. Hence no change to the reader's verdict.","tokens_in":16907,"tokens_out":6987,"duration_ms":65328,"concrete_test":"Reconstruct the Diplomatic corpus from the cited public source (https://www.fmprc.gov.cn/). For each of the 500 test source sentences, compute the maximum similarity against all remaining 5,028 demonstration source sentences using the paper's hybrid retrieval score (Eq. 1) and also chrF/BLEU n-gram overlap. Report the fraction of test sentences with a near-identical match (e.g., chrF ≥ 80 or cosine ≥ 0.95). Then remove every demonstration pair whose source is a near-duplicate of any test sentence and rerun the full Diplomatic evaluation (all four LLMs, both directions, with the same hyperparameters). If the BLEU gap between AFSP and Few-shot drops by more than half, or if AFSP no longer dominates KNN Few-shot, the superiority claim is contaminated; if the gap is essentially unchanged, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main quantitative evidence for AFSP's superiority comes from a split that may not be clean. In Experiment Settings, the authors state that for both datasets they 'randomly selected 500 parallel translation pairs to serve as the test set' and used the remaining pairs as the demonstration corpus. This is a random split of a single 5,528-sentence Diplomatic corpus of press-conference Q&A (2022–2023), a genre with heavy formulaic repetition. AFSP retrieves top-k demonstrations by semantic similarity to the test source (Eq. 1, Hybrid Demonstration Retrieval). If a test source sentence or a close paraphrase exists in the 5,028-sentence demonstration pool, the retrieval module will surface that pair, putting the human reference translation into the prompt. The LLM can then copy or lightly revise that reference, yielding inflated BLEU/COMET scores that reflect leakage rather than the proposed adaptive-prompting mechanism. No deduplication, overlap analysis, or near-duplicate rate is reported anywhere in the paper, despite the central claim of 'effectiveness and superiority' resting on these numbers. Because the gaps over few-shot baselines reach 4–9 BLEU points, even a 10–20% near-duplicate rate among the 500 test sentences could explain a large share of the improvement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adaptive Few-shot Prompting (AFSP), an inference-time framework for machine translation with large language models. AFSP first retrieves top-k translation demonstrations using a hybrid relevance score built from dense, sparse, and multi-vector embeddings computed with the deployed LLM's own embedding layer, then samples multiple candidate translations from the few-shot-prompted LLM and reranks them with a BERT-based reranker trained in a self-supervised manner on perturbed references. The method is evaluated on a newly constructed Diplomatic Chinese-English parallel corpus and a Chinese-English subset of the UN Open Corpus, across four LLMs, and is reported to outperform zero-shot, fixed few-shot, and kNN-based few-shot prompting on BLEU, METEOR, ROUGE, CHRF, and COMET-Kiwi.","tokens_in":17180,"tokens_out":3611,"duration_ms":35547,"significance":"If the reported gains hold up, AFSP would be a practically useful, fine-tuning-free way to improve LLM translation, and the design choice of grounding demonstration retrieval in the deployed LLM's own embeddings is interesting and worth testing. The self-supervised reranker idea is also attractive because it avoids human annotation, and the new Diplomatic corpus could be a useful resource. However, the current evaluation contains several load-bearing gaps: possible test/demonstration leakage in the Diplomatic split, hyperparameters tuned on the test set, no variance or significance testing on 500-sentence test sets, and a reranker training signal with no demonstrated correlation to translation quality. These issues prevent confidence in the central claim.","major_comments":[{"comment":"For the Diplomatic corpus, the 500 test sentences are randomly drawn from the same 5,528-sentence corpus whose remaining sentences form the demonstration pool, and no deduplication or overlap analysis is reported. This is a press-conference Q&A corpus with heavy formulaic repetition; because retrieval scores by semantic similarity to the test source, the top-k demonstrations may include near-copies of the test sentence together with its reference translation, allowing the model to copy or lightly revise the reference. That could explain a substantial share of the reported 4-9 BLEU gains over fixed few-shot prompting. Please report exact and near-duplicate rates between the test set and the demonstration pool, and either re-run the main comparison on a deduplicated split or evaluate on a separate held-out corpus.","section":"Experiment Settings"},{"comment":"The hyperparameters alpha1, alpha2, alpha3 (set to 0.4, 0.4, 0.2) and the demonstration count k=3 are selected via ablations in Table 7 and Table 8 conducted on the same 500-sentence test sets used for the main results in Tables 3 and 4. Consequently, the main comparison is in-sample for these hyperparameters, and the statement that 0.4/0.4/0.2 achieves the best performance is a claim about the test set, not a validated configuration. Please either fix these hyperparameters before looking at the test set, use a separate validation split, or report the sensitivity of the main findings to the choice of alpha and k.","section":"Ablation on Weights of Hybrid Demonstration Retrieval; Implementation Details"},{"comment":"All results are point estimates on 500-sentence test sets, with no bootstrap confidence intervals, paired significance tests, or variance reporting. Several improvements over KNN Few-shot are small relative to the expected sampling variance at this test size (e.g., Table 3, ChatGLM3-6B English-to-Chinese COMET-Kiwi 89.12 vs 88.33, and various ROUGE scores differing by less than 1 point). The claim that AFSP 'always' produces higher-quality translations is therefore not supported for every metric. Please add confidence intervals and significance tests over multiple test samples or bootstrap resamples.","section":"Main Results (Tables 3 and 4)"},{"comment":"The reranker is trained to predict the invented score 1 - 0.2*|b|, where b is a combination of degeneration operations. This score is not shown to correlate with human translation quality, BLEU, or COMET, so it is unclear what property of the candidates the reranker actually optimizes. Without validation of the synthetic supervision signal, the contribution of the reranking component to translation quality is not established. Please validate the reranker's ranking against human judgments or established metrics, and report how often the reranker selects the candidate with the highest BLEU/COMET among the sampled outputs.","section":"Result Re-ranking, Eq. (2)"}],"minor_comments":[{"comment":"Several typos appear in the tables: 'BLUE-4' should be 'BLEU-4' throughout Tables 3 and 4; Table 5's header 'ASFP' should be 'AFSP'; and in Table 3, the ChatGPT-3.5 English-to-Chinese row for CHRF contains a stray '0' before the value 58.67.","section":"Tables 3-5"},{"comment":"In the description of multi-vector retrieval, the formula for the multi-vector score is labeled 'ssparse = ...' but should be 'smulti = ...' to match the notation in Eq. (1).","section":"Hybrid Demonstration Retrieval"},{"comment":"The human evaluation (Table 5) is based on only 5 randomly selected examples per translation direction per dataset. This is a very small sample and the confidence intervals would be extremely wide; please state this limitation explicitly in the text and, ideally, expand the evaluation or report it as a pilot study.","section":"Human Evaluation"},{"comment":"The paper does not state whether any filtering or cleaning was applied to the crawled Diplomatic corpus beyond aligning the parallel sentences; please clarify how the 5,528 pairs were extracted and whether duplicates within the corpus were removed before the random split.","section":"Experiment Settings"},{"comment":"The claim that AFSP 'mitigates semantic bias caused by probabilistic sampling' would benefit from an example or analysis showing that the reranker's selected output is indeed more semantically consistent than a randomly chosen candidate; a qualitative case study would help the reader understand the mechanism beyond the aggregate metrics.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core idea—retrieving demonstrations with the deployed LLM's own embeddings and reranking sampled outputs with a cheap self-supervised model—is a reasonable inference-time enhancement, but the current experimental validation is not yet convincing. The most serious issue is the unchecked overlap between the Diplomatic demonstration pool and test set, which could undermine the headline BLEU gains; the authors must address this with deduplication analysis and, if needed, a clean split. The other issues (test-set hyperparameter tuning, absent significance testing, arbitrary reranker labels) are fixable but require substantial additional experiments. If the authors provide a clean evaluation, the paper could become suitable for publication. I also note the manuscript is formatted as an AAAI submission; if it is intended for a journal, the framing and related-work discussion should be expanded accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the AFSP recipe is plausible and worth knowing about—retrieve translation demonstrations using the deployed LLM's own hybrid embeddings, then rerank sampled outputs with a cheap self-supervised reranker—but the reported BLEU gains are not trustworthy until the authors rule out reference leakage from the demonstration pool into the test set.\n\nWhat is new: using the deployed LLM's embedding layer for dense/sparse/multi-vector retrieval is a fair twist on the BGE-M3 recipe, and the reranker trained on synthetic scores from text perturbations (back-translation, typos, etc.) is a low-cost way to get a ranking model without human annotation. The Diplomatic corpus (5,528 press-conference Q&A pairs, 2022–2023) could be a useful testbed if released.\n\nWhat it does well: four LLMs, both translation directions, two datasets; the improvements are directionally consistent. The embedding-model ablation shows a real advantage for the deployed-LLM embeddings over standard embedders. The paper is clearly written and the framework is easy to reproduce in principle.\n\nSoft spots, in order:\n\n1. Train/test leakage. The Diplomatic test set is a random split of the same 5,528 sentences, with the rest serving as the demonstration pool. No dedup or near-duplicate analysis is reported. Press-conference Q&A is formulaic; semantically similar sentences are common. Since retrieval selects by similarity, a test sentence will often have a near-twin in the pool carrying its reference translation. That can inflate absolute scores and can inflate the gap over fixed few-shot. It can also inflate the gap over kNN few-shot if AFSP's retrieval is better at finding near-duplicates. This is the load-bearing concern.\n\n2. Test-set-tuned hyperparameters. The α weights, k, and the reranker's penalty coefficient are chosen via ablations on the same 500-sentence test set, so the final numbers are in-sample. The α grid differences are small, so this is a mild issue, but it should be fixed with a held-out dev split.\n\n3. No significance testing or confidence intervals. The main gaps are large enough that they would likely survive, but with 500 sentences the paper should report bootstrap intervals.\n\n4. The human evaluation is 5 examples per direction, one model. Supportive, not load-bearing.\n\n5. No code or data. For a paper that trains a reranker and introduces a dataset, that is a reproducibility gap.\n\nAlso, for ChatGPT the authors use ChatGLM3-6B for retrieval, so the 'deployed LLM's own embedding' claim only holds for open models. The kNN few-shot baseline is underspecified, making the comparison to it hard to interpret.\n\nBottom line: the method is reasonable and the paper deserves a serious referee, but the evidence as reported does not establish the claimed superiority. Send it to review with a request for major revision: deduplication analysis, clean dev/test separation, significance tests, and code/data release. I would not cite the numbers as they stand.","headline":"The AFSP recipe is plausible and the reranker idea is clever, but the reported gains rest on a possibly leaked Diplomatic split and test-set-tuned hyperparameters.","tokens_in":17711,"tokens_out":5442,"would_cite":false,"duration_ms":51109,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adaptive few-shot prompting that retrieves demonstrations with the deployed LLM's own hybrid embeddings and reranks multiple candidates consistently improves LLM-based Chinese-English and English-Chinese translation over zero-shot, fixed…","keywords":["adaptive few-shot prompting","machine translation","in-context learning","demonstration retrieval","hybrid embeddings","translation reranking","large language models","diplomatic parallel corpus"],"falsifier":"Remove from the Diplomatic demonstration corpus any source sentence with high n-gram overlap with a test sentence (for example, BLEU above 0.8 against a test source or its reference), rerun the AFSP versus kNN-few-shot comparison, and check whether the BLEU gap collapses; if it does, the claimed advantage is mostly leakage.","tokens_in":16685,"feed_emoji":"🌐","tokens_out":8079,"duration_ms":65868,"temperature":0.7,"pith_summary":"Large language models translate better when the few-shot examples in the prompt are chosen for each input rather than fixed in advance. This paper proposes an adaptive few-shot prompting framework that retrieves translation demonstrations using the same LLM that will perform the translation, combining dense, sparse, and multi-vector embeddings from its own embedding layer, and then reranks several sampled candidate translations with a small BERT model trained in a self-supervised way on degraded reference texts. The authors claim this consistently outperforms zero-shot, fixed few-shot, and kNN-based few-shot prompting for Chinese-English and English-Chinese translation across four LLMs, on a newly collected diplomatic corpus and on the UN Parallel Corpus. The work also contributes a 5,528-sentence diplomatic Chinese-English parallel dataset as a testbed for recent-language translation.","feed_headline":"Per-input example retrieval lifts LLM translation by 9 BLEU","feed_subtitle":"Hybrid LLM-embedding retrieval plus self-trained reranking beats fixed few-shot prompts across four LLMs.","key_machinery":"The load-bearing component is a train-free hybrid demonstration retrieval module built on the deployed LLM's embedding layer. For a query source sentence, it computes dense embeddings by max-pooling the LLM's token embeddings, sparse weights via a linear projection with ReLU, and multi-vector embeddings via a projection of the full embedding sequence; the final relevance score is $s_{\\text{rank}} = \\alpha_1 s_{\\text{dense}} + \\alpha_2 s_{\\text{sparse}} + \\alpha_3 s_{\\text{multi}}$ with weights $0.4$, $0.4$, $0.2$, and the top-$k$ parallel pairs are inserted into a fixed prompt template. The second component is a reranker: a BERT-based small language model that scores candidate translations, trained with mean squared error on self-generated labels $s'_i = 1 - 0.2|b_i|$, where $b_i$ is the number of applied degeneration operations (parallel text, back translation, source insertion, spelling mistakes, repeated translation, synonym replacement). The reranker selects the best of $n$ sampled outputs from the translation LLM, and the paper sets $k=3$ demonstrations in the main experiments.","core_discovery":"The paper's central claim is that translation demonstrations for in-context learning should be chosen per input rather than fixed, and that the deployed LLM itself supplies the best representation for that choice. It argues that a hybrid retriever combining dense, sparse, and multi-vector embeddings derived from the translation model's own embedding layer selects more semantically relevant example pairs than third-party embedding models, and that sampling multiple candidate translations and reranking them with a small BERT-based model—trained in a self-supervised way by degrading reference translations—yields additional gains. Across ChatGLM3-6B, InternLM2-7B, Llama3-8B, and ChatGPT-3.5, on a newly collected diplomatic corpus and on the UN Chinese-English corpus, the approach reports higher BLEU, METEOR, ROUGE, CHRF, and COMET-Kiwi scores than zero-shot, fixed few-shot, and kNN few-shot prompting.","pith_inferences":["Editorial inference: Because the demonstration pool and test set come from the same 5,528-sentence corpus without reported deduplication, part of the reported gain may come from retrieving near-duplicate reference translations rather than from the prompting strategy itself.","Editorial inference: The method's design assumes the demonstration corpus is domain-matched to the test input; with a mismatched pool, retrieval quality and hence translation quality would likely drop, separating the value of domain overlap from the value of the retrieval algorithm.","Editorial inference: The self-supervised reranker could be applied to other text-generation tasks, such as summarization or grammatical error correction, where reference-style texts can be automatically degraded to create training labels.","Editorial inference: For the closed-source model, the paper uses ChatGLM3-6B embeddings as a proxy; whether the benefit persists when the proxy model is from a different family than the translation model is untested."],"forward_implications":["Inference-time demonstration retrieval and reranking can raise LLM translation quality without fine-tuning the translation model.","The deployed LLM's own embedding layer is sufficient for high-quality demonstration retrieval, outperforming third-party embedding models in the reported experiments.","A reranker trained purely on the parallel corpus via text-perturbation negative sampling adds further gains on top of retrieval.","The gains appear across open-source and closed-source LLMs, suggesting the method transfers across model families."],"supporting_citations":[{"why":"Demonstrates that LLM translation quality depends on the prompt and example selection, motivating adaptive demonstration retrieval.","marker":"Zhang, Haddow, and Birch 2023a"},{"why":"Introduces the hybrid dense-sparse-multi-vector embedding retrieval that AFSP adapts to the LLM's own embedding layer.","marker":"Chen et al. 2024"},{"why":"Documents output diversity from probabilistic sampling, motivating the multiple-candidate reranking step.","marker":"Kirk et al. 2023"},{"why":"Studies strategies for picking translation few-shot examples, the selection problem AFSP automates.","marker":"Vilar et al. 2022"},{"why":"Shows few-shot examples alone can produce strong translation, the bar retrieval-based selection aims to beat.","marker":"García et al. 2023"},{"why":"Provides the BERT backbone used as the reranker's small language model.","marker":"Devlin et al. 2019"},{"why":"Defines the kNN-based few-shot prompting baseline that AFSP compares against.","marker":"Nori et al. 2023"},{"why":"Supplies the BLEU metric used as the primary translation-quality measure.","marker":"Papineni et al. 2002"},{"why":"Supplies the COMET-Kiwi quality-estimation metric used alongside n-gram metrics.","marker":"Rei et al. 2022"}],"fun_headline_variants":["Per-input example retrieval lifts LLM translation","Adaptive few-shot prompting with LLM-embedding retrieval","Self-trained reranking beats fixed prompts in MT","Hybrid retriever from LLM's own embeddings boosts BLEU","Dynamic demonstration selection improves machine translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The demonstration corpus and the test set are drawn from the same 5,528-sentence Diplomatic corpus with no overlap or near-duplicate analysis, so the reported gains could partly reflect the model copying reference translations from retrieved near-duplicates rather than better prompting.","fun_headline_variants_meta":{"raw":{"variants":["Per-input example retrieval lifts LLM translation","Adaptive few-shot prompting with LLM-embedding retrieval","Self-trained reranking beats fixed prompts in MT","Hybrid retriever from LLM's own embeddings boosts BLEU","Dynamic demonstration selection improves machine translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1421,"prompt_tokens":1006,"completion_tokens":415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":339}},"tokens_in":622,"tokens_out":415,"duration_ms":4445,"temperature":1.0,"reasoning_tokens":339,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:22:02.701463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Remove from the Diplomatic demonstration corpus any source sentence with high n-gram overlap with a test sentence (for example, BLEU above 0.8 against a test source or its reference), rerun the AFSP versus kNN-few-shot comparison, and check whether the BLEU gap collapses; if it does, the claimed advantage is mostly leakage.","supporting_citations":[],"review_version":1}