{"id":"5959f369-af88-4061-8a5c-ae1f4a51b6b7","arxiv_id":"2505.08411","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning multilingual retrieval models on an even mix of native and Latin-transliterated training queries largely closes the retrieval gap caused by transliterated queries, with full closure only when scripts overlap.","lead":"Neural search engines lose accuracy when users type queries in Latin letters instead of the language's native script. The authors show that fine-tuning on a mix of native and Latinized queries closes most of this script gap for Russian and reduces it for Chinese.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The script-gap claim is trained and evaluated only on Uroman canonical romanizations; if real user transliterations deviate, the reported bridging may not transfer.","rationale":"The reader and I identify the same vulnerability: the only transliteration distribution in the paper is Uroman's. The method is a data augmentation recipe whose success depends on the augmented queries matching the queries seen at test time. Since Uroman is used on both sides, the experiment cannot distinguish 'the model learned to map any Latinization of a Russian/Chinese query to native documents' from 'the model learned Uroman's particular mapping.' This matters because the motivating examples (Greeklish, Arabizi) are exactly cases where users deviate from canonical forms. The paper's own limitation statements—preliminary tokenization sensitivity, loss of nuance, and the call for a 'proper transliteration procedure'—reinforce rather than resolve this concern. I do not think this is an internal inconsistency; the numbers are self-consistent and significance testing is reported. The strongest internal overstatement is the abstract's 'nearly the same performance' given that Chinese transliterated MRR remains far below native even after fine-tuning, but the authors partially acknowledge this in §4.2. That overstatement is secondary; the primary risk to the central contribution is external validity. Independent support: the authors provide a GitHub repository, use standard collections (mMARCO, NeuCLIR), and report significance tests; these are real evidence for the in-distribution claim. However, fine-tuning details (steps, batch size, negatives, seeds) are omitted, and no formal verification or released checkpoints are indicated, so independent replication is not yet possible. The conditional verdict is appropriate; no verdict change is needed.","tokens_in":10991,"tokens_out":5060,"duration_ms":51693,"concrete_test":"Construct a user-typed transliteration test set: have native Chinese and Russian speakers (or mined search/social-media data) rewrite the mMARCO/NeuCLIR query sets in Latin script without constraints, then run the released BGE-M3(50) and mT5(50) checkpoints on these queries. If MRR@10/nDCG@20 on these human transliterations is close to the Uroman transliterated results, the concern is resolved; if it drops materially (e.g., more than 20% relative), the bridge is an artifact of train/test Uroman matching. A supplementary check: retrain with transliteration augmentation (drop vowels, substitute digits, vary spacing) and see whether robustness to human input improves.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (RQ2, §4.2) is that fine-tuning on an even native/transliterated mixture closes the script gap. The evidence, however, only covers transliterations generated by Uroman (§3, §4). Uroman is a deterministic, canonical romanizer; it does not model the vowel omissions, ad hoc spellings, digit substitutions, homophones, or code-mixing found in Greeklish, Arabizi, and casual pinyin/translit. Consequently, training and evaluation share the same transform, so the mixed model learns Uroman-specific regularities. The paper's own out-of-domain numbers already hint at fragility: on Chinese neuCLIR, transliterated nDCG@20 rises only from 0.0028 to 0.0698, and native performance drops (Table 2). The qualitative analysis shows transliterations create ambiguities, and §4.1 reports that pre-tokenizing queries changes results inconsistently, which is another sign that surface form matters. The abstract's 'nearly the same performance' is also not supported for Chinese even in-domain: Table 1 leaves a 0.126 MRR@10 residual gap (0.2642 native vs 0.1382 transliterated). All of this points to a method that may close the gap for Uroman-style input while remaining untested for the heterogeneous transliterations real users type.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how multilingual neural IR systems behave when queries are transliterated from a non-Latin script (Chinese, Russian) into Latin script. Using Uroman as the romanization tool, the authors build transliterated versions of mMARCO/v2 and NeuCLIR/1 queries and fine-tune BGE-M3 and an mT5-based reranker under three training configurations: native only (N), an even native/transliterated mixture (50), and transliterated only (T). They report that off-the-shelf models degrade sharply on transliterated queries, that the mixed configuration substantially improves transliterated-query effectiveness in-domain while retaining native-query performance for Russian, that the transliterated-only configuration behaves differently across languages and models, and that out-of-domain evaluation on NeuCLIR shows gains for Russian but not for Chinese. The paper concludes that a 'transliterate-train' procedure can bridge the script gap on the query side.","tokens_in":11252,"tokens_out":6186,"duration_ms":63224,"significance":"The proposed recipe is simple and potentially useful: adding transliterated training queries to the fine-tuning mixture improves robustness to Uroman-style romanized queries in both tested languages, with Russian reaching near parity and Chinese showing large in-domain gains. The evaluation is on standard held-out benchmarks (mMARCO dev, TREC NeuCLIR), covers two model families (dense retriever and cross-encoder reranker), and uses significance testing with Bonferroni correction. There are no fitted parameters or circular derivations: the method is an augmentation strategy evaluated on held-out data. The main significance caveat is that the evidence is limited to Uroman's deterministic romanizations, so the headline claim about bridging the script gap for real users' transliterations is not yet established. The paper is a reasonable short-paper contribution if the claims are scoped to Uroman-style transliterations and if the abstract is aligned with the Chinese residual gap.","major_comments":[{"comment":"The abstract's claim that models 'can perform this cross-script matching at nearly the same performance as when the query was formulated in the native script' is only true for Russian. For Chinese under the mixed configuration, Table 1 shows MRR@10 of 0.1382 for transliterated queries versus 0.2642 for native queries with BGE-M3, and 0.1105 versus 0.2520 with the mT5 cascade; a residual gap of roughly half remains. The conclusion in §4.2 that 'we can bridge the gap between the transliterated and native scripts' should therefore be qualified to say that the gap is fully bridged for Russian and only partially bridged for Chinese.","section":"Abstract / §4.2, Table 1"},{"comment":"The statement that 'the mixed configuration is the only fine-tuning configuration which preserves the native text performance and significantly enhances the robustness of both models on the transliterated queries' is contradicted by the Russian rows of Table 1. For BGE-M3, the transliterated-only (T) configuration preserves native MRR@10 (0.2785 vs. the 0.2444 baseline) and improves transliterated MRR@10 from 0.1244 to 0.2679; for mT5, the T configuration also preserves native MRR@10 (0.2662 vs. 0.2552) and improves transliterated MRR@10 from 0.1948 to 0.2522. The uniqueness claim is only true for Chinese; the conclusion should be rewritten as a language-dependent finding.","section":"§4.2, Table 1"},{"comment":"All training and evaluation transliterations are generated by Uroman, a deterministic romanizer, yet the introduction motivates the problem with heterogeneous real-world transliterations such as Greeklish and Arabizi, which contain vowel drops, digit substitutions, and non-canonical spellings. Because the training and evaluation data share the same transform, the reported gains may not transfer to the transliterations real users actually type. The authors should either evaluate on real transliteration data (for example, FIRE transliterated-search queries or Greeklish samples) or explicitly scope the contribution to Uroman-style romanizations and state this as a limitation.","section":"§3 / §4"},{"comment":"The manuscript reports that pre-tokenizing Chinese queries before transliteration yields inconsistent results between BGE-M3 and mT5 and then excludes this variant from the main experiments. Since Uroman's Chinese output appears to have no word boundaries and the residual Chinese script gap is large, the tokenization choice could materially affect the central comparison. The authors should report the pre-tokenization results (even in an appendix or the linked repository) or provide a more direct justification that the choice is neutral for the claims.","section":"§4.1 / §4.2"}],"minor_comments":[{"comment":"The paper does not report fine-tuning hyperparameters (epochs, learning rate, batch size, number of training queries, random seeds, or number of runs). Since the contribution is a training recipe, these details are needed for reproducibility; they should be added to the text or referenced to a specific committed version of the GitHub repository.","section":"§4"},{"comment":"The significance testing is described only as 'Bonferroni correction' with a dagger denoting significance. Please specify the underlying test (e.g., paired t-test or permutation test) and clarify whether the dagger marks significant improvement, significant difference in either direction, or significance after correction; this is currently ambiguous in cells where the metric is lower than the baseline.","section":"Tables 1 and 2"},{"comment":"The qualitative analysis is based on ten sampled queries and informal overlap counts. It is useful as illustration, but the paper should state explicitly that this is not a systematic evaluation and should avoid drawing strong conclusions from the number of common documents across scripts.","section":"§4.3"},{"comment":"The FIRE transliterated-search and mixed-script tasks are cited in related work but not used for evaluation. A sentence explaining why these collections were not suitable (for example, different languages or BM25-era evaluation practices) would help readers understand the experimental scope.","section":"§2"},{"comment":"The Chinese query examples in the submitted text appear as broken glyphs (e.g., the strings around 'qid 32' and 'qid 26'). Please ensure the final PDF renders the native-script examples correctly.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a SIGIR short paper and the experiments are straightforward. My main concern is that the abstract and the RQ2 conclusion overstate what Table 1 shows for Chinese; the residual gap is roughly half, and the 'only configuration' claim is false for Russian. These are central-claim issues that can be fixed by rewriting the conclusions to match the evidence. I would not require a new large-scale experiment for this venue, but the claims must be scoped to Uroman-style transliterations unless real transliteration data are added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a solid short paper with a genuinely new empirical result. The authors adapt translate-train to transliterated queries, fine-tune BGE-M3 and mT5 on mixtures of native and Uroman-romanized mMARCO queries, and show the script gap closes for Russian and shrinks for Chinese. As far as I can tell from their related work, no one had tested this before. The experimental design is standard, significance testing is appropriate, and the paper is honest about the Chinese residual gap in §4.2.\n\nWhat it does well: clear RQs, sensible baselines, in-domain and out-of-domain evaluation, and a small qualitative analysis that shows transliterations lose nuance. The FIRE shared-task background is cited properly, and the only self-citation [4] is used as motivation, not as evidence.\n\nThe soft spots are real but not fatal. First, the abstract says fine-tuning on an even mixture yields 'nearly the same performance' as native queries. That is true for Russian (MRR@10 0.2633 vs 0.2770) but not for Chinese (0.1382 vs 0.2642 on the mixed config, still a 47% gap). The paper's own §4.2 acknowledges this, but the abstract doesn't. Second, the whole pipeline relies on Uroman's canonical romanizations. Real Greeklish, Arabizi, and casual pinyin involve vowel drops, ad hoc spellings, and digit substitutions that Uroman does not model. So the recipe is proven for one transliteration convention, not for the messy user input that motivates the introduction. That limits external validity; the OOD results on NeuCLIR (Chinese transliterated nDCG@20 only 0.0698, with native performance dropping) already hint at fragility. Third, hyperparameters, epoch counts, learning rates, and seeds are missing from the paper. There is a GitHub link, so this may be available, but the text doesn't say.\n\nOverall, the central claim holds for what the paper actually tests: fine-tuning on Uroman-style transliterations can bridge the script gap for Russian and reduce it for Chinese in-domain. The abstract overclaims, and the real-user transliteration question is a genuine limitation. I'd send this to peer review; a revision that fixes the abstract and adds reproducibility details would be worth citing. If you work on multilingual or dense retrieval, this is a useful data point.\n\nRecommendation: engage with it, but don't take the 'nearly same performance' at face value.","headline":"Honest short paper with a genuinely new result for Russian, but the abstract's 'nearly same performance' doesn't hold for Chinese, and the whole recipe is only tested on canonical Uroman romanizations, not real user transliterations.","tokens_in":11756,"tokens_out":3575,"would_cite":true,"duration_ms":35794,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning multilingual rankers on an even mix of native and romanized queries nearly closes the retrieval gap between native and transliterated queries.","keywords":["Transliteration","Neural information retrieval","Script gap","Multilingual retrieval","Cross-script retrieval","Dense retrieval","Romanization","Fine-tuning"],"falsifier":"Take a set of queries actually typed in Latin letters by Russian and Chinese speakers (for example, from chat logs or real search sessions), transliterate the same queries with Uroman, and compare retrieval quality of the mixed-configuration models on both sets. If human-typed transliterations perform no better than the unfine-tuned baseline while the Uroman versions reproduce the paper's gains, the bridge does not extend to real user input.","tokens_in":10817,"feed_emoji":"🔤","tokens_out":8012,"duration_ms":72237,"temperature":0.7,"pith_summary":"Most search users in non-Latin-script languages sometimes type queries in Latin letters (Greeklish, Arabizi, pinyin-style Chinese), and this paper shows that current multilingual neural retrievers fail on those queries: BGE-M3's retrieval effectiveness drops by 97% for Chinese and 49% for Russian when queries are transliterated. The paper argues that this 'script gap' can be largely closed by fine-tuning on an even mixture of native and Uroman-romanized queries, while documents stay in the native script. With this 'transliterate-train' recipe, Russian transliterated queries reach nearly native-level performance, Chinese transliterated queries improve substantially, and native-script performance is preserved. The gains transfer to out-of-domain Russian collections but only partially to Chinese, where transliteration also flattens query nuance. If right, this is a cheap data-augmentation fix for a real usability problem.","feed_headline":"Mixed-script training nearly closes the transliteration gap","feed_subtitle":"A 50/50 mix of native and romanized queries lifts Latin-typed Russian and Chinese retrieval toward native-script levels.","key_machinery":"The central mechanism is the 'transliterate-train' procedure, an adaptation of translate-train: take a language's native training queries, romanize them with the Uroman tool, and fine-tune the ranker on a 50/50 mix of native and romanized queries paired with unchanged native-script documents. This exposes the model to pairs of surface forms for the same information need so that the embedding and reranking layers learn to treat the two scripts as referring to the same text. The argument largely rests on the measured performance gap between native and transliterated queries before and after this fine-tuning on mMARCO Chinese and Russian, plus out-of-domain checks on TREC NeuCLIR.","core_discovery":"On the paper's own terms: fine-tuning a multilingual dense retriever (BGE-M3) and a multilingual cross-encoder reranker (mT5) on an even mix of native-script and transliterated queries, against native-script documents, lets the models match transliterated queries to relevant documents at nearly the same effectiveness as native queries in-domain. The mixed configuration is the only fine-tuning scheme tested that preserves native-text performance while significantly improving transliterated-query performance; training only on transliterations trades away native Chinese performance, and training only on native script does nothing for transliterations. Out-of-domain, the mixed model closes most of the Russian script gap but fails to do so for Chinese, and qualitative inspection shows romanized queries drift to generic results because transliteration flattens lexical nuance.","pith_inferences":["A direct test of the practical value would use human-typed transliterations from real interactions; Uroman's output is a canonical romanization, so real Greeklish, Arabizi, or pinyin-style queries with vowel drops and informal spellings may not reproduce the reported gains.","The authors' own qualitative results suggest the remaining Chinese out-of-domain gap is partly semantic ambiguity, so coupling transliterate-train with a script-alignment or contrastive objective could push generalization further.","Tokenizer behavior may be a lever: the authors report inconsistent results when Chinese queries were pre-tokenized before romanization, and the interaction between segmentation and romanization is not yet pinned down.","A production system could apply the same idea invisibly by romanizing queries at input time rather than asking users to type in a particular script, which would make the script gap a deployment detail rather than a user burden."],"forward_implications":["Romanized Russian queries can reach near-native retrieval quality after fine-tuning on mixed data, so users would not need to switch keyboards to get good results.","Existing multilingual dense retrievers and rerankers can be adapted with data augmentation alone; no new architecture is required.","The same augmentation is less effective when the native script and Latin share little surface similarity, so languages like Chinese need extra measures.","Out-of-domain collections may not see the full benefit, especially for Chinese; deployment should verify against target-domain queries.","Transliteration can blur query nuance (e.g., 'peanut allergy treatment' drifting to generic medicine), so script matching alone does not guarantee meaning preservation."],"supporting_citations":[{"why":"Supplies Uroman, the romanizer that produces every transliterated query used in training and evaluation.","marker":"[9]"},{"why":"Supplies the Chinese and Russian mMARCO training and dev collections with relevance labels.","marker":"[3]"},{"why":"Defines translate-train, the paradigm that transliterate-train adapts.","marker":"[20]"},{"why":"Supplies BGE-M3, the dense retriever whose single-vector embeddings are fine-tuned and evaluated.","marker":"[5]"},{"why":"Supplies mT5, the multilingual reranker fine-tuned and evaluated in the cross-script setting.","marker":"[29]"},{"why":"Supplies the TREC NeuCLIR track collections and queries used for out-of-domain evaluation.","marker":"[12]"},{"why":"Supplies MS MARCO, whose human judgements underlie the automatically translated mMARCO labels.","marker":"[21]"}],"fun_headline_variants":["50/50 native + romanized queries nearly erase script gap","Mixed native and Latin training shrinks IR transliteration gap","Fine-tuning on half native, half romanized bridges script gap","Even mix of scripts recovers transliterated search accuracy","Bilingual fine-tuning closes transliteration gap in IR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Uroman's canonical romanizations adequately represent how real users type transliterated queries; if real Greeklish, Arabizi, or Latin-typed Chinese differ in spelling and style, the fine-tuning gains may not transfer to actual users.","fun_headline_variants_meta":{"raw":{"variants":["50/50 native + romanized queries nearly erase script gap","Mixed native and Latin training shrinks IR transliteration gap","Fine-tuning on half native, half romanized bridges script gap","Even mix of scripts recovers transliterated search accuracy","Bilingual fine-tuning closes transliteration gap in IR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1253,"prompt_tokens":960,"completion_tokens":293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":211}},"tokens_in":576,"tokens_out":293,"duration_ms":3532,"temperature":1.0,"reasoning_tokens":211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:54:25.039231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of queries actually typed in Latin letters by Russian and Chinese speakers (for example, from chat logs or real search sessions), transliterate the same queries with Uroman, and compare retrieval quality of the mixed-configuration models on both sets. If human-typed transliterations perform no better than the unfine-tuned baseline while the Uroman versions reproduce the paper's gains, the bridge does not extend to real user input.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies mT5, the multilingual reranker fine-tuned and evaluated in the cross-script setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MS MARCO, whose human judgements underlie the automatically translated mMARCO labels."}],"review_version":1}