{"id":"bfc0b397-d4c9-4e44-91c5-d695946d29df","arxiv_id":"2412.08090","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A translated temporal reasoning dataset plus a cross-lingual example retriever that outperforms semantic-alignment baselines on low-resource language temporal questions.","lead":"This paper introduces mTEMPREASON, a Romanian, German, and French translation of the TEMPREASON temporal reasoning benchmark, and CLiTSSA, a retriever that selects in-context examples by transferring in-language semantic similarity into cross-lingual space. In tests on four LLMs, CLiTSSA improves temporal question answering F1 over cross-lingual baselines by 1.4 to 5.3 F1 points depending on language.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The retriever is trained on similarity labels produced by the same multilingual SBERT model it fine-tunes; the paper never shows those labels track whether a demonstration actually improves downstream temporal QA.","rationale":"The reader's weakest assumption is exactly that the in-language similarity scores produced by the base multilingual Sentence-BERT model are a valid training signal for cross-lingual retrieval. I agree this is the most load-bearing gap. The paper's method is a supervised distillation from a model's own monolingual similarity judgments into its cross-lingual space; unless those judgments correlate with downstream LLM benefit, the entire approach lacks a principled foundation. Table 1 is suggestive but covers only L1 with LLaMA3-8B and does not by itself validate the label quality for other tasks and models. The reported p-value of 0.05 and the absence of error bars make the empirical gains fragile, but that is a secondary robustness concern. The missing oracle/usefulness validation is more fundamental because it targets the causal chain: similarity labels -> retriever -> retrieved examples -> improved LLM answers. A direct correlation check between retrieval scores and oracle usefulness would settle whether the proposed mechanism actually operates. I do not recommend changing the verdict: the paper is already conditional, and this concern is addressable with one additional validation experiment rather than a fatal flaw.","tokens_in":17735,"tokens_out":6145,"duration_ms":67344,"concrete_test":"On a random sample of 200 French L1 test queries, construct a candidate pool of 50 English demonstrations from the training set. For each (query, candidate) pair, measure oracle usefulness as the change in LLaMA3-8B F1 when that candidate replaces a fixed neutral demonstration in a 3-shot prompt, averaged over two or three neutral baselines. Then compute the Spearman rank correlation between CLiTSSA's retrieval score (and the original in-language similarity label) and oracle usefulness, with a confidence interval. If rho is close to zero or negative, the training signal does not track example usefulness and the reported gains are not attributable to semantic alignment; if rho is clearly positive, the load-bearing assumption is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism of CLiTSSA rests on using cosine similarities from distiluse-base-multilingual-cased-v1, computed in the low-resource monolingual space between queries and T5-translated English examples, as supervised labels (Eq. 3) to fine-tune the same model with CoSENT loss (Eq. 4). The paper never validates that these scores track example usefulness for the LLM. Table 1 demonstrates only one task (L1), one LLM (LLaMA3-8B), and top-3 retrieval; it does not establish label quality for L2/L3 or for the other LLMs in Table 5. Because the labels are generated by the same embedding model that is being adapted, the fine-tuning could learn a shallow transformation of the model's own biases rather than a general in-language semantic signal. Without an oracle or independent check, the modest F1 gains in Tables 4 and 5 are not tied to the proposed mechanism: they could reflect lexical overlap between the query and translated English candidates, or self-distillation artifacts, rather than semantically aligned retrieval. The absence of a direct retrieval-quality evaluation and the lack of a non-semantic baseline (e.g., BM25 over translated examples) leave this assumption untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that multilingual LLMs 'inherently reward' in-language semantically aligned cross-lingual demonstrations over direct cross-lingual semantic alignment for time-sensitive queries, based on a comparison of two retrieval scoring procedures (Table 1). It introduces mTEMPREASON, a T5-translated version of TEMPREASON in Romanian, German, and French, and proposes CLiTSSA, a retriever fine-tuned from multilingual Sentence-BERT with CoSENT loss. The training pairs for CLiTSSA are cross-lingual query pairs labeled by cosine similarity in the low-resource monolingual embedding space. The paper reports F1 and EM improvements over X-ICL and X-InSTA across three languages, three temporal tasks, and four LLMs, with mean F1 gains of 5.32, 1.62, and 1.43 for French, German, and Romanian, respectively, and appendix analyses of embedding-space shifts, cross-task transfer, and monolingual-vs-cross-lingual gaps.","tokens_in":17956,"tokens_out":3517,"duration_ms":38092,"significance":"If the central claims are supported, the paper would make a useful contribution: mTEMPREASON addresses a genuine gap in multilingual temporal reasoning evaluation, and the proposed retrieval training signal is simple and potentially transferable. The paper also ships code and data, and the experimental scope is respectable, spanning multiple languages, tasks, and models. However, the strongest claim in the title and abstract—that multilingual LLMs inherently reward in-language semantic alignment—is currently supported by a very narrow experiment, and the mechanism behind CLiTSSA's gains lacks independent validation. The paper's value would be substantially increased by strengthening these two points.","major_comments":[{"comment":"The central generalization that multilingual LLMs 'inherently reward' in-language semantic alignment over cross-lingual alignment rests on a single experiment: Table 1 reports only the L1 (time–time) task, one LLM (LLaMA3-8B), top-3 retrieval, and no significance testing. This is too narrow to support the title's universal claim. The authors should either extend Table 1 to L2/L3 tasks and additional LLMs (or at least report the same comparison for the other models in Table 5), or substantially qualify the claim as specific to the tested setting.","section":"Table 1 and 'Our Proposed Method'"},{"comment":"The training signal for CLiTSSA is generated by the same multilingual Sentence-BERT model that is then fine-tuned: the labels f(s)_u,v in Eq. (3) come from distiluse-base-multilingual-cased-v1, and the same model is optimized with CoSENT loss in Eq. (4). The appendix histograms show that after fine-tuning, positive pairs (English query-translation pairs) move closer and negative pairs move apart, but this is largely a check that the model learned its own training objective. The paper provides no external validation that the pre-fine-tuning in-language similarity scores track whether a demonstration actually improves downstream temporal QA. Without such validation—for example, an oracle-based retrieval-quality evaluation, a correlation analysis between label score and downstream gain, or a non-semantic baseline such as BM25 over translated examples—the modest F1 gains in Tables 4 and 5 are not tied to the proposed mechanism; they could arise from lexical overlap or self-distillation artifacts.","section":"Method, Eq. (3)-(4), and Appendix Figures 5-7"},{"comment":"The statistical support for the main comparison is thin. The paper reports a one-tailed Mann-Whitney U p-value of 0.05 for comparing mean F1 of CLiTSSA with X-InSTA 'across languages and tasks,' but this does not establish that each of the per-cell gains in Table 4 is significant, and no correction is applied for the multiple comparisons implicit in claiming gains for each language and task. The authors should report per-condition significance tests with a clearly described null distribution (e.g., across the three top_p runs, or across bootstrapped samples) and correct for multiple testing, or explicitly state which comparisons are not individually significant.","section":"Table 4 statistics"},{"comment":"The robustness claim across LLMs is not backed by significance testing. The mean F1 improvement over X-InSTA is 9.68 for L1, 2.33 for L2, and 0.58 for L3; the L3 improvement is within the range of run-to-run noise one would expect from varying top_p among {1.0, 0.8, 0.6}. Without per-model per-task tests or confidence intervals, the conclusion that CLiTSSA is robust across LLMs is stronger than the evidence.","section":"Table 5, 'Robustness Across LLMs'"}],"minor_comments":[{"comment":"Equation (4) contains an unfinished placeholder ('exp ...') and is not a complete mathematical expression; since this is the central training loss, the complete CoSENT loss should be written out.","section":"Eq. (4)"},{"comment":"The sentence 'This study used the This study used the Huggingface repository...' contains a duplicated phrase and should be edited.","section":"Appendix, 'Hyperparameters'"},{"comment":"In Table 11, the Zero-Shot row for L1 appears to have misaligned numbers under the F1/EM columns, and the Δ rows are difficult to interpret; the table should be reformatted with clear column separators and explicit labels for the reported deltas.","section":"Table 11"},{"comment":"The phrase 'the procedure stats with the extraction' should read 'starts with the extraction.'","section":"Method, sentence after Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a NLP conference and the dataset/release is a concrete asset. The main risk is that the central 'inherently rewards' claim and the CLiTSSA mechanism are both supported by evidence that is narrower than the claims. I would encourage the editor to ask for the Table 1 extension and a retrieval-quality/oracle experiment, rather than reject, because the core idea is plausible and the resource contribution is useful. Also, the authors may want to position CLiTSSA more carefully against XAMPLER (Lin, Martins, and Schütze 2024), which already learns a cross-lingual retriever for in-context examples."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper gives the community something real: mTEMPREASON, a translated version of TEMPREASON for Romanian, German, and French, with quality checks, and CLiTSSA, a retriever that distills in-language similarity scores into a cross-lingual space for temporal queries. The gains over X-InSTA are consistent across three languages, three tasks, and four LLMs—not huge, but steady. Second, the headline claim that multilingual LLMs 'inherently reward' in-language alignment is wider than the evidence. Table 1, the only direct support, covers one task (L1) and one model (LLaMA3-8B). That is a meaningful observation, but it does not support 'inherently.'\n\nWhat the paper does well: the dataset construction is transparent, with TSR and back-translation BLEU-3 reported, and the code/data are public. The method is simple to understand and reproduce: use multilingual SBERT to score low-resource queries against translated English examples, then fine-tune the same model with CoSENT to make cross-lingual pairs produce those scores. The cross-task and cross-LLM experiments are a useful addition, and the few-shot and h/w ablations are honest.\n\nThe soft spots are real but not fatal. Statistical support is thin: a single Mann-Whitney U with p=0.05, no confidence intervals, and three runs that only vary top_p. The baseline set is too narrow: XAMPLER, a learned cross-lingual retriever, is cited but not included, and there is no BM25 or lexical baseline. So the gains over X-InSTA could come from any learned retriever, not necessarily the specific in-language distillation idea. The stress-test is partly right. The labels are produced by the same base model, so the appendix histogram is unsurprising. But note the fine-tuning inputs are cross-lingual pairs (low-resource query + English example) while the labels come from the monolingual space (low-resource query + translated English). That is a genuine transfer task, not pure self-distillation. The missing piece is whether those scores track usefulness for the LLM; the paper doesn't show that, and the downstream gains, while consistent, don't isolate the mechanism.\n\nThe paper itself flags some of these limits in the error analysis and the conclusion's future work, which I read as honest. The dataset alone is a usable contribution, and the method is reasonable, but the 'inherently' claim should be tempered to 'this method works in these setups' until the label-quality question is answered.\n\nWho is this for? Researchers working on cross-lingual ICL, multilingual QA, or temporal reasoning will find it useful, and the dataset is worth citing. It deserves a serious referee—the issues are fixable with more baselines, per-run variance reporting, and a more careful claim. I'd recommend sending it to review, with the expectation of major revision.","headline":"A useful multilingual temporal-reasoning dataset and a plausible cross-lingual retriever, undercut by an overstated 'inherently rewards' claim and thin validation of the retrieval mechanism.","tokens_in":18522,"tokens_out":3827,"would_cite":true,"duration_ms":36598,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multilingual LLMs reward in-language similarity scoring over cross-lingual scoring for temporal few-shot retrieval, and CLiTSSA transfers that signal into a retriever that lifts mean F1 by 5.32, 1.62, and 1.43 points in French, German…","keywords":["temporal reasoning","cross-lingual in-context learning","low-resource languages","semantic alignment","retrieval-augmented prompting","multilingual LLMs","mTEMPREASON","CLiTSSA"],"falsifier":"Retrieve examples for a held-out set of low-resource temporal queries using three scorers: in-language similarity, direct cross-lingual similarity, and an oracle score equal to the downstream F1 gain each English example actually produces for the LLM. If a retriever trained on the oracle or on direct cross-lingual scores matches or beats CLiTSSA, or if the rank correlation between in-language similarity and oracle usefulness is near zero, the paper's central claim would fail.","tokens_in":17501,"feed_emoji":"🌍","tokens_out":9019,"duration_ms":76092,"temperature":0.7,"pith_summary":"Temporal reasoning questions posed in a low-resource language are best answered when the English few-shot examples shown to the LLM are chosen by first translating those examples into the query's language, scoring semantic similarity there, and only then retrieving. The authors demonstrate that this in-language scoring outperforms direct cross-lingual similarity in multilingual embedding space, and they turn that observation into a method: CLiTSSA fine-tunes a multilingual Sentence-BERT retriever with CoSENT loss to reproduce in-language similarity scores from cross-lingual query–example pairs. They also contribute mTEMPREASON, a French/German/Romanian translation of the TEMPREASON temporal benchmark with three levels of temporal tasks. In three-shot settings across four LLMs, CLiTSSA improves mean F1 over the X-InSTA semantic-alignment baseline by 5.32, 1.62, and 1.43 points for French, German, and Romanian, with the largest gains on the simplest time–time task. If the finding holds, retrieval for low-resource in-context learning should be trained to imitate monolingual similarity rather than rely on the encoder's raw cross-lingual similarity.","feed_headline":"In-language scoring lifts low-resource temporal QA by up to 5.3 F1 points","feed_subtitle":"A retriever trained on monolingual similarity scores picks better English examples for French, German, and Romanian.","key_machinery":"The central object is the in-language similarity score, f(s)_u,v, computed by a multilingual Sentence-BERT encoder between a low-resource query ql_u and a T5-translated English query qr'_v in the low-resource language's monolingual embedding space. CLiTSSA uses these scores as supervised labels to fine-tune the same encoder (with CoSENT loss) on cross-lingual pairs (ql_u, qr_v), so the retriever learns to rank English examples by how semantically close they are to the query when both are read in the query's language. The CoSENT loss is the training mechanism that distills the monolingual ranking into the cross-lingual space.","core_discovery":"The central discovery is that multilingual LLMs intrinsically reward in-language semantic alignment for time-sensitive cross-lingual in-context learning: for the L1 time–time task, retrieving the top-3 English examples by in-language similarity (query vs. translated English) gives F1 of 43.48, 64.77, and 60.50 for Romanian, German, and French, versus 33.65, 56.63, and 46.62 for direct cross-lingual similarity. CLiTSSA operationalizes this by constructing a training set of (low-resource query, English example) pairs labeled with the cosine similarity between the query and the T5-translated English example in the multilingual Sentence-BERT space, subsampling top-h and random-w pairs per query, and fine-tuning the retriever with CoSENT loss so that cross-lingual embeddings reproduce those in-language scores. The fine-tuned retriever then selects English demonstrations that improve LLaMA3-8B's temporal QA across three tasks and three languages, and the gains transfer across four LLMs, with the largest relative improvements on the French L1 task.","pith_inferences":["A likely consequence not tested in the paper: the in-language advantage may reflect a weakness of the base multilingual encoder's cross-lingual alignment for temporal vocabulary rather than a property of LLM inference itself; using a stronger cross-lingual encoder as the teacher could shrink the gap or eliminate it.","Because the training labels come from T5 translations, the method inherits translation noise; in lower-resource languages without reliable machine translation, the same recipe would need pivot languages or seed parallel data, which the paper does not address.","The transfer idea is not obviously specific to temporality; a parallel experiment on arithmetic or commonsense QA could reveal whether in-language scoring helps all low-resource ICL or only queries with time-sensitive semantics.","The paper's cross-task result suggests that a single retriever fine-tuned on the easiest temporal task could serve as a general semantic aligner for a low-resource language, a possibility the authors raise but do not fully explore."],"forward_implications":["Cross-lingual retrievers for low-resource QA should be trained with in-language similarity labels, not direct cross-lingual cosine similarity, when the goal is downstream LLM accuracy.","Translation is a legitimate and useful preprocessing step for building retrieval training data, even for languages the LLM can already process.","Temporal reasoning in low-resource languages can be improved without modifying the LLM itself, purely by changing which English demonstrations are placed in the prompt.","Fine-tuning the retriever on the simplest time–time task (L1) transfers to harder time–event and event–event tasks, so the alignment signal is partly task-agnostic within temporal reasoning.","The mTEMPREASON dataset provides a reusable benchmark for future work on low-resource temporal reasoning across French, German, and Romanian."],"supporting_citations":[{"why":"Defines X-InSTA, the semantic-alignment baseline that CLiTSSA extends and improves upon.","marker":"(Tanwar et al. 2023)"},{"why":"Supplies the multilingual Sentence-BERT encoder that computes in-language similarity scores and serves as the base retriever for fine-tuning.","marker":"(Reimers and Gurevych 2019)"},{"why":"Introduces cross-lingual in-context learning with random English examples, the starting baseline the paper compares against.","marker":"(Winata et al. 2021)"},{"why":"Provides TEMPREASON, the English temporal reasoning dataset that mTEMPREASON is translated from.","marker":"(Tan, Ng, and Bing 2023)"},{"why":"The T5 model used to translate TEMPREASON into French, German, and Romanian, producing mTEMPREASON and the translated English examples used for similarity scoring.","marker":"(Raffel et al. 2023)"},{"why":"Inspiration for transferring within-language semantic knowledge into a cross-lingual space, the core idea behind CLiTSSA.","marker":"Yamada and Ri (2024)"},{"why":"Provides the top-h plus random-w subsampling strategy used to keep CLiTSSA training data tractable.","marker":"(Rubin, Herzig, and Berant 2022)"},{"why":"LLaMA3-8B is the primary LLM on which CLiTSSA's gains are measured across languages and tasks.","marker":"(AI@Meta 2024)"}],"fun_headline_variants":["In-language alignment beats cross-lingual for low-resource temporal QA","Train retriever on translated scores to lift multilingual temporal reasoning","Low-resource temporal QA improved by in-language example scoring","Counterintuitive win: in-language scores pick better English demos","CLiTSSA: boosting French, German, Romanian temporal QA with in-language scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the semantic similarity scores a multilingual sentence-encoder computes in the low-resource language—between a query and a translated English example—genuinely track which English demonstrations will improve the LLM's answer.","fun_headline_variants_meta":{"raw":{"variants":["In-language alignment beats cross-lingual for low-resource temporal QA","Train retriever on translated scores to lift multilingual temporal reasoning","Low-resource temporal QA improved by in-language example scoring","Counterintuitive win: in-language scores pick better English demos","CLiTSSA: boosting French, German, Romanian temporal QA with in-language scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1696,"prompt_tokens":1067,"completion_tokens":629,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":535}},"tokens_in":683,"tokens_out":629,"duration_ms":6674,"temperature":1.0,"reasoning_tokens":535,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:13:40.371670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrieve examples for a held-out set of low-resource temporal queries using three scorers: in-language similarity, direct cross-lingual similarity, and an oracle score equal to the downstream F1 gain each English example actually produces for the LLM. If a retriever trained on the oracle or on direct cross-lingual scores matches or beats CLiTSSA, or if the rank correlation between in-language similarity and oracle usefulness is near zero, the paper's central claim would fail.","supporting_citations":[{"cited_title":"I.; Madotto, A.; Lin, Z.; Liu, R.; Yosinski, J.; and Fung, P","cited_arxiv_id":null,"evidence_quote":"Introduces cross-lingual in-context learning with random English examples, the starting baseline the paper compares against."},{"cited_title":"LEIA: Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data Augmentation","cited_arxiv_id":"2402.11485","evidence_quote":"Inspiration for transferring within-language semantic knowledge into a cross-lingual space, the core idea behind CLiTSSA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the top-h plus random-w subsampling strategy used to keep CLiTSSA training data tractable."}],"review_version":1}