{"id":"38023a66-8d44-4799-98e7-6a871b0ce008","arxiv_id":"2412.02626","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Time-reversed language models, which score queries given responses, improve best-of-N reranking, citation attribution, retrieval, and jailbreak filtering over forward perplexity baselines.","lead":"The paper trains language models to read and score text in reverse token order, from response to query, and uses their scores to rerank LLM answers, find citations, retrieve documents, and filter harmful prompts. If the approach holds up, it offers a way to improve LLM outputs and safety without human preference labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline reranking result lacks a matched forward baseline: Forward Baseline is only pretrained while TRLM-Ba is FLaN-tuned, so the 5–8 point LC win-rate gains may reflect instruction tuning, not reverse token order.","rationale":"The paper's idea is interesting, and the empirical pattern across tasks is consistent. The ablation family (TRLM-Fo, -Ba, -FoBa) is a real strength, and the within-family ordering (TRLM-Ba > TRLM-Fo) is suggestive that reverse pre-training contributes beyond direction prompting. My concern is narrower: the headline number that the abstract and conclusion cite ('5% over self log-perplexity reranking') is not backed by a matched forward baseline. The Forward Baseline row, which is supposed to be the controlled comparison, is not documented as FLaN-tuned, while every TRLM variant is. Since FLaN instruction tuning alone typically changes both length behavior and answer quality on AlpacaEval, the 32.44 vs 24.27 gap cannot be attributed to token direction. Self-scoring uses Gemini-Pro-1.0, a different model family, so it is also unmatched. This does not disprove the paper's thesis; TRLM-Ba vs TRLM-Fo still favors reverse pre-training, and the retrieval/citation results point in the same direction. But it means the central claim's strength is currently uncertain. The missing control is easy to run (one additional model and one reranking pass) and would settle the issue. Hence I keep the reader's CONDITIONAL verdict unchanged, with the condition sharpened to require this control.","tokens_in":21768,"tokens_out":12302,"duration_ms":135271,"concrete_test":"Run Table 2 with an additional baseline: a forward-token-order PALM2-Otter model, same pretraining corpus and compute, FLaN fine-tuned, scoring Query->Response with the same prompt formatting used for the TRLM rows (and, for completeness, also scoring Response->Query). If this FLaN-tuned forward baseline reaches an LC win rate at or above TRLM-Fo's 29.19 (or TRLM-Ba's 32.44), then the reverse-direction gains in the headline table are explained by instruction tuning or model family rather than by time-reversed scoring. Report bootstrap confidence intervals for all rows.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is Table 2's best-of-N reranking on AlpacaEval: TRLM-Ba reaches 32.44 LC win rate versus 27.05 for Gemini-Pro self log-perplexity and 24.27 for Forward Baseline. What has to be true for this to establish that reverse token-order pre-training provides useful feedback is that the comparison isolates the scoring direction or token direction. It does not. Forward Baseline (Table 1) is described only as a 'conventional forward model trained for next-token prediction on the same training corpus and model class as TRLM'; Section 3 states FLaN instruction fine-tuning only for TRLM variants. The 8.17-point gap over Forward Baseline is therefore potentially an instruction-tuning gap, not a reverse-model gap. Self-scoring is not a matched control either: it uses a different generator (Gemini-Pro-1.0) whose likelihood is a known weak reranker. Within TRLM variants, TRLM-Ba beats TRLM-Fo by 3.25 points, but both score response->query and both are FLaN-tuned, so this only isolates reverse pre-training given the response->query direction; it does not validate the paper's headline comparison against conventional forward scoring. Thus the strongest evidence for 'reverse scoring complements forward models' rests on a missing control: an equally instruction-tuned, forward-token model scoring Query->Response.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Time Reversed Language Models (TRLMs), a family of models that score and generate in the response-to-query direction. Three variants are proposed: TRLM-Fo, a forward-pretrained model that is prompted to reverse the direction; TRLM-Ba, pretrained from scratch in reversed token order; and TRLM-FoBa, pretrained in both directions. The authors evaluate reverse-direction scoring for best-of-N reranking on AlpacaEval, citation attribution on CNN/Daily Mail, and passage retrieval on MS MARCO and NF-Corpus, and evaluate reverse-direction generation for jailbreak defense. They report large gains over forward scoring baselines, supported by a theoretical result in a stylized bipartite-graph hallucination model (Section 4, Appendix A).","tokens_in":22069,"tokens_out":9209,"duration_ms":92041,"significance":"If the headline comparisons were properly matched, the paper would contribute a practical unsupervised feedback mechanism: reverse scoring can be used to rerank forward generations without preference data, and the ablations TRLM-Ba versus TRLM-Fo suggest that reverse token pretraining adds value within the response-to-query direction. The paper is transparent about the stylized nature of its theoretical model and evaluates on public benchmarks across several tasks, which is a strength. However, the key quantitative claims are weakened by unmatched baselines: the AlpacaEval comparison varies instruction tuning and generator in the same comparison, the citation and retrieval tables vary both scoring direction and model quality, and the jailbreak defense is evaluated on very small samples without uncertainty quantification. The central idea is promising, but the current evidence does not yet isolate the contribution of reverse-direction scoring.","major_comments":[{"comment":"The central AlpacaEval result is confounded as described. Section 3 states that TRLM variants are FLaN fine-tuned, while Table 1 describes Forward Baseline only as a conventional forward model trained for next-token prediction on the same corpus and model class, without stating whether it receives the same FLaN instruction tuning; Self-scoring uses Gemini-Pro-1.0's own log-perplexity, a known weak reranker. Therefore the 8.17-point LC win-rate gap between TRLM-Ba and Forward Baseline, and even the 4.92-point gap for TRLM-Fo, could reflect instruction tuning or generator identity rather than the reverse scoring direction. The within-family comparison TRLM-Ba versus TRLM-Fo (3.25 points, both Response→Query) is valid evidence that reverse pretraining helps for that direction, but it does not validate the paper's headline claim against conventional forward scoring. Please add a matched control: an equally FLaN-tuned forward-token model scoring Query→Response, and report bootstrap confidence intervals for the 805-question win rates.","section":"§5.1.1, Tables 1–2"},{"comment":"The citation and retrieval tables vary both the model and the scoring direction. In Table 3, TRLM models score A→S while Forward Baseline and Backward Baseline score S→A; in Table 5, TRLM models score D→Q while the baselines score Q→D. Consequently, the claimed 44.15% citation gain and the 44.19-point NF-Corpus NDCG gain do not isolate the direction of scoring; they could be due to the difference between a FLaN-tuned reverse-pretrained model and an untuned forward log-perplexity scorer. Since the thesis is that the response→query direction itself is valuable, the baselines should include a forward model scoring in the same direction as the TRLM models (A→S or D→Q), and ideally a reverse-pretrained model scoring in the baseline direction, so that direction and model quality are not confounded.","section":"§5.2–§5.3, Tables 3 and 5"},{"comment":"The jailbreak defense evaluation is based on very small samples with no uncertainty quantification. As described in Appendix F.1, new-HA is the subset of 43 human-annotated toxic questions that pass the GPT-3.5 input filter (25.58% FNR), leaving roughly 11 questions; JBB is stated as 72 questions in one place and 68 in another; and the H and E sets contain about 48–49 questions each. With these denominators, a single misclassification changes FNR or FPR by up to about 9 percentage points, so the claimed drastic FNR reduction with negligible FPR impact is not statistically supported. Please report exact denominators for every cell, add confidence intervals such as Clopper–Pearson intervals, and consider evaluating on larger or resampled sets.","section":"§5.4, Table 6, Appendix F.1"}],"minor_comments":[{"comment":"The text reports '5%' and '8%' improvements, but Table 2 shows differences of 5.39 and 8.06 percentage points; please clarify whether these are absolute percentage-point gains or relative gains and use one convention consistently.","section":"§5.1.1, Table 2"},{"comment":"The text first says 'only 72 are declared as safe' and then says 'this set of 68 questions forms our JBB Dataset'; the discrepancy should be reconciled, and the exact denominators used for each FNR and FPR cell in Table 6 should be stated.","section":"Appendix F.1, Table 6"},{"comment":"The indexing in line 3 appears to be missing parentheses: s + ⌈t−s/2⌉ should likely be s + ⌈(t−s)/2⌉, and the control flow contains typos; please correct these formatting issues.","section":"Appendix D, Algorithm 8"},{"comment":"Lemma 2 writes PTRLM-Ba(Q|A), but the actual score used in Algorithm 2 is a log-likelihood over reversed token sequences; the paper should make explicit that this is not a direct conditional distribution over natural-language Q given A and should discuss the calibration implications of the full reversal.","section":"Section 4, Algorithm 2"},{"comment":"The number of generated queries N, the sampling temperature, and the exact threshold grid are not reported for the results in Table 6, so the defense procedure cannot be reproduced from the current description.","section":"§5.4, Algorithm 12"},{"comment":"The paper does not state whether code, models, or checkpoints will be released; for a contribution involving from-scratch pretraining, this information materially affects reproducibility and should be included.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for a top ML venue and the core idea is worth pursuing, but the missing matched forward baseline is the central issue: as written, the AlpacaEval, citation, and retrieval claims all conflate scoring direction with model instruction tuning and model class. The within-family TRLM-Ba versus TRLM-Fo results are the strongest evidence for the thesis and should be foregrounded. The safety evaluation also needs larger samples or explicit uncertainty quantification before the FNR/FPR claims can be accepted. These are fixable with additional experiments rather than a fundamental flaw."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the paper has a real idea—pretrain from scratch on reversed token order and use P(Q|A) as an unsupervised feedback score—but the headline AlpacaEval result overstates it because the Forward Baseline is not instruction-tuned. TRLM-Ba is FLaN-tuned; the baseline isn't. So the 8.17-point LC win-rate gap mostly reflects instruction tuning, not reverse pretraining. The within-family comparison (TRLM-Ba vs TRLM-Fo, both FLaN-tuned, both scoring response→query) gives a credible but smaller 3.25-point gain, and that is the cleanest evidence for the paper's central claim.\n\nWhat's new: prior work did forward-then-reverse training (Golovneva et al.) or used reverse scoring only as a decoding regularizer (Li et al.). Pre-training a model from scratch in reverse order and using it as a reranker/retriever/safety filter is not in those papers. The paper also provides a stylized theoretical argument (a corollary of Yang et al. 2024) and is honest about its assumptions.\n\nWhat it does well: consistent gains across tasks, four model variants, detailed prompts and compute, and a clever defense idea—generate queries from a response and run them through the input filter. The limitations section is candid.\n\nSoft spots, in proportion:\n\n1. The missing matched control. An equally instruction-tuned forward-token model scoring Query→Response is needed to isolate the direction effect. This is fixable and is the main issue.\n\n2. Retrieval and citation baselines are log-perplexity forward scoring, a weak relevance signal. The 44% gains are against a low bar; the forward baseline NDCG@10 on NF-Corpus is 3.89, so the relative gain is hard to interpret.\n\n3. Safety: 11–72 samples, no significance testing, some numbers move erratically with threshold. Suggestive, not conclusive.\n\n4. No code or models released, so the key results aren't independently reproducible yet.\n\nThe theory is stylized by the authors' own admission; that's acceptable for a toy model.\n\nBottom line: this deserves a serious referee and a major revision. I'd accept it only after a matched baseline, a larger safety evaluation, and ideally a model release.","headline":"The reverse-pretraining idea is real, but the headline AlpacaEval comparison confounds token direction with instruction tuning; the within-family comparison is the honest evidence.","tokens_in":22627,"tokens_out":5111,"would_cite":true,"duration_ms":49916,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that time-reversed language models, which score the query given the response, provide useful unsupervised feedback for improving LLM outputs.","keywords":["time-reversed language models","unsupervised feedback","response-to-query scoring","best-of-N reranking","citation attribution","passage retrieval","jailbreak defense"],"falsifier":"On a held-out set of question-answer pairs with human preference labels, compute the Spearman correlation between TRLM-Ba's reverse score and human ratings; if this correlation is statistically indistinguishable from zero, the claim that reverse scoring provides meaningful unsupervised feedback would be falsified.","tokens_in":21582,"feed_emoji":"🔁","tokens_out":6379,"duration_ms":57473,"temperature":0.7,"pith_summary":"Large language models are normally trained to predict the next token in forward order. This paper introduces Time Reversed Language Models (TRLMs), which are pre-trained from scratch to predict in the reversed token order, and therefore naturally score or generate the query given the response. The central claim is that this reverse-direction scoring — response to query — supplies a form of unsupervised feedback that complements the forward model, and that re-ranking forward generations by TRLM scores improves quality. Concretely, TRLM-Ba raises length-controlled win rates on AlpacaEval by about 5 points over self-perplexity reranking, improves citation attribution and retrieval markedly, and cuts false negative rates of input safety filters against jailbreak attacks. The authors present a stylized theoretical model showing that reverse scoring can shift the answer distribution away from hallucinated nearby answers, and they caution that this model and the demonstrated gains are limited to short-query, long-answer settings.","feed_headline":"Reverse-trained LLMs lift AlpacaEval scores 5 points","feed_subtitle":"Scoring queries from responses, with models pre-trained backward, also sharpens retrieval, citations, and jailbreak filters.","key_machinery":"The central object is the Time Reversed Language Model, specifically TRLM-Ba: a model pre-trained from scratch on the same corpus as a PALM2-Otter-style model but in reversed token order, so its next-token prediction is the previous token in the original text. At inference it scores a candidate response $A$ to query $Q$ by computing $\\log P_{\\text{TRLM-Ba}}(\\text{Reverse}(SP+Q) | \\text{Reverse}(CP+A))$, with $SP$ a scoring prompt (e.g. 'Question:') and $CP$ a conditioning prompt (e.g. '? Answer:'). A second variant, TRLM-Fo, is a forward model prompted to score in reverse, and TRLM-FoBa is trained in both directions. The key mechanism that carries the argument is that reverse scoring induces a distribution shift different from temperature scaling: Lemma 2 gives the aligned policy proportional to $P_{\\text{Fw}}(A|Q) P^\\alpha_{\\text{TRLM}}(Q|A)$, and the stylized bipartite-graph model of Appendix A shows this can collapse the support of an imperfect forward model from answers of neighboring questions to the true answer set.","core_discovery":"The paper's central discovery is that a language model pre-trained to predict tokens in reverse — reading 'Answer: ... ?' instead of 'Question: ...' — produces a conditional probability $P_{\\text{TRLM-Ba}}(\\text{Reverse}(\\text{Scoring Prompt} + \\text{Query}) | \\text{Reverse}(\\text{Conditioning Prompt} + \\text{Answer}))$ that works as a scoring function for whether a response plausibly answers a question. The authors claim this reverse score is not just a re-parameterization of the forward log-perplexity: in the KL-constrained alignment framework, using TRLM log-perplexity as reward yields an optimal policy proportional to $P_{\\text{Fw}}(A|Q) P^\\alpha_{\\text{TRLM}}(Q|A)$, whereas forward log-perplexity only rescales temperature. Empirically, using this score to re-rank sixteen Gemini-Pro-1.0 generations against GPT4-1106-Preview gives a length-controlled win rate of 32.44%, versus 27.05% for self log-perplexity reranking; on citation attribution the reverse direction lifts Gecko cosine accuracy by about 44 points; on NF-Corpus retrieval NDCG@10 rises by about 44 points. The same generative reverse model, applied to defense, reduces false negatives of a GPT-3.5 input filter on jailbreak attacks with negligible false-positive impact.","pith_inferences":["If reverse scores are semantically meaningful, they could be used as a training signal (e.g., DPO or RLHF reward) without any human preference labels, potentially making alignment cheaper for new domains; the paper does not explicitly test this.","The success of reverse pre-training suggests an analogous trick for other structured prediction tasks where 'query given response' has a natural inverse, such as summarization or code generation; the paper only demonstrates short-query, long-answer settings.","The dramatic NF-Corpus gains suggest the reverse direction helps whenever documents are much more complex than queries; one could test TRLM as a general retrieval ranker on a broader set of corpora than the two benchmarks reported.","The defense method implicitly assumes that TRLM-generated queries preserve the toxic content classification of the original intent; this could be tested by measuring how often generated queries flip the safety label."],"forward_implications":["If reverse scoring is used as reward in KL-constrained RL, the optimal policy is a product of forward likelihood and TRLM likelihood, not a temperature rescaling; this opens a new axis of alignment without preference data.","Best-of-N reranking with TRLM-Ba achieves a 32.44% length-controlled win rate on AlpacaEval with 16 Gemini-Pro generations, about 5 points above self log-perplexity reranking and about 8 points above a single generation.","Scoring in the direction document-to-query yields 44.19-point gains in NDCG@10 on NF-Corpus and 22.48% recall gains on MS-MARCO versus forward baselines.","Citation attribution using TRLM reverse scoring improves Gecko cosine similarity by roughly 44% on CNN-Daily Mail, with binary and exclusion search reducing inference calls to $O(\\log N)$.","The same TRLM generative capability projects responses back to query space, reducing false negatives of an input safety filter by about 70% on a human-annotated jailbreak dataset while keeping false positives near zero."],"supporting_citations":[{"why":"Supplies Lemma 1, the KL-constrained alignment result that the paper extends to reverse scoring in Lemma 2.","marker":"[Yang et al., 2024b]"},{"why":"Provides the AlpacaEval benchmark and the length-controlled win-rate metric used in the main reranking experiments.","marker":"[Dubois et al., 2024]"},{"why":"Provides the FLaN instruction-tuning dataset used to fine-tune all TRLM variants.","marker":"[Longpre et al., 2023]"},{"why":"Defines the PALM2-Otter pre-training recipe and model family that TRLM pre-training follows, except for token order.","marker":"[Anil et al., 2023b]"},{"why":"Supplies the JailbreakBench attack data and the human-annotated dataset used to evaluate the safety-filter defense.","marker":"[Chao et al., 2024]"}],"fun_headline_variants":["Reverse-trained LLMs add 5 points to AlpacaEval win rate","Backward LLMs score queries from answers, lifting AlpacaEval","Time-reversed LLMs improve reranking, retrieval, and jailbreak defense","Reading LLMs backwards yields better scores and safer filters","Pre-training in reverse gives LLMs unsupervised query-scoring skill"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reverse conditional probability — scoring a question given an answer with a model pre-trained on reversed text — reflects genuine semantic plausibility rather than token-order artifacts, because if that score is just an artifact of reversed language statistics, the reranking, retrieval, and safety gains would vanish.","fun_headline_variants_meta":{"raw":{"variants":["Reverse-trained LLMs add 5 points to AlpacaEval win rate","Backward LLMs score queries from answers, lifting AlpacaEval","Time-reversed LLMs improve reranking, retrieval, and jailbreak defense","Reading LLMs backwards yields better scores and safer filters","Pre-training in reverse gives LLMs unsupervised query-scoring skill"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000321,"raw_usage":{"total_tokens":1894,"prompt_tokens":1117,"completion_tokens":777,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":733,"completion_tokens_details":{"reasoning_tokens":697}},"tokens_in":733,"tokens_out":777,"duration_ms":7529,"temperature":1.0,"reasoning_tokens":697,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:14:09.564732+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of question-answer pairs with human preference labels, compute the Spearman correlation between TRLM-Ba's reverse score and human ratings; if this correlation is statistically indistinguishable from zero, the claim that reverse scoring provides meaningful unsupervised feedback would be falsified.","supporting_citations":[],"review_version":1}