Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that LLM preference alignment is retriever optimization, and that the resulting method, LARPO, beats eight offline baselines and iterative DPO with 38.9% and 13.7% averaged relative gains on AlpacaEval2 and MixEval-Hard.

desk verdict A sensible extension of DPO with listwise IR objectives and iterative on-policy sampling, but the headline gains are selected over hyperparameters tuned on the test benchmarks, so the practical claim is overstated. read the letter →

arxiv 2502.03699 v3 pith:LFLZX2EM submitted 2025-02-06 cs.CL cs.AIcs.IR

Bowen Jin , Jinsung Yoon , Zhen Qin , Ziqi Wang , Wei Xiong , Yu Meng , Jiawei Han , Sercan O. Arik This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords LLMalignmentdirectpreferenceoptimizationlearningtorankretrieverhardnegativeminingcandidatelistconstructionrewardmodelsLARPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that aligning a large language model is the same kind of problem as training a retrieval system, and that information-retrieval techniques can therefore be imported directly into alignment. In the proposed mapping, the LLM plays the role of a retriever that draws candidate responses, and the reward model plays the role of a reranker that scores them; preference optimization becomes a distillation from reranker to retriever. Building on that mapping, the authors introduce LARPO, an iterative alignment method that uses listwise learning-to-rank objectives, deliberately hard negative responses, and carefully constructed candidate lists. They report that LARPO outperforms eight offline baselines and iterative DPO, with 38.9% and 13.7% averaged relative improvements on AlpacaEval2 and MixEval-Hard using the same reward model. If the results hold, alignment becomes simpler to implement and gains access to decades of retriever-optimization knowledge.

What carries the argument

The load-bearing identity is the DPO implicit-reward equation $r(x,y) = \beta \log(\pi_\theta(y|x)/\pi_{\text{ref}}(y|x)) + \beta \log Z$, which lets the paper convert any ranking assumption about reward scores into a policy loss. The paper deploys three ranking assumptions borrowed from IR: contrastive (softmax over one positive versus $m$ negatives), LambdaRank (product of pairwise sigmoids over the full list), and ListMLE (sequential softmax over the list). Around these objectives, the LARPO loop is the retriever-optimization pipeline: generate $k$ candidate responses per prompt, score them with the reward model, split into chosen and rejected responses by rank, update the policy with the ranking loss, and repeat, while keeping previous iterations' responses in the candidate pool to make the list more informative. The retriever-reranker analogy itself carries the argument, since it justifies why hard negatives, list construction, and listwise losses should transfer from IR to alignment.

What would settle it

Retrain LARPO and the DPO-style baselines with all hyperparameters, including generation temperature, number of candidate responses, and the number of chosen and rejected responses, selected only on a held-out development set that is disjoint from AlpacaEval2 and MixEval-Hard, and then evaluate on those benchmarks; if LARPO's 38.9% and 13.7% advantages shrink to noise, the central claim is benchmark-overfit. A complementary check is to evaluate LARPO on an unseen instruction-following benchmark with no re-tuning.

Watch

Extended reading notes

Core claim

The central claim is that direct LLM preference optimization is best understood as retriever optimization: each generation step is a token-level retrieval over the vocabulary, the LLM is a bi-encoder retriever over responses, and the reward model is a cross-encoder reranker. On this analogy, supervised fine-tuning is direct retriever optimization, and preference optimization is reranker-to-retriever distillation. The paper's proposed method, LARPO, operationalizes three IR principles in an iterative loop: it optimizes a ranking loss (contrastive, LambdaRank, or ListMLE) instead of a pairwise DPO loss; it mines hard negatives by sampling incorrect responses at low temperature; and it builds candidate lists that are inclusive and carry over previous iterations' responses. Under the standard implicit-reward identity $\gamma(y|x) = \beta \log(\pi_\theta(y|x)/\pi_{\text{ref}}(y|x))$, each ranking assumption yields a concrete policy objective. The authors claim this consistently beats DPO, SimPO, and the other baselines on AlpacaEval2 and MixEval-Hard.

Load-bearing premise

The reported gains assume that the generation temperatures and candidate-list sizes chosen by searching on AlpacaEval2 and MixEval-Hard (for example, temperature 0.8 or 1.0 and ten candidate responses) are not overfit to those benchmarks, so the same choices would generalize to new prompts, models, and tasks.

Editorial extensions

If this is right

  • Practitioners can replace a dedicated reward-model training pipeline with an off-the-shelf reranker-style reward model and still improve alignment, because LARPO's reported gains use the same reward model as the baselines.
  • Listwise objectives (LambdaRank and ListMLE) and the contrastive objective outperform pairwise DPO-style losses across models, since they exploit more preference information per prompt.
  • Sampling rejected responses at lower temperature produces harder negatives, and harder negatives yield better final policies, giving a concrete recipe for constructing preference data.
  • Growing the candidate list and carrying previous iterations' responses improves alignment, implying that the data-generation side of alignment matters as much as the loss function.
  • The same framework explains Best-of-N inference as retriever-reranker scaling, so alignment training and inference-time scaling can be understood under one lens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the generation temperatures and candidate-list sizes in the main experiments were chosen by searching on AlpacaEval2 and MixEval-Hard; a fair generalization test would fix these hyperparameters on held-out prompts before evaluation, which the paper does not report.
  • Editorial extension: the listwise objectives should transfer to other preference domains such as safety, coding, or multilingual tasks, wherever multiple candidate responses can be ranked by a reward model or a rule-based checker, and the same hard-negative scaling curve is expected.
  • Editorial extension: the retriever-reranker analogy suggests a symmetric pipeline in which the reward model is itself iteratively updated from the policy's ranked outputs, effectively closing the loop as online retriever-reranker co-training; the paper keeps the reward model fixed.
  • Editorial extension: because low-temperature sampling creates harder negatives, one could design a curriculum over sampling temperatures across iterations rather than a fixed searched temperature, potentially making LARPO robust to this hyperparameter choice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper draws an analogy between LLM alignment and information retrieval, treating the LLM as a retriever and the reward model as a reranker. It derives three preference-optimization objectives (contrastive, LambdaRank, ListMLE) by substituting DPO's reward reparameterization into standard IR ranking losses, and introduces LARPO, an iterative algorithm that combines these objectives with hard-negative mining and candidate-list construction. The authors report large gains on AlpacaEval2 and MixEval-Hard, claiming 38.9% and 13.7% averaged relative improvements over baselines. The paper also contains auxiliary studies on Pass@N/Recall@N analogies, hard-negative hardness, and candidate-list size.

Significance. If the reported gains were robust, the paper would be a valuable practical and conceptual contribution: it shows how well-known IR techniques such as listwise objectives, hard negatives, and candidate-list construction can be ported to LLM alignment, and the algebraic derivations in Appendix F are straightforward and checkable. The Pass@N/Recall@N analogy and the empirical analyses of candidate-list size and hard negatives are interesting and potentially reusable. However, the central empirical claim is not currently established. The headline numbers appear to be selected maxima over objective variants and hyperparameters, and the comparison is confounded with the number of sampled responses. The significance of the contribution is therefore conditional on a rigorous re-evaluation that controls for these factors.

major comments (3)
  1. [Appendix H.1 and Table 2] The paper does not report the selection procedure behind the headline improvements, and Appendix H.1 discloses that the generation temperature was searched over {0.8, 0.9, 1.0, 1.1, 1.2} and that the top-k choices (top-1/bottom-3 for contrastive; top-2/bottom-2 for LambdaRank and ListMLE) were set empirically. No held-out validation split is described. If this search was performed on AlpacaEval2 and MixEval-Hard themselves, the reported 38.9% and 13.7% averaged improvements are selected maxima over temperatures, objectives, and top-k configurations, not protocol-general results. The paper must either fix all hyperparameters before evaluation on a held-out split, or report a proper validation-based selection and provide the full grid of configurations tried, so that the abstract's central claim can be assessed.
  2. [Section 5, Table 2] The claim that LARPO consistently outperforms baselines is not supported by all variants. On Mistral-Instruct AlpacaEval2 LC, LARPO (ListMLE) achieves 29.7, below SimPO's 32.1, and LARPO (Contrastive) at 32.7 is only marginally above; the 38.9% average appears to be obtained by selecting the best variant per cell and comparing only against the strongest baseline, SimPO. The paper does not state this selection rule or report the exact computation of the averaged improvements. Please disclose the per-cell best variant, the comparison baseline set, and a summary that accounts for the number of configurations tried.
  3. [Algorithm 1 and Appendix H.1] The comparison between LARPO and iterative DPO is confounded by the number of generated responses: LARPO generates and reward-scores 10 responses per prompt per iteration, whereas iterative DPO generates only 2 responses, and the offline baselines use static preference data. The observed gains could therefore be driven by a larger candidate list and more on-policy reward-model feedback rather than by the IR-based objective. A controlled experiment that matches the number of generated responses (e.g., iterative DPO with 10 responses, or a 2-response variant of LARPO) is necessary to attribute the improvement to the contrastive/listwise objective.
minor comments (5)
  1. [Abstract and Section 4] The method name is spelled inconsistently as 'LarPO' in the abstract and 'LARPO' elsewhere; please unify the notation.
  2. [Appendix G] The baseline description says 'DPO (Guo et al., 2024)', but DPO is introduced in Rafailov et al. (2024); the reference list entry for Guo et al. is a different paper on online AI feedback. The citation should be corrected.
  3. [Appendix F.3] In the statement of the ListMLE objective in Appendix F.3, the denominator is written as exp(gamma(y_i)) + sum_{j=i}^m exp(gamma(y_j)), which double-counts the numerator term; this appears to be a typo because the proof line and the main-text Table 1 use j=i+1.
  4. [Section 3.2] The claim that lower temperatures produce harder negatives is justified partly by reference to Figure 2(b) (Pass@1), but Pass@1 is not a direct measure of negative hardness; the argument would be clearer if the paper presented a direct metric of similarity between positive and negative responses.
  5. [Section 5] The sentence 'with the same reward model as the baselines' is imprecise because the offline baselines are not trained with a reward model; the paper should clarify that the same reward model is used for the on-policy iterative DPO baseline and for LARPO's candidate-list scoring.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: LARPO's objectives are algebraic substitutions of DPO's reward reparameterization into standard ranking losses, and benchmark claims are empirical rather than derived from self-citations.

full rationale

The derivation chain examined here is: (i) the retriever/reranker analogy of Section 2, (ii) the objective construction of Section 3.1 and Table 1, and (iii) the empirical claim of 38.9% and 13.7% improvement in Section 5. Step (ii) is not circular: Eq. (4) is DPO's stated optimal-reward reparameterization, imported from Rafailov et al. (2024), and Appendix F substitutes it algebraically into the contrastive, LambdaRank, and ListMLE likelihoods. The resulting objectives are the paper's method, not a premise used to establish their own outcomes; Table 2 and Table 3 are measurements, not derivations. Step (i) is a framing analogy and does no deductive work: the paper states 'we can conceptualize this process through an IR lens' (Section 2.2) and uses IR notions only to motivate design choices. The hard-negative ordering and candidate-list studies (Sections 6.2-6.3) are experiments whose outcomes are not entailed by the definitions. The paper does cite overlapping-author work, notably Xiong et al. (2024) for iterative preference learning and Liu et al. (2024) for LiPO; these are used as baselines and related work and as motivation for iteration, but LARPO's objective derivation and headline numbers do not depend on accepting those papers' conclusions. Appendix H.1 discloses that generation temperature was searched over {0.8, 0.9, 1.0, 1.1, 1.2} and that candidate and top-k choices were made empirically; this is a legitimate generalization-risk concern about benchmark-reported tuning, but it is not a circularity because no fitted value is renamed as a prediction and no equation reduces to its own input.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The retriever-reranker mapping is an analogy, not an invented mechanism. The free parameters listed above are all chosen by hand or by search on the evaluation sets, representing the main cost the paper pays for its empirical gains.

free parameters (7)
  • Generation temperature for negative sampling (Mistral-7b-base) = 1.0
    Searched among 0.8, 0.9, 1.0, 1.1, 1.2 and selected based on downstream benchmark performance (Appendix H.1).
  • Generation temperature for negative sampling (Mistral-7b-it) = 0.8
    Searched among 0.8, 0.9, 1.0, 1.1, 1.2 and selected based on downstream benchmark performance (Appendix H.1).
  • Hard-negative temperature for GSM8K study = 0.7 (low) and 1.0 (high)
    Chosen to define negative hardness levels in Figure 4(a), Appendix H.5.
  • Number of candidate responses k (Table 2) = 10
    Set for inclusiveness; the paper shows larger candidate lists improve performance up to 10 responses (Figure 4c), so the choice is an empirical optimum on the evaluation tasks.
  • Top-k selection for chosen/rejected = contrastive: top-1/bottom-3; LambdaRank/ListMLE: top-2/bottom-2
    Hyperparameters of the loss construction in Appendix H.1, not justified by theory.
  • Number of responses in Table 3 objective study = pairwise: 2; others: 4
    Appendix H.2; chosen per experiment without a principled criterion.
  • Beta (KL penalty) = not reported
    Standard DPO beta is used but its value for LARPO is not given, so the loss scale is a hidden free parameter.
assumptions (5)
  • standard math The reward model r(x,y) is the optimal reward under the KL-constrained RLHF objective, so r(x,y) = beta * log(pi/pi_ref) + beta * log Z (Equation 4).
    Inherited from DPO (Rafailov et al., 2024), used in Appendix F to derive all LARPO objectives. It assumes the policy pi is the optimal solution to the RLHF objective.
  • domain assumption Preferences over responses follow one of the ranking models in Table 1: softmax (contrastive), product of pairwise sigmoids (LambdaRank), or top-1 chain (ListMLE).
    The paper posits these probability models for the reward-induced ranking (Eqs. 5-7). If the true preference distribution is not of this form, the MLE objectives are misspecified.
  • ad hoc to paper LLM generation can be treated as retrieval from a response corpus, so IR techniques (hard negatives, candidate lists, listwise objectives) transfer to LLM alignment.
    This is the central analogy in Section 2.2 and is assumed, not derived, but it underlies the entire method.
  • domain assumption Lower-temperature generations are harder negatives for a well-initialized policy (Section 3.2).
    Based on Figure 2(b) for GSM8K; the paper states 'assuming a well-initialized policy LLM... low temperatures tend to produce harder negatives.' This may not transfer across tasks.
  • domain assumption Including all previous iterations' responses (memorization) in the candidate list improves training and does not introduce harmful distribution shift.
    Algorithm 1 accumulates D_s across iterations; Table 4 supports it empirically, but the paper gives no theoretical justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Alignment as Retriever Optimization: An Information Retrieval Perspective." pith.science (2026). https://pith.science/paper/LFLZX2EM

@misc{pith2026250203699,
  author       = {Pith},
  title        = {Pith review of: LLM Alignment as Retriever Optimization: An Information Retrieval Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LFLZX2EM}},
  note         = {Machine review of arXiv:2502.03699}
}
read the original abstract

Large Language Models (LLMs) have revolutionized artificial intelligence with capabilities in reasoning, coding, and communication, driving innovation across industries. Their true potential depends on effective alignment to ensure correct, trustworthy and ethical behavior, addressing challenges like misinformation, hallucinations, bias and misuse. While existing Reinforcement Learning (RL)-based alignment methods are notoriously complex, direct optimization approaches offer a simpler alternative. In this work, we introduce a novel direct optimization approach for LLM alignment by drawing on established Information Retrieval (IR) principles. We present a systematic framework that bridges LLM alignment and IR methodologies, mapping LLM generation and reward models to IR's retriever-reranker paradigm. Building on this foundation, we propose LLM Alignment as Retriever Preference Optimization (LarPO), a new alignment method that enhances overall alignment quality. Extensive experiments validate LarPO's effectiveness with 38.9 % and 13.7 % averaged improvement on AlpacaEval2 and MixEval-Hard respectively. Our work opens new avenues for advancing LLM alignment by integrating IR foundations, offering a promising direction for future research.

Figures

Figures reproduced from arXiv: 2502.03699 by the authors.

Figure 1
Figure 1. Architecture connection between retriever/LLM (bi-encoder) and reranker/reward model (cross-encoder). Bi-encoder models process each query/prompt and passage/response separately and often calculate their alignment score via a dot product operator, while cross-encoder models take both query/prompt and passage/response as input and score them directly. Bi-encoder models can be more efficient (i.e., large-scale text ma… view at source ↗
Figure 2
Figure 2. Analogy between evaluating retriever with Recall@N and LLM with Pass@N. As the number (N) of retrieved pas￾sages/generated responses increases, the retriever and LLM have a similar increasing trend. This highlights the importance of infer￾ence time scaling (e.g., Best-of-N) for LLM similar to retriever￾reranker scaling in IR. Retriever: e5; LLM: Mathstral-7b-it. formance under N “ 1 may underestimate the model’s ful… view at source ↗
Figure 3
Figure 3. The connection between iterative LLM alignment (Xiong et al., 2024) and iterative retriever optimization (Xiong et al., 2020) Iterative learning is a common technique in retriever opti￾mization (Xiong et al., 2020), where results from the newly￾trained model are used to generate new training data, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Hard negative and candidate list study. (a) Hard negative study with Lpair on GSM8K with Mathstral-7b-it model. We explore four negative settings: (1) a random response not related to the given prompt; (2) a response to a related prompt; (3) an incorrect response to th…
Figure 5
Figure 5. Figure 5: Evaluate the LLM as a retriever with Recall@N (Pass@N). As the number (N) of retrieved responses increases, the retrieval recall increases. The higher the temperature is, the broader spectrum the retrieved responses are, and thus the higher the recall is. F.1. Contrast…
Figure 6
Figure 6. Figure 6: Training temperature study with Lpair on Gemma2-2b-it and Alpaca Eval 2. Within a specific range (ą 0.9), lower temperature leads to harder negative and benefit the trained LLM. However, temperature lower than this range can cause preferred and rejected responses non-d…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Format rewards help base LLMs more than intermediate retrieval rewards, general-purpose backbones beat reasoning-distilled ones, and stronger search engines stabilize RL training for LLM search agents.

Reference graph

Works this paper leans on

40 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    w. current

    Within a specific range (ą 0.9), lower temperature leads to harder negative and benefit the trained LLM. However, temperature lower than this range can cause preferred and rejected responses non-distinguishable and lead to degrade training. H.3. Table 4 We adopt Gemma2-2b-it as the initial model. All the models are trained with iterative DPO for 3 iterati...

    work page 2023

  3. [5]

    As the number (N) of retrieved responses increases, the retrieval recall increases

    Evaluate the LLM as a retriever with Recall@N (Pass@N). As the number (N) of retrieved responses increases, the retrieval recall increases. The higher the temperature is, the broader spectrum the retrieved responses are, and thus the higher the recall is. F.1. Contrastive ranking Theorem F.1. Let x be a prompt andpyw, yp1q l , ..., ypmq l q be the respons...

    work page 2024

  4. [6]

    in-batch negatives

    We observe that both SFT and DPO improve both retrieval and reranking, with SFT being more effective. Adding DPO after SFT further improves performance on both tasks. This is consistent with information retrieval principles that both direct retriever optimization and reranker-retrieval distillation can enhance the retriever performance, while the latter o...

    work page 2021

  5. [7]

    For AlpacaEval2, we report the result with both opensource LLM evaluator alpaca eval llama3 70b fn and GPT4 evaluator alpaca eval gpt4 turbo fn

    Preference optimization objective study on AlpacaEval2 and MixEval. For AlpacaEval2, we report the result with both opensource LLM evaluator alpaca eval llama3 70b fn and GPT4 evaluator alpaca eval gpt4 turbo fn. SFT corresponds to the initial chat model. AlpacaEval 2 (opensource LLM) AlpacaEval 2 (GPT-4) MixEval MixEval-Hard Method LC Winrate Winrate LC ...

    work page 2024

  6. [8]

    Dubois, Y ., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475,

    arXiv

  7. [9]

    Kto: Model alignment as prospect theoretic optimization

    Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306,

    arXiv

  8. [10]

    Direct language model alignment from online ai feedback.arXiv preprint arXiv:2402.04792,

    9 LLM Alignment as Retriever Optimization: An Information Retrieval Perspective Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y ., Piot, B., et al. Direct language model alignment from online ai feedback.arXiv preprint arXiv:2402.04792,

    arXiv

Show all 40 references
  1. [11]

    Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring

    Humeau, S. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv preprint arXiv:1905.01969,

    1905 arXiv

  2. [12]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a. Jiang, D., Ren, X., and Lin, B. Y . Llm-blender: Ensembling large language ...

    arXiv

  3. [13]

    Dense passage retrieval for open-domain question answering

    Karpukhin, V ., O˘guz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906,

    2004 arXiv

  4. [14]

    Y ., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y ., et al

    Lambert, N., Pyatkin, V ., Morrison, J., Miranda, L., Lin, B. Y ., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y ., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787,

    arXiv

  5. [15]

    From matching to generation: A sur- vey on generative information retrieval

    Li, X., Jin, J., Zhou, Y ., Zhang, Y ., Zhang, P., Zhu, Y ., and Dou, Z. From matching to generation: A sur- vey on generative information retrieval. arXiv preprint arXiv:2404.14851,

    arXiv

  6. [17]

    Ranked list truncation for large language model-based re-ranking

    Meng, C., Arabzadeh, N., Askari, A., Aliannejadi, M., and de Rijke, M. Ranked list truncation for large language model-based re-ranking. In Proceedings of the 47th In- ternational ACM SIGIR Conference on Research and De- velopment in Information Retrieval, pp. 141–151, 2024a. ...

    arXiv

  7. [19]

    and Cho, K

    Nogueira, R. and Cho, K. Passage re-ranking with bert. arXiv preprint arXiv:1901.04085,

    1901 arXiv

  8. [20]

    Document ranking with a pretrained sequence-to-sequence model

    Nogueira, R., Jiang, Z., and Lin, J. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713,

    2003 arXiv

  9. [22]

    Disentan- gling length from quality in direct preference optimiza- tion

    Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. arXiv preprint arXiv:2403.19159,

    arXiv

  10. [23]

    X., Dong, D., Wu, H., and Wang, H

    Qu, Y ., Ding, Y ., Liu, J., Liu, K., Ren, R., Zhao, W. X., Dong, D., Wu, H., and Wang, H. Rocketqa: An optimized train- ing approach to dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2010.08191,

    2010 arXiv

  11. [24]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  12. [25]

    Sessa, P. G., Dadashi, R., Hussenot, L., Ferret, J., Vieil- lard, N., Ram ´e, A., Shariari, B., Perrin, S., Friesen, A., 10 LLM Alignment as Retriever Optimization: An Information Retrieval Perspective Cideron, G., et al. Bond: Aligning llms with best-of-n distillation. arXiv ...

    arXiv

  13. [26]

    I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

    Team, G., Georgiev, P., Lei, V . I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024a. Team, G., Riviere, M., Patha...

    arXiv

  14. [27]

    Self-consistency im- proves chain of thought reasoning in language models

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171,

    arXiv

  15. [28]

    Aligning large language models with human: A survey

    Wang, Y ., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., and Liu, Q. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966,

    arXiv

  16. [30]

    Xu, H., Sharaf, A., Chen, Y ., Tan, W., Shen, L., Van Durme, B., Murray, K., and Kim, Y . J. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417, 2024a. Xu, W., Li, J., Wang, W. Y ., and Li, L. ...

    2024 arXiv

  17. [31]

    Rrhf: Rank responses to align language mod- els with human feedback without tears

    Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language mod- els with human feedback without tears. arXiv preprint arXiv:2304.05302,

    arXiv

  18. [32]

    Curriculum learning for dense retrieval distillation

    Zeng, H., Zamani, H., and Vinay, V . Curriculum learning for dense retrieval distillation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1979–1983,

    1979

  19. [34]

    Zhao, Y ., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023b. 11 LLM Alignment as Retriever Optimization: An Information Retrieval Perspective Zhu, Y ., Yuan, H., Wang...

    arXiv

  20. [37]

    We consider two base models: Mistral-7b-base and Mistral-7b-it

    and MixEval (Ni et al., 2024). We consider two base models: Mistral-7b-base and Mistral-7b-it. For Mistral-7b-base, we first conduct supervised finetuning following Meng et al. (2024b) before the preference optimization. 17 LLM Alignment as Retriever Optimization: An Informati...

    2024

  21. [40]

    We generate 4/6/8/10 responses with the LLM and score the responses with the off-the-shelf reward model (Dong et al., 2024)

    in iterative preference optimization. We generate 4/6/8/10 responses with the LLM and score the responses with the off-the-shelf reward model (Dong et al., 2024). The top-1 scored response is adopted as the positive response and the other responses are treated as the negative ...

    2024

  22. [1952]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  23. [2008]

    Approximate nearest neigh- bor negative contrastive learning for dense text retrieval

    Xiong, L., Xiong, C., Li, Y ., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., and Overwijk, A. Approximate nearest neigh- bor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808,

    2007 arXiv

  24. [2010]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

  25. [2017]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  26. [2019]

    Raft: Re- ward ranked finetuning for generative foundation model alignment

    Dong, H., Xiong, W., Goyal, D., Zhang, Y ., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767,

    arXiv

  27. [2020]

    Oord, A. v. d., Li, Y ., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv:1807.03748,

    arXiv

  28. [2021]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023a. Zhao, W. X., Liu, J., Ren, R., and Wen, J.-R. Dense text retrieval based on pretrained langu...

    arXiv

  29. [2022]

    Lipo: Listwise preference optimization through learning- to-rank

    Liu, T., Qin, Z., Wu, J., Shen, J., Khalman, M., Joshi, R., Zhao, Y ., Saleh, M., Baumgartner, S., Liu, J., et al. Lipo: Listwise preference optimization through learning- to-rank. arXiv preprint arXiv:2402.01878,

    arXiv

  30. [2023]

    Rlhf workflow: From reward modeling to online rlhf

    Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y ., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863,

    arXiv

  31. [2024]

    Training a helpful and harmless assistant with rein- forcement learning from human feedback

    Bai, Y ., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862,

    arXiv

  32. [2025]

    Accessed: 2025- 01-16

    URL https:// mistral.ai/news/mathstral/. Accessed: 2025- 01-16. Ni, J., Xue, F., Yue, X., Deng, Y ., Shah, M., Jain, K., Neu- big, G., and You, Y . Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures. arXiv preprint arXiv:2406.06565,

    2025 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.