Pith. sign in

REVIEW 2 major objections 5 minor 49 references

Tiny Reward Models

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 400-million-parameter bidirectional masked language model, trained with cloze-style instruction prompting, low-rank adaptation, and layer freezing, matches a 70-billion-parameter reward model on reasoning preference judgments.

desk verdict A promising recipe for cheap reward models, but the 'rivals 175x larger' claim rests on incompatible evaluation protocols and needs re-scoping. read the letter →

arxiv 2507.09973 v1 pith:YABZKNAW submitted 2025-07-14 cs.CL cs.AI

Sarah Pan This is my paper
classification cs.CLcs.AI
keywords rewardmodelingRLHFbidirectionalmaskedlanguagemodelsparameter-efficientfinetuningDoRAFLAN-stylepromptingBenchefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that reward models used to steer large language models do not need to be giant decoder-only networks. It presents TinyRM, a family of bidirectional masked language models with 150 million and 400 million parameters, and reports that the 400 million parameter specialist reaches 91.2 percent on RewardBench's Reasoning category, essentially matching the 90.6 percent of a 70 billion parameter decoder-based reward model, with Safety close behind at 89.3 versus 92.8 percent. The method rephrases preference modeling as a fill-in-the-blank cloze task, then finetunes with weight-decomposed low-rank adaptation (DoRA) and frozen lower layers, training separate specialists for each domain. If these results hold, preference signals for routing, data filtering, and step-by-step supervision could be produced at a small fraction of today's inference cost, while a single generalist model remains an open problem.

What carries the argument

The load-bearing object is the FLAN-style cloze reformulation of reward modeling: given an instruction prefix, the problem statement, and both the chosen and rejected responses, the model predicts a masked preference token, and the probability of the correct token serves as the preference score. DoRA (weight-decomposed low-rank adaptation) splits weights into magnitude and direction and applies low-rank updates to the direction; the paper applies it to the Reasoning specialists and credits it for the gains there. Freezing the lower layers preserves general language representations while the task-specific upper layers are tuned. This combination converts a pretrained bidirectional encoder into a reward model without a classification head, in a single forward pass that sees both candidate responses.

What would settle it

Run the 400 million parameter Reasoning specialist through RewardBench's official pointwise evaluation, scoring each response separately and comparing the two scores. If its Reasoning accuracy falls well below the 70 billion parameter model's official 90.6, the claim that it rivals a model over 175 times larger loses its support.

Watch

Extended reading notes

Core claim

The paper's central claim is that small bidirectional encoders can serve as effective reward models, rivaling models over 175 times larger on reasoning and safety preference modeling. On RewardBench, its 400 million parameter Reasoning specialist reaches 91.2 percent accuracy against 90.6 percent for a 70 billion parameter decoder-based reward model, and its Safety specialist reaches 89.3 percent against 92.8 percent. The paper also reports that the 150 million parameter specialists beat a 400 million parameter all-at-once generalist, that DoRA specifically benefits the Reasoning domain, and that open-ended conversation remains the weak spot where these models lag.

Load-bearing premise

TinyRM is trained and evaluated with both the chosen and rejected responses visible in the prompt, while the leaderboard baselines it is compared with are scored with official RewardBench's pointwise setup, and the paper reports no TinyRM numbers under that official setup.

Editorial extensions

If this is right

  • Reasoning and safety reward modeling can be done with sub-billion-parameter encoders, cutting per-token inference cost by roughly two orders of magnitude compared with 70B-class decoders.
  • Parameter-efficient methods such as DoRA can elicit reasoning preferences from small models more effectively than full finetuning, so scaling the number of trainable parameters is not the only route to better reward models.
  • Domain-specific specialists are stronger than a single all-at-once model at the same or larger size; a 150M specialist outperforms a 400M generalist.
  • Small encoders still trail on conversational preference modeling, and one epoch of supervised finetuning on conversational data recovers some but not all of the gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the pairwise setup is not only a threat to comparability but also a deployment recipe, since any reranking pipeline that has two candidate responses can feed both to TinyRM and use the reported operating point.
  • My inference: DoRA's outsized effect on reasoning suggests the preference judgment is already largely present in the pretrained bidirectional representations; probing where low-rank updates change attention could localize that knowledge.
  • My inference: a successful generalist tiny reward model may come from training the cloze objective on mixed-domain instruction data rather than averaging specialist weights, which the paper reports failing.
  • My inference: the largest practical payoff would be in inference-time supervision and agentic routing, where the reward model is invoked many times per task; the paper names these uses but does not benchmark end-to-end latency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes TinyRM, a family of reward models built from ModernBERT-Base and ModernBERT-Large (150M and 400M parameters), trained with FLAN-style masked prediction on preference pairs, augmented with DoRA and layer freezing. It reports RewardBench accuracies for domain-specific specialists, an all-at-once model, and comparisons to a 70B LLaMA-3-SteerLM-RM and other baselines. The central claim, stated in the abstract and Section 5, is that 400M specialists rival models over 175 times larger on reasoning and safety preference modeling. The authors acknowledge in Section 3.2 that their training and evaluation format shows both responses to the model, which is not the official RewardBench setup, and they explicitly call the comparison "not apples-to-apples."

Significance. If the headline comparison survived a matched evaluation protocol, the result would be a meaningful efficiency contribution: a 400M bidirectional encoder trained with FLAN-style masking, DoRA, and layer freezing could serve as a much cheaper reward model for reasoning and safety tasks, with potential impact on test-time decoding and routing. The paper is transparent about its main caveat, uses public datasets, and reports a concrete training recipe with hyperparameters in Appendix A. It does not offer a mathematical derivation, but it also does not rely on circular definitions; the decisive question is whether the measurement supports the cross-model comparison. The lack of an apples-to-apples evaluation currently prevents the paper's strongest claim from being established.

major comments (2)
  1. [Section 3.2 and Table 1] The headline comparison is invalid as stated because TinyRM is trained and evaluated with both the chosen and rejected responses visible in the prompt, while the official RewardBench scores for LLaMA-3-SteerLM-RM 70B and the other baselines are obtained under pointwise scoring. The paper concedes this in Section 3.2, and the deployment argument that follows addresses deployability, not measurement validity. Since the abstract and Section 5 claim that TinyRM models "rival" models over 175 times larger based on the numbers in Table 1, the central claim is currently unsupported. Please provide an evaluation of TinyRM under the official pointwise RewardBench protocol, or evaluate the comparison models under the pairwise protocol, and report both sets of numbers.
  2. [Appendix A and Table 1] The hyperparameter sweeps in Appendix A are reported without stating which RewardBench split was used for model selection, and the phrase "configurations that yielded the results shown" suggests that the evaluation set itself may have been used for selection. Each TinyRM row in Table 1 is the best result of a separate domain-specific sweep, whereas the 70B baseline is a single generalist model, so the comparison conflates model capability with test-set selection. Please clarify the selection procedure, use a held-out validation split if one was not used, and report the full sweep results or variance across seeds so the reader can judge whether the 91.2 versus 90.6 Reasoning difference is robust.
minor comments (5)
  1. [Abstract and Section 5] The caveat from Section 3.2 should appear in the abstract and conclusion; the unqualified statement that TinyRM models "rival" models over 175 times larger is stronger than what the current evaluation supports.
  2. [Section 3.1] The sentence claiming that "DoRA provided significant performance gains" for reasoning is not backed by an ablation table, and Section 3 states that ablations are left to future work; please soften the claim or include a minimal comparison with and without DoRA.
  3. [Appendix A] Please report the exact RewardBench split used for the hyperparameter sweeps, the selection criterion, and the number of random seeds; without this information the reported accuracies cannot be interpreted as stable estimates.
  4. [Figure 3] The axes and plotted points are not fully specified; please state which models are included, which RewardBench categories are aggregated, and how "proportionate loss in accuracy" is defined.
  5. [Section 4] The phrase "84% percentile" should be "84th percentile."

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TinyRM's claims are empirical and evaluated on RewardBench; related-group citations are used as external building blocks, not as the conclusion.

full rationale

The paper contains no derivation in which a predicted quantity is defined in terms of its inputs, and no fitted parameter is renamed as a prediction. TinyRM's performance is an empirical result obtained by fine-tuning ModernBERT with a FLAN-style cloze objective and measuring accuracy on the external RewardBench benchmark. The two building blocks from the author's affiliated group—ModernBERT (Warner et al., 2024) and the FLAN-style masked classifier (Clavié et al., 2025)—are cited as pretrained infrastructure and are independently reproducible public artifacts; the paper's central claim does not reduce to those citations. The paper explicitly discloses its main evaluation caveat: 'our models are trained and evaluated with visibility of both options in their contexts, which is not the case for official RewardBench evaluation' and 'we admit ours is not an apples-to-apples comparison.' That is a methodological validity limitation, not circularity, because the conclusion is still measured against an external benchmark rather than defined into existence. Hyperparameter sweeps could create selection effects, but the paper reports observed configurations and does not present model-selection outcomes as theoretically forced predictions. Therefore no load-bearing step reduces to its own input by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No invented entities. The free parameters are the hyperparameters selected by sweep, plus the unreported Chat/Chat-Hard weighting. The axioms are standard empirical-evaluation assumptions plus the paper's own protocol-comparability assumption, which it flags as not apples-to-apples.

free parameters (5)
  • Learning rate per domain/model = Large: Chat 7.865e-5, Reasoning 2.327e-5, Safety 5.845e-5, AAO 9.388e-5; Base: Chat 7.642e-5, Reasoning 8.354e-5…
    Chosen by Bayesian hyperparameter sweep; the reported results use the best configuration found on RewardBench.
  • Number of frozen layers per domain/model = Large: Chat 26, Reasoning 12, Safety 7, AAO 5; Base: Chat 2, Reasoning 17, Safety 12
    Chosen by sweep; freezing lower layers is a core claimed contributor to performance.
  • Instruction prefix per domain = Chat: 'Select the best response.'; Reasoning: 'Which response is more correct?'; Safety: 'Which response is safer?'…
    A pool of instruction prefixes was part of the hyperparameter search; the winning prefixes are reported in Appendix A, but the AAO prompt is not specified.
  • DoRA rank = 128 (Reasoning specialists only)
    Rank chosen by sweep; DoRA was applied only to reasoning models.
  • Chat/Chat-Hard weighted-average weights = Not reported
    The paper merges Chat and Chat-Hard into a single Chat score with a weighted average but does not state the weights, so the exact aggregate cannot be reproduced.
assumptions (5)
  • domain assumption RewardBench accuracy is a valid proxy for reward-model quality.
    The entire evaluation is on RewardBench; the central claim is defined by scores on this benchmark.
  • domain assumption The pairwise protocol (both responses in the prompt) and the pointwise protocol used for leaderboard models are commensurable.
    The paper admits 'ours is not an apples-to-apples comparison' (Section 3.2), yet the headline comparison to the 70B model assumes the scores are comparable.
  • domain assumption The public preference datasets (Skywork decontaminated, WebGPT, HH-RLHF, PKU-SafeRLHF) are sufficient to train capable reward specialists.
    No analysis of data coverage or saturation is provided; the paper relies on these datasets without showing they adequately represent RewardBench's distribution.
  • domain assumption The FLOPs approximation FLOPs = 2N_params + 6N_params*L/H accurately captures inference cost.
    Appendix C uses this formula to claim two-orders-of-magnitude savings; it ignores latency, memory bandwidth, and other practical costs.
  • domain assumption The masked-token prediction P(m | x, y_w, y_l) correctly encodes preference.
    The method converts preference modeling to a cloze task; there is no calibration analysis showing the predicted token probability reliably orders responses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tiny Reward Models." pith.science (2026). https://pith.science/paper/YABZKNAW

@misc{pith2026250709973,
  author       = {Pith},
  title        = {Pith review of: Tiny Reward Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YABZKNAW}},
  note         = {Machine review of arXiv:2507.09973}
}
read the original abstract

Large decoder-based language models have become the dominant architecture for reward modeling in reinforcement learning from human feedback (RLHF). However, as reward models are increasingly deployed in test-time strategies, their inference costs become a growing concern. We present TinyRM, a family of small, bidirectional masked language models (MLMs) with as few as 400 million parameters, that rival the capabilities of models over 175 times larger on reasoning and safety preference modeling tasks. TinyRM combines FLAN-style prompting, Directional Low-Rank Adaptation (DoRA), and layer freezing to achieve strong performance on RewardBench, despite using significantly fewer resources. Our experiments suggest that small models benefit from domain-specific tuning strategies, particularly in reasoning, where lightweight finetuning methods are especially effective. While challenges remain in building generalist models and conversational preference modeling, our preliminary results highlight the promise of lightweight bidirectional architectures as efficient, scalable alternatives for preference modeling.

Figures

Figures reproduced from arXiv: 2507.09973 by the authors.

Figure 1
Figure 1. Overview of our finetuning pipeline: DoRA and layer freezing combined with FLAN-style, cloze prompting convert a pretrained MLM into a reward model. Recent approaches to training strong reward models rely on scaling up decoder-based LLMs as RMs, under the as￾sumption that bigger models generalize better (Nvidia et al., 2024; Wang et al., 2024; Winata et al., 2025). However, while large RMs represent a one-time expen… view at source ↗
Figure 3
Figure 3. TinyRM cuts inference costs by two orders of magnitude without suffering a proportionate loss in accuracy. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 6 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, ...

    arXiv 2022

  3. [3]

    PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding

    Chaffin, A., Claveau, V., and Kijak, E. Ppl-mcts: Constrained textual generation through discriminator-guided mcts decoding, 2022. URL https://arxiv.org/abs/2109.13582

    work page Pith review arXiv 2022

  4. [4]

    Rm-r1: Reward modeling as reasoning, 2025

    Chen, X., Li, G., Wang, Z., Jin, B., Qian, C., Wang, Y., Wang, H., Zhang, Y., Zhang, D., Zhang, T., Tong, H., and Ji, H. Rm-r1: Reward modeling as reasoning, 2025. URL https://arxiv.org/abs/2505.02387

    arXiv 2025

  5. [5]

    B., Martic, M., Legg, S., and Amodei, D

    Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences, 2023

    work page 2023

  6. [6]

    W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S

    Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts,...

    arXiv 2022

  7. [7]

    It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifiers

    Clavié, B., Cooper, N., and Warner, B. It's all in the [mask]: Simple instruction-tuning enables bert-like masked language models as generative classifiers, 2025. URL https://arxiv.org/abs/2502.03793

    work page Pith review arXiv 2025

  8. [8]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805

    arXiv 2019

Show all 49 references
  1. [9]

    and Li, Y

    Eldan, R. and Li, Y. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https://arxiv.org/abs/2305.07759

    2023 arXiv

  2. [10]

    L., and Vigliocco, G

    Fernandez Monsalve, I., Frank, S. L., and Vigliocco, G. Lexical surprisal as a general predictor of reading time. In Daelemans, W. (ed.), Proceedings of the 13th Conference of the E uropean Chapter of the Association for Computational Linguistics , pp.\ 398--408, Avignon, Fran...

    2012

  3. [11]

    Scaling laws for reward model overoptimization, 2022

    Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization, 2022. URL https://arxiv.org/abs/2210.10760

    2022 arXiv

  4. [12]

    Making pre-trained language models better few-shot learners

    Gao, T., Fisch, A., and Chen, D. Making pre-trained language models better few-shot learners. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen...

    2021 doi

  5. [13]

    Ghanizadeh, M. A. and Dousti, M. J. Towards data-efficient language models: A child-inspired approach to language learning, 2025. URL https://arxiv.org/abs/2503.04611

    2025 arXiv

  6. [14]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gre...

    2024 arXiv

  7. [15]

    Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need, 2023. ...

    2023 arXiv

  8. [16]

    Does rlhf scale? exploring the impacts from data, model, and method, 2024

    Hou, Z., Du, P., Niu, Y., Du, Z., Zeng, A., Liu, X., Huang, M., Wang, H., Tang, J., and Dong, Y. Does rlhf scale? exploring the impacts from data, model, and method, 2024. URL https://arxiv.org/abs/2412.06000

    2024 arXiv

  9. [17]

    and Ruder, S

    Howard, J. and Ruder, S. Universal language model fine-tuning for text classification, 2018. URL https://arxiv.org/abs/1801.06146

    2018 arXiv

  10. [18]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  11. [19]

    Pku-saferlhf: Towards multi-level safety alignment for llms with human preference, 2024

    Ji, J., Hong, D., Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Li, B., and Yang, Y. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference, 2024. URL https://arxiv.org/abs/2406.15513

    2024 arXiv

  12. [20]

    M., Stanley, O., Nagyfi, R., ES, S., Suri, S., Glushkov, D., Dantuluri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A

    Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., ES, S., Suri, S., Glushkov, D., Dantuluri, A., Maguire, A., Schuhmann, C., Nguyen, H., and Mattick, A. Openassistant conversations -- democratizin...

    2023 arXiv

  13. [21]

    Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N

    Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Rewardbench: Evaluating reward models for language modeling, 2024. URL https://arxiv.org/abs/2403.13787

    2024 arXiv

  14. [22]

    What would elsa do? freezing layers during transformer fine-tuning, 2019

    Lee, J., Tang, R., and Lin, J. What would elsa do? freezing layers during transformer fine-tuning, 2019. URL https://arxiv.org/abs/1911.03090

    2019 arXiv

  15. [23]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019

    Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019. URL https://arxiv.org/abs/1910.13461

    2019 arXiv

  16. [24]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. URL https://arxiv.org/abs/2005.11401

    2021 arXiv

  17. [25]

    Let's verify step by step, 2023

    Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step, 2023. URL https://arxiv.org/abs/2305.20050

    2023 arXiv

  18. [26]

    Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y

    Liu, C. Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in llms, 2024 a . URL https://arxiv.org/abs/2410.18451

    2024 arXiv

  19. [27]

    F., Cheng, K.-T., and Chen, M.-H

    Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation, 2024 b . URL https://arxiv.org/abs/2402.09353

    2024 arXiv

  20. [28]

    Routing to the expert: Efficient reward-guided ensemble of large language models, 2023

    Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C., and Zhou, J. Routing to the expert: Efficient reward-guided ensemble of large language models, 2023. URL https://arxiv.org/abs/2311.08692

    2023 arXiv

  21. [29]

    Improve mathematical reasoning in language models by automated process supervision, 2024

    Luo, L., Liu, Y., Liu, R., Phatale, S., Guo, M., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., and Rastogi, A. Improve mathematical reasoning in language models by automated process supervision, 2024. URL https://arxiv.org/abs/2406.06592

    2024 arXiv

  22. [30]

    Webgpt: Browser-assisted question-answering with human feedback, 2022

    Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. Webgpt: Browser-assisted question-answering with human feedbac...

    2022 arXiv

  23. [31]

    and Cho, K

    Nogueira, R. and Cho, K. Passage re-ranking with bert, 2020. URL https://arxiv.org/abs/1901.04085

    2020 arXiv

  24. [32]

    Nvidia, :, Adler, B., Agarwal, N., Aithal, A., Anh, D. H., Bhattacharya, P., Brundyn, A., Casper, J., Catanzaro, B., Clay, S., Cohen, J., Das, S., Dattagupta, A., Delalleau, O., Derczynski, L., Dong, Y., Egert, D., Evans, E., Ficek, A., Fridman, D., Ghosh, S., Ginsburg, B., Gi...

    2024 arXiv

  25. [33]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to f...

    2022 arXiv

  26. [34]

    Let's reinforce step by step, 2023

    Pan, S., Lialin, V., Muckatira, S., and Rumshisky, A. Let's reinforce step by step, 2023. URL https://arxiv.org/abs/2311.05821

    2023 arXiv

  27. [35]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiv.org/abs/1910.10683

    2023 arXiv

  28. [36]

    Warm: On the benefits of weight averaged reward models, 2024

    Ramé, A., Vieillard, N., Hussenot, L., Dadashi, R., Cideron, G., Bachem, O., and Ferret, J. Warm: On the benefits of weight averaged reward models, 2024. URL https://arxiv.org/abs/2401.12187

    2024 arXiv

  29. [37]

    Berts are generative in-context learners, 2024

    Samuel, D. Berts are generative in-context learners, 2024. URL https://arxiv.org/abs/2406.04823

    2024 arXiv

  30. [38]

    and Schütze, H

    Schick, T. and Schütze, H. Exploiting cloze questions for few shot text classification and natural language inference, 2021 a . URL https://arxiv.org/abs/2001.07676

    2021 arXiv

  31. [39]

    and Schütze, H

    Schick, T. and Schütze, H. It's not just size that matters: Small language models are also few-shot learners, 2021 b . URL https://arxiv.org/abs/2009.07118

    2021 arXiv

  32. [40]

    The trickle-down impact of reward (in-)consistency on rlhf, 2023

    Shen, L., Chen, S., Song, L., Jin, L., Peng, B., Mi, H., Khashabi, D., and Yu, D. The trickle-down impact of reward (in-)consistency on rlhf, 2023. URL https://arxiv.org/abs/2309.16155

    2023 arXiv

  33. [41]

    R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., and Shwartz-Ziv, R

    Skean, O., Arefin, M. R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., and Shwartz-Ziv, R. Layer by layer: Uncovering hidden representations in language models, 2025. URL https://arxiv.org/abs/2502.02013

    2025 arXiv

  34. [42]

    D., and Su, W

    Song, Z., Cai, T., Lee, J. D., and Su, W. J. Reward collapse in aligning large language models, 2023. URL https://arxiv.org/abs/2305.17608

    2023

  35. [43]

    Large gpt-like models are bad babies: A closer look at the relationship between linguistic competence and psycholinguistic measures, 2023

    Steuer, J., Mosbach, M., and Klakow, D. Large gpt-like models are bad babies: A closer look at the relationship between linguistic competence and psycholinguistic measures, 2023. URL https://arxiv.org/abs/2311.04547

    2023 arXiv

  36. [44]

    How to fine-tune bert for text classification?, 2020

    Sun, C., Qiu, X., Xu, Y., and Huang, X. How to fine-tune bert for text classification?, 2020. URL https://arxiv.org/abs/1905.05583

    2020 arXiv

  37. [45]

    J., Sreedhar, M

    Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., and Kuchaiev, O. Helpsteer2: Open-source dataset for training top-performing reward models, 2024. URL https://arxiv.org/abs/2406.08673

    2024 arXiv

  38. [46]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024

    Warner, B., Chaffin, A., Clavié, B., Weller, O., Hallström, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., Cooper, N., Adams, G., Howard, J., and Poli, I. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and l...

    2024 arXiv

  39. [47]

    Y., Guu, K., Yu, A

    Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners, 2022. URL https://arxiv.org/abs/2109.01652

    2022 arXiv

  40. [48]

    I., Anugraha, D., Susanto, L., Kuwanto, G., and Wijaya, D

    Winata, G. I., Anugraha, D., Susanto, L., Kuwanto, G., and Wijaya, D. T. Metametrics: Calibrating metrics for generation tasks using human preferences, 2025. URL https://arxiv.org/abs/2410.02381

    2025 arXiv

  41. [49]

    How transferable are features in deep neural networks?, 2014

    Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks?, 2014. URL https://arxiv.org/abs/1411.1792

    2014 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.