Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

A model's expressed values are largely set during supervised fine-tuning, and subsequent preference optimization rarely re-aligns them unless the preference data carries an explicit value contrast. This claim is established by tracing stanc

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:06 UTC pith:BDKYUUXF

load-bearing objection A well-documented empirical study showing SFT sets value priors and standard preference data barely moves them; the headline claim is a bit overstated, but the core result is legitimate and should be refereed. the 3 major comments →

arxiv 2510.26707 v2 pith:BDKYUUXF submitted 2025-10-30 cs.CL cs.CYcs.LG

Value Drifts: Tracing Value Alignment During LLM Post-Training

Mehar Bhatia , Shravan Nayak , Gaurav Kamath , Marius Mosbach , Karolina Sta\'nczak , Vered Shwartz , Siva Reddy This is my paper
classification cs.CL cs.CYcs.LG
keywords value alignmentpost-training dynamicssupervised fine-tuningpreference optimizationvalue driftLLM stancesUltraFeedbackHH-RLHF
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that, in the standard two-stage post-training pipeline, the supervised fine-tuning (SFT) phase is the dominant force shaping an LLM's values, and that preference optimization (PPO, DPO, SimPO) only weakly corrects those values when trained on popular preference datasets. It argues that these datasets fail to re-align values because the chosen and rejected responses in each preference pair exhibit nearly identical stance distributions, a property the paper calls a small value-gap. Using a synthetic preference dataset with a deliberately large value-gap, the paper shows that preference optimization algorithms can re-align values, but in different ways: PPO preserves the SFT values, DPO amplifies stances already favored by the SFT prior, and SimPO produces smaller, slower drifts. If correct, these findings give practitioners a concrete rule: to change a model's values, the strongest lever is the SFT corpus, not the subsequent preference optimization stage.

Core claim

The paper's central discovery is that value formation in LLM post-training is largely completed during supervised fine-tuning. By sampling checkpoints throughout training and measuring stance distributions over a curated set of value-laden prompts (the V-PRISM set), the authors observe that SFT rapidly and strongly shifts a model's stances toward the value distribution of the instruction-tuning data, and that subsequent preference optimization with standard datasets (UltraFeedback and HH-RLHF) barely moves those stances. The paper attributes this to a small value-gap in standard preference data: chosen and rejected response pairs show nearly identical stance distributions, providing a weak t

What carries the argument

The central machinery is the operationalization of values as stance distributions: a model's value on a topic is defined as the expected probability of support, neutral, or oppose stances across generated responses, measured by GPT-4o classification of responses to a curated prompt set (V-PRISM). Two metrics, drift magnitude and drift time, quantify how much and how quickly a model's stance distribution changes between checkpoints. To explain why preference optimization often has little effect, the paper introduces the value-gap: the Euclidean distance between the stance distributions of the chosen and rejected responses in a preference pair. The synthetic preference dataset, built by genera

Load-bearing premise

The central measurement assumption is that a GPT-4o three-way stance label (support/neutral/oppose), averaged over five generations, captures a model's latent values independently of response style; if the judge's 'neutral' category mostly absorbs hedged or refusal-style text, then the small value-gap and the drift-to-neutral results could be partly artifacts of the measurement rather than genuine value properties.

What would settle it

Re-run the value-drift measurements using a stance classifier that separates style from content (for example, asking annotators to judge the underlying position even when the response is hedged, or using a continuous stance scale rather than the three-way neutral-inclusive scheme). If, on standard preference datasets like UltraFeedback and HH-RLHF, the chosen and rejected responses then show large value gaps, and if PPO or DPO on those datasets produces substantial stance shifts, the paper's central claim that SFT fixes values and preference optimization rarely moves them would be refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Data curation: to steer a model's values, the SFT corpus is the primary control; preference data without explicit stance contrast will not correct the SFT-imposed value profile.
  • Algorithm selection: if SFT has already produced the desired values, PPO is a safe choice because it preserves those values; DPO is preferable when the goal is to amplify a value the model already leans toward.
  • Preference data design: a small value-gap in preference pairs is a measurable diagnosis for why a preference optimization run may fail to change model values.
  • Evaluation practice: checkpoint-level value drift during SFT can be used for early attribution of a model's eventual stances, enabling detection of value misalignment before the final model is deployed.
  • Hyperparameter effects: the KL coefficient in PPO and the β in DPO control the strength of value drift, with smaller coefficients allowing more drift, giving practitioners a tuning lever for value alignment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: the small value-gap in standard preference datasets may partly be a measurement artifact, since the GPT-4o judge's neutral category absorbs hedged, refusal-like responses; re-measuring with a style-controlled judge could reveal larger value contrasts than the paper reports.
  • The paper's finding implies that SFT is not merely teaching instruction-following but is the de facto value-alignment stage; this reframes the role of the SFT dataset in any safety or pluralism effort, suggesting that SFT data should be audited for value biases directly.
  • Looking beyond the paper, the same measurement framework could be applied to pretraining checkpoints to test whether SFT is imprinting new values or simply exposing latent stances already acquired during pretraining; if the latter, value alignment begins even earlier than this study shows.
  • The algorithm-specific divergence on synthetic data suggests a practical recipe for controlled value editing: combine a large-gap preference dataset with DPO to amplify a chosen stance, or with a small KL penalty in PPO to override the SFT prior, and validate using the paper's drift metrics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces a methodology (V-PRISM, drift magnitude, drift time) for tracing how LLM stances on value-laden topics change during post-training. Using Llama-3 and Qwen-3 models, SFT on WildChat/Alpaca, and PPO/DPO/SIMPO on UltraFeedback/HH-RLHF, the authors report three findings: (1) SFT rapidly establishes the model's value profile; (2) standard preference optimization induces little additional value drift, which they attribute to a small value-gap between chosen and rejected responses; and (3) a synthetic preference dataset with a deliberately large value-gap reveals algorithm-specific behavior (PPO preserves SFT values, DPO amplifies SFT-consistent stances, SIMPO gives smaller/slower drifts). The paper includes extensive appendices with per-topic tables, hyperparameter ablations, and downstream-task checks.

Significance. If the measurement is valid, the central claim—that SFT is the dominant locus of value formation and preference optimization is a weak corrector under typical data—is an important, actionable result for alignment research and data curation. The paper's strengths are its controlled synthetic-preference manipulation, transparent methodology with detailed appendices (Tables 9–16), explicit hyperparameter ablations (§J), and the stated plan to release code, checkpoints, and data. The main risk is that the stance classifier may conflate response style with latent value, which would undermine the empirical basis for the central claim.

major comments (3)
  1. [§2.1, Eq. (1); §D; §K.3] The central measurement v_θ(T) is based on GPT-4o three-way stance labels with 'neutral' defined as 'neither supports nor opposes, or avoids expressing a stance.' Appendix D (Tables 5–7) shows that WildChat-SFT models produce long, hedged, refusal-like responses that the judge likely maps to neutral. Thus the large SFT drift toward neutral (e.g., M_neutral = 0.38–0.76 in Tables 9–12) and the apparent stability during PO may be style imprinting, not latent value change. The same confound affects the value-gap analysis in §K.3: if most responses collapse to neutral, the near-zero Euclidean distances may be a floor effect. The Ethics Statement acknowledges semantic flattening but not this style confound. Please provide robustness evidence—e.g., a forced-choice or style-controlled annotation, or conditioning the value-gap analysis on response length/hedging—to show that the low value-gap and
  2. [§5.2, Table 1] The claim of 'minimal to no value drift' overstates the paper's own numbers. Table 1 reports SIMPO climate-change support drift magnitude 0.27 and PPO 0.20; these are comparable to many SFT drift magnitudes in Tables 9–12. The drift-time value for SIMPO climate-change oppose is 0.84, which contradicts the statement that 'drift time is also low.' The conclusion should be re-calibrated to 'small but non-negligible drifts, with magnitudes varying by algorithm and topic' rather than 'minimal to no drift,' unless a quantitative threshold for 'minimal' is specified.
  3. [§6.1, Table 4] Table 4 lists the same value for the 'Value (Support)' and 'Value (Oppose)' columns for Climate Change Opinions ('support climate change action' in both). This is either a typo or a serious error in the synthetic data generation: the oppose-aligned condition for climate change would not actually oppose the value. Since Tables 13–16 report full results across all topics, this affects the validity of the climate-change rows in the synthetic experiment and needs to be corrected and the affected results re-examined.
minor comments (4)
  1. [§3, drift time] The definition of drift time uses the 95% confidence interval of the extremum, but it is unclear how the extremum is determined when the trajectory is noisy or non-monotonic. Please clarify the estimation procedure and any smoothing.
  2. [§K.1] The cosine-similarity threshold of 0.5 used to retrieve dataset datapoints is not motivated. A brief sensitivity analysis (e.g., thresholds 0.4/0.6) would strengthen the value-gap conclusion.
  3. [Appendix K, Fig. 17] The caption for Fig. 17 is incomplete: it ends with 'On the other hand,' and the sentence is cut off. Please complete the caption.
  4. [Table 4] Aside from the climate-change row, consider adding a note explaining how the value mapping was validated (e.g., the manual verification of 100 pairs described in §6.1) to preempt concerns about prompt adherence.

Circularity Check

0 steps flagged

No circular derivation; claims are empirical measurements with disclosed controls.

full rationale

The paper's central claims—SFT establishes a model's value profile and preference optimization rarely re-aligns it—are empirical findings about measured stance distributions (Eq. 1), not consequences of how those distributions are defined. v_theta(T) is computed from GPT-4o stance labels on model generations (§3), independently of the SFT/PO objectives, so no 'prediction' is equivalent to a fitted input. The value-gap analysis (§K.3) measures a property of preference datasets with the same judge; even if the judge's neutral bucket creates a floor effect (a validity concern, acknowledged in the Ethics Statement as stance proxy oversimplification), it is not a construction that forces the drift results. The synthetic experiment (§6.1) is a controlled manipulation: support/oppose responses are generated by an external model and labels are flipped; the observed algorithm differences (PPO preserving SFT values, DPO amplifying SFT-consistent stances, SIMPO moderate drifts) are not entailed by the dataset construction and are supported by hyperparameter ablations (§J). The one self-citation in the introduction (Stańczak et al., 2025) is a framing citation for the importance of human-value alignment and is not load-bearing; no uniqueness theorem, ansatz, or known-result renaming is used. Consequently there is no circular step by the paper's own equations or citation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical or architectural entities are postulated. Free parameters are confined to measurement design choices (retrieval threshold, checkpoint cadence, judge configuration). The axioms are measurement and generalization assumptions typical of an empirical LLM evaluation study; the most fragile is the GPT-4o-stance operationalization, which the paper itself acknowledges in the Ethics Statement only at the semantic level, not the style-confound level.

free parameters (3)
  • Dataset retrieval cosine similarity threshold = 0.5
    §K.1: datapoints are retrieved from WildChat/Alpaca/UltraFeedback/HH-RLHF to approximate their stance distributions using cosine similarity >= 0.5. Hand-chosen; this threshold underlies the 'small value-gap' comparison in Fig. 17 and the dataset-distribution claims in Fig. 16.
  • Checkpoint cadence for drift-time measurement = 500 steps (WildChat), 100 steps (Alpaca); 3 epochs
    §4.1 and §3 (drift time definition): the cadence sets drift-time resolution to ~0.09 for WildChat models, so 'values established within first 9% of training' means 'changed by the first checkpoint'. The 'very early' drift-time claims are resolution-limited.
  • GPT-4o stance label softmax over 3 options = temperature 0.7 sampling, 5 generations, softmax of option logits
    §3: all drift magnitudes and value-gap distances are computed from this judge configuration. It is verified on 100 samples but not calibrated against a broader human-judgment distribution or across checkpoints; the trinary neutral-heavy collapse could produce near-zero value-gap distances artifactually (§K.3).
axioms (4)
  • domain assumption Stance (support/neutral/oppose) on value-laden prompts is a valid operationalization of latent values
    §2.1, Eq. 1; the Ethics Statement concedes it flattens distinct values (economic vs. cultural opposition to immigration are both 'oppose'). Load-bearing for all three findings.
  • domain assumption GPT-4o stance labels approximate human judgments across all checkpoints, models, and response styles
    §3: verified on a manual sample of 100 prompt-generation pairs; not calibrated per checkpoint or per model. Since WildChat-fine-tuned outputs are longer and more hedged, the judge's neutral label may track style rather than latent value.
  • domain assumption The studied setup (Llama-3 3B/8B, Qwen-3 4B/8B; WildChat/Alpaca SFT; UltraFeedback/HH-RLHF preference data) represents the standard post-training pipeline
    §4.1, §5.1: WildChat and Alpaca are not the instruction mixtures used in production alignment recipes; the generalization of 'SFT sets values, preference optimization rarely re-aligns' to real pipelines is untested.
  • domain assumption Checkpoint sampling at 500/100-step intervals captures the value evolution trajectory
    §3 (drift time), §4.1: the first WildChat checkpoint is at step 500 of ~5000, so any drift occurring and reversing within 500 steps is unobserved; drift-time ~0.09 means 'by the first checkpoint'.

pith-pipeline@v1.3.0-alltime-deepseek · 45528 in / 15774 out tokens · 184103 ms · 2026-08-04T07:06:57.500148+00:00 · methodology

0 comments
read the original abstract

As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems. Therefore, studying the alignment of LLMs with human values has become a crucial field of inquiry. Prior work, however, mostly focuses on evaluating the alignment of fully trained models, overlooking the training dynamics by which models learn to express human values. In this work, we investigate how and at which stage value alignment arises during the course of a model's post-training. Our analysis disentangles the effects of post-training algorithms and datasets, measuring both the magnitude and time of value drifts during training. Experimenting with Llama-3 and Qwen-3 models of different sizes and popular supervised fine-tuning (SFT) and preference optimization datasets and algorithms, we find that the SFT phase generally establishes a model's values, and subsequent preference optimization rarely re-aligns these values. Furthermore, using a synthetic preference dataset that enables controlled manipulation of values, we find that different preference optimization algorithms lead to different value alignment outcomes, even when preference data is held constant. Our findings provide actionable insights into how values are learned during post-training and help to inform data curation, as well as the selection of models and algorithms for preference optimization to improve model alignment to human values.

Figures

Figures reproduced from arXiv: 2510.26707 by Gaurav Kamath, Karolina Sta\'nczak, Marius Mosbach, Mehar Bhatia, Shravan Nayak, Siva Reddy, Vered Shwartz.

Figure 1
Figure 1. Figure 1: Post-training can cause value drift, shifting the stance of model generations from a neutral to support, when asked a value-probing question such as “Should we close the gates and stop immi￾gration?” In this paper, we analyze how post-training reshapes these values. largely opaque. This motivates our central research question: How does the underlying training data, algorithms, and their interaction shape t… view at source ↗
Figure 2
Figure 2. Figure 2: SFT-induced values for Llama-3-3B and Qwen-3-4B models trained on WildChat and Alpaca for the topic of immigration. Each line represents the mean stance probability of support, neutral, and oppose stances, with 95% confidence intervals. In all cases, SFT leads to changes in stance distribution, often very early in training; WildChat leads to a high proportion of neutral responses, while on Alpaca leads to … view at source ↗
Figure 3
Figure 3. Figure 3: Values on the topic of abortion induced by training [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Value drifts induced by different preference optimization algorithms. Each line represents [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Prompt used to elicit stance distribution for each generated response. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Preference optimization on the Llama-3-3B SFT WildChat using HHRLHF, shown for the topic abortion. Each line depicts the mean stance probability for support, neutral, and op￾pose stances, with 95% confidence intervals. Across PPO, DPO, and SIMPO, stance probabilities remain stable post-SFT with low drift magnitudes and low drift time, indicating that preference op￾timization reinforces existing behaviours … view at source ↗
Figure 7
Figure 7. Figure 7: System prompt used for generating synthetic preference data. The prompt instructs the [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: DPO-induced value drifts for Llama3 3B and Qwen3 4B models for Setup 1 and Setup 2, [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: DPO-induced value drifts for Llama3 3B and Qwen3 4B models for Setup 1 and Setup 2, [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: SIMPO-induced value drifts for Llama3 3B and Qwen3 4B models for Setup 1 and [PITH_FULL_IMAGE:figures/full_fig_p026_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: SIMPO-induced value drifts for Llama3 3B and Qwen3 4B models for Setup 1 and Setup [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Effect of the PPO hyperparameter kl on model stance distributions across topics. Each plot shows how varying kl influences the proportion of support stances predicted by Llama3-3B SFT WildChat on three topics. base 200 400 600 800 0.0 0.2 0.4 0.6 0.8 1.0 SFT training DPO training Llama3 3B DPO (SFT WildChat) Support (def. =0.1) Neutral (def. =0.1) Oppose (def. =0.1) Support ( =0.5) Neutral ( =0.5) Oppose … view at source ↗
Figure 13
Figure 13. Figure 13: Effect of the DPO hyperparameter β on model stance distributions across topics. Each plot shows how varying β influences the proportion of support stances predicted by Llama3-3B SFT WildChat on three topics. J.1 PPO: EFFECT OF KL PENALTY COEFFICIENT For PPO, we vary the KL penalty coefficient to study its impact on value drifts during training. The resulting effects for the topics of Abortion, Immigration… view at source ↗
Figure 14
Figure 14. Figure 14: Effect of the SIMPO hyperparameter γ on model stance distributions across topics. Each plot shows how varying γ influences the proportion of support stances predicted by Llama3-3B SFT WildChat on three topics. K.1 METHODOLOGY To approximate the data distribution, we first embed the entire dataset and all value-laden questions in V-PRISM using all-mpnet-base-v2 sentence transformer. Then, for each question… view at source ↗
Figure 15
Figure 15. Figure 15: Prompt used to elicit stance distribution for each retrieved datapoint. [PITH_FULL_IMAGE:figures/full_fig_p028_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Comparison of stance distributions for the WildChat (left) and Alpaca (right) SFT [PITH_FULL_IMAGE:figures/full_fig_p029_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Histogram of Euclidean distances between preference pairs in (a) UltraFeedback, (b) HH [PITH_FULL_IMAGE:figures/full_fig_p029_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Epistemic Goggles: A Pretrained Module that Induces an Epistemic Frame via Gradient Editing

    cs.AI 2026-07 unverdicted novelty 7.0

    Goggles is a gradient-editing module trained once per base model and frame that, when applied frozen during finetuning, causes LLMs to treat unannotated documents with a specified epistemic stance (e.g., as fiction) a...

  2. Agents of Chaos

    cs.AI 2026-02 unverdicted novelty 6.0

    An exploratory red-teaming study documents eleven cases of security, privacy, and governance failures in autonomous language-model agents with tool access and persistent memory.

Reference graph

Works this paper leans on

93 extracted references · 3 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  3. [3]

    Foundational challenges in assuring alignment and safety of large language models

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024. URL https://arxiv.org/abs/2404.09932

    Pith/arXiv arXiv 2024

  4. [4]

    Explicitly unbiased large language models still form biased associations

    Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 122 0 (8): 0 e2416228122, 2025. URL https://www.pnas.org/doi/10.1073/pnas.2416228122

    doi:10.1073/pnas.2416228122 2025

  5. [5]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. URL https://arxiv.org/abs/2204.05862

    Pith/arXiv arXiv 2022

  6. [6]

    Managing extreme AI risks amid rapid progress

    Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et al. Managing extreme AI risks amid rapid progress. Science, 384 0 (6698): 0 842--845, 2024. URL https://www.science.org/doi/10.1126/science.adn0117

    doi:10.1126/science.adn0117 2024

  7. [7]

    Piqa: Reasoning about physical commonsense in natural language, 2019

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https://arxiv.org/abs/1911.11641

    Pith/arXiv arXiv 2019

  8. [8]

    Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35: 0 3663--3678, 2022

    Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. Picking on the same person: Does algorithmic monoculture lead to outcome homogenization? Advances in Neural Information Processing Systems, 35: 0 3663--3678, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/17a234c91f746d9625a75cf8a8731ee2-Abstract-Confer...

    2022

  9. [9]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952. URL https://doi.org/10.2307/2334029

    doi:10.2307/2334029 1952

  10. [10]

    Density-based clustering based on hierarchical density estimates

    Ricardo JGB Campello, Davoud Moulavi, and J \"o rg Sander. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pp.\ 160--172. Springer, 2013. URL https://link.springer.com/chapter/10.1007/978-3-642-37456-2_14

    doi:10.1007/978-3-642-37456-2_14 2013

  11. [11]

    How people use chatgpt

    Aaron Chatterji, Thomas Cunningham, David J Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. How people use chatgpt. Working Paper 34255, National Bureau of Economic Research, September 2025. URL http://www.nber.org/papers/w34255

    2025

  12. [12]

    Chatbot arena: An open platform for evaluating LLMs by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating LLMs by human preference. In Forty-first International Conference on Machine Learning, 2024. URL https://dl.acm.org/doi/abs/10.5555/3692070.3692401

    arXiv 2024

  13. [13]

    Reward model interpretability via optimal and pessimal tokens

    Brian Christian, Hannah Rose Kirk, Jessica AF Thompson, Christopher Summerfield, and Tsvetomira Dumbalska. Reward model interpretability via optimal and pessimal tokens. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 1048--1059, 2025. URL https://dl.acm.org/doi/10.1145/3715275.3732068

    arXiv 2025

  14. [14]

    Ultrafeedback: Boosting language models with high-quality feedback, 2023

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023. URL https://openreview.net/forum?id=pNkOx3IVWI

    2023

  15. [15]

    Towards measuring the representation of subjective global opinions in language models

    Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language models...

    2024

  16. [16]

    Towards analyzing and understanding the limitations of DPO : A theoretical perspective

    Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of DPO : A theoretical perspective. arXiv preprint arXiv:2404.04626, 2024. URL https://arxiv.org/abs/2404.04626

    Pith/arXiv arXiv 2024

  17. [17]

    Artificial intelligence, values, and alignment

    Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30 0 (3): 0 411--437, 2020

    2020

  18. [18]

    The delta learning hypothesis: Preference tuning on weak data can yield strong gains

    Scott Geng, Hamish Ivison, Chun-Liang Li, Maarten Sap, Jerry Li, Ranjay Krishna, and Pang Wei Koh. The delta learning hypothesis: Preference tuning on weak data can yield strong gains. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025. URL https://openreview.net/forum?id=cVlY21dIVE

    2025

  19. [19]

    Donoho, and Sanmi Koyejo

    Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, Dan Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. Is model collapse inevitable? B reaking the curse of recursion by accumulating real and synthetic data. In First Conference on La...

    2024

  20. [20]

    Mixed preference optimization: Reinforcement learning with data selection and better reference model

    Qi Gou and Cam-Tu Nguyen. Mixed preference optimization: Reinforcement learning with data selection and better reference model. arXiv preprint arXiv:2403.19443, 2024. URL https://arxiv.org/abs/2403.19443

    Pith/arXiv arXiv 2024

  21. [21]

    Alignment faking in large language models

    Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, et al. Alignment faking in large language models. arXiv preprint arXiv:2412.14093, 2024. URL https://arxiv.org/abs/2412.14093

    Pith/arXiv arXiv 2024

  22. [22]

    Assessing the alignment of large language models with human values for mental health integration: Cross-sectional study using schwartz’s theory of basic values

    Dorit Hadar-Shoval, Kfir Asraf, Yonathan Mizrachi, Yuval Haber, and Zohar Elyoseph. Assessing the alignment of large language models with human values for mental health integration: Cross-sectional study using schwartz’s theory of basic values. JMIR Mental Health, 11: 0 e55988, 2024. URL https://pubmed.ncbi.nlm.nih.gov/38593424/

    arXiv 2024

  23. [23]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    Pith/arXiv arXiv 2021

  24. [24]

    Collective constitutional AI : Aligning a language model with public input

    Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. Collective constitutional AI : Aligning a language model with public input. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 1395--1417, 2024 a . URL https://dl.acm.org/doi/10.1145/3630106.3658979

    arXiv 2024

  25. [25]

    Values in the wild: Discovering and analyzing values in real-world language model interactions

    Saffron Huang, Esin Durmus, Miles McCain, Kunal Handa, Alex Tamkin, Jerry Hong, Michael Stern, Arushi Somani, Xiuruo Zhang, and Deep Ganguli. Values in the wild: Discovering and analyzing values in real-world language model interactions. arXiv preprint arXiv:2504.15236, 2025. URL https://arxiv.org/abs/2504.15236

    Pith/arXiv arXiv 2025

  26. [26]

    The n+ implementation details of RLHF with PPO : A case study on TL ; DR summarization

    Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of RLHF with PPO : A case study on TL ; DR summarization. In First Conference on Language Modeling, 2024 b . URL https://openreview.net/forum?id=kHO2ZTa8e3

    2024

  27. [27]

    Smith, Yejin Choi, and Hannaneh Hajishirzi

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking DPO and PPO : Disentangling best practices for learning from preference feedback. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=JMBWTlazjW

    2024

  28. [28]

    Evaluating and inducing personality in pre-trained language models

    Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems, 36: 0 10622--10643, 2023. URL https://dl.acm.org/doi/10.5555/3666122.3666588

    arXiv 2023

  29. [29]

    Can machines learn morality? The D elphi experiment

    Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, et al. Can machines learn morality? The D elphi experiment. arXiv preprint arXiv:2110.07574, 2021. URL https://arxiv.org/abs/2110.07574

    Pith/arXiv arXiv 2021

  30. [30]

    The PRISM alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models

    Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gomez, Juan Ciro, Max Bartolo, Adina Williams, He He, et al. The PRISM alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. Advances i...

    arXiv 2024

  31. [31]

    Understanding the effects of RLHF on LLM generalisation and diversity

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. arXiv preprint arXiv:2310.06452, 2023. URL https://arxiv.org/abs/2310.06452

    Pith/arXiv arXiv 2023

  32. [32]

    What are human values, and how do we align ai to them? arXiv preprint arXiv:2404.10636, 2024

    Oliver Klingefjord, Ryan Lowe, and Joe Edelman. What are human values, and how do we align ai to them? arXiv preprint arXiv:2404.10636, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    You are what you eat-- AI alignment requires understanding how data shapes structure and generalisation

    Simon Pepin Lehalleur, Jesse Hoogland, Matthew Farrugia-Roberts, Susan Wei, Alexander Gietelink Oldenziel, George Wang, Liam Carroll, and Daniel Murfet. You are what you eat-- AI alignment requires understanding how data shapes structure and generalisation. arXiv preprint arXiv:2502.05475, 2025. URL https://arxiv.org/abs/2502.05475

    Pith/arXiv arXiv 2025

  34. [34]

    Beyond probabilities: Unveiling the misalignment in evaluating large language models

    Chenyang Lyu, Minghao Wu, and Alham Aji. Beyond probabilities: Unveiling the misalignment in evaluating large language models. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pp.\ 109--131, 2024. URL https://aclanthology.org/2024.knowllm-1.10/

    2024

  35. [35]

    Treleaven, and Miguel Rodrigues Rodrigues

    Reem Masoud, Ziquan Liu, Martin Ferianc, Philip C. Treleaven, and Miguel Rodrigues Rodrigues. Cultural alignment in large language models: An explanatory analysis based on H ofstede ' s cultural dimensions. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st Internatio...

    2025

  36. [36]

    How people use claude for support, advice, and companionship, 2025

    Miles McCain, Ryn Linthicum, Chloe Lubinski, Alex Tamkin, Saffron Huang, Michael Stern, Kunal Handa, Esin Durmus, Tyler Neylon, Stuart Ritchie, Kamya Jagadish, Paruul Maheshwary, Sarah Heck, Alexandra Sanderford, and Deep Ganguli. How people use claude for support, advice, and companionship, 2025. URL https://www.anthropic.com/news/how-people-use-claude-f...

    2025

  37. [37]

    UMAP : Uniform manifold approximation and projection

    Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP : Uniform manifold approximation and projection. Journal of Open Source Software, 3 0 (29): 0 861, 2018. doi:10.21105/joss.00861. URL https://doi.org/10.21105/joss.00861

    doi:10.21105/joss.00861 2018

  38. [38]

    Sim PO : Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Sim PO : Simple preference optimization with a reference-free reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=3Tzcot1LKb

    2024

  39. [39]

    S em E val-2016 task 6: Detecting stance in tweets

    Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. S em E val-2016 task 6: Detecting stance in tweets. In Steven Bethard, Marine Carpuat, Daniel Cer, David Jurgens, Preslav Nakov, and Torsten Zesch (eds.), Proceedings of the 10th International Workshop on Semantic Evaluation ( S em E val-2016) , pp.\ 31--41, San Diego, Ca...

    doi:10.18653/v1/s16-1003 2016

  40. [40]

    Jared Moore, Tanvi Deshpande, and Diyi Yang. Are large language models consistent over value-laden questions? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 15185--15221, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https://aclan...

    2024

  41. [41]

    Reinforcement learning finetunes small subnetworks in large language models

    Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur, and Hao Peng. Reinforcement learning finetunes small subnetworks in large language models. arXiv preprint arXiv:2505.11711, 2025. URL https://arxiv.org/abs/2505.11711

    arXiv 2025

  42. [42]

    Value imprint: A technique for auditing the human values embedded in RLHF datasets

    Ike Obi, Rohan Pant, Srishti Shekhar Agrawal, Maham Ghazanfar, and Aaron Basiletti. Value imprint: A technique for auditing the human values embedded in RLHF datasets. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=fq7WmnJ3iV

    2024

  43. [43]

    Attributing mode collapse in the fine-tuning of large language models

    Laura O'Mahony, Leo Grinsztajn, Hailey Schoelkopf, and Stella Biderman. Attributing mode collapse in the fine-tuning of large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL https://openreview.net/forum?id=3pDMYjpOxk

    2024

  44. [44]

    Help OpenAI fix over-refusals! https://community.openai.com/t/help-openai-fix-over-refusals/409799, October 2023

    OpenAI . Help OpenAI fix over-refusals! https://community.openai.com/t/help-openai-fix-over-refusals/409799, October 2023. Accessed: 2025-09-23

    2023

  45. [45]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022. URL https://dl.acm.org/doi/10.5555/3600270.3602281

    arXiv 2022

  46. [46]

    Does writing with language models reduce content diversity? arXiv preprint arXiv:2309.05196, 2023

    Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? arXiv preprint arXiv:2309.05196, 2023. URL https://arxiv.org/abs/2309.05196

    Pith/arXiv arXiv 2023

  47. [47]

    Smaug: Fixing failure modes of preference optimisation with DPO -positive

    Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with DPO -positive. arXiv preprint arXiv:2402.13228, 2024. URL https://arxiv.org/abs/2402.13228

    Pith/arXiv arXiv 2024

  48. [48]

    Do LLMs possess a personality? M aking the MBTI test an amazing evaluation for large language models

    Keyu Pan and Yawen Zeng. Do LLMs possess a personality? M aking the MBTI test an amazing evaluation for large language models. arXiv preprint arXiv:2307.16180, 2023. URL https://arxiv.org/abs/2307.16180

    Pith/arXiv arXiv 2023

  49. [49]

    What matters in data for DPO ? arXiv preprint arXiv:2508.18312, 2025

    Yu Pan, Zhongze Cai, Guanting Chen, Huaiyang Zhong, and Chonghuan Wang. What matters in data for DPO ? arXiv preprint arXiv:2508.18312, 2025. URL https://arxiv.org/abs/2508.18312

    arXiv 2025

  50. [50]

    Enhancing alignment using curriculum learning & ranked preferences

    Pulkit Pattnaik, Rishabh Maheshwary, Kelechi Ogueji, Vikas Yadav, and Sathwik Tejaswi Madhusudhan. Enhancing alignment using curriculum learning & ranked preferences. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 12891--12907, Miami, Florida, USA, November 2024. Ass...

    doi:10.18653/v1/2024.findings-emnlp.754 2024

  51. [51]

    AI psychometrics: Assessing the psychological profiles of large language models through psychometric inventories

    Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. AI psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science, 19 0 (5): 0 808--826, 2024. URL https://journals.sagepub.com/doi/10.1177/17456916231214460

    doi:10.1177/17456916231214460 2024

  52. [52]

    Discovering language model behaviors with model-written evaluations

    Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 13387--13434, Toronto, Canada, July 2023. Association f...

    2023

  53. [53]

    The lock-in hypothesis: Stagnation by algorithm

    Tianyi Qiu, Zhonghao He, Tejasveer Chugh, and Max Kleiman-Weiner. The lock-in hypothesis: Stagnation by algorithm. In ICLR 2025 Workshop on Bidirectional Human-AI Alignment, 2025. URL https://openreview.net/forum?id=4CRMWP1tYc

    2025

  54. [54]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 0 53728--53741, 2023. URL https://dl.acm.org/doi/10.5555/3666122.3668460

    arXiv 2023

  55. [55]

    Balancing the budget: Understanding trade-offs between supervised and preference-based finetuning

    Mohit Raghavendra, Junmo Kang, and Alan Ritter. Balancing the budget: Understanding trade-offs between supervised and preference-based finetuning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 25702...

    work page doi:10.18653/v1/2025.acl-long.1248 2025

  56. [56]

    Close Encounters of the AI Kind: A Survey of Public Sentiment About Artificial Intelligence

    Lee Rainie. Close Encounters of the AI Kind: A Survey of Public Sentiment About Artificial Intelligence . Report, Elon University - Imagining the Digital Future Center and Pew Research Center, March 2025. URL https://imaginingthedigitalfuture.org/reports-and-publications/close-encounters-of-the-ai-kind/

    2025

  57. [57]

    Scalpel vs

    Neel Rajani, Aryo Pradipta Gema, Seraphina Goldfarb-Tarrant, and Ivan Titov. Scalpel vs. hammer: Grpo amplifies existing capabilities, sft replaces them, 2025. URL https://arxiv.org/abs/2507.10616

    Pith/arXiv arXiv 2025

  58. [58]

    Sentence- BERT : Sentence embeddings using S iamese BERT -networks

    Nils Reimers and Iryna Gurevych. Sentence- BERT : Sentence embeddings using S iamese BERT -networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.\ 3982--399...

    2019

  59. [59]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA : A graduate-level google-proof Q&A benchmark, 2023. URL https://arxiv.org/abs/2311.12022

    Pith/arXiv arXiv 2023

  60. [60]

    Sutherland

    Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tPNHOoZFl9

    2025

  61. [61]

    The nature of human values

    Milton Rokeach. The nature of human values. NSF Award, 72 0 (7205473): 0 5473, 1972. URL https://philpapers.org/rec/ROKTNO

    1972

  62. [62]

    Political compass or spinning arrow? T owards more meaningful evaluations for values and opinions in large language models

    Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. Political compass or spinning arrow? T owards more meaningful evaluations for values and opinions in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association fo...

    2024

  63. [63]

    Unintended impacts of LLM alignment on global representation

    Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of LLM alignment on global representation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 16121--16140, Bangkok, Thailand, August 2024. Association for Computational...

    2024

  64. [64]

    Personal values across cultures

    Lilach Sagiv and Shalom H Schwartz. Personal values across cultures. Annual review of psychology, 73 0 (1): 0 517--546, 2022. URL https://www.annualreviews.org/content/journals/10.1146/annurev-psych-020821-125100

    work page doi:10.1146/annurev-psych-020821-125100 2022

  65. [65]

    A note on the pure theory of consumer's behaviour

    Paul A Samuelson. A note on the pure theory of consumer's behaviour. In The Foundations of Price Theory Vol 4, pp.\ 101--116. Routledge, 2024. URL http://www.jstor.org/stable/2548836

    arXiv 2024

  66. [66]

    Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML'23

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023. URL https://dl.acm.org/doi/10.5555/3618408.3619652

    arXiv 2023

  67. [67]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    Pith/arXiv arXiv 2017

  68. [68]

    Extending the cross-cultural validity of the theory of basic human values with a different method of measurement

    Shalom H Schwartz, Gila Melech, Arielle Lehmann, Steven Burgess, Mari Harris, and Vicki Owens. Extending the cross-cultural validity of the theory of basic human values with a different method of measurement. Journal of cross-cultural psychology, 32 0 (5): 0 519--542, 2001. URL https://journals.sagepub.com/doi/10.1177/0022022101032005001

    doi:10.1177/0022022101032005001 2001

  69. [69]

    Personality traits in large language models

    Gregory Serapio-Garc \' a, Mustafa Safdari, Cl \'e ment Crepy, Luning Sun, Stephen Fitz, Marwa Abdulhai, Aleksandra Faust, and Maja Matari \'c . Personality traits in large language models. 2023. URL https://arxiv.org/abs/2307.00184

    Pith/arXiv arXiv 2023

  70. [70]

    Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In The Twelfth Internati...

    2024

  71. [71]

    AI models collapse when trained on recursively generated data

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. AI models collapse when trained on recursively generated data. Nature, 631 0 (8022): 0 755--759, 2024. URL https://www.nature.com/articles/s41586-024-07566-y

    2024

  72. [72]

    Recognizing stances in ideological on-line debates

    Swapna Somasundaran and Janyce Wiebe. Recognizing stances in ideological on-line debates. In Diana Inkpen and Carlo Strapparava (eds.), Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text , pp.\ 116--124, Los Angeles, CA, June 2010. Association for Computational Linguistics. URL https://acla...

    2010

  73. [73]

    Position: A roadmap to pluralistic alignment

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: A roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 46280--46302, 2024. URL https://dl.acm.org/doi/10.5555/3692070.3693952

    arXiv 2024

  74. [74]

    Value profiles for encoding human variation

    Taylor Sorensen, Pushkar Mishra, Roma Patel, Michael Henry Tessler, Michiel Bakker, Georgina Evans, Iason Gabriel, Noah Goodman, and Verena Rieser. Value profiles for encoding human variation. arXiv preprint arXiv:2503.15484, 2025. URL https://arxiv.org/abs/2503.15484

    arXiv 2025

  75. [75]

    Lillicrap, Ana Marasović, Sylvie Delacroix, Gillian K

    Karolina Stańczak, Nicholas Meade, Mehar Bhatia, Hattie Zhou, Konstantin Böttinger, Jeremy Barnes, Jason Stanley, Jessica Montgomery, Richard Zemel, Nicolas Papernot, Nicolas Chapados, Denis Therien, Timothy P. Lillicrap, Ana Marasović, Sylvie Delacroix, Gillian K. Hadfield, and Siva Reddy. Societal alignment frameworks can improve LLM alignment. arXiv pr...

    Pith/arXiv arXiv 2025

  76. [76]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford A lpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  77. [77]

    A deep dive into the trade-offs of parameter-efficient preference alignment techniques

    Megh Thakkar, Quentin Fournier, Matthew Riemer, Pin-Yu Chen, Amal Zouaq, Payel Das, and Sarath Chandar. A deep dive into the trade-offs of parameter-efficient preference alignment techniques. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

    work page doi:10.18653/v1/2024.acl-long.311 2024

  78. [78]

    Zephyr: Direct distillation of LM alignment

    Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Cl \'e mentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. In First Conference on Language Modeling, 2024. URL https://openrev...

    2024

  79. [79]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2023

  80. [80]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=gEZrGCozdqR

    2022

Showing first 80 references.