REVIEW 4 major objections 6 minor 23 references
Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignment
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Public will can be turned into a measurable target for language model alignment.
desk verdict Genuinely new normative-empirical factoring, but the public-will measurement claim outruns the evidence: the RBR is validated against experts, not the public. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The chain of alignment factorization and its reward formula. The paper defines normative objectives J (good outcomes, bad outcomes to avoid, and deontological values) as the link to public will, and empirical rules R as the link to behavior. For each domain, experts rate each rule's expected effect on each objective, producing a rule–objective alignment φ(r, J); the RBR in Eq. (1) is the weighted average of an LM grader's rule-adherence scores φ({x,y}, r), with those alignments as weights. The normative–empirical decoupling is what lets nonexperts set goals and experts pick behaviors without asking the public to predict outcomes.
What would settle it
Take a set of mental-health prompts in which two rules are chosen so that following one reverses the other's effect (for example, a rule demanding immediate acknowledgment and a rule demanding brevity in crisis triage). Have experts rate the full set of responses holistically, compute the CoA RBR on the same set, and test whether the RBR's ranking matches the experts. If the independence assumption is materially false, the RBR should misrank the interaction-heavy cases even though each rule alone is aligned with the objectives.
Extended reading notes
Core claim
The central claim is that alignment between public will and language model behavior can be factored into a chain of three links—public will to normative objectives, normative objectives to empirical rules, and empirical rules to observable model behavior—and that the resulting rule-based reward yields a usable, scalable approximation of will-alignment. The paper argues that eliciting preferences directly on model outputs conflates what people want (normative judgments) with how they predict outputs will affect the future (empirical judgments), and that this conflation hides underlying consensus. Instead, the public votes only on objectives, experts assess which observable rules best achieve those objectives, and an LM grader checks whether a response follows the rules. The validation uses mental-health experts' averaged ratings of 65 responses across three risk-graded domains as ground truth, reporting Pearson's r = 0.841 and AUC = 0.964 for the RBR's ordering and classification.
Load-bearing premise
The whole reward rests on assuming that each rule's effect on the chance of achieving an objective is independent of all other rules and linear in how well the response follows it; if rules interact or the dose–response is nonlinear, the weighted sum in Eq. (1) can be wrong even when every rule individually is well chosen.
Editorial extensions
If this is right
- The CoA RBR can be evaluated at scale, so it can generate fine-tuning datasets, provide online oversight by blocking low-scoring outputs, and serve as a pre-release safety check.
- Because the objectives carry near-unanimous public support (96–98% overall, 92–96% bridging support), the RBR doubles as an approximate measure of alignment with public will, not just with expert preferences.
- The normative–empirical split means the public does not need to predict how a behavior will affect outcomes; experts supply that prediction, which makes agreement easier to find.
- Ablation shows rule–objective weighting outperforms a uniform-weight version (r = 0.833 vs 0.842), suggesting the expert alignments add signal even when all rules are positively aligned.
- The same structure could be extended to AI systems whose behavior outruns human understanding by swapping human experts for more capable models, since the public agency is carried by the objectives.
Reading between the lines
- My inference: the linear independence assumption is the natural pressure point; a version that learns rule-interaction weights from a large ground-truth set would settle how much the simplicity costs.
- My inference: the public-support numbers likely depend on the specific dialogue design and ratification thresholds (75% overall, 66% bridging), so re-running the process with different deliberation formats would reveal how robust the near-unanimity is.
- My inference: the approach's usefulness for pre-release safety checks depends on the domain taxonomy being stable; cross-domain prompts that straddle MH1–MH3 would test whether the RBR degrades when the domain label is ambiguous.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces 'Chain of Alignment' (CoA), a method for building rule-based rewards (RBRs) for language model behavior by factoring the alignment problem into normative objectives endorsed by the public and empirical rules created by domain experts. The authors report a public-input pipeline (collective dialogues, bridging-based ranking, and ratification thresholds) that yields normative objectives with 96–98% US public support across three mental-health domains, and expert-created rules that, when converted into an RBR via weighted rule-adherence scores (Eq. 1), correlate with expert holistic ratings of response-objective alignment (Pearson's r = 0.841, AUC = 0.964). The paper's central claim is that because the objectives have near-unanimous public support and the RBR tracks expert judgments of objective alignment, the RBR approximates alignment between LM behavior and public will.
Significance. If fully validated, the CoA framework would be a valuable contribution to pluralistic alignment: it provides a legible, decomposable alternative to black-box reward models and a concrete procedure for integrating nonexpert public input with expert knowledge. The paper is commendably transparent: it includes detailed appendices on the participant pipeline (A.2), demographic limitations (A.4), an ablation of the weighting scheme (A.5), a technical analysis of the normative-empirical conflation (A.7), and an explicit list of limitations in Section 3. The main validation step—correlating the RBR against expert judgments on 65 responses—is a genuine, non-circular check of the rule-weighted scheme, since the rule-objective weights were fixed before the comparison. However, the headline inference from objective-level public support to response-level public will is not directly tested, and the ablation results weaken the link between the RBR and the normative objectives that carry the public-will content.
major comments (4)
- [Abstract and Section 2.3] The central claim that the CoA RBR 'provide[s] an approximate measure of alignment between LM behavior and public will' rests on an unvalidated link. The public voted on objective texts (Section 2.1), while the RBR was validated against expert holistic ratings of response-objective alignment (Section 2.3). Nothing in the paper measures public judgments of model responses or tests whether expert operationalization of the objectives matches how the public would apply them. The paper presents the expert evaluations as 'ground truth' for objective alignment, but that conflates expert judgment with public will. Section 3 acknowledges that 'public support is an imperfect measure of public will' but does not address this response-level gap. To support the headline claim, the authors need either a direct public-rating study or a clearly stated and defended argument that expert response judgments are a faithful proxy for the public's will—absent that, the conclusion should be reframed as measuring expert-consistency rather than public-will alignment.
- [A.5 and Eq. (1)] The ablation in Appendix A.5 reports that removing the rule-objective alignment weights ϕ(r,J) from Eq. (1) changes the Pearson correlation with ground truth from r=0.842 to r=0.833, and the authors note the improvement is not statistically significant at N=65. Since the ϕ(r,J) weights are the only component of the RBR that connects rule adherence to the normative objectives (and hence to public will), this result indicates that the RBR's predictive power is essentially carried by the raw rule-adherence scores, not by the objectives-alignment weighting. The paper should either demonstrate that the weighting effect is statistically significant with a larger sample or acknowledge that the RBR is effectively measuring adherence to expert-authored rules, not alignment with the public-endorsed objectives.
- [A.4 and Section 2.3] The quantitative evidence for both the public-support claim and the expert-correlation claim lacks appropriate uncertainty quantification. The public sample is described in A.4 as 'reasonably well balanced' on gender, age, and political party but skewed on ethnicity, education, religion, household income, and AI opinion; despite this, the abstract reports support as '96% ± 2%' without stating whether the quoted margin is a sampling margin or a demographic-bounding range. For the central correlation, N=65 with no confidence intervals on r or AUC; a reported r=0.841 could have a wide interval at this sample size. The authors should provide bootstrap confidence intervals or equivalent, and should report the demographic-adjusted range of public support rather than a simple margin of error.
- [A.7, Eq. (15) and Section 3 (Rule-based reward)] The RBR formula Eq. (1) inherits the linear, independent rule-aggregation assumption stated in Eq. (15) of A.7: each rule's impact on objective probabilities is assumed independent and proportional to rule adherence. The paper labels this a limitation in Section 3, but it is load-bearing because the RBR is the only quantitative bridge from model behavior to the public-endorsed objectives. If rules interact or have nonlinear effects—plausible in mental-health contexts where multiple constraints may compound—the weighted sum in Eq. (1) can misrepresent objective achievement even when individual rules are well chosen. The authors should either provide evidence that the linear independence assumption holds for their rule sets (e.g., a residual analysis or a comparison with a learned aggregation on a larger ground-truth set) or restrict the paper's claims to the specific, non-interacting regime they assume.
minor comments (6)
- [Figure 1 caption] The word 'liklelihood' is a typo; it should be 'likelihood'.
- [Section 3] 'princlpled' should be 'principled'.
- [A.2.1] There are several typos: 'sucide' should be 'suicide', and 'domian' should be 'domain'.
- [A.5] 'we're assessed' should be 'were assessed', and 'Person's r' should be 'Pearson's r'.
- [A.2.3] The footnote describing the ratification thresholds as 'somewhat arbitrary' and 'handwavey' is candid but undercuts the precision of the high public-support numbers; the authors should either justify the thresholds more rigorously or soften the presentation of the ratification as a validated procedure.
- [A.1] The description of the three domains is clear, but the table in A.1 duplicates content from the main text; consider consolidating to avoid redundancy.
Circularity Check
No significant circularity: the RBR validation is an independent expert benchmark, though the public-will inference rests on an unmeasured bridging step.
full rationale
The CoA RBR is not fitted to the response-level expert ratings: Section 2.2 fixes rule-objective alignment weights from expert rule evaluations before Section 2.3 compares the RBR to expert holistic response-objective ratings on 65 responses, so r=0.841 and AUC=0.964 are genuine out-of-sample checks of the weighting scheme rather than a restatement of its inputs. The Appendix A.7 derivation is explicitly assumption-based: Eq. (15) assumes independent linear rule effects, and Eq. (20) defines the rule-objective weight as the implied least-squares coefficient, but the paper does not estimate these weights from the response-objective alignment data; it elicits them from separate expert rule assessments. Thus the derivation is conditional, not circular. Self-citations ([2], [18], [20]) supply definitions and process templates, but the public-support numbers (96-98%) and expert validation are new empirical results generated in this paper, so those citations are not load-bearing in a circularity sense. The remaining gap - public approval was elicited on objective texts, not on model responses, so equating objective support with response-level public will is an untested bridge - is a validity limitation (the paper acknowledges public support is an imperfect measure in Section 3), not a circularity.
Assumptions & free parameters
free parameters (3)
- Ratification thresholds for objectives =
75% overall, 66% bridging
- Statement selection threshold (max-min bridging) =
about 50%
- Rule-objective alignment weights phi(r,J) =
0.13 to 0.65, average 0.35
assumptions (5)
- domain assumption Objectives can mediate between will and future states: phi(h,f) is approximated as a weighted sum over a small set of objectives, Eq. (8) in A.7.
- domain assumption Rules' effects on objective achievement are independent and linear, Eq. (15) in A.7.
- domain assumption Expert-assessed rule-objective alignment phi(j,r) can stand in for the regression coefficient defined in Eq. (20) of A.7.
- domain assumption Public support for objectives, measured by votes, is a valid measure of public will.
- domain assumption The grader LM's adherence scores phi({x,y},r) are valid measures of rule following.
Cite this review
Pith. "Pith review of Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignment." pith.science (2026). https://pith.science/paper/UCOPT4MN
@misc{pith2026241110534,
author = {Pith},
title = {Pith review of: Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/UCOPT4MN}},
note = {Machine review of arXiv:2411.10534}
}
abstract
We introduce a method to measure the alignment between public will and language model (LM) behavior that can be applied to fine-tuning, online oversight, and pre-release safety checks. Our `chain of alignment' (CoA) approach produces a rule based reward (RBR) by creating model behavior $\textit{rules}$ aligned to normative $\textit{objectives}$ aligned to $\textit{public will}$. This factoring enables a nonexpert public to directly specify their will through the normative objectives, while expert intelligence is used to figure out rules entailing model behavior that best achieves those objectives. We validate our approach by applying it across three different domains of LM prompts related to mental health. We demonstrate a public input process built on collective dialogues and bridging-based ranking that reliably produces normative objectives supported by at least $96\% \pm 2\%$ of the US public. We then show that rules developed by mental health experts to achieve those objectives enable a RBR that evaluates an LM response's alignment with the objectives similarly to human experts (Pearson's $r=0.841$, $AUC=0.964$). By measuring alignment with objectives that have near unanimous public support, these CoA RBRs provide an approximate measure of alignment between LM behavior and public will.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Beyond preferences in ai alignment, 2024
Tan Zhi-Xuan, Micah Carroll, Matija Franklin, and Hal Ashton. Beyond preferences in ai alignment, 2024. URL https://arxiv.org/abs/2408.16984
arXiv 2024
-
[2]
Deliberative technology for alignment, 2023
Andrew Konya, Deger Turan, Aviv Ovadya, Lina Qui, Daanish Masood, Flynn Devine, Lisa Schirch, Isabella Roberts, and Deliberative Alignment Forum. Deliberative technology for alignment, 2023. URL https://arxiv.org/abs/2312.03893
arXiv 2023
-
[3]
Manning, and Chelsea Finn
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model,
-
[4]
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study.arXiv preprint arXiv:2404.10719, 2024
arXiv 2024
-
[5]
Direct preference-based policy optimization without reward modeling
Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim, and Hyun Oh Song. Direct preference-based policy optimization without reward modeling. Advances in Neural Information Processing Systems, 36:70247–70266, 2023
work page 2023
-
[6]
Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.16240, 2023
arXiv 2023
-
[7]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- bac...
work page 2022
-
[8]
Learning to summarize with human feedback
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020
2020
Show all 23 references
-
[9]
Deep reinforcement learning from human preferences
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017
2017
-
[10]
Improving alignment of dialogue agents via targeted human judgements
Amelia Glaese, Nat McAleese, Maja Tr˛ ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022
2022 arXiv
-
[11]
Training a helpful and harmless assistant with reinforcement learning from human feedback
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022
2022 arXiv
-
[12]
Jury learning: Integrating dissenting voices into machine learning models
Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing System...
2022
-
[13]
Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan P...
2022 arXiv
-
[14]
Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli
Saffron Huang, Divya Siddarth, Liane Lovitt, Thomas I. Liao, Esin Durmus, Alex Tamkin, and Deep Ganguli. Collective constitutional ai: Aligning a language model with public input. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’24. ACM, June 20...
2024
-
[15]
Rule- based rewards for language model safety, 2024
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule- based rewards for language model safety, 2024. URL https://cdn.openai.com/ rule-based-rewards-for-language-model-safety.pdf . Pr...
2024
-
[16]
Specific versus general principles for constitutional ai
Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, et al. Specific versus general principles for constitutional ai. arXiv preprint arXiv:2310.13798, 2023
-
[17]
Rule-based reinforcement learning for efficient robot navigation with space reduction
Yuanyang Zhu, Zhi Wang, Chunlin Chen, and Daoyi Dong. Rule-based reinforcement learning for efficient robot navigation with space reduction. IEEE/ASME Transactions on Mechatronics, 27(2):846–857, 2021
2021
-
[18]
Democratic policy development using collective dialogues and ai, 2023
Andrew Konya, Lisa Schirch, Colin Irwin, and Aviv Ovadya. Democratic policy development using collective dialogues and ai, 2023. URL https://arxiv.org/pdf/2311.02242.pdf
2023 arXiv
-
[19]
Inverse constitutional ai: Compressing preferences into principles
Arduin Findeis, Timo Kaufmann, Eyke Hüllermeier, Samuel Albanie, and Robert Mullins. Inverse constitutional ai: Compressing preferences into principles. arXiv preprint arXiv:2406.06560, 2024
2024 arXiv
-
[20]
Qiu, Michael Varga, and Aviv Ovadya
Andrew Konya, Yeping L. Qiu, Michael Varga, and Aviv Ovadya. Elicitation inference optimiza- tion for multi-principal-agent alignment. In NeurIPS 2022: Foundation Models for Decision Making Workshop, 2022. URL https://neurips.cc/virtual/2022/59639
2022
-
[21]
Mémoire sur les élections au scrutin
Jean-Charles de Borda. Mémoire sur les élections au scrutin. Histoire de l’Académie Royale des Sciences, pages 657–665, 1781
-
[22]
super-majority
Stefan Palan and Christian Schitter. Prolific.ac—a subject pool for online experiments. Journal of Behavioral and Experimental Finance , 17:22–27, 2018. ISSN 2214-6350. doi: https: //doi.org/10.1016/j.jbef.2017.12.004. URL https://www.sciencedirect.com/science/ article/pii/S22...
2018 doi
-
[2024]
URL https://arxiv.org/abs/2305.18290
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.