{"id":"00d081d9-023a-4f8d-a013-fe9c64af72a9","arxiv_id":"2502.03397","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SPRI generates per-query guiding principles through a generate-critique-refine loop and uses them to steer responses, matching expert principles on cognitive reappraisal and improving TruthfulQA for base models.","lead":"This paper presents SPRI, a framework that writes a fresh set of behavioral principles for every incoming user query, then uses a critic model to refine both the principles and the final answer. Tests on emotional support, LLM-as-a-judge rubrics, and synthetic fine-tuning data show results near those of expert-written principles, with notable caveats.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The TruthfulQA claim is contradicted by Table 4: for Mistral-7B-v0.3-Instruct, SPRI fine-tuning yields 56.43% and 57.99% versus the off-the-shelf 66.11%, yet Section 5.3 claims consistent gains over the off-the-shelf model.","rationale":"The reader's verdict is CONDITIONAL, and the reader's rationale already identifies the unqualified TruthfulQA claim as conflicting with the Mistral-instruct result in Table 4. I agree with that assessment, and I consider it the most load-bearing concern because it is a direct internal inconsistency in a headline result, not a matter of external calibration or subjective judgment. The paper's central claim has three components: on-par performance with expert principles, superior LLM-as-a-judge rubrics, and substantial truthfulness gains from SFT. The third component is falsified as stated for a meaningful subset of the evaluated models. A one-line qualification would fix the text, but without it the abstract and conclusion overstate the framework's generality. Other concerns, such as the unvalidated critic scores or the reliance on seed examples, are real but less decisive because the paper provides end-to-end task evaluations that partially validate the pipeline; the Table 4 contradiction is directly checkable and already present in the paper. The paper otherwise deserves credit for releasing code, providing ablations, and reporting seeds and costs. Verdict remains CONDITIONAL: the framework may be sound, but the truthfulness claim needs to be corrected and re-reported with model-wise deltas before the paper should be accepted as written.","tokens_in":41520,"tokens_out":7378,"duration_ms":74492,"concrete_test":"Recompute Table 4 grouped by base versus post-trained models and compute per-model deltas between SPRI and the off-the-shelf baseline. If Mistral-7B-v0.3-Instruct shows negative deltas on both Dolly (-9.68) and MixInstruct (-8.12), and Gemma-2-9B-it shows a negative delta on MixInstruct (-0.72), then the sentence 'SPRI consistently outperforms the off-the-shelf model' must be revised to report gains only for base models or to list the post-trained exceptions explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"One of the three central evaluation pillars is the claim that SPRI-generated synthetic SFT data leads to substantial improvement on truthfulness. Section 5.3 states that 'SPRI consistently outperforms the off-the-shelf model as well as other synthetic response generation methods on the TruthfulQA dataset' and that instruction-tuned models benefit from gains. Table 4 directly contradicts this. For Mistral-7B-v0.3-Instruct, the off-the-shelf model scores 66.11%, while SPRI fine-tuned on Dolly scores 56.43% and on MixInstruct scores 57.99%—decreases of 9.68 and 8.12 points. Gemma-2-9B-it also slightly regresses on MixInstruct (59.75% vs. 60.47%). The blanket statement is therefore internally inconsistent with the paper's own reported numbers. This matters because the abstract and conclusion present truthfulness improvement as a headline contribution of SPRI, not merely a result for base models. The finding can likely be rescued by qualifying the claim to base (non-instruction-tuned) models or to specific model-dataset cells, but as written the broad claim is false for a substantial subset of the evaluated post-trained models. This is not a disagreement with external consensus; it is a factual mismatch within the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SPRI, a two-stage framework in which a base LLM generates context-specific guiding principles for each user query, a critic model scores and refines those principles until a threshold is met, and the refined principles then guide response generation, which is also critic-refined. SPRI is evaluated on three tasks: cognitive reappraisal against expert-written RESORT constitutions (§4.1), instance-specific rubric generation on BiGGen Bench (§4.2), and synthetic SFT data for TruthfulQA and related benchmarks (§5). The paper claims on-par performance with expert-crafted principles in the first two tasks and substantial truthfulness gains from SPRI-generated SFT data, with code and model generations released.","tokens_in":41811,"tokens_out":6200,"duration_ms":57598,"significance":"The three task settings are well chosen and cover generation, evaluation, and data synthesis, and the released code plus the cost analysis in Appendix B are valuable. The ablation over seed conditions in Table 3 is a strength because it lets the reader see how much the oracle seed contributes. If fully supported, SPRI would be a practical contribution to alignment and LLM-as-a-judge research. However, the claim in §5.3 is internally contradicted by Table 4 for two instruction-tuned models, and the §4.1 result relies on an expert seed drawn from the same taxonomy used as the evaluation metric. These issues must be resolved before the broad conclusions in the abstract and conclusion can be accepted.","major_comments":[{"comment":"The main reappraisal experiment is partly circular and not robust to its own ablation. §4.1 'SPRI Method' states that SPRI is given a single oracle RESORT constitution as a seed example, while 'Evaluation & Criteria' uses 'Alignment with Reappraisal Constitutions' as the key metric. The seed and the yardstick thus come from the same expert-knowledge family. Table 3 shows that removing the seed ('-seed=[none]') lowers GPT-4o-mini alignment to 7.67, below the vanilla baseline of 7.90, so the sentence 'SPRI consistently outperforms methods that lack access to oracle principles' is not robust to the ablation. In addition, no significance test against the oracle condition is reported, so the headline 'on-par with oracle guidance' is not statistically established. Please report the no-seed condition as primary, or justify why one expert seed is within the 'minimal human effort' budget, and add explicit non-inferiority or equivalence tests against the oracle condition.","section":"§4.1, Table 1, Table 3"},{"comment":"The blanket truthfulness claim is contradicted by the paper's own table. §5.3 states that 'SPRI consistently outperforms the off-the-shelf model as well as other synthetic response generation methods on the TruthfulQA dataset,' but Table 4 shows that Mistral-7B-v0.3-Instruct scores 66.11% off-the-shelf, while SPRI fine-tuned on Dolly scores 56.43% and on MixInstruct scores 57.99%. Gemma-2-9B-it also regresses on MixInstruct (59.75% vs. 60.47%). The abstract's 'substantial improvement on truthfulness' and the conclusion's 'improves synthetic data generation' are therefore internally inconsistent with the reported numbers. The claim should be restricted to base models, where the gains are consistent, or the post-trained regressions should be explained.","section":"§5.3, Table 4"},{"comment":"The stopping rule in both stages is a critic score of at least 4 on a 1-5 Likert scale, but no calibration evidence is provided that Prometheus-2's scores track human judgments of principle quality or response-principle alignment. If the critic rewards superficial properties such as length or generic reassuring phrasing, SPRI could stop early on weak principles or over-refine on strong ones, and the apparent parity with oracle conditions in §4.1 and §4.2 could be an artifact of the critic. Please include a small human validation of the critic's scores on generated principles and responses, or an explicit sensitivity analysis to the threshold τ.","section":"§3, Eqs. (2) and (5)"}],"minor_comments":[{"comment":"The dataset definition has a typographical error: D = {(p1, r1), ..., (pn, rn} should be D = {(p1, r1), ..., (pn, rn)}.","section":"§5.1"},{"comment":"The instruction 'Reset M, C' is not defined; please state whether this clears the dialogue history of the base and critic models between iterations.","section":"Algorithm 1"},{"comment":"The phrase 'SPRI consistently outperforms methods that lack access to oracle principles' should be qualified as 'SPRI with an oracle seed', since the no-seed ablation in Table 3 shows a regression for GPT-4o-mini.","section":"§4.1"},{"comment":"The column headers 'Scale of10' and 'Scale of5' are missing spaces and should be 'Scale of 10' and 'Scale of 5'.","section":"Table 1"},{"comment":"The cost tables report 'Total Cost' without stating the unit or the number of responses over which the cost is averaged; please clarify the unit in the captions.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth your time: SPRI generates context-specific principles for each input on the fly, using a critic model to iteratively refine both the principles and the response. That per-query combination is genuinely new relative to Constitutional AI, ConstitutionalExperts, and ConstitutionMaker, and the paper evaluates it across three different use cases with real care. The cognitive reappraisal results (Table 1) are the strongest evidence: SPRI closes most of the gap to oracle psychological constitutions using only one seed example, and the ablation in Table 3 shows that seeds matter but even generic seeds beat no seeds. The BiGGen Bench rubric generation results (Table 2) are also convincing; SPRI beats the MT-Bench and FLASK baselines consistently and gets within 0.07 Pearson correlation of human-curated rubrics. The cost tables and the released code are a plus, and the paper is honest about stopping criteria and iteration counts.\n\nThe soft spot is the truthfulness claim, and it is not minor. Section 5.3 says SPRI 'consistently outperforms the off-the-shelf model' on TruthfulQA, but Table 4 shows the opposite for two of the three instruction-tuned models: Mistral-7B-v0.3-Instruct drops from 66.11% off-the-shelf to 56.43% (Dolly) and 57.99% (MixInstruct), and Gemma-2-9B-it also regresses on MixInstruct (59.75% vs 60.47%). The abstract's 'substantial improvement on truthfulness' is only true for base models. This is an internal contradiction, not a disagreement with external consensus. The fix is straightforward: qualify the claim to base models, or report per-model rather than blanket averages. The absence of confidence intervals across all three result tables also makes it hard to know which differences are real, and the reliance on Prometheus-2's 1–5 scores as a stopping target is never calibrated against human judgments of principle quality. On the reappraisal seed side, using one oracle RESORT constitution as the seed and then evaluating with RESORT-based criteria does create some circularity, though the paper discloses it and the ablation partially addresses it.\n\nWho should read this: anyone working on scalable alignment, LLM-as-a-judge, or synthetic SFT data. The reappraisal and rubric experiments are solid enough to deserve referee time. My recommendation: send it to peer review, but require revision that fixes the TruthfulQA claim, adds variance estimates, and discusses critic calibration. The framework is not groundbreaking, but it is a clean, reproducible step forward.","headline":"A real increment in per-query alignment principles, but the TruthfulQA headline claim is contradicted by the paper's own Table 4 and needs qualification before this is ready.","tokens_in":42356,"tokens_out":1283,"would_cite":true,"duration_ms":14165,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SPRI generates per-query principles that rival expert-written alignment guidance.","keywords":["LLM alignment","principle-based alignment","constitutional AI","LLM-as-a-judge","cognitive reappraisal","synthetic data generation","critique-refine","TruthfulQA"],"falsifier":"A direct test is to sample the queries where SPRI's critic assigns a score of 4 or higher and have human experts independently rate the same principles and responses. If the critic's 'good enough' judgments frequently disagree with expert judgments—for example, marking vague or generic principles as sufficient—the claimed equivalence to oracle guidance would not survive.","tokens_in":41326,"feed_emoji":"⚙️","tokens_out":5041,"duration_ms":41080,"temperature":0.7,"pith_summary":"The paper introduces SPRI, a framework that replaces generic, hand-written alignment principles with automatically generated ones tailored to each individual input query. Its central claim is that these context-situated principles can achieve performance on par with expert-crafted principles, even for a task as complex as psychologically grounded emotional support, while requiring minimal or no human effort. The paper demonstrates this claim in three settings: cognitive reappraisal, where SPRI matches oracle constitution guidance; LLM-as-a-judge, where SPRI-generated rubrics beat prior instance-agnostic frameworks; and synthetic data generation, where fine-tuning on SPRI outputs substantially improves TruthfulQA. If the claim holds, alignment can be scaled to domains that currently depend on costly human expertise.","feed_headline":"LLMs write their own alignment principles, rivaling experts","feed_subtitle":"Two-stage critique-refine loop produces per-query guidelines that match expert-crafted ones with minimal human effort.","key_machinery":"The central mechanism is the two-stage iterative critique-refine loop with a threshold-based stopping rule. Stage I synthesizes context-situated principles K for the user input T and refines them until the critic model assigns a score of at least 4 on a 1-5 Likert scale; Stage II generates and refines a response R against those principles until the same critic threshold is met, with a maximum of four iterations in each stage. The same principles serve as the evaluation criteria in Stage II, making the framework self-contained apart from optional seed examples. This loop is what replaces human-authored constitutions.","core_discovery":"SPRI's central discovery is that a two-stage critique-refine loop lets a base language model and a separate critic model jointly produce and use principles that are specific to each user query. In Stage I the base model generates initial principles for the query, the critic scores and critiques them, and the base model refines them until the critic gives a score of at least 4 or four iterations pass. In Stage II the same loop is repeated for the response, with the finalized principles serving as the rubrics for the critic's scoring. The paper shows that this procedure produces reappraisal responses that match oracle-guided outputs, evaluation rubrics whose correlation with human labels outperforms generic rubrics, and SFT data that yields large gains on truthfulness benchmarks.","pith_inferences":["A natural extension is to apply SPRI to non-textual or multimodal inputs, where per-instance principles could guide vision-language responses.","The framework's dependence on the critic's 1-to-5 score suggests an upper bound: if the critic cannot detect a flawed principle, the loop stops on a weak guide; calibrating the critic to human judgments of principle quality would likely strengthen the method.","SPRI could be used as a data-generation engine for RLHF preference pairs, where the generated principles provide a per-query rationale for which response is better.","One testable prediction is that SPRI's advantage grows as instances become more heterogeneous, because generic principles degrade exactly where per-query tailoring matters most."],"forward_implications":["In domains where expert-written principles exist, SPRI can substitute for them with comparable measured alignment and empathy, reducing the need for contracted specialists.","SPRI-generated instance-specific rubrics improve LLM-as-a-judge correlation with human ratings over generic and domain-level rubrics, with a gap to oracle rubrics of only 0.07 Pearson correlation.","Fine-tuning base models on SPRI-generated synthetic data raises TruthfulQA scores by an average of 24.76% over off-the-shelf models and 19.09% over oracle-response fine-tuned models.","The critic model does not need to be larger than the base model, so the framework is usable with smaller open-weights models.","SPRI requires fewer model calls than self-refine on complex tasks while performing better, so the per-query overhead is modest."],"supporting_citations":[{"why":"Introduced Constitutional AI, the generic-principles baseline that SPRI replaces with per-query principles.","marker":"Bai et al., 2022b"},{"why":"Provides 16 manually devised guiding principles, a key instance-agnostic baseline for self-alignment.","marker":"Sun et al., 2023"},{"why":"Supplies the expert-crafted reappraisal constitutions and evaluation schema used as oracle in Section 4.1.","marker":"Zhan et al., 2024"},{"why":"Prometheus-2, the critic model used as C in every SPRI experiment.","marker":"Kim et al., 2024"},{"why":"BiGGen Bench, the source of instance-specific human rubrics and gold labels in Section 4.2.","marker":"Kim et al., 2025"},{"why":"Self-refine, the iterative refinement baseline that SPRI compares against and extends.","marker":"Madaan et al., 2023"},{"why":"TruthfulQA, the benchmark used in Section 5 to measure SPRI's SFT gains.","marker":"Lin et al., 2022"}],"fun_headline_variants":["Self-generated principles match human-crafted alignment","SPRI: Auto principles beat generic rubrics","LLM-driven principles outperform LLM-as-a-judge","Two-stage loop for self-alignment without experts","Context-specific principles via critique-refine loop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire method rests on the critic model's 1-to-5 score being a trustworthy measure of whether the generated principles are genuinely useful and the response genuinely adheres to them; if that score is miscalibrated or rewards superficial traits, the loop stops early on weak guidance.","fun_headline_variants_meta":{"raw":{"variants":["Self-generated principles match human-crafted alignment","SPRI: Auto principles beat generic rubrics","LLM-driven principles outperform LLM-as-a-judge","Two-stage loop for self-alignment without experts","Context-specific principles via critique-refine loop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2903,"prompt_tokens":909,"completion_tokens":1994,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1923}},"tokens_in":525,"tokens_out":1994,"duration_ms":15285,"temperature":1.0,"reasoning_tokens":1923,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T04:52:00.798077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to sample the queries where SPRI's critic assigns a score of 4 or higher and have human experts independently rate the same principles and responses. If the critic's 'good enough' judgments frequently disagree with expert judgments—for example, marking vague or generic principles as sufficient—the claimed equivalence to oracle guidance would not survive.","supporting_citations":[],"review_version":1}