{"id":"9617f367-5a73-4157-96aa-121a644d9e91","arxiv_id":"2507.09037","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ALIGN is an open-source framework that aligns large language models to user-selected attributes via prompts and compares alignment methods across two decision-making domains.","lead":"The ALIGN framework lets users steer and compare how different large language models make decisions by aligning them to fine-grained attributes such as 'high fairness' or 'college graduate', and the paper demonstrates it on public-opinion and medical-triage datasets. A generalist might care because it is an open-source tool for testing whether AI decision aids can be personalized to reflect diverse human values rather than one fixed default.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical headline is not yet statistically grounded: tables report no sample sizes, confidence intervals, or tests, and the MTA unaligned baseline sits at exactly 50.0 in 11 of 12 cells, so the claimed alignment gains are plausibly within noise.","rationale":"The reader's weakest assumption focused on construct validity: whether the accuracy metric and dataset labels actually measure alignment to user preferences. My concern is adjacent but distinct: even granting the metric and labels, the reported accuracy values lack the statistical reporting needed to support the quantitative headline. The framework itself is the more credible contribution; the architecture, structured generation, configurable ADMs, and UI clearly support side-by-side comparison. The paper's own Limitations section concedes that results were not linked to real outcomes and that the attribute sets are pre-defined, which further supports a conditional rather than accepting verdict. I do not see an internal inconsistency or a deceptive presentation; the issue is that the empirical evidence is under-specified. A focused reproduction with per-cell Ns and significance tests would settle whether the claimed alignment improvements are real or noise, and would also clarify whether the MTA baseline's uniform 50.0 is a degenerate artifact. Therefore the appropriate verdict remains CONDITIONAL: the software contribution can stand, but the quantitative claims should be treated as preliminary until the statistical foundation is supplied.","tokens_in":11353,"tokens_out":4380,"duration_ms":57609,"concrete_test":"Reproduce Tables 1 and 2 from the released configurations on the exact dataset subset, then report per-attribute item counts, 95% confidence intervals (e.g., Clopper-Pearson), and a paired permutation or McNemar test comparing baseline vs prompt-aligned per question for each backbone-attribute cell. If the mean differences are not significant at alpha=0.05, the abstract should be softened from 'higher alignment accuracy' to 'directional improvements', or the quantitative claims should be explicitly flagged as preliminary.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that a zero-shot prompt-aligned ADM reliably outperforms the unaligned baseline rests entirely on Tables 1 and 2. Those tables report no per-attribute item counts, no confidence intervals, and no significance tests. Without N, the exact 50.0 repeated across nearly all MTA unaligned cells is consistent with chance-level behavior on a two-choice task, and even a 25-point improvement over that baseline could be a small-N artifact. On OpinionQA, the Llama-3.3-70B mean improves by only 0.2 points (58.6 to 58.8) while one attribute drops from 70.0 to 56.4; that pattern does not support 'higher average alignment accuracy' without knowing the item counts and variance. In addition, the OpinionQA subset taken from Feng et al. is unspecified, so the results may not generalize to the full dataset. The framework architecture, configuration management, and UI are plausibly useful and the qualitative example is illustrative, but the quantitative demonstration advertised in the abstract is not yet established as reliable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ALIGN, an open-source framework for comparing LLM-based decision-makers that are aligned to fine-grained attributes via prompting. The system combines a Trame-based interactive UI, Hydra-based configuration management, structured output generation with reasoning via Outlines, and pluggable ADM implementations: an unaligned baseline, a prompt-aligned ADM, and two Kaleido variants. The authors demonstrate the framework on two domains, demographic alignment using a subset of OpinionQA and value alignment using the MTA medical triage dataset, across four LLM backbones, and report per-attribute and mean alignment accuracy. The central quantitative claim is that a zero-shot prompt-aligned ADM achieves higher average alignment accuracy than the unaligned baseline, with Kaleido showing the best overall MTA performance.","tokens_in":11504,"tokens_out":4124,"duration_ms":47241,"significance":"If the empirical claims held, ALIGN would be a useful community resource: it lowers the barrier to comparing alignment methods, exposes prompts and reasoning traces for qualitative inspection, and supports swapping backbones and attributes in a reproducible way. The paper's concrete strengths are its open-source implementation, modular backend, configuration tracking, deterministic decoding with fixed seeds, and an interactive UI for side-by-side comparison. The quantitative evaluation, however, is not yet statistically grounded, and the demographic results rely on an unspecified subset of OpinionQA. The contribution is therefore currently a promising system demonstration rather than a validated claim about alignment accuracy; the framework's value is more clearly established than the empirical headline.","major_comments":[{"comment":"The load-bearing claim that the prompt-aligned ADM achieves 'higher mean alignment accuracy' compared to the baseline is not statistically supported. No sample sizes, confidence intervals, or significance tests are reported for any cell, so the reader cannot tell whether the differences reflect real alignment or small-sample noise. For example, the OpinionQA Llama-3.3-70B means differ by only 0.2 points (58.6 vs 58.8), and the Inc100k attribute drops from 70.0 to 56.4; on MTA, the unaligned values are exactly 50.0 in 11 of 12 cells, which is consistent with chance on a small two-choice task. Please report the number of items per attribute, confidence intervals, and appropriate tests, or explicitly reframe the quantitative results as an illustrative demonstration rather than a validated comparison.","section":"Section 6, Tables 1 and 2"},{"comment":"The OpinionQA subset is unspecified. The paper states that it uses 'a subset of the data converted into a steerable benchmark' from Feng et al. (2024), but it does not report how many questions were used, which questions were selected, or how the subset was constructed. Because all OpinionQA results in Table 1 depend on this subset, the results cannot be reproduced or compared with other work, and they may not generalize to the full dataset. Please specify the subset construction, provide the question counts, or use the full steerable benchmark.","section":"Section 4.1 and Appendix A.1"},{"comment":"The Kaleido comparison is incomplete as a benchmark. The table does not report unaligned metrics for the Kaleido models, and the caption states that these metrics are not computed because Kaleido requires alignment attributes. As a result, the claim that Kaleido models 'demonstrate the best overall performance' is not supported by a comparison against a common baseline, and the reader cannot judge whether the Kaleido advantage over the prompt-aligned ADM is meaningful. Please either compute a comparable baseline for Kaleido or soften the claim to describe absolute accuracy only.","section":"Section 6 and Table 2"}],"minor_comments":[{"comment":"The sentence 'The entire ALIGN framework is open source' appears nearly verbatim in both the abstract and the introduction; consider stating it once and referring to the GitHub repositories in the other place.","section":"Abstract and Section 1"},{"comment":"There is a missing space in the full text after 'system.' in the sentence ending '...resulting in a more interpretable system.' followed immediately by 'Outlines also provides...'; please fix the typographical error.","section":"Section 3.1"},{"comment":"The tables would be easier to interpret if the captions stated the number of scenarios per attribute and the exact scoring rule (e.g., binary match versus partial credit), since the reported percentages otherwise float without a denominator.","section":"Section 4 and Tables 1-2"},{"comment":"In the definition of Fairness, the text reads 'Fairness (F)is treating...' with a missing space before 'is'; please correct this typo.","section":"Appendix A.2"},{"comment":"The evaluation partly uses the authors' own prompt-aligned ADM from Hu et al. (2024) and the MTA dataset from the same group, with overlapping authors. This is not a circularity problem because the alignment metric comes from prior benchmarks, but the self-evaluation overlap should be stated explicitly for transparency in the experimental setup.","section":"Full manuscript"}],"recommendation":"major_revision","confidential_remarks":"The paper is best read as a systems contribution: the framework, UI, and modular backend are genuinely useful, and the open-source release is a strength. The main gap is that the quantitative claims in the abstract and Section 6 are not yet statistically grounded, and the OpinionQA subset is underspecified. These issues are fixable by adding item counts, confidence intervals, and significance tests, and by clarifying the subset. I would recommend asking the authors directly for the per-attribute denominators and a description of how the OpinionQA subset was selected, since the repeated 50.0 values in Table 2 suggest very small test sets. No concerns about citation ethics beyond the transparency point raised in the minor comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The framework is real, and it fills a gap. ALIGN gives pluralistic-alignment researchers a modular harness: Hydra-based configs, structured output generation via Outlines, swappable LLM backbones, and a trame-based UI for side-by-side comparison of prompts and reasoning traces. It is open source, and the architecture is clean enough that adding a new ADM is a one-function job. That is a real contribution. The qualitative moral-desert example is a good demonstration: the baseline treats the thief, the aligned model favors the would-be rescuer, and the UI makes the difference legible.\n\nThe quantitative section is where the paper falls short. Tables 1 and 2 report no sample sizes, no confidence intervals, and no significance tests. On MTA, the unaligned baseline is exactly 50.0 in 11 of 12 cells—chance on a two-choice task—and the aligned gains, while large in some cells, are not anchored to any measure of uncertainty. On OpinionQA, the Llama-3.3-70B aligned mean improves by 0.2 points, and the Inc100K attribute drops from 70.0 to 56.4. Without N and variance, \"higher average alignment accuracy\" is a claim about sign, not size or reliability. The stress-test note is on point here; I do not think it is being pedantic.\n\nTwo smaller issues: the OpinionQA subset taken from Feng et al. is not specified, which limits generalization, and Kaleido has no unaligned baseline, though that is defensible since Kaleido requires an attribute target. The self-citation overlap with Hu et al. is not a problem; the metric comes from prior benchmarks and the ADM is their own prior work.\n\nThe paper also states its limitations plainly—no link to real-world outcomes, fixed choice sets, single-attribute alignment only. That honesty helps.\n\nBottom line: this is a workshop systems paper, and the open-source framework is worth a serious referee. The empirical claims need a revision with per-attribute N, confidence intervals or tests, and at least a comment on the MTA baseline distribution. I would recommend the editor send it to review and let the process push on the statistics, not desk-reject on the basis of the missing tests.","headline":"A genuinely useful open-source framework for comparing aligned decision-makers, but the empirical headline is not yet statistically supported.","tokens_in":12097,"tokens_out":2737,"would_cite":true,"duration_ms":29415,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ALIGN is an open-source framework that uses prompt-based alignment to make LLM decision-makers follow fine-grained user attributes, and shows this steering improves measured alignment accuracy across backbones and domains.","keywords":["LLM alignment","prompt-based personalization","attribute alignment","pluralistic values","medical triage","public opinion surveys","structured output generation","modular evaluation framework"],"falsifier":"Run a new evaluation where the same ALIGN pipeline is applied to a held-out set of survey questions and triage scenarios with independently collected attribute labels, while paraphrasing each system prompt; if the prompt-aligned decision-maker fails to beat the unaligned baseline on mean alignment accuracy under those conditions, the central claim that prompt-based attribute alignment reliably steers LLM decision-makers would be falsified.","tokens_in":11119,"feed_emoji":"⚖️","tokens_out":8017,"duration_ms":86834,"temperature":0.7,"pith_summary":"The paper argues that LLM-based decision aids should be judged not only on factual accuracy but on whether their choices can be personalized to a user's values, and it offers ALIGN as a domain-agnostic testbed for that kind of alignment. Its central claim is that a zero-shot, prompt-aligned decision-maker—an LLM given a system prompt that names an attribute target such as 'high moral desert' or 'college graduate'—achieves higher mean alignment accuracy than an unaligned baseline across four LLM backbones. That claim is demonstrated in two settings that lack a single correct answer: demographic alignment on public opinion surveys and value alignment in medical triage. The wider point is that alignment can be treated as a modular, inspectable property of a decision-making system rather than a fixed model characteristic.","feed_headline":"Prompt-only steering moves LLM decisions toward user values","feed_subtitle":"Open-source ALIGN compares unaligned, prompt-aligned, and Kaleido decision-makers on public-opinion and triage tasks.","key_machinery":"The load-bearing mechanism is the aligned system prompt: a zero-shot instruction that embeds a target attribute into the decision-making context and asks the model to reason before choosing. Around that prompt, ALIGN layers three pieces of machinery: a dataset interface that turns each scenario into a structured question with choices; structured output generation, via the Outlines library, that constrains the model to emit a JSON answer plus an explicit reasoning trace; and a configuration manager, built on Hydra, that makes backbones, attributes, prompt templates, and decoding settings swappable. The Kaleido ADM is the second alignment mechanism, probing a pluralistic value model for attribute relevance and valence per choice. Together these let the same scenario be run under different alignment conditions and compared side by side in a Trame-based interface.","core_discovery":"On its own terms, the paper's central discovery is that fine-grained attribute alignment can be implemented and measured entirely at the prompt level, without fine-tuning, and that this prompt-level steering changes decisions in the direction of the target attribute. ALIGN packages this discovery as a comparison framework: the same dataset interface and configuration system hold the baseline, prompt-aligned, and Kaleido decision-makers fixed while only the system prompt and alignment target change. The quantitative backbone of the claim is the alignment-accuracy metric, which scores whether the selected choice is the one closest to the target attribute; the prompt-aligned ADM beats the baseline on mean accuracy for both OpinionQA demographics and MTA value attributes across all four tested backbones, and the Kaleido models achieve the highest MTA scores.","pith_inferences":["A natural extension the paper leaves implicit is that combining multiple attributes in one prompt, such as fairness plus risk aversion, may yield better personalization than single-attribute steering, since the paper's own results show single attributes can underperform the baseline on some items.","The reported accuracy metric tests whether the model picks the label-endorsed choice, not whether a real user feels represented; a direct follow-up would be to run ALIGN-generated prompts with human raters and compare satisfaction against the metric.","Because the OpinionQA portion uses an unspecified subset of the dataset, the quantitative results are difficult to reproduce externally; releasing the exact question subset and templates would make the framework's claims independently checkable.","The framework's ability to swap backbones exposes a testable question: does prompt-based alignment transfer across model generations, and do smaller models need stronger attribute phrasing to reach the same alignment accuracy?"],"forward_implications":["If prompt-based alignment works as reported, users can personalize an LLM decision-maker simply by editing the system prompt, with no retraining or fine-tuning, which makes personalization cheap and fast.","ALIGN's modular design means other researchers can drop in new alignment algorithms, datasets, or LLM backbones and directly compare alignment accuracy on identical scenarios, effectively standardizing how alignment methods are evaluated.","Because outputs are constrained to a structured schema with an explicit reasoning trace, the framework makes it possible to see not only which choice an aligned model made but why, allowing failures to be traced to specific scenarios.","The reported per-attribute results imply that alignment gains are not uniform: some attributes and backbones improve more than others, and a few attributes actually drop relative to baseline, suggesting single-attribute steering is only a partial solution."],"supporting_citations":[{"why":"Supplies the Medical Triage Alignment (MTA) dataset, the six triage value attributes, the prior prompt-aligned ADM implementation, and the alignment-accuracy metric the paper reuses.","marker":"Hu et al., 2024"},{"why":"Provides the OpinionQA subset formatted for steerability analysis and is a second source for the alignment-accuracy metric.","marker":"Feng et al., 2024"},{"why":"Contributes the OpinionQA dataset of Pew public-opinion questions with demographic labels used for demographic alignment.","marker":"Santurkar et al., 2023"},{"why":"Defines the Value Kaleidoscope model that ALIGN adapts into the Kaleido ADM for attribute-based decision-making.","marker":"Sorensen et al., 2024a"},{"why":"Provides the Outlines library used for structured generation, constraining LLM outputs to a JSON schema with reasoning.","marker":"Willard & Louf, 2023"},{"why":"Provides the Hydra configuration framework that lets ALIGN swap backbones, attributes, and ADM parameters.","marker":"Yadan, 2019"},{"why":"Provides the Trame framework used to build the side-by-side comparison user interface.","marker":"Jourdain et al., 2025"},{"why":"Motivates the pluralistic-alignment framing that ALIGN operationalizes for decision-making tasks.","marker":"Sorensen et al., 2024b"}],"fun_headline_variants":["Prompt-level alignment steers LLMs to user values without fine-tuning","Open-source ALIGN aligns LLM decisions to attributes via prompts","Prompt-based attribute alignment beats baseline in surveys and triage","Align LLM choices to user values with prompts alone, no retraining","Prompt-only steering improves LLM decision alignment across domains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the alignment-accuracy score derived from the dataset labels measures real alignment to user preferences; if the attribute definitions, answer labels, or the particular OpinionQA subset do not track actual user values, the reported accuracy differences would show prompt compliance rather than true personalization.","fun_headline_variants_meta":{"raw":{"variants":["Prompt-level alignment steers LLMs to user values without fine-tuning","Open-source ALIGN aligns LLM decisions to attributes via prompts","Prompt-based attribute alignment beats baseline in surveys and triage","Align LLM choices to user values with prompts alone, no retraining","Prompt-only steering improves LLM decision alignment across domains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1299,"prompt_tokens":893,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":320}},"tokens_in":509,"tokens_out":406,"duration_ms":5614,"temperature":1.0,"reasoning_tokens":320,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:05:25.859886+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a new evaluation where the same ALIGN pipeline is applied to a held-out set of survey questions and triage scenarios with independently collected attribute labels, while paraphrasing each system prompt; if the prompt-aligned decision-maker fails to beat the unaligned baseline on mean alignment accuracy under those conditions, the central claim that prompt-based attribute alignment reliably steers LLM decision-makers would be falsified.","supporting_citations":[{"cited_title":"Y., Choi, Y., and Tsvetkov, Y","cited_arxiv_id":null,"evidence_quote":"Provides the OpinionQA subset formatted for steerability analysis and is a second source for the alignment-accuracy metric."},{"cited_title":"Whose opinions do language models reflect? International Conference on Machine Learning (ICML), 2023","cited_arxiv_id":null,"evidence_quote":"Contributes the OpinionQA dataset of Pew public-opinion questions with demographic labels used for demographic alignment."},{"cited_title":"Hydra - a framework for elegantly configuring complex applications","cited_arxiv_id":null,"evidence_quote":"Provides the Hydra configuration framework that lets ALIGN swap backbones, attributes, and ADM parameters."},{"cited_title":"Trame: Platform ubiquitous, scalable integration framework for visual analytics","cited_arxiv_id":null,"evidence_quote":"Provides the Trame framework used to build the side-by-side comparison user interface."}],"review_version":1}