{"id":"dd607c1d-c591-40d7-920b-cd9bd8dc898e","arxiv_id":"2411.09341","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A variational Bayesian inverse RL objective, AVA, models token-level rewards and outperforms reward-difference baselines in LLM alignment on standard benchmarks.","lead":"This paper applies Bayesian inverse reinforcement learning to align large language models, using a new training objective that models token-level rewards instead of only whole-sentence preference scores. The method reportedly improves reward modeling and downstream fine-tuning on standard alignment benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline empirical claim lacks statistical support: several reported gains are within noise (e.g., Table 1 Harmless +0.25, Table 2 Harmless +0.2), and the paper's own reproducibility checklist says no variance or significance tests were run.","rationale":"The reader's verdict was CONDITIONAL, with the weakest assumption identified as the identifiability of the reward via the TD-error constraint. I agree that identifiability is a serious theoretical concern, and the missing Technical Appendix for the AVA-p ELBO is also an omitted proof. However, the strongest claim in the paper is empirical: that AVA outperforms baselines. The most load-bearing condition for that claim is whether the reported differences are real or noise. The paper's own checklist admits that no variance or significance testing was performed, and several differences (e.g., Table 1: 0.25 accuracy points; Table 3: 0.8 win-rate points) are smaller than typical seed-to-seed variation in LLM fine-tuning. If those differences are not significant, the headline claim fails across the board, even if the theory were correct. If the differences survive significance testing, the empirical claim holds, and the theoretical issues would be grounds for revision, not rejection. Therefore, the concrete check is to add proper statistical evaluation. I recommend keeping the reader's CONDITIONAL verdict: the paper should be accepted only if the authors provide variance estimates, significance tests, and the missing theoretical derivation.","tokens_in":13485,"tokens_out":12036,"duration_ms":118811,"concrete_test":"Run each reported experiment with at least 5 independent seeds (varying data order and initializations) and report mean ± 95% confidence interval. For the pairwise win-rate evaluations (Tables 2-5), apply a paired significance test such as Wilcoxon signed-rank over per-prompt outcomes and report p-values. If for the small-margin comparisons listed above the 95% confidence intervals overlap or p > 0.05, then the claimed superiority is not statistically supported in those settings, and the abstract's global 'outperforms' claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that AVA outperforms existing alignment approaches. The evidence for this claim is tables of accuracy and win rates with no confidence intervals, no standard deviations, and no significance tests. The paper's Reproducibility Checklist explicitly answers 'no' to 'Analysis of experiments ... include measures of variation, confidence, or other distributional information' and 'no' to 'The significance of any improvement or decrease in performance is judged using appropriate statistical tests.' Several reported gains are small enough to be seed noise: Table 1 on Harmless with GPT-2 shows 70.27 vs 70.02 (Δ=0.25); Table 2 on Harmless BoN vs Bradley-Terry shows 28.8 vs 28.6 (Δ=0.2); Table 3 on Helpful PPO vs Bradley-Terry shows 1.8 vs 1.0 (Δ=0.8); Table 4 on Harmless direct vs DPO shows 13.7 vs 12.5 (Δ=1.2). Without paired tests or variance estimates, the claim 'AVA outperforms' is not established for these settings. This is a direct threat to the strongest claim, whereas theoretical concerns (e.g., identifiability of the TD-error reward) would remain secondary even if the empirical gains were confirmed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Approximated Variational Alignment (AVA), a Bayesian Inverse Reinforcement Learning (BIRL) objective for LLM alignment. AVA is instantiated in two forms: AVA-d for demonstration data and AVA-p for preference data, both built on an Approximated Variational Reward Imitation Learning (AVRIL) objective. The method introduces token-level intermediate rewards, a Transformer architecture with Q-value and reward heads (TQR), and supports both reward-modeling and direct-optimization pipelines. Experiments on Anthropic-Harmless, Anthropic-Helpful, OpenAI-Summary, Alpaca-GPT-4, and Math-GPT-4o report reward accuracy and win rates against Bradley-Terry, Preference Transformer, DPO, and AfD baselines, with ablation studies. The central claim is that AVA outperforms existing alignment approaches in reward modeling, RL fine-tuning, and direct optimization.","tokens_in":13869,"tokens_out":3606,"duration_ms":35970,"significance":"If the results hold, the paper would make a useful contribution by reframing LLM alignment as BIRL, enabling token-level intermediate reward modeling and direct reward estimation on individual demonstrations rather than only pairwise reward differences. The approach is flexible across data formats and training pipelines, and the ablation study is a genuine strength. The paper ships pseudocode and claims code availability, and it evaluates on standard public datasets against common baselines, which is good empirical practice in principle. However, the significance is currently limited by two load-bearing issues: the central ELBO derivation is deferred to an absent Technical Appendix, and the empirical comparisons lack any measure of variation or statistical testing, so the headline superiority claim is not established from the reported numbers.","major_comments":[{"comment":"The derivation of Eq. 16 from Eq. 15 is deferred to a Technical Appendix that is not present in the preprint. This is not a routine algebraic step: minimizing the sum of two KL divergences with respect to p(R|P+) and p(R|P−) is claimed to yield an ELBO containing log[1 − p(P−|R)], which requires independence or complementarity assumptions that are not stated. Moreover, the set P− used in Eq. 15 is not formally defined (the text says it denotes demonstrations that differ from P−). Because Eq. 16 is the basis for the AVA-p objective in Eq. 17, the theoretical foundation of the method is not verifiable from the manuscript as written.","section":"Eq. 15-16, §Alignment from Preference"},{"comment":"The empirical claim that AVA outperforms baselines is not statistically supported. The Reproducibility Checklist explicitly answers 'no' to including measures of variation and 'no' to using statistical significance tests, and all tables report single numbers without confidence intervals or error bars. Several reported gains are small enough to be seed noise: Table 1 (Harmless, GPT-2) shows 70.27 vs 70.02 for AVA-p vs Bradley-Terry (Δ=0.25); Table 2 (Harmless BoN) shows 28.8 vs 28.6 (Δ=0.2); Table 3 (Helpful PPO) shows 1.8 vs 1.0 (Δ=0.8); Table 4 (Harmless direct optimization) shows 13.7 vs 12.5 (Δ=1.2). Without paired tests or variance estimates, the abstract's claim that 'AVA outperforms existing LLM alignment approaches' is not established for these settings.","section":"Reproducibility Checklist; Tables 1-5"},{"comment":"In Eq. 13, the term log B(yt+1|y1:t; Qθ) from Eq. 5 is replaced by β log softmax(απw(yt+1|y1:t)), omitting the log-partition function of the Boltzmann distribution, −log Σ_{y′} exp(β Qw(y1:t, y′)). Because Qw is itself a log-softmax (Eq. 12), this omitted term is not a constant independent of the policy parameters, and no argument is given for why this approximation leaves the optimization objective qualitatively unchanged. Since this substitution is used in both AVA-d and AVA-p, the soundness of the training objective depends on this unstated approximation.","section":"Eq. 13, §AVA-d"},{"comment":"The reward distribution qφ is trained to match the TD error of a Q-function that is simultaneously fit to the demonstration likelihood. In the demonstration-only setting there is no independent reward signal, so the learned reward is operationally defined by the parameterization and initialization rather than identified as the 'true reward' stated in the Abstract and Section 'Alignment from Demonstration'. This is a correctness-risk concern: if the TD-error constraint does not identify a unique reward, the claim of direct reward modeling is unsupported even if the empirical win rates hold. The paper would benefit from a formal identifiability argument or a sensitivity analysis over initializations.","section":"Eq. 4-7, §Preliminaries and §AVA-d"}],"minor_comments":[{"comment":"The notation for negative demonstrations is confusing: the text defines P− as 'demonstrations that differ from P−', but Eq. 15 and the preceding sentence use P− inconsistently, and P− is never formally defined as a set complement or otherwise.","section":"Eq. 15 and surrounding text"},{"comment":"Eq. 17 uses δt(θ) in the TD-error term, but in the preference setting the Q-function is parameterized through the policy πw (Eq. 14), so the notation should be δt(w); this inconsistency makes it unclear which parameters are being updated in the term.","section":"Eq. 17"},{"comment":"The definition of the reward weight wt in Eq. 11 is hard to parse: the index t appears both as the summation index and as the output index, and the softmax normalization is not expressed clearly; please rewrite with distinct indices.","section":"Eq. 11"},{"comment":"The rows 'AVA-p w/o neg' and 'AVA-d' have identical accuracy values for several columns (e.g., 70.54, 70.36, 69.75, 69.15, 62.06, 58.65/59.00), which looks like a copy-paste artifact or a coincidence that should be clarified.","section":"Table 1"},{"comment":"The AfD baseline is cited as Sun and van der Schaar (2024) 'Inverse-RLignment', but the text calls it 'Alignment from Demonstrations (AfD)'; please ensure the reference name and acronym match the cited work.","section":"References"},{"comment":"The checklist states that all source code is included in a code appendix and will be made public, but no link or anonymized repository is provided in the manuscript; please add a URL or clearly state where the code will be available.","section":"Reproducibility Checklist"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is formatted as an AAAI submission but is posted as an arXiv preprint; it repeatedly refers to a Technical Appendix that is not included. The editor should verify that the final version contains the appendix and the statistical analysis promised by the revision. The empirical claims are currently the weakest point: the absence of variance estimates or significance tests is a blocking issue for a paper whose main claim is empirical superiority. The theoretical derivation is also unverifiable in the preprint. These issues are fixable, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes AVA, an AVRIL/BIRL-style objective for LLM alignment that models token-level intermediate rewards and works on both preference and demonstration data. It's a reasonable synthesis of AVRIL, Preference Transformer's attention weighting, and contrastive learning. But as it stands, the empirical claim that AVA outperforms existing approaches is not established: the paper's own reproducibility checklist says no variance or significance tests, and several reported gains are tiny (e.g., Table 1 Harmless GPT-2 70.27 vs 70.02; Table 2 Harmless BoN 28.8 vs 28.6). Those could easily be seed noise. The central derivation for the contrastive preference objective (Eq. 15 to Eq. 16) is deferred to a Technical Appendix that is not in the preprint, so the BIRL framing can't be verified from the available text.\n\nWhat is new and good: the idea of directly modeling the reward per demonstration instead of just the reward difference, plus intermediate rewards, is a sensible direction. The TQR architecture with reward and Q-value heads is a clean way to implement it. The authors are honest about the missing variance measures in the checklist, and they provide ablations that isolate the components. The method is flexible: it can be used for reward modeling or direct optimization, and on demonstration or preference data. That flexibility is genuinely useful.\n\nThe soft spots: the lack of error bars and statistical tests is the load-bearing flaw, because the strongest claims rest on small deltas. Some differences are larger (e.g., Summary BART accuracy 64.31 vs 59.27; PPO Summary win rate 33.9 vs 18.5), so the method may indeed help, but the paper as written overclaims. The missing appendix makes the derivation unverifiable; that should be fixed before publication. The TD-error reward is defined via a jointly trained Q-function, so the 'true reward' is not independently identified — that's a conceptual limitation, but it's secondary to the empirical issue; the method could still work as a regularizer even if the reward interpretation is loose. Eq. 14's notation is ambiguous (looks like a division by a gamma-exponentiated softmax when it should be a difference in log-space).\n\nWho it's for: researchers working on RLHF/DPO alternatives, especially those interested in token-level rewards and IRL-based alignment. It deserves a serious referee, but it needs a major revision with proper statistics and the missing derivation. I'd bring it to a reading group for discussion, but I wouldn't cite it yet in its current form.\n\nRecommendation: send to peer review, but the authors should be told that acceptance hinges on adding variance/statistical tests and the technical appendix.","headline":"Plausible new alignment objective, but the headline 'outperforms' claim is not statistically supported and the key derivation is missing; worth a serious referee after revision.","tokens_in":14263,"tokens_out":2563,"would_cite":false,"duration_ms":24263,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By casting LLM alignment as Bayesian inverse reinforcement learning, AVA fits a per-token reward distribution for each demonstration and reports higher reward accuracy and win rates than Bradley-Terry, Preference Transformer, DPO, and AfD.","keywords":["large language model alignment","Bayesian inverse reinforcement learning","variational inference","reward modeling","direct preference optimization","intermediate reward","TD-error constraint","transformer architectures"],"falsifier":"A direct test would train AVA-d on demonstrations whose tokens are randomly permuted (so no true intermediate reward structure exists) and then evaluate the learned reward on held-out preference pairs. If the model still achieves high reward accuracy, the TD-error constraint is not in fact identifying a meaningful reward and the gains must come from the policy objective alone.","tokens_in":13329,"feed_emoji":"🤖","tokens_out":8127,"duration_ms":108107,"temperature":0.7,"pith_summary":"This paper proposes a training objective called Approximated Variational Alignment (AVA) for aligning large language models with human feedback. Instead of learning a scalar reward difference between a chosen and a rejected sentence, AVA treats the alignment task as Bayesian inverse reinforcement learning and models a per-token reward distribution for every single demonstration. The authors argue that this makes fuller use of the feedback signal, supports intermediate rewards, and reduces reward hacking. They report that AVA outperforms Bradley-Terry, Preference Transformer, DPO, and AfD in reward modeling, RL fine-tuning, and direct optimization on standard preference and demonstration datasets.","feed_headline":"Bayesian token-level reward objective beats DPO on LLM alignment","feed_subtitle":"AVA models per-token rewards from single demonstrations, beating standard alignment baselines.","key_machinery":"The load-bearing machinery is the Approximated Variational Reward Imitation Learning (AVRIL) objective for Bayesian inverse reinforcement learning, adapted to autoregressive text generation. The adaptation rests on three per-token terms: the log-probability of the ground-truth token under a Boltzmann policy built from a Q-function, a KL term that keeps the reward distribution near its prior, and a TD-error constraint that forces the Q-values' temporal-difference error to match the sampled Gaussian reward. The TQR architecture operationalizes this by adding a Q-value head and a mean-variance reward head to a pre-trained Transformer decoder, with attention-based weights re-weighting each token's reward. For preference data, the paper adds a Contrastive Bayesian inverse RL term and a Contrastive Expected Return objective so the chosen demonstration's reward exceeds the rejected one's.","core_discovery":"The central discovery the paper is trying to establish is that the reward function for aligning a language model is better learned as a posterior distribution over per-token rewards than as a scalar comparison between one whole response and another. In the AVA framework, the demonstration likelihood is expressed through a Boltzmann policy over a Q-function, and the reward is constrained to equal the temporal-difference error of that Q-function at each step. Optimizing the evidence lower bound of the Bayesian inverse RL posterior with this constraint yields an objective that directly models the reward of every single token and of every single demonstration, without requiring pairwise preferences. On the reported benchmarks, this objective outperforms the standard Bradley-Terry and Preference Transformer reward models and the DPO and AfD direct optimizers, in both reward accuracy and LLM win rates.","pith_inferences":["One open question the paper leaves implicit is whether the learned 'reward' is identifiable: in the demonstration-only setting, no independent reward signal exists, so the reward distribution may simply reflect the Q-function parameterization and initialization rather than a ground-truth reward.","A natural extension is to apply the per-token TD-error credit assignment to other dense-feedback-free sequential domains, such as sparse-reward robot control, where the same variational machinery could distribute credit across actions.","Because the paper's ablations show the TD-error constraint (the 'irl' term) matters, an explicit comparison against a per-token Bradley-Terry variant with dense rewards would isolate whether the benefit comes from dense rewards or from the variational posterior itself."],"forward_implications":["AVA-d learns a reward from single demonstrations, so LLM alignment no longer requires pairwise preference labels.","Token-level intermediate rewards should generalize better to data with similar intermediate state distributions but different complete sentences.","Because the policy and the reward are trained jointly under a TD-error consistency constraint, the learned reward is less likely to separate from policy behavior, reducing reward hacking in downstream RL.","The same AVA objective can drive both a reward-modeling pipeline (with a small backbone and downstream PPO) and a direct-optimization pipeline (with a large policy), replacing separate Bradley-Terry and DPO/AfD objectives."],"supporting_citations":[{"why":"Supplies the AVRIL variational objective for Bayesian IRL that AVA adapts to text generation.","marker":"Chan and van der Schaar 2021"},{"why":"Establishes the BIRL formulation of learning a posterior over rewards from demonstrations.","marker":"Ramachandran and Amir 2007"},{"why":"Baseline ranking model for reward modeling that AVA-p is compared against.","marker":"Bradley and Terry 1952"},{"why":"Preference Transformer, the reward-modeling baseline and the inspiration for attention-based per-token reward weights.","marker":"Kim et al. 2023"},{"why":"DPO, the direct-preference-optimization baseline AVA-p targets in direct optimization.","marker":"Rafailov et al. 2024"},{"why":"AfD, the demonstration-based direct-optimization baseline AVA-d is compared against.","marker":"Sun and van der Schaar 2024"},{"why":"Defines inverse reinforcement learning from demonstrations, underlying the MDP formulation of text generation.","marker":"Ng, Russell et al. 2000"}],"fun_headline_variants":["Per-token reward model beats DPO in LLM alignment","Bayesian inverse RL improves LLM reward learning","AVA: Direct token-level reward learning for alignment","LLM alignment via approximated variational reward imitation","Beyond DPO: Bayesian reward modeling for safer LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the true reward can be recovered by forcing the temporal-difference error of a Q-function fit to the demonstrations to match the learned reward; without any independent reward signal, the reward distribution is identified only by model choice and initialization, not by the data.","fun_headline_variants_meta":{"raw":{"variants":["Per-token reward model beats DPO in LLM alignment","Bayesian inverse RL improves LLM reward learning","AVA: Direct token-level reward learning for alignment","LLM alignment via approximated variational reward imitation","Beyond DPO: Bayesian reward modeling for safer LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000533,"raw_usage":{"total_tokens":2546,"prompt_tokens":905,"completion_tokens":1641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1576}},"tokens_in":521,"tokens_out":1641,"duration_ms":11407,"temperature":1.0,"reasoning_tokens":1576,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:46:03.309704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would train AVA-d on demonstrations whose tokens are randomly permuted (so no true intermediate reward structure exists) and then evaluate the learned reward on held-out preference pairs. If the model still achieves high reward accuracy, the TD-error constraint is not in fact identifying a meaningful reward and the gains must come from the policy objective alone.","supporting_citations":[{"cited_title":"J.; and van der Schaar, M","cited_arxiv_id":null,"evidence_quote":"Supplies the AVRIL variational objective for Bayesian IRL that AVA adapts to text generation."}],"review_version":1}