{"id":"78e933ea-3537-4908-b7ff-e811b68521fe","arxiv_id":"2502.09597","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"PrefEval, a benchmark of 3,000 preference-query pairs in multi-session conversations up to 100k tokens, finds that most LLMs follow user preferences poorly beyond a few turns, though fine-tuning helps.","lead":"The paper introduces PrefEval, a benchmark of 3,000 preference-query pairs that tests whether LLMs infer, remember, and follow user preferences across long multi-session conversations. It finds that most models' preference-following accuracy drops sharply after a few turns, and that fine-tuning on the benchmark improves performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'below 10% at 10 turns across most models' is contradicted by the paper's own explicit-preference tables; the claim only holds for implicit-choice or pooled numbers, not as stated.","rationale":"The reader's weakest assumption is LLM-judge reliability, which is a legitimate concern, but the paper already provides human validation on 300 samples (Appendix A.17) and reports high agreement rates, so that concern is partially mitigated. The more decisive issue is internal: the abstract's headline threshold ('below 10% at 10 turns across most evaluated models') is contradicted by Table 5 for explicit preferences, where most models score 12–17%. This is directly checkable from the paper's own numbers and does not require new experiments. The benchmark remains useful, and the qualitative finding that LLMs fail proactive preference following is supported; the required correction is to qualify the central claim by preference form and report the actual per-form numbers. Because the reader already returned CONDITIONAL, this stress-test does not move the verdict, but it strengthens the reason for conditioning acceptance on correcting the abstract and conclusion.","tokens_in":50429,"tokens_out":7245,"duration_ms":75778,"concrete_test":"Recompute the 10-turn zero-shot generation accuracy separately for explicit, choice-based, and persona-driven preferences, and for the pooled average, using the released PrefEval data and the paper's evaluator prompts. Report per-model percentages for all three forms and the pooled value. If fewer than half of the evaluated models are below 10% in the pooled or explicit columns, revise the abstract and conclusion to say 'below 10% for implicit preferences' or 'averaged over preference forms,' with the actual per-form numbers. As a secondary check, disaggregate Table 2 by topic to confirm that the 0.07 and 0.50 entries are topic-specific rather than general.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is the abstract's statement that 'in zero-shot settings, preference following accuracy falls below 10% at merely 10 turns (~3k tokens) across most evaluated models.' This is not supported by the paper's own generation-task tables. Table 5 (explicit preferences, 10 turns, zero-shot) gives per-model averages: Claude 3 Sonnet 13.41, Claude 3 Haiku 15.78, Llama3 8B 9.67, Llama3 70B 12.46, Mistral 7B 17.35, Mistral 8x7B 15.18. Only one of six models is below 10%. Table 2 additionally reports GPT-o1 at 50% on the travel-restaurant topic at 10 turns. The below-10% statement is accurate for implicit choice-based preferences (Table 8: all six models 6.4–8.0%) and partially for persona-driven preferences (Table 11: five of six below 10%, with Mistral 7B at 10.72%). Averaging the three preference forms can also yield means below 10% for some models, but 'across most evaluated models' is not supported. This is an internal numerical inconsistency, not an external judgment call: the headline threshold depends on how preference forms are pooled and on whether one averages over models. The qualitative conclusion that LLMs struggle with proactive preference following is credible and survives correction; the specific 'below 10% / most models' phrasing does not.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces PrefEval, a benchmark with 3,000 preference-query pairs in three preference forms (explicit, implicit choice-based, implicit persona-driven) and two tasks (generation and classification), intended to measure whether LLMs can infer, remember, and proactively follow user preferences in multi-session conversations of up to 100k tokens. The authors evaluate six main open-source and proprietary models across five methods (zero-shot, Reminder, Self-Critic, few-shot CoT, RAG), supplement with results on GPT-4o, o1-preview, Claude 3.5 Sonnet, and Gemini-1.5-Pro on a single topic, analyze error types, study multiple and conflicting preferences, and show that supervised fine-tuning on PrefEval improves Mistral-7B's preference following and length generalization. The manuscript's central qualitative conclusion is that current LLMs struggle with proactive preference following, but several headline quantitative claims in the abstract are not supported by the paper's own tables.","tokens_in":50871,"tokens_out":5754,"duration_ms":60659,"significance":"If the numerical claims are corrected, PrefEval is a valuable resource: it is a substantial, publicly released benchmark targeting an underevaluated capability, it includes both a generation task and a fast classification task with a reported correlation of 0.73, it covers a wide range of models and methods, and it provides human-LLM agreement checks as well as fine-tuning experiments that show meaningful improvements. The main risk is not the benchmark itself but the overstatement of the findings: the abstract's 'below 10% at 10 turns across most models' and 'still deteriorates in long-context conversations' claims are contradicted by specific tables in the paper. These are internal numerical inconsistencies rather than external judgment calls, and they can be fixed by rewording the claims to match the data.","major_comments":[{"comment":"The abstract's claim that 'in zero-shot settings, preference following accuracy falls below 10% at merely 10 turns (~3k tokens) across most evaluated models' is not supported by the paper's own generation-task results for explicit preferences. Table 5 gives 10-turn zero-shot averages of 13.41, 15.78, 9.67, 12.46, 17.35, and 15.18 percent for Claude 3 Sonnet, Claude 3 Haiku, Llama3 8B, Llama3 70B, Mistral 7B, and Mistral 8x7B, so only Llama3 8B is below 10%. The claim holds for implicit choice-based preferences (Table 8, where all six models are between 6.4% and 8.0%) and partially for persona-driven preferences (Table 11, where five of six models are below 10%), but it is not true as stated for explicit preferences or for the benchmark as a whole. Please reword the abstract and Section 3.2 to state which preference form and which model subset the threshold applies to.","section":"Abstract and Table 5"},{"comment":"The statement that 'even with advanced prompting and retrieval methods, preference following still deteriorates in long-context conversations' is too strong as written. Table 4 shows that GPT-o1-preview with the Reminder method achieves 0.98 at both 10 and 300 turns, and Claude-3.5-Sonnet with Reminder drops from 0.45 to 0.02 while Gemini-1.5-Pro drops from 0.91 to 0.05, so the direction and magnitude of the long-context deterioration are model- and method-dependent. The claim should be qualified as 'generally deteriorates' or restricted to the specific models and methods for which the tables show the effect.","section":"Abstract and Table 4"},{"comment":"The headline single-digit accuracy numbers rest on an LLM-as-a-judge evaluation. Section 2.5 reports validation on only 200 randomly sampled evaluations with a 5% error rate, and Appendix A.17 reports per-checker human-LLM agreement rates on 100 samples per preference form. Because the reported values are around 6-17%, a systematic judge error of a few percentage points could change the ordering of models and could move some models across the 'below 10%' threshold. The paper should either provide a stratified judge-error analysis by model, topic, and conversation length, or explicitly state that the precise numerical threshold is only a point estimate subject to evaluator error. The classification task provides useful convergent evidence for the qualitative conclusion, but it does not by itself validate the exact generative accuracy numbers.","section":"Section 2.5 and Appendix A.17"}],"minor_comments":[{"comment":"The paper repeatedly says it evaluated '10 state-of-the-art LLMs,' but the full cross-topic evaluation across preference forms and methods is reported for only six models; Table 4 covers four additional models on a single travel-restaurant topic. Please qualify the model count in the abstract and the contribution list.","section":"Abstract and Section 1"},{"comment":"The text calls the dataset 'manually curated,' while Appendix A.13 describes a pipeline in which preferences and queries are generated by GPT-4/Claude and then manually filtered and rated. Please make the generation-and-filtering pipeline explicit in the main text so readers do not infer that all 3,000 pairs are purely human-written.","section":"Section 2.2 and Appendix A.13"},{"comment":"The many reported accuracy values are percentages over roughly 50-60 preference instances per topic, but no confidence intervals or variance estimates are provided. A sentence on variance or a small number of bootstrap intervals would make the cross-model and cross-length comparisons easier to interpret.","section":"Figures 4 and Tables 5-13"},{"comment":"There are minor typographical errors in the appendix prompts, including 'respomse' in the Figure 18 prompt and 'shnmp' in Figure 1. The PDF-extracted title also contains rendering artifacts; please provide a clean text version.","section":"Appendix Figures 15-19"},{"comment":"The fine-tuning result is demonstrated on a single model (Mistral-7B) with a single topic split and no multiple seeds; the conclusion should note this limitation when claiming that fine-tuning on PrefEval 'significantly improves performance.'","section":"Section 3.7"}],"recommendation":"major_revision","confidential_remarks":"The core benchmark and the qualitative finding that LLMs have limited proactive preference following are credible, and the paper already contains much of the evidence needed to support a carefully worded version of its claims. The main barrier is that the abstract and several summary sentences state quantitative results that the paper's own tables contradict. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The LLM-as-a-judge concern raised by the stress-test is partially addressed by Appendix A.17 and the classification-task correlation, but it should be addressed in the revision because the absolute accuracy values are the paper's headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read this one. First, the benchmark is a real resource: PrefEval gives you 3,000 preference-query pairs across 20 topics, multi-session long-context conversations up to 100k tokens, explicit plus two implicit preference forms, and both generation and classification tasks. That combination is genuinely new relative to LAMP, RPBench, TIMECHARA, and RoleLLM, and the 0.73 correlation between classification and generation accuracy makes cheap evaluation practical. Second, the headline claim is overstated. The abstract says zero-shot preference-following accuracy falls below 10% at 10 turns (~3k tokens) across most evaluated models. The paper's own Table 5 (explicit preferences, 10 turns, zero-shot) gives per-model averages of 9.67-17.35, with only Llama3 8B below 10%. The claim is accurate for implicit choice-based preferences (Table 8: 6.4-8.0%) and roughly for persona-driven ones (five of six below 10%). So it holds only when you pool preference forms, and as written it is contradicted by the explicit-preference table. That is an internal numerical inconsistency, not a judgment call.\n\nThe qualitative story survives the correction. Zero-shot models lose track of explicitly stated preferences within a few turns, a simple Reminder prompt recovers most of the loss at short context, and RAG helps early but degrades as context grows. That message matters for anyone deploying chatbots. The companion claim that even advanced methods deteriorate in long context is also too broad: o1-preview with Reminder holds 98% at both 10 and 300 turns in Table 4, though the paper does footnote the test-time compute caveat.\n\nJudge validation is thinner than I would like — 200 samples in the main text, 100 per preference form in the appendix, agreement 0.86-0.98 — but the failure finding is robust to judge leniency, and the classification task provides an independent signal. The release status is confusing: the abstract says the dataset is available, the reproducibility statement says 'plan to release in the future.'\n\nBottom line: read the tables, not just the abstract. The benchmark is usable, the overclaims are correctable, and the fine-tuning result on held-out topics with length generalization is a legit bonus. Anyone working on personalized dialogue should know this paper, and it deserves serious refereeing.","headline":"PrefEval is a genuinely useful benchmark and the qualitative story holds, but the abstract's 'below 10% at 10 turns across most models' contradicts the paper's own explicit-preference tables.","tokens_in":51259,"tokens_out":5357,"would_cite":true,"duration_ms":48400,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State-of-the-art LLMs fall below 10% accuracy at following a user's stated preferences after only 10 conversation turns when left unprompted.","keywords":["preference following","LLM personalization","long-context conversation","benchmark","implicit preference inference","LLM-as-a-judge","retrieval-augmented generation","supervised fine-tuning"],"falsifier":"Take a random sample of several hundred generated responses spanning models, methods, preference forms, and turn counts, and have human annotators apply the paper's own four error-type checks; if human-scored preference-following accuracy at 10 turns lands well above 10% for most models, the central claim fails. A cheaper check runs the same protocol with a different judge model and compares verdicts.","tokens_in":50250,"feed_emoji":"🎯","tokens_out":7048,"duration_ms":57582,"temperature":0.7,"pith_summary":"PrefEval asks whether a chatbot can infer, remember, and apply a user's stated preferences across a long, multi-topic conversation, and answers that current LLMs mostly cannot. The benchmark pairs 1,000 manually curated preferences with queries engineered so that a generic answer would violate the preference, then inserts real conversational distractors of up to 100k tokens between the two. In zero-shot settings, preference-following accuracy across most of the 10 evaluated models falls below 10% once the preference is a mere 10 turns (about 3,000 tokens) old. A simple reminder sentence or retrieval augmentation helps considerably at short distances, but all methods deteriorate as the conversation grows, and error analysis shows models either ignore the preference, hallucinate it, or refuse to answer. Fine-tuning on PrefEval markedly improves adherence, transfers to longer and unseen contexts, and raises attention to preference-relevant tokens.","feed_headline":"LLMs drop below 10% preference accuracy after 10 turns","feed_subtitle":"New benchmark shows chatbots quietly ignore stated preferences a few thousand tokens in.","key_machinery":"The load-bearing object is the PrefEval dataset and its evaluation protocol. Each of 1,000 underlying preferences is rendered in three forms — an explicit one-sentence statement, a two-turn implicit dialogue in which the user reveals the preference by choosing among options, and a 4–8 turn persona-driven dialogue in which the preference is mentioned in passing — yielding 3,000 preference-query pairs across 20 topics. Real user conversations from LMSYS-Chat-1M are inserted between the preference and the query as distractors to create contextual distances up to 100k tokens. The generation task is scored by four independent LLM-judge binary checks (violation, acknowledgment, hallucination, helpfulness) aggregated into four error types, with preference-following accuracy defined as the absence of any error; the classification task asks the model to pick the one of four options that respects the preference, and correlates with generation accuracy at $r=0.73$.","core_discovery":"The paper's central claim is that proactive preference following is a distinct, largely unmet capability of state-of-the-art LLMs, and that PrefEval measures it. On the paper's own numbers, zero-shot generation accuracy in the explicit-preference setting drops from roughly 80% at 0–1 turns to below 30% at 5 turns and below 10% at 10 turns (about 3,000 tokens) across most models, approaching zero at 300 turns (about 100k tokens); the strongest deployed models (Claude 3.5 Sonnet, Gemini 1.5 Pro, GPT-o1-preview) all score at or below 50% at 10 turns without prompting. Prompting and retrieval interventions recover much of the short-context performance — a single reminder sentence matches or beats more complex self-critique and chain-of-thought methods, and RAG is the most consistently strong — yet still decay with context length, with the reminder falling to single digits at 300 turns for several models. Error analysis attributes the zero-shot failures largely to preference-unaware violations, while prompting shifts failures toward hallucinated preferences and unhelpful refusals. Two further findings round out the claim: introducing multiple preferences, even conflicting ones, improves adherence to the original preference; and supervised fine-tuning on PrefEval raises zero-shot adherence above the best prompting baselines while generalizing to unseen topics, longer contexts, and implicit preference forms.","pith_inferences":["The success of a single reminder sentence implies the underlying capability is latent: models can retrieve and apply preferences when explicitly told to, so the measured failure is largely a proactiveness deficit rather than a capacity deficit.","The appendix's 'lost in the middle' analogue suggests preference adherence will worsen as services interleave more history; memory systems that surface preferences near the query may behave like the reminder intervention.","The finding that conflicting preferences improve adherence is striking but fragile-looking; a direct test would vary whether the later preference agrees or conflicts with the first while holding topic density fixed, to separate reinforcement of attention from topic priming.","A testable extension: fine-tuning on PrefEval might be improving a general 'attend to user-stated constraints' behavior, in which case the same model should also improve on instruction-following and long-context QA benchmarks — a cheap experiment to run."],"forward_implications":["Without prompting, current LLMs should not be trusted to carry a user's stated preferences across even a short conversation: zero-shot accuracy is below 10% at 10 turns for most of the ten models tested.","A one-sentence reminder to honor earlier discussion is the strongest cheap intervention, matching or beating retrieval-augmented generation at short distances and far outperforming self-critique and chain-of-thought.","All evaluated interventions deteriorate with context length, so long-context personalization remains open; RAG and reminder degrade to near single-digit accuracy at 300 turns for several models.","Supervised fine-tuning on PrefEval lifts zero-shot adherence above the best prompting baselines, and the gain transfers to unseen topics, longer contexts, and implicit preference settings.","The multiple-choice classification variant tracks generation accuracy closely ($r=0.73$), so cheap automatic evaluation of preference following is feasible without an LLM judge."],"supporting_citations":[{"why":"LMSYS-Chat-1M supplies the real interleaved conversation turns used as distractors between preference and query, which create the long-context setup.","marker":"Zheng et al. 2023"},{"why":"Retrieval-augmented generation is one of the five benchmarked methods and the most consistently strong intervention.","marker":"Lewis et al. 2020"},{"why":"SimCSE sentence embeddings serve as the retriever in the RAG baseline.","marker":"Gao et al. 2021"},{"why":"Iterative feedback from constitutional AI is adapted as the Self-Critic baseline method.","marker":"Bai et al. 2022"},{"why":"The 'lost in the middle' finding is extended to preference placement and used to explain mid-context degradation.","marker":"Liu et al. 2024b"},{"why":"LAMP is the prior personalization benchmark that PrefEval positions itself against by adding long-context, multi-turn lifestyle preferences.","marker":"Salemi et al. 2023"}],"fun_headline_variants":["Chatbots forget your preferences in 10 turns, new benchmark shows","LLMs ignore user preferences after ~3k tokens, PrefEval reveals","Preference following: LLMs fall below 10% accuracy at 10 turns","New benchmark PrefEval exposes LLMs' preference blindness","LLMs can't keep track of what you like in long chats"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 'below 10% at 10 turns' figure depends on the LLM judge (Claude 3 Sonnet) being as accurate on the full 3,000 evaluation instances as it was on the roughly 200–300 samples where it was compared to human annotators.","fun_headline_variants_meta":{"raw":{"variants":["Chatbots forget your preferences in 10 turns, new benchmark shows","LLMs ignore user preferences after ~3k tokens, PrefEval reveals","Preference following: LLMs fall below 10% accuracy at 10 turns","New benchmark PrefEval exposes LLMs' preference blindness","LLMs can't keep track of what you like in long chats"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1265,"prompt_tokens":1106,"completion_tokens":159,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":722,"completion_tokens_details":{"reasoning_tokens":63}},"tokens_in":722,"tokens_out":159,"duration_ms":3274,"temperature":1.0,"reasoning_tokens":63,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:51:58.512959+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of several hundred generated responses spanning models, methods, preference forms, and turn counts, and have human annotators apply the paper's own four error-type checks; if human-scored preference-following accuracy at 10 turns lands well above 10% for most models, the central claim fails. A cheaper check runs the same protocol with a different judge model and compares verdicts.","supporting_citations":[],"review_version":1}