{"id":"37e532ac-ec47-4f17-89a0-7f5f6fb8d9cc","arxiv_id":"2607.20485","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Current LLMs satisfy only about half of the expectations real users reveal in follow-up turns, and a latent-expectation module produces only modest improvements.","lead":"The paper builds ExpectBench, a benchmark that extracts user expectations from follow-up messages in 12,000 real ChatGPT conversations, and finds that even GPT-4o is scored only 2.72 out of 5 on satisfying them. It also introduces LENS, a small module that injects latent expectations into a frozen LLM, improving scores by about 0.2 points.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ExpectBench's headline low scores and 40% coverage ceiling rest on treating every follow-up correction or new request as an expectation the initial response should have anticipated; extraction is LLM-generated with no per-item human agreement, so the misalignment may be an artifact of the extraction","rationale":"The paper's strongest claim is a quantitative one: current LLMs systematically fail to satisfy and anticipate user expectations, with GPT-4o at 2.72/5 and ~40% coverage even at a prediction budget of 10. For this claim to be true, the expectations in ExpectBench must be things users genuinely wanted from the initial response. The paper gives independent support for the scoring step—GPT-4o correlates with human judgments on expectation-level satisfaction (A.5.2), and the qualitative case studies illustrate genuine mismatches. But the extraction step itself is not human-validated with per-item agreement. My concern is a sharpening of the reader's weaker assumption: it is not just that the LLM extractor might paraphrase or invent expectations; the methodology of deriving expectations from follow-up messages can systematically conflate post-hoc requests with pre-existing expectations. In the cover-letter example, the follow-up 'add that I have just returned to America' may be a new requirement rather than an unstated expectation from the original query. If this is widespread, both the low satisfaction scores and the low coverage ceiling are partly artifacts of the benchmark construction, not evidence of a fundamental alignment gap. This does not refute the existence of expectation misalignment—the case studies support it—but it undermines the precision and generality of the headline numbers. The reader's CONDITIONAL verdict is exactly right: the paper should be accepted only if the extraction construct is validated or the claims are softened. My proposed test would settle whether the concern lands: if extraction agreement is high and 'new request' prevalence is low, the central claim is robust; if not, the benchmark needs revision before its results can be interpreted as measuring real user expectations.","tokens_in":33563,"tokens_out":4350,"duration_ms":49732,"concrete_test":"Sample 300 EXPECTBENCH instances. For each, give two independent human annotators the initial query and the follow-up turns (in the same form as the extraction prompt) and ask them to (1) extract expectations exactly in the paper's format and (2) label each extracted item as 'pre-existing expectation' vs. 'new/extended request introduced during interaction'. Measure per-instance agreement on extracted expectation sets (e.g., Cohen's κ on items, or item-level match with the released benchmark using the BART-MNLI matcher at τ=0.8). If mean pairwise agreement is low (κ<0.6) or if >30% of benchmark expectations are labeled 'new request' by the majority of annotators, then the 2.72/5 scores and 40% coverage ceiling are not established as measuring real user expectations; the benchmark would need a revised extraction protocol or a re-scoring under filtered expectations. If agreement is high an","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that current LLMs fail to satisfy and anticipate real user expectations (GPT-4o 2.72/5 in §4.1; ~40% coverage at budget 10 in §5.2)—requires that the extracted expectation set E(q) faithfully represents expectations the user actually had about the initial response, not later refinements or new requests introduced during the interaction. The paper validates only the scoring step: Appendix A.5.2 reports GPT-4o vs. four human annotators on 283 expectations (Cohen's κ=0.626, r=0.963), but that measures agreement on how to score already-given expectations. It does not validate extraction: Appendix A.1 describes only a general manual review, and no per-item extraction agreement is reported. The extraction prompt (A.2.1) explicitly instructs the LLM not to infer unexpressed preferences or include 'new or extended request[s]', but the running example (A.2.2) itself includes follow-ups like 'Can you add that I have just returned to America...'—a request that may be a post-hoc addition rather than a pre-existing expectation. If many benchmark instances encode post-hoc requests as 'expectations', then (a) satisfaction scores are artificially low because no initial response can satisfy a requirement not yet stated, and (b) the expectation-prediction experiment in §5 becomes 'guess future revision requests' rather than 'anticipate user expectations', making the 40% ceiling an artifact. The BART-MNLI semantic matcher with τ=0.8 (§C.1) adds a further arbitrary component: coverage numbers would shift with τ, and no sensitivity analysis is reported. Thus the 'fundamental misalignment' conclusion is load-bearing on an unvalidated extraction construct.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that standard benchmarks overstate LLM competence because they ignore the diverse, often implicit expectations real users bring to open-ended interactions. It introduces ExpectBench, constructed from 4.8M WildChat conversations by extracting user expectations from follow-up messages using an LLM, filtering to 12,000 multi-turn instances with 34,876 expectations. The authors report that six LLMs achieve low expectation satisfaction (GPT-4o scores 2.72/5 on a 1–5 scale) and that even with a prediction budget of 10, the best model covers only about 40% of gold expectations, concluding that current models fundamentally fail to anticipate and satisfy user expectations. They then propose LENS, a two-stage framework in which a frozen observer produces latent expectation tokens, a trained projector feeds them into a frozen main LLM, and claim consistent expectation-satisfaction improvements (e.g., 2.17→2.37 for LLaMA-3.1-8B, 2.39→2.50 for Mistral-7B).","tokens_in":33938,"tokens_out":3847,"duration_ms":43779,"significance":"If the construct validity of the extracted expectations holds, ExpectBench would be a valuable and novel resource: it uses real user follow-up behavior rather than expert rubrics, provides a multilingual, multi-turn corpus, separates the evaluation target from the judge, and reports a human-consistency check for the scoring step (κ=0.626, r=0.963 on 283 expectations). LENS is also a lightweight and reproducible recipe that keeps the base LLM frozen and trains only a projector. However, the significance of all three contributions—the benchmark, the diagnosis of low satisfaction/coverage, and the LENS gains—depends on whether the gold expectation set E(q) faithfully represents expectations the user held before seeing the initial response, an assumption that is not directly validated. The paper is transparent about its LLM-based extraction pipeline and its human check of scoring only, which makes the gap clear but unresolved.","major_comments":[{"comment":"The benchmark's ground truth is LLM-extracted, and no per-item human agreement on extraction is reported. Appendix A.5.2 validates only the scoring of already-given expectations; Appendix A.1 describes a general manual review, not a measurement of extractor precision/recall. The running example in A.2.2 itself includes 'Can you add that I have just returned to America...' as an extracted expectation, which the prompt's own rule ('Do not represent a new or extended request') would seem to exclude. If many extracted expectations are requests first made after the initial response, then the low satisfaction scores (§4.1) and the 40% coverage ceiling (§5.2) measure the model's inability to predict future turns, not its failure to satisfy pre-existing user expectations. Please report per-item extraction agreement on a sample and quantify the fraction of expectations that are expressed or infer","section":"§3.1, Appendix A.2.1, A.2.2"},{"comment":"The coverage metric and the headline '40% coverage ceiling' are computed with a semantic matcher (BART-MNLI) using a threshold τ=0.8. No sensitivity analysis or human validation of this matching threshold is provided. Because both the gold expectations and model predictions are free-form paraphrases, the coverage value is directly controlled by τ; a different threshold could materially change the reported ceiling. Please report coverage across a range of τ values and/or validate a sample of predicted/gold pairs against human judgment. Without this, the claim that expectation prediction is 'intrinsically challenging' is not robust.","section":"Eq. (8), Appendix C.1"},{"comment":"There is a partial circularity in the evaluation loop: GPT-4o is used to extract gold expectations, to score satisfaction, and (via the observer model prompted with the same extraction rubric) to define the latent signal for LENS; the only human validation is the scoring step on 283 expectations. The LENS gains in Table 2 may therefore reflect better alignment with GPT-4o's scoring preferences rather than with real users. The manuscript should include a human evaluation of LENS outputs on a held-out sample, or at least a per-dimension human audit of the extracted expectations used to construct the benchmark.","section":"§3.5, §6, Table 2"},{"comment":"The LENS improvements are small (overall +0.20 and +0.11 on a 5-point scale) and are reported without confidence intervals or significance tests. Given that the evaluator's Cohen's κ is 0.626, the consistency claim for LENS (RQ3) is not statistically established. Please report bootstrap confidence intervals or per-model significance tests for the overall and dimension-level gains, and clarify whether the improvements are consistent across the 283 human-validated expectations.","section":"Table 2, §6.2"},{"comment":"The claim that 'the best-performing model still achieves limited coverage' is interpreted as a fundamental limitation, but no human upper bound or chance baseline is reported. Users on average express 2.91 expectations, and the model is asked to predict 10; a coverage of 40% means matching roughly 1.2 of 2.91 expectations per query. Without a human-prediction baseline on the same task, the conclusion that models are 'intrinsically' poor at anticipation is not yet supported. A small human study (e.g., annotators given only the initial query and asked to list likely expectations) would sharpen the diagnostic claim.","section":"§5.2, Figure 7"}],"minor_comments":[{"comment":"The text says the benchmark is limited to 'the top 10 most frequent languages', but Figure 4(b) and the surrounding text list nine languages. Please reconcile.","section":"Appendix A.4"},{"comment":"The model is called 'DeepSeek-R1-7B' in the main text but 'DeepSeek-R1-Distill-Qwen-7B' in Table 3; please use a single consistent name.","section":"§4 and Table 3"},{"comment":"Several figures contain unreadable character-substitution artifacts (e.g., '/uni000...' sequences), making them impossible to interpret. Please regenerate with proper font embedding.","section":"Figures 5, 11, 12"},{"comment":"The heatmap in Figure 10 has empty bins and the color scale is not defined; a normalized agreement matrix or a weighted kappa would be easier to interpret.","section":"Appendix A.5.2"},{"comment":"The dimension-discovery process is described as iterative, but the number of subsets (15) and the stop criterion for convergence are not given in the main text; please state them in Appendix A.3.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical architecture and a potentially useful benchmark, but the central claim rests on the validity of LLM-extracted expectations. The lack of per-item extraction validation is the key blocker; the coverage-threshold sensitivity and the absence of a human upper bound for prediction further weaken the quantitative conclusions. If the authors can add a focused human study on extraction, a threshold sensitivity analysis, and a small human LENS evaluation, the paper would be much stronger. The 'first systematic study' framing should be softened until extraction validity is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper is worth taking seriously as a new evaluation direction, but the headline claim—that LLMs score 2.72/5 and cap out at ~40% expectation coverage—rests on an extraction step the paper never validates. You could fix that with a few experiments, but as written the strong conclusion outruns the evidence.\n\nWhat's new: ExpectBench is built from 12k real multi-turn WildChat interactions, and instead of model-generated checklists or simulated users, it extracts user expectations from the follow-up turns and uses them as per-instance rubrics. That's a real departure from WildBench and the user-simulation line, and the thought that misalignment shows up in post-hoc corrections is well taken. The taxonomy (practicality, compliance, empathy, etc.) is plausible and the dataset size is substantial. The GPT-4o-as-judge reliability check on 283 expectations (kappa .626, r .963) is fine as far as it goes—but it only validates scoring given the expectations, not the expectations themselves.\n\nThe soft spot is the extraction. The prompt tells the LLM not to include new or extended requests, but the paper's own running example does exactly that: 'Can you add that I have just returned to America from China' is a post-hoc request, not a pre-existing expectation. If a nontrivial fraction of the benchmark is built this way, then the low satisfaction scores and the 40% coverage ceiling are partly an artifact—no initial response can satisfy a request that was only made later. There's no per-item human agreement on extraction, only a general manual review. The prediction experiment then becomes 'guess future revision requests,' which is interesting but different from anticipating expectations.\n\nOther weaknesses: the same GPT-4o family does extraction, scoring, and LENS's observer, so there's a same-judge loop; the BART-MNLI matcher with tau=0.8 is arbitrary and unswept; LENS gains are small (2.37 vs 2.17, 2.50 vs 2.39) without error bars or significance tests; and the write-up is rough in places (some corrupted glyphs in appendices). Nothing here is fatal to the core idea, but the abstract's 'fundamental source of misalignment' needs the extraction validated first.\n\nBottom line: for people working on user-centric LLM evaluation, this is a useful paper to engage with. It deserves peer review rather than desk rejection, but it needs major revision. I'd want the data and code released, per-item extraction agreement, a sensitivity analysis on tau, and significance tests before relying on it. I wouldn't cite it yet.","headline":"A genuinely new evaluation angle, undercut by an unvalidated extraction step: the paper's strongest numbers likely overstate how badly models miss real user expectations.","tokens_in":34493,"tokens_out":3770,"would_cite":false,"duration_ms":40977,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes a measurable alignment gap: even the strongest LLM averages 2.72/5 on real user expectations, and expectation coverage tops out near 40 percent.","keywords":["user expectations","expectation alignment","LLM evaluation","multi-turn conversation","benchmark","latent expectation","expectation prediction","response generation"],"falsifier":"Take a random sample of ExpectBench instances and have the original users (or blind human annotators) judge each extracted expectation: were these 'expectations' things they actually wanted from the first response? If a large share are judged inventions, restatements of the query, or post-hoc rationalizations, then the low scores and LENS gains measure the extractor's construct rather than real user expectations. A cheaper version: measure inter-annotator agreement on expectation extraction from the same follow-up messages; if agreement is low, the rubrics are unstable.","tokens_in":33415,"feed_emoji":"🎯","tokens_out":7741,"duration_ms":73564,"temperature":0.7,"pith_summary":"This paper argues that standard LLM benchmarks overstate real-world usefulness because they ignore user expectations—the often implicit preferences that surface only in follow-up messages. To test this, the authors build ExpectBench, a benchmark of 12,000 real multi-turn conversations with 34,876 expectations extracted from user follow-ups, spanning nine languages and ten expectation dimensions. On it, even the strongest tested model averages only 2.72 out of 5 in expectation satisfaction, and when asked to predict user expectations from the initial query alone, the best model covers only about 40 percent of them even with a budget of ten predictions. The paper's constructive claim is that explicitly modeling expectations helps: LENS, a lightweight framework that learns a latent expectation representation and conditions a frozen LLM on it, raises satisfaction scores on both tested base models. A sympathetic reader would take away that alignment with real users is a distinct, measurable capability that current training objectives do not supply.","feed_headline":"Best LLM scores 2.72/5 on real user expectations","feed_subtitle":"New benchmark: even top LLMs satisfy only ~54% of real user expectations and anticipate just 40%.","key_machinery":"Expectation extraction is the load-bearing mechanism: a prompted LLM converts post-hoc user follow-ups into standalone criteria ('be concise,' 'mention my resume,' 'use subtle humor'), which then serve as rubrics instead of expert checklists or user simulations. ExpectBench is the benchmark built on those rubrics; its ten discovered dimensions (practicality, compliance, empathy, etc.) carry the analysis. LENS is the generative mechanism: a small observer model encodes the query into latent expectation tokens, a lightweight trainable projector maps them into the main LLM's representation space, and the frozen main LLM generates conditioned on that signal, with only the projector trained. Expe","core_discovery":"The central discovery is a systematic gap between benchmark competence and what users actually want. The authors mine 4.8 million real human–AI conversations, extract expectations from follow-up messages (corrections, clarifications, complaints), and turn them into rubrics for ExpectBench (12,000 conversations, 34,876 expectations). Six models score low: averages run from about 2.0 to 2.7 out of 5, with the strongest at 2.72. Because an LLM judge agrees with human raters when given explicit expectations (κ≈0.63, r≈0.96), the low scores are treated as genuine. Predicting expectations from the initial query alone reaches about 40% coverage even with ten predictions, versus an average of 2.91 t","pith_inferences":["If the extraction assumption holds, current post-training recipes that optimize benchmarks or human-preference proxies may not transfer to real-user satisfaction; expectation-aware data collection could become a standard ingredient.","Expectation coverage could be repurposed as a standalone, query-only capability test: a model that cannot anticipate expectations at near-ceiling coverage is unlikely to satisfy them, so coverage may predict downstream usefulness without needing generation.","The 40% coverage ceiling despite a budget of 10 (versus 2.91 average true expectations) suggests the bottleneck is not output length but prior knowledge about users; conditioning on user history or demographic context is the natural next experiment.","One can test LENS's mechanism directly: if the latent tokens truly carry expectation information, ablating or scrambling them should degrade satisfaction on expectation-heavy dimensions more than on surface dimensions; the paper does not report this ablation."],"forward_implications":["Benchmark scores and user satisfaction can diverge sharply: a model can be factually correct and still fail the user's actual criteria.","Satisfaction and anticipation are separable capabilities; generation improvements alone will not close the gap if the model cannot predict what users value, since coverage at a budget of 10 stays near 40%.","Expectation-level evaluation is feasible at scale: an LLM judge given explicit expectation rubrics agrees with human raters (κ≈0.63, r≈0.96), enabling large-scale user-centric evaluation.","Lightweight latent conditioning works: training only a small projector while freezing the main model yields consistent satisfaction gains of about 0.2 on a 5-point scale across two 7–8B model families and most dimensions.","Dimension-level results identify where to focus: surface criteria like coherence and consistency are comparatively easier, while practical grounding, versatility, and compliance remain the hardest to satisfy and to predict."],"fun_headline_variants":["Top LLM scores 2.72/5 on real user expectations","LLMs anticipate only 40% of user expectations","ExpectBench: even best LLMs score 2.72/5 on user wants","Real user expectations: LLMs fall short with 2.72/5 score","New benchmark: LLMs miss most user expectations"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that expectations extracted by an LLM from follow-up messages match what the user actually expected from the original response; no per-item human agreement is reported for extraction itself.","fun_headline_variants_meta":{"raw":{"variants":["Top LLM scores 2.72/5 on real user expectations","LLMs anticipate only 40% of user expectations","ExpectBench: even best LLMs score 2.72/5 on user wants","Real user expectations: LLMs fall short with 2.72/5 score","New benchmark: LLMs miss most user expectations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000599,"raw_usage":{"total_tokens":2620,"prompt_tokens":713,"completion_tokens":1907,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":1815}},"tokens_in":457,"tokens_out":1907,"duration_ms":16152,"temperature":1.0,"reasoning_tokens":1815,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:25:48.677591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of ExpectBench instances and have the original users (or blind human annotators) judge each extracted expectation: were these 'expectations' things they actually wanted from the first response? If a large share are judged inventions, restatements of the query, or post-hoc rationalizations, then the low scores and LENS gains measure the extractor's construct rather than real user expectations. A cheaper version: measure inter-annotator agreement on expectation extraction from the same follow-up messages; if agreement is low, the rubrics are unstable.","supporting_citations":[],"review_version":1}