{"id":"284af3b8-8d60-481a-bc3d-baa6fb46cc21","arxiv_id":"2605.29711","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Presents a memory-augmented turn-level satisfaction evaluator and PersTurnBench benchmark that improve agreement with human judgments over generic LLM judges and enable controlled model comparisons via replay.","lead":"This paper builds an AI evaluator that scores how satisfied a specific user is with one turn in a conversation, by combining a compact memory of that user's past interactions with the current context. It also releases PersTurnBench, a benchmark that lets researchers compare different response-generation models on personalized satisfaction using replay without collecting new human ratings for each model.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Human satisfaction annotations lack reported inter-annotator agreement or personalization validation, undermining meta-evaluation of ordinal agreement gains.","rationale":"The reader's weakest assumption directly identifies the same dependency on human labels and replay isolation. No deeper internal inconsistency is visible from the provided abstract and claim; the concern is empirical reliability of the evaluation pipeline rather than a logical flaw.","tokens_in":1705,"tokens_out":289,"duration_ms":12152,"concrete_test":"Compute and report inter-annotator agreement (Cohen's or Fleiss' kappa) on the human satisfaction annotations used for meta-evaluation; if kappa < 0.6 on the ordinal scale or on the binary dissatisfied-turn label, re-run the meta-evaluation tables after majority-vote or noise-robust aggregation and check whether the claimed gains over baselines remain statistically significant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim requires that human turn-level satisfaction labels serve as stable, unbiased ground truth. Personalization is inherently subjective; if annotators were not given the same user memory state as the model, or if labels exhibit low agreement due to differing interpretations of 'satisfaction,' then measured improvements in ordinal agreement and dissatisfied-turn detection over baselines could be artifacts of label noise rather than genuine evaluator superiority. The replay isolation in PersTurnBench inherits this risk because the 'verified evaluator' is itself meta-evaluated against those labels.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces a personalized turn-level conversation satisfaction evaluator that combines compact user memories with target-turn context to output satisfaction scores and dissatisfaction rationales. Meta-evaluation against human annotations is claimed to show that personalized memory plus post-hoc calibration improves ordinal agreement and dissatisfied-turn detection over supervised, retrieval-based, and generic LLM-as-a-judge baselines. The work further presents PersTurnBench, which uses the verified evaluator to score generation models via replay while holding state fixed, thereby enabling model comparisons without new human labels for each candidate.","tokens_in":1801,"tokens_out":454,"duration_ms":18002,"significance":"If the meta-evaluation results are robust, the work provides a practical route to assessing personalized satisfaction without repeated annotation collection, and the replay design in PersTurnBench is a clear methodological strength for controlled comparisons of memory-augmented versus generic generators.","major_comments":[{"comment":"Meta-evaluation section: the central claim of improved ordinal agreement and dissatisfied-turn detection rests on human satisfaction annotations serving as stable ground truth, yet the manuscript provides no inter-annotator agreement statistics, no description of how annotators were given equivalent user memory state, and no validation that labels reflect personalization rather than generic quality; without these, measured gains over baselines cannot be distinguished from label noise.","section":"Meta-evaluation"},{"comment":"PersTurnBench section: the replay protocol is presented as isolating generation-model effects, but the evaluator itself is meta-evaluated against the same unvalidated annotations; any systematic bias in those labels propagates directly into the benchmark scores, undermining the claim that the benchmark enables label-free model comparison.","section":"PersTurnBench"}],"minor_comments":[{"comment":"The abstract and methods would benefit from an explicit statement of the annotation collection protocol (number of annotators, instructions, payment, quality controls) even if moved to an appendix.","section":null},{"comment":"Notation for the post-hoc calibration step should be defined once and used consistently when reporting the calibrated versus uncalibrated scores.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments highlighting important aspects of the meta-evaluation and PersTurnBench. We address each major comment below and indicate planned revisions.","responses":[{"response":"We agree that inter-annotator agreement (IAA) statistics are not reported, which is a limitation of the current manuscript; we will compute and report IAA in the revision to demonstrate annotation stability. The annotation protocol provided each annotator with the same compact user memories used by the model to ground judgments in personalization, and we will add a detailed description of this protocol to the meta-evaluation section. For validation that labels capture personalization rather than generic quality, the consistent outperformance of the memory-augmented evaluator over generic LLM baselines offers supporting evidence, though we will include an additional analysis of label correlations with personalization features in the revision.","revision_made":"yes","referee_comment":"[Meta-evaluation] Meta-evaluation section: the central claim of improved ordinal agreement and dissatisfied-turn detection rests on human satisfaction annotations serving as stable ground truth, yet the manuscript provides no inter-annotator agreement statistics, no description of how annotators were given equivalent user memory state, and no validation that labels reflect personalization rather than generic quality; without these, measured gains over baselines cannot be distinguished from label noise."},{"response":"We recognize that any systematic bias in the human annotations can propagate through the evaluator into PersTurnBench scores. The replay protocol nevertheless enables controlled, state-fixed comparisons across models without requiring fresh human labels for each candidate, which remains a methodological contribution. We will revise the PersTurnBench section to explicitly acknowledge this propagation risk as a limitation while clarifying the benchmark's intended use case.","revision_made":"partial","referee_comment":"[PersTurnBench] PersTurnBench section: the replay protocol is presented as isolating generation-model effects, but the evaluator itself is meta-evaluated against the same unvalidated annotations; any systematic bias in those labels propagates directly into the benchmark scores, undermining the claim that the benchmark enables label-free model comparison."}],"tokens_in":1320,"tokens_out":442,"duration_ms":25322,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core offering is PersTurnBench, a replay-based benchmark for turn-level personalized satisfaction, paired with an evaluator that folds in compact user memories and post-hoc calibration. The abstract positions this as better than generic LLM judges, retrieval baselines, and supervised models on ordinal agreement and dissatisfied-turn detection.\n\nThe practical piece that stands out is the replay setup: fix the conversation state and swap in different generation models without needing fresh human labels each time. That design choice makes controlled comparisons feasible and addresses a real bottleneck in personalized system testing.\n\nThe soft spot is the ground truth. The meta-evaluation rests entirely on human satisfaction annotations, yet the abstract supplies no inter-annotator agreement figures, no description of how annotators received the same memory context as the model, and no check on whether labels reflect stable personalization rather than individual rater bias. Personal satisfaction is inherently variable; if label noise is high, the reported improvements could shrink or disappear. The stress-test concern lands because nothing in the provided abstract contradicts it.\n\nThis work is for researchers building or evaluating conversational systems who need turn-level, user-specific metrics. A reader already working on LLM-as-a-judge variants or benchmark construction could extract the replay idea and the memory-plus-calibration pattern. It is coherent on its own terms and shows clear engagement with the evaluation literature, so it deserves a serious referee even though the current evidence on label quality is thin.\n\nRecommendation: send it to review with a request for annotation details and any available agreement statistics.","headline":"The paper introduces PersTurnBench and a memory-augmented evaluator that claims gains on meta-evaluation metrics, but the human satisfaction labels have no reported agreement or personalization checks.","tokens_in":2288,"tokens_out":384,"would_cite":false,"duration_ms":16558,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Personalized user memory plus post-hoc calibration improves agreement with human turn-level satisfaction judgments over generic LLM judges and baselines.","keywords":["personalized satisfaction evaluation","turn-level conversation","user memory","LLM judge","meta-evaluation","replay benchmark","dissatisfied turn detection"],"falsifier":"Collect fresh human satisfaction labels on the exact same replayed conversations for two different generation models and check whether the PersTurnBench rankings match the new human rankings.","tokens_in":2605,"feed_emoji":"💬","tokens_out":587,"duration_ms":20995,"temperature":0.7,"pith_summary":"The paper builds an evaluator that stores compact user memories and combines them with the current turn to output satisfaction scores along with dissatisfaction rationales. Meta-evaluation on human annotations shows this raises ordinal agreement and better flags dissatisfied turns than supervised models, retrieval methods, or plain LLM judges. A post-hoc calibration step further refines the scores. The authors release PersTurnBench, which replays fixed conversation states through the evaluator to score different generation models without new human labels for each candidate. This setup lets researchers compare generic and memory-augmented systems on personalized satisfaction directly.","feed_headline":"Personalized memory lifts turn-level satisfaction scoring","feed_subtitle":"A fixed evaluator and replay benchmark let models be compared on user-specific satisfaction without new human labels for each candidate.","key_machinery":"The memory-augmented turn-level satisfaction evaluator that fuses compact user history with the current turn to produce calibrated scores and rationales.","core_discovery":"The paper claims that a satisfaction evaluator built from compact user memories and target-turn context, after post-hoc calibration, produces higher ordinal agreement with human annotations and stronger dissatisfied-turn detection than supervised, retrieval-based, or generic LLM-as-a-judge baselines, and that this evaluator can serve as a fixed judge in a replay benchmark that compares generation models on personalized satisfaction without requiring fresh human labels for every new model.","pith_inferences":["The replay approach could be applied to other turn-level metrics such as task success or engagement to reduce labeling costs across evaluation settings.","If the evaluator's rationales prove consistent, they might be used directly for model debugging or user-facing explanations.","One could test whether the fixed evaluator introduces its own systematic bias when the generation model changes dramatically in style or length.","Extending the memory representation beyond compact summaries might further lift agreement on long conversations."],"forward_implications":["The benchmark supports direct comparison of generic generation models against memory-augmented personalized systems on user-specific satisfaction.","Post-hoc calibration raises the reliability of ordinal rankings and dissatisfied-turn detection.","New generation models can be scored on personalized satisfaction without collecting additional user feedback for each model.","The evaluator supplies both numeric scores and explicit rationales for why a turn is dissatisfying.","Controlled replay removes the need to re-annotate every candidate model."],"fun_headline_variants":["User memories refine turn satisfaction evaluation","Memory context improves satisfaction scoring accuracy","Personalized evaluator detects dissatisfied turns better","Replay benchmark tests models without fresh labels","Fixed judge compares personalized generation systems"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Human satisfaction annotations form a stable unbiased ground truth and replaying fixed conversation states isolates the effect of the generation model without evaluator-induced confounds.","fun_headline_variants_meta":{"raw":{"variants":["User memories refine turn satisfaction evaluation","Memory context improves satisfaction scoring accuracy","Personalized evaluator detects dissatisfied turns better","Replay benchmark tests models without fresh labels","Fixed judge compares personalized generation systems"]},"model":"grok-4.3","cost_usd":0.006364,"raw_usage":{"total_tokens":2971,"prompt_tokens":636,"num_sources_used":0,"completion_tokens":56,"cost_in_usd_ticks":63637000,"prompt_tokens_details":{"text_tokens":636,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2279,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":636,"tokens_out":56,"duration_ms":19388,"temperature":1.0,"reasoning_tokens":2279,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T07:35:28.922693+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Collect fresh human satisfaction labels on the exact same replayed conversations for two different generation models and check whether the PersTurnBench rankings match the new human rankings.","supporting_citations":[],"review_version":1}