{"id":"491f2bed-7b62-4270-b84d-5cec4af11adc","arxiv_id":"2505.14106","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"PERSONACONVBENCH is a new Reddit-based benchmark showing that LLMs predict sentiment, community scores, and next replies better when given a user's multi-turn conversation history, and it releases public data and code.","lead":"Researchers built PERSONACONVBENCH, a large new benchmark from Reddit discussions that tests whether AI language models can learn an individual user's style across multiple conversation threads. The paper reports that giving models a user's past conversation history improves their predictions on sentiment, community score, and next-message tasks, though several headline gains look inflated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The temporal-consistency guarantee may be violated: user-history conditioning uses Cu \\ C without a documented t < tτ filter, and the Appendix prompt template reveals the few-shot demonstration is drawn from the same post (u1p1) as the test target, risking leakage of the target's own…","rationale":"The reader's weakest_assumption already identified the temporal-consistency guarantee as the key load-bearing condition, and my reading of the manuscript text strongly agrees. The formal definition of Cu in Section 2.1 has no time bound; Section 3.1 includes Cu \\ C with no explicit t < tτ filter; Section 2.3 promises temporal consistency but does not demonstrate its implementation; and the Appendix prompt templates (Tables A–D) demonstrably draw the few-shot demonstration from the same post ID as the test target, which is the most concrete evidence that leakage is not merely possible but plausibly realized. This is decisive for the central claim because the entire benchmark's value proposition is that conditioning on conversational user history — not on future behavior — drives the gains. The paired t-tests in Table F establish only that P-Conv > P-NonConv on the actual prompts as constructed; they cannot distinguish 'personalized context' from 'leaked future messages'. If the temporal filter is missing, the qualitative finding (conversation helps) may survive in a weaker form (GPT-4.1's accuracy 0.9122 vs 0.7862 remains large even if MCC shrinks), which is why I recommend CONDITIONAL rather than REJECT: the benchmark's existence, tasks, and non-leaking variants are still valuable, but the headline magnitude and its causal interpretation are contingent on the code audit. The concrete test is cheap and decisive: re-run a filtered subset against the released code path. I do not see a stronger internal-inconsistency objection than this; other concerns (best-of-10 generation, regression hint) are real but affect secondary claims and effect sizes, not the causal integrity of the benchmark design.","tokens_in":28482,"tokens_out":2387,"duration_ms":20283,"concrete_test":"Pin the exact commit of the released code (github.com/PERSONA-bench/PERSONA/tree/Latest). Instrument the instance-construction function to record, for a random sample of ≥100 test instances, whether every message in Cu \\ C (and in the few-shot demonstration's referenced post and profile) has a timestamp strictly earlier than tτ. Then recompute Table 3's P-Conv metrics twice: (a) with a hard temporal filter t < tτ applied to all user-history and demonstration content, and (b) with the demonstration guaranteed to come from a different post (different post ID) than the target. If GPT-4.1's MCC drops materially below 0.6770 (or below P-NonConv's 0.2266 plus a meaningful margin) in either recomputation, the headline 198% relative gain no longer supports 'personalized multi-turn context' as the causal driver.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The benchmark's central empirical claim is that P-Conv's gains come from personalized conversational context (Table 3, GPT-4.1 MCC 0.2266→0.6770). The paper explicitly promises temporal consistency (Section 2.3: 'only prior context is accessible... preventing any leakage from future content'), and Section 3.1 says the model receives Cu \\ C. But the formal definition of Cu (Section 2.1) is 'all trajectories containing messages authored by u', with no timestamp bound relative to target mτ. Section 3.1's 'Context and User History' lists {mt ∈ C | tt < tτ} for the current trajectory, then 'the rest of the user's data Cu \\ C' with no explicit filtering. If the released code implements Cu \\ C without a t < tτ filter, the conditioning set includes the user's later posts and, critically, replies the same user made to other branches of the same post — including messages that occur temporally after the target. The prompt templates (Tables A–D, Figure D) confirm this worry: the few-shot demonstration is drawn from the same post ID 'u1p1' as the ACTUAL TASK target (also 'u1p1'), so the demonstration's 'dialogue tree context' and the model's available user history can contain the target post's full conversation tree, including the target reply's siblings and even later turns. In Reddit threads, the original poster typically replies to multiple comment branches at similar times; a 'user trajectory' defined per post contains both temporally prior and temporally posterior replies by the same user. If such later replies leak into the prompt, classification/regression/generation scores inflate: e.g., in sentiment classification the model could read the user's later same-thread replies ('Thanks...', etc.) that share sentiment with the target. The paper's Appendix C.1 states the user's 'profile' includes 'the full conversation tree rooted at the user's original post', which is exactly the leakage channel.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PersonaConvBench, a benchmark built from Reddit conversation trees across 10 domains, with three tasks: personalized conversational sentiment classification, impact forecasting, and follow-up text generation. The authors evaluate five LLMs under a Personalized Conversational condition (P-Conv, which receives the current thread prefix plus the user's other trajectories), a Personalized Non-Conversational condition (P-NonConv, which receives only the root post and a fixed demonstration), and for two models a Non-Personalized Conversational condition (NP-Conv). The central empirical claim is that P-Conv substantially outperforms P-NonConv on all three tasks, headlined as a 198% relative MCC gain for GPT-4.1 on sentiment classification (MCC 0.2266 to 0.6770), an 11.5% regression gain, and a 35.1% generation gain. The paper also releases the dataset and code.","tokens_in":28723,"tokens_out":7140,"duration_ms":76445,"significance":"If the headline results survive scrutiny, PersonaConvBench would be a useful community resource: it is one of the first benchmarks to combine user-level personalization with multi-user, graph-structured conversation, it covers three tasks across 10 domains at reasonable scale (19,215 posts, 111,239 conversations, 3,878 users), and it uses a unified prompting protocol across five diverse LLMs. The open release of data and code, and the paired significance testing for the generation task, are positive features. However, the central empirical contribution currently rests on two load-bearing issues: the temporal-consistency guarantee in Section 2.3 is not implemented in the formal definitions or the prompt templates, and the generation scores in Appendix C.3 are produced by selecting the candidate with the highest SBERT similarity to the ground truth, which is an oracle procedure over the reported metric. These issues must be resolved and the experiments rerun before the claimed gains can be attributed to personalized conversational context.","major_comments":[{"comment":"The temporal-consistency guarantee is not enforced by the written definitions. Section 2.1 defines the user trajectory set Cu as 'all trajectories containing messages authored by u' with no timestamp bound relative to the target mτ, and Section 3.1 states that the prompt includes 'the rest of the user's data Cu \\ C' without a t < tτ filter. This contradicts Section 2.3's promise that 'only prior context is accessible ... preventing any leakage from future content.' The risk is concrete: in Tables A-D the few-shot demonstration is taken from the same post (u1p1) as the actual task target (also u1p1), and in Table A the demonstration reply u1p1-c3-c4-c5-c6 lies in the same branch as the target u1p1-c3-c4. If the released code implements Cu \\ C literally, the P-Conv condition can see the user's later replies, including replies in the same post tree, so the reported P-Conv gains (e.g., GPT-4.1 MCC 0.2266 to 0.6770 in Table 3) may reflect future-content leakage rather than personalization. Please enforce and document a strict temporal cutoff on all conditioning data, verify that no demonstration or history item shares the target post tree, and rerun the full comparison; the revised numbers should be reported for all models and tasks.","section":"Section 2.1, Section 2.3, Section 3.1, Tables A-D"},{"comment":"The generation evaluation is an oracle over the headline metric. Appendix C.3 states: 'The final score for each input was computed by selecting, from those 10 candidates, the one with the highest SBERT score as the “Best Response.”' Because SBERT similarity to the ground truth is also the reported semantic metric, the generation scores in Table 3 are obtained by optimizing the evaluation metric over a candidate set. This makes the reported SBERT numbers an upper-bound estimate rather than a measure of a deployed sampling strategy, and the lexical metrics computed on the selected candidate inherit the same selection effect. The P-Conv vs P-NonConv comparison is same-treatment, but the absolute numbers and the 35.1% generation gain cannot be interpreted as standard single-output quality. Please report greedy decoding or average-of-samples results as the primary numbers, and present best-of-k selection as a separate analysis.","section":"Appendix C.3; Section 2.3.3; Table 3"},{"comment":"The headline '198% relative gain' is computed as (0.6770 - 0.2266) / 0.2266 on the MCC scale. MCC is a bounded correlation coefficient, not a ratio-scale quantity, so a relative percentage change is not meaningful and overstates the improvement; the actual change is 0.45 MCC points. Please report absolute differences with confidence intervals, and avoid percentage-change language for MCC and other bounded metrics.","section":"Abstract; Section 4.2; Table 3"},{"comment":"Paired significance testing is reported only for the generation task. The central classification and regression comparisons in Table 3 are presented without significance tests or confidence intervals, even though instances are paired by construction across the P-Conv and P-NonConv conditions. Please provide paired tests or bootstrap confidence intervals for all three tasks and all five models, and report effect sizes rather than relying on large-sample point estimates alone.","section":"Appendix E.1; Table 3"}],"minor_comments":[{"comment":"The benchmark construction uses several thresholds (θ, Ns = 3, Ns2 = 2, Nu = 4, Nr = 4, Np = 3, and the bottom-55% positive-score filter), but the paper reports no sensitivity analysis. At minimum, please state whether the headline gains persist under reasonable variations of these thresholds, since they define the entire dataset.","section":"Section 2.2.1; Appendix C.1"},{"comment":"The NP-Conv value for DeepSeek-R1 impact forecasting in Table Q is RMSE 551.67, far outside the range of all other RMSE values (roughly 282–354), and the table notes that NP-Conv data is marked as '-' for these models in the provided source. Please verify this number and correct any inconsistency.","section":"Table Q; Table E"},{"comment":"Section 3.1 says that for classification and regression 'the text xτ is revealed,' while Section 2.3 says 'the content of the target message mτ is masked during evaluation.' Please clarify exactly which fields are masked for each of the three tasks; the current wording is contradictory.","section":"Section 3.1 vs Section 2.3"},{"comment":"The caption states that RMSE and MAE are normalized to [0,1] using RMSEscaled = (360 - RMSE)/70 and MAEscaled = (120 - MAE)/30, but these formulas can produce values outside [0,1] for the reported ranges. Please use a proper min-max normalization or describe the scaling as a heuristic.","section":"Figure 2 caption"},{"comment":"Claude was limited to one generated candidate 'to preserve performance,' while all other models generated 10 candidates. This creates a model-dependent evaluation protocol; please justify this choice or treat Claude's generation scores separately in the comparisons.","section":"Appendix C.3"}],"recommendation":"major_revision","confidential_remarks":"The benchmark has clear potential value, and the breadth of the collaboration is evident, but the written manuscript and the supplied prompt templates contradict the paper's own temporal-consistency guarantee. The authors may respond that 'historical' was intended or that the code filters timestamps, but the definitions in Section 2.1 and the prompts in Tables A-D do not say that, and the released code must be audited before the headline numbers can be trusted. A rerun under a strict t < tτ filter, plus a non-oracle generation protocol, is essential. I do not see evidence of deliberate misconduct; the issues are consistency and documentation failures, but they are load-bearing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2505.14106. First, it is a real and useful artifact: a benchmark combining user trajectory sets with graph-structured multi-turn conversation across classification, regression, and generation, over 10 Reddit domains, with data and code released, and evaluation on five LLMs. That combination is not present in LaMP, LongLaMP, PersoBench, or the dialogue benchmarks it cites. Second, the headline numbers (198% MCC gain, 35.1% ROUGE-L gain) are inflated by the evaluation protocol, and the paper's own temporal-consistency promise is likely broken.\n\nLet me get the soft spots in proportion. The most important is temporal leakage. Cu is defined as all trajectories containing messages authored by u, with no timestamp bound relative to the target m_tau. Section 3.1 feeds Cu \\ C with no explicit t < t_tau filter. Appendix C.2 says the user profile includes the full conversation tree rooted at the user's original post. The prompt templates in Table A show the few-shot demonstration and the actual task both referencing the same post ID (u1p1), and the profile includes the whole thread. In a Reddit thread, a user typically replies to multiple branches over time; if those later replies land in Cu \\ C, the model can read the user's later sentiment and behavior before predicting the target. Section 2.3 promises 'only prior context is accessible,' but the formal definition does not guarantee it and the code is not yet verifiable. This is a load-bearing concern, not cosmetic.\n\nSecond, generation is scored by generating 10 candidates and keeping the one with the highest SBERT similarity (Appendix C.3). That mechanically inflates SBERT and gives models with 10 candidates an advantage over Claude's single candidate. Third, the regression prompt in Table B tells the model 'the score is expected to be positive,' leaking the sign and overstating regression gains. This one is easy to fix and I'd call it a bug. Fourth, the 198% relative gain is computed against an MCC near chance (0.2266); the direction of the finding is robust across models, but that particular headline is a bit of a showpiece.\n\nWhat holds up: the qualitative conclusion that personalized conversational context helps is credible and consistent across five models and ten domains. The benchmark construction is documented at a level that makes it checkable, and the authors did the right thing in releasing data and code. The paired t-tests give statistical support, though without p-values or effect sizes.\n\nBottom line: this is a solid first cut at a useful benchmark, and it deserves a serious referee rather than a desk rejection. For the version that I would trust with real numbers, the authors need to: pin the code with a commit hash, enforce a strict t < t_tau filter on all user history (and demonstrate it in the released code), remove or justify the regression hint, and report single-sample generation scores alongside the best-of-10 ones. If those changes land, I'd cite the benchmark and consider it a standard testbed for user-adaptive dialogue evaluation. Until then, treat the magnitudes with caution but the artifact as worth engaging with.","headline":"Useful new benchmark, but the headline gains are inflated by temporal leakage and evaluation choices; fixable.","tokens_in":29570,"tokens_out":3839,"would_cite":false,"duration_ms":35989,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-07T15:41:01.743563+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}