{"id":"36dd30b9-3ee2-4905-b124-b620ec42ba39","arxiv_id":"2505.17571","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Large reasoning models underperform general chat models on personalization tasks, but a structured template plus self-checking and self-referencing restores and improves performance.","lead":"This paper tests whether reasoning-focused AI models (LRMs) are better than standard chat models at generating personalized content, and finds they usually are not, especially when pulling in user history. The authors then build a prompt framework called R2P that nudges reasoning models to follow a step-by-step personalization template, and show it beats several simple baselines on a standard benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'significant outperformance' claim is not backed by statistics: Table 7 gives only three-run averages without variance, and several key R2P-vs-RAG gaps are tiny or negative.","rationale":"I read the paper in good faith: it offers a systematic comparison of LRMs and general models on LaMP, a plausible training-free framework (R2P), and an ablation study. The strongest claim, however, is that R2P 'significantly outperforms existing techniques.' That claim rests on Table 7, which reports only three-run means. The reader's listed weakest assumption was the unspecified RPI detection mechanism; I agree that is a real reproducibility gap, and it appears in the ablation (removing RPI degrades performance), so it matters for understanding why R2P works. But the more load-bearing issue is statistical: the central claim is empirical, and the reported numbers do not demonstrate that the differences are outside noise. The paper's own results show R2P is not uniformly better (e.g., LaMP-2N k=1 and LaMP-2M k=1 favor RAG on some metrics), yet the abstract and conclusion assert consistent superiority. The correct response is conditional acceptance with a requirement for error bars, significance testing, and code release; these requirements were already in the reader's verdict, so my read does not change the verdict. I set agreement_with_reader to 'partial' because the reader identified the RPI omission as the weakest assumption, whereas I see the missing variance/error bars as the most load-bearing; the reader did note 'add error bars' in the rationale, but not as the primary concern. My concrete test targets exactly the numbers that would settle whether the superiority claim is real.","tokens_in":14180,"tokens_out":3375,"duration_ms":33997,"concrete_test":"Re-run the Table 7 comparison with at least 10 independent generations per condition (or 5 seeds with per-seed evaluation), and report mean ± std as well as paired bootstrap 95% confidence intervals for every R2P-vs-RAG difference. If the LaMP-1 k=1 Acc gap (0.673 vs 0.650) and the LaMP-5 k=1 R-1 gap (0.412 vs 0.406) have CIs that include zero, the 'significantly outperforms' claim fails and the conclusion must be softened to 'mixed, mostly within noise.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that R2P 'significantly outperforms' RAG, PAG, and self-verification on LaMP tasks. The evidence in Table 7 is averages over three runs (LRM at temperature 0.6), with no standard deviations, confidence intervals, or per-run values. Several decisive comparisons are small: LaMP-1 k=1 Acc 0.673 vs 0.650; LaMP-5 k=1 R-1 0.412 vs 0.406; LaMP-2N k=4 Acc 0.720 vs 0.722 (R2P numerically worse). Only a few cells carry a p<0.05 marker, and the statistical test is not described (paired over what unit? 200 users? 3 runs?). Without variance information, the claimed gains could easily be within run-to-run noise, especially since LRM decoding uses temperature 0.6. The RPI mechanism is also underspecified, but even a perfect specification would not rescue an unquantified measurement. The paper's own abstract says 'significantly outperforms,' yet the table shows mixed results across tasks and metrics; this overstatement is the most load-bearing weakness because it directly affects whether the core contribution is established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a systematic evaluation of large reasoning models (LRMs) versus general-purpose LLMs on the LaMP personalization benchmark, reporting that LRMs do not consistently outperform LLMs, particularly under retrieval-intensive settings. It proposes R2P, a training-free framework with three components: a hierarchical reasoning thought template (HRT), a reasoning process intervention (RPI), and a self-referencing module (SRM). The main experimental claim is that R2P significantly outperforms RAG, PAG, and self-verification baselines on most LaMP tasks, supported by Table 7 and ablations in Figure 3.","tokens_in":14462,"tokens_out":2845,"duration_ms":26759,"significance":"If the results hold, the paper would make a useful contribution by identifying limitations of LRMs in personalization and offering a light-weight, training-free remedy that also reduces output token length. The preliminary comparison across multiple model families (Llama, Qwen, DeepSeek) is a informative starting point for future work. However, the significance is substantially undercut by the absence of variance information or a described statistical test in the main results, and by the underspecified RPI mechanism, which is a core claimed component.","major_comments":[{"comment":"The central claim that R2P “significantly outperforms” baselines is not supported by the reported statistics. Table 7 gives only three-run averages with no standard deviations, confidence intervals, or per-run values, and the paired t-test used to mark asterisks is not described (paired over users, runs, or something else?). Several decisive comparisons are tiny or negative: at k=1 on LaMP-2N, R2P's accuracy is 0.697 vs. self-verification's 0.713; at k=1 on LaMP-2M, R2P's accuracy is 0.282 vs. RAG's 0.332; at k=4 on LaMP-2N, R2P's accuracy is 0.720 vs. PAG's 0.722. The abstract's “significantly outperforms” is therefore an overstatement. Please report variance or confidence intervals, specify the test, and temper the claims to what the data support.","section":"§5.2, Table 7"},{"comment":"The Reasoning Process Intervention is a load-bearing component of R2P, but its implementation is not specified. The text says the output is evaluated “against a checklist derived from the HRT” and that deviations are detected, yet it never states whether this evaluation is performed by a separate LLM judge, by rule-based heuristics, or by manual inspection. Without this detail, the reader cannot determine whether the intervention is automatic, reproducible, or merely a prompt-engineering artifact. Please specify the detection mechanism, including how the checklist is instantiated and how false positives/negatives are handled, and provide evidence that the intervention behaves as claimed.","section":"§4.2"},{"comment":"The ablation study that is meant to verify the contribution of each module is reported only as a figure with no numeric values, no error bars, and no significance testing. The text states that “removing the reasoning process intervention can degrade model performance” but does not quantify the degradation or show that it is outside noise. Since the framework has three interacting components, the ablation should include the actual scores and, ideally, the same statistical treatment as the main table.","section":"§5.3, Figure 3"}],"minor_comments":[{"comment":"The abstract describes an “intervention method in the reasoning process,” but §4.2 states that the intervention occurs “after the LRM completes the reasoning chain.” The wording should be adjusted to avoid implying an interleaved, mid-chain intervention, or the method should be changed accordingly.","section":"Abstract and §4.2"},{"comment":"The TAKEAWAY I statement that the general-purpose LLM “outperforms the LRM across most tasks in retrieval-intensive settings” should be reconciled with Table 1, where DeepSeek-Llama3 is better on several tasks (e.g., LaMP-5 R-1) even at k=4. Please phrase the takeaway with the actual per-task counts.","section":"§3.2, Table 1"},{"comment":"The term “self referencing” is written inconsistently; it should be hyphenated as “self-referencing” throughout, including in the module name and figure captions.","section":"Throughout"},{"comment":"The reference number n=2 is chosen for the main experiments, but the paper does not discuss how sensitive the results are to this choice beyond Figure 4. Since n is the only hyperparameter, please state whether the significance in Table 7 changes for n=1 or n=3.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core empirical claim is not yet established because Table 7 lacks variance information and the RPI mechanism is presented at a level that cannot be reproduced. These are fixable in principle: add error bars/confidence intervals and a precise description of RPI. If the authors cannot provide such evidence, the paper would not meet the bar for acceptance in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the good part: I think the paper is worth reading for its evaluation alone. It is the first systematic comparison of reasoning-distilled models (DeepSeek-R1-Distill against Llama and Qwen bases) on LaMP, and the finding that LRMs don't consistently beat their base models, especially when retrieval-augmented, is a real and counterintuitive result. The three qualitative limitations they identify (divergent thinking, format alignment, poor use of retrieved context) are plausible and the example in Figure 6 shows the contrast nicely. The R2P framework is simple: a handcrafted thought template, a correction step, and a self-referencing vote. Nothing revolutionary, but the combination is new and the prompt template in Appendix B is concrete enough to reproduce.\n\nNow the soft spots. The central claim of 'significantly outperforms' is not supported by the numbers in Table 7. The table reports averages over three runs at temperature 0.6, with no standard deviations, confidence intervals, or per-run values. The significance markers are unexplained (paired over what unit?), and several key comparisons are tiny or actually negative (LaMP-2N at k=4, R2P Acc 0.720 vs RAG 0.722). Without variance, most of the claimed gains could be noise. That is the load-bearing weakness. Second, the Reasoning Process Intervention is underspecified: how does the framework detect deviation from the template? An LLM judge, heuristics, manual annotation? The paper doesn't say, and the ablation figure suggests RPI helps but gives no numbers. Third, the hyperparameter n is tuned on the same benchmark and then used to report the main results, which is a mild leak; not fatal, but worth flagging.\n\nThe authors do include an honest limitation section (limited model set, training-free focus), which I appreciate. And the token-length analysis in Section 3.4 and Figure 5 is a nice attempt to connect compute cost to performance, though the overall cost of R2P is acknowledged to be higher.\n\nWho is this for? Anyone working on personalization with reasoning models or on prompt engineering for LRMs. The evaluation table is a useful reference even if the framework's gains are not statistically confirmed. I would not cite it until the numbers have variance, but I would give it a serious peer review, and I would ask for code, error bars, and a precise description of RPI before publication.","headline":"Useful first evaluation of LRMs on LaMP, but the claimed significant gains lack error bars and the core intervention is underspecified.","tokens_in":14930,"tokens_out":2594,"would_cite":false,"duration_ms":23710,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large reasoning models underperform on personalization; a training-free framework closes the gap.","keywords":["large reasoning models","personalization","LaMP benchmark","retrieval-augmented generation","hierarchical reasoning template","reasoning process intervention","self-referencing","training-free framework"],"falsifier":"Run R2P on LaMP tasks with the intervention stage replaced by a no-op that never injects corrective instructions, keeping the template and self-referencing identical; if the no-op variant matches or exceeds full R2P on the reported metrics, the intervention is not doing the work the paper claims.","tokens_in":14030,"feed_emoji":"🧠","tokens_out":4831,"duration_ms":36007,"temperature":0.7,"pith_summary":"The paper argues that large reasoning models (LRMs), despite their strong performance on math and coding, are not naturally good at personalization: on the LaMP benchmark they often lose to general-purpose LLMs, especially when four retrieved user examples are provided. It attributes this to three shortcomings: reasoning that converges too narrowly, responses that ignore required formats, and poor use of retrieved user context. To fix this without any fine-tuning, it proposes R2P, a pipeline that guides the model through a hierarchical reasoning thought template, injects corrective instructions when the chain deviates, and synthesizes multiple candidate outputs by self-referencing. On LaMP tasks, R2P beats RAG, profile-augmented prompting, and self-verification baselines on most metrics, with the largest gains in regression and generation tasks.","feed_headline":"Training-free framework lifts reasoning models past RAG","feed_subtitle":"A structured prompt, corrective intervention, and self-referencing improve LaMP personalization without retraining.","key_machinery":"The load-bearing object is the hierarchical reasoning thought template, a seven-step prompt that decomposes any personalization query into question analysis, user profile integration, retrieval-augmented generation, content generation, format alignment, conciseness, and final output. Around it, R2P wraps two mechanisms: reasoning process intervention (RPI), a feedback loop that evaluates the chain against a checklist and injects a corrective instruction at the end if a sub-step is missing, and a self-referencing module (SRM), which generates n candidate answers and prompts the model to merge them into one consistent output. The template's role is to stop divergent thinking and force retrieved context into the answer; RPI enforces adherence; SRM stabilizes outputs across exploratory samples.","core_discovery":"The central discovery is that LRMs' reasoning strengths do not transfer to personalization: generating hundreds of reasoning tokens can hurt rather than help when the goal is a format-constrained, user-specific answer. The paper's diagnosis is that LRMs engage in convergent reasoning on open-ended personalization tasks, skip over retrieved user history, and emit verbose chains that break the strict output format. R2P, the paper's proposed remedy, consists of three ingredients: a handcrafted hierarchical reasoning thought template that forces question analysis, user-profile synthesis, retrieval use, format checking, and conciseness steps; a reasoning process intervention that checks the generated chain against the template's checklist and injects a corrective instruction such as \"Wait, let me analyze the user profile\" when a sub-step is missing; and a self-referencing module that generates several candidate answers and then asks the model to synthesize one final response. With the full pipeline, the paper reports that R2P achieves the best scores on most LaMP tasks and improves on regression and generation metrics at k=4.","pith_inferences":["The intervention's benefit rests on how deviations are detected, which the paper leaves unspecified; a natural testable extension is to compare RPI variants that use an explicit judge prompt, keyword heuristics, or no detection at all.","Because the paper evaluated only the Llama/Qwen/DeepSeek distill families, an untested implication is that the framework transfers to other reasoning models such as OpenAI o1; if it does, the template is a general interface for reasoning models rather than a fix specific to certain distill weights.","A practical extension would be to apply R2P's template-plus-intervention pattern to dynamic user interactions or multimodal personalization, which the paper explicitly flags as outside the LaMP benchmark's coverage."],"forward_implications":["If R2P is correct, personalization does not require retraining a reasoning model; a structured prompt plus a feedback loop is enough to beat retrieval-augmented prompting alone.","The template alone reduces token length relative to bare RAG, while improving output quality, so cheaper inference can come with better personalization.","Increasing retrieved context from k=1 to k=4 helps R2P more than it helps baselines, so the framework converts additional user history into better answers.","The self-referencing parameter n=2 gives the best trade-off across classification, regression, and generation tasks; n=1 is best on the simpler citation-identification task."],"supporting_citations":[{"why":"Defines the LaMP benchmark, its tasks, user-based separation, and evaluation metrics used throughout.","marker":"[25]"},{"why":"Provides retrieval-augmented generation, the main baseline and context-setting approach the paper compares against.","marker":"[17]"},{"why":"Introduces profile-augmented prompting (PAG), one of the baseline methods R2P is measured against.","marker":"[23]"},{"why":"Supplies the self-verification baseline method that R2P outperforms.","marker":"[33]"},{"why":"Describes DeepSeek-R1, the large reasoning model family whose personalization behavior the paper evaluates.","marker":"[9]"},{"why":"Provides Llama 3, the general-purpose LLM compared against the reasoning models.","marker":"[7]"},{"why":"Introduces the Qwen2.5 series, used for the model-scaling comparison.","marker":"[35]"},{"why":"Supplies BM25, the retrieval algorithm used to fetch user-specific examples in all RAG settings.","marker":"[24]"}],"fun_headline_variants":["Why reasoning models fail at personalization and how to fix it","R2P realigns reasoning models for personalized output","Training-free R2P boosts reasoning models on personalization","No retraining: R2P improves reasoning models for personalization","Structured reasoning template lifts reasoning models in personalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole claimed benefit of the reasoning process intervention depends on an unstated way of detecting whether the model's chain really deviates from the template; if that detection is unreliable, circular, or manual, the intervention's contribution to the reported gains is unproven.","fun_headline_variants_meta":{"raw":{"variants":["Why reasoning models fail at personalization and how to fix it","R2P realigns reasoning models for personalized output","Training-free R2P boosts reasoning models on personalization","No retraining: R2P improves reasoning models for personalization","Structured reasoning template lifts reasoning models in personalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001101,"raw_usage":{"total_tokens":4595,"prompt_tokens":946,"completion_tokens":3649,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":3566}},"tokens_in":562,"tokens_out":3649,"duration_ms":21473,"temperature":1.0,"reasoning_tokens":3566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:43:34.888818+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run R2P on LaMP tasks with the intervention stage replaced by a no-op that never injects corrective instructions, keeping the template and self-referencing identical; if the no-op variant matches or exceeds full R2P on the reported metrics, the intervention is not doing the work the paper claims.","supporting_citations":[{"cited_title":"Salemi, S","cited_arxiv_id":null,"evidence_quote":"Defines the LaMP benchmark, its tasks, user-based separation, and evaluation metrics used throughout."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the self-verification baseline method that R2P outperforms."},{"cited_title":"Robertson, H","cited_arxiv_id":null,"evidence_quote":"Supplies BM25, the retrieval algorithm used to fetch user-specific examples in all RAG settings."}],"review_version":1}