{"id":"0fa6164c-8eb3-4ad0-8258-ce59d391f163","arxiv_id":"2507.05880","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A reimplementation of RecRanker shows its pointwise variant's high top-k scores come from ground-truth data leakage in the prompts, and the new RecRankerEval framework finds listwise tuning, DBSCAN sampling, XSimGCL, and Llama3 improve results once the leak is fixed.","lead":"Researchers re-ran RecRanker, an LLM-based movie recommender, and found its strongest training method secretly peeks at the correct answers through hint scores hidden in the prompts. They built a modular framework, RecRankerEval, to fairly re-test the model's parts, and found simpler ranking prompts plus better base recommenders and stronger LLMs work better once the leak is removed.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Leakage claim is confounded: removing hint scores removes both ground-truth content and any hint signal; a predicted-score control is needed to isolate leakage.","rationale":"The reader's CONDITIONAL verdict already hinges on the absence of a control that keeps a hint while removing its ground-truth content; my analysis identifies the same experimental gap as the most load-bearing issue. The paper's headline contribution is the leakage attribution, and the current evidence is a single comparison that removes the hint entirely. This does not distinguish ground-truth leakage from the general value of a hint signal, especially since the other prompt variants legitimately use initial-recommender hints. A predicted-score control would settle the attribution. The framework and component-level findings remain useful regardless, so a conditional accept with this required control is appropriate. I agree with the reader's weakest_assumption and would not move the verdict.","tokens_in":20000,"tokens_out":4075,"duration_ms":44131,"concrete_test":"Construct a pointwise variant 'pointwise_HintPred' identical to the original pointwise prompt except that the hint score is replaced by the initial recommender's predicted score for the target item (e.g., the MF/LightGCN score or the rank-derived score used in U_retrieval). Train and evaluate on ML-100K with MF and LightGCN under the same protocol as Tables 3 and 4. If H@3 is close to the original pointwise value (~0.0807), the claimed leakage is not the cause of the boost. If H@3 is close to pointwise_Fix (~0.0488), the boost is attributable to ground-truth content. Either result adjudicates the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 5.4) is that RecRanker's pointwise results are inflated by data leakage because pointwise prompts include explicit ground-truth scores. The supporting evidence is the drop from RecRanker_pointwise (H@3 0.0807) to pointwise_Fix (H@3 0.0488) when hint scores are removed. But this manipulation conflates two variables: (a) the presence of a numeric hint as a useful input signal, and (b) the ground-truth content of that hint. The listwise and pairwise prompts also include hints from the initial recommender, and those are not treated as leakage; a score hint from the initial recommender would be a legitimate, non-leaking signal. Without a control that replaces the ground-truth score with the initial recommender's predicted score (keeping a hint but removing ground-truth content), the paper cannot attribute the performance drop to ground-truth leakage rather than to the general usefulness of a numeric hint. If the predicted-score control also yields high performance, the original pointwise advantage is not evidence of leakage; if it yields low performance comparable to pointwise_Fix, the leakage story is confirmed. Section 5.4 does not report this control, so the central causal claim is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reimplements RecRanker, an LLM-based top-k recommender, and packages the reimplementation as RecRankerEval, a modular framework spanning user sampling, initial recommendation models, LLM backbones, datasets, and instruction tuning variants. The authors report that their reimplementation reproduces RecRanker's pairwise and listwise results, while the pointwise variant's high performance is due to data leakage from ground-truth scores in the prompts; removing those scores (pointwise_Fix) causes a sharp drop in H@3/N@3. They further compare alternative samplers, stronger initial recommenders, and different LLMs, concluding that DBSCAN sampling, listwise tuning, and stronger backbones generally improve performance, and that BookCrossing is not reproducible because the original paper lacks timestamp information.","tokens_in":20183,"tokens_out":5129,"duration_ms":58812,"significance":"The paper is a useful reproducibility and analysis contribution: it provides a modular framework with preprocessing and prompt-construction scripts, covers a broad configuration space (three samplers, three initial recommenders, three LLMs, four datasets), and identifies a plausible leakage risk in RecRanker's pointwise prompts that, if confirmed, would be an important caution for instruction-tuning evaluations in LLM-based recommendation. The comparative results across sampling strategies and initial recommenders are also a useful reference for practitioners. The main weakness is that the two central claims—the leakage explanation and the reproduction of the original numbers—are not yet backed by fully controlled comparisons: the leakage experiment removes the hint signal entirely, and no side-by-side table of reproduced versus original RecRanker numbers is provided.","major_comments":[{"comment":"The evidence for the leakage claim is the drop from RecRanker_pointwise (H@3 0.0807) to pointwise_Fix (H@3 0.0488) when hint scores are removed from the pointwise prompts. This manipulation removes both the ground-truth content and the numeric hint signal simultaneously. Because the listwise and pairwise prompts retain hints from the initial recommender and are not treated as leakage, the paper should include a control that replaces the ground-truth score with the initial recommender's predicted score, keeping a non-ground-truth hint in the prompt. Without that control, the performance drop cannot be attributed to ground-truth leakage rather than to the general usefulness of a numeric hint, so the central causal claim of Section 5.4 is underdetermined.","section":"Section 5.4, Table 4"},{"comment":"The reproduction claim is not backed by a direct numerical comparison: Table 3 reports only the authors' reimplemented numbers, and the original RecRanker values are never shown in a table or listed against the reproduced ones. The reimplementation also uses LoRA on a single NVIDIA RTX A6000, whereas the original RecRanker uses full fine-tuning on 16 A800 GPUs (Section 5.3). The paper should provide a side-by-side table of reproduced versus original H@k and N@k values and discuss how LoRA and the reduced GPU setup might affect the comparison; otherwise RQ1 cannot be assessed from the presented evidence.","section":"Section 5.3, Table 3, RQ1"},{"comment":"The BookCrossing preprocessing is claimed to follow the original paper, but the reproduced dataset differs materially in size: 1,777 vs 1,820 users, 22,288 vs 2,030 items, and 90,818 vs 41,456 interactions. This is not a small discrepancy, and it may explain why all RecRanker variants underperform the baseline on BookCrossing in Table 4. The paper should clarify whether these statistics result from a different filtering rule or whether Table 2 contains an error, and should not attribute the failure solely to timestamp simulation without addressing this inconsistency.","section":"Section 5.1, Table 2"},{"comment":"The paper reports that pointwise_Fix aligns with pairwise and listwise performance, but it does not provide a statistical test of the difference between RecRanker_pointwise and pointwise_Fix; the asterisks in Tables 3 and 4 are comparisons to the base model only. Since the leakage conclusion rests entirely on this drop, the authors should report a paired significance test (or confidence interval) for the pointwise versus pointwise_Fix comparison, and confirm that the drop is not driven by the changed prompt format alone.","section":"Section 5.4, Tables 3 and 4"}],"minor_comments":[{"comment":"There is a typo: 'best perfrmance' should be 'best performance'.","section":"Section 5.6"},{"comment":"The sentence 'Furthermore, they the results show the importance...' is grammatically broken and should be rewritten.","section":"Section 5.7"},{"comment":"The table is split into three subtables with repeated column headers, which makes cross-model comparison difficult; a unified table with the initial recommender as a row or column group would be clearer.","section":"Table 4"},{"comment":"The text says ML-100K and ML-1M results are 'directly reproduced from the original RecRanker's supplied resources,' but Section 4.1 describes RecRankerEval as a reimplementation; the wording should clarify that these are reimplemented configurations, not copied outputs.","section":"Section 4.5"},{"comment":"The abstract says BookCrossing is not reproducible 'due to the lack of timestamp information,' while Section 5.1 says the authors simulate timestamps and Section 5.4 says the lack of real timestamps 'may introduce noise'; the phrasing should be consistent about whether the issue is absence of timestamps or noisy simulated timestamps.","section":"Abstract and Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the overall direction is sound. The central leakage finding is the kind of result worth publishing, but it currently rests on a confounded experiment; the predicted-score control is a clean, feasible addition that would substantially strengthen the paper. I would also encourage the editor to note the high concentration of self-citations in the reference list; it is not disqualifying, but a transparency concern for a reproducibility-oriented paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zeyuan, here's my read on 2507.05880.\n\nThe paper does two useful things. First, it identifies a likely data-leakage bug in RecRanker's pointwise prompt: the 'hint' score is literally the ground-truth rating for interacted items and zero for non-interacted ones. That is a real finding and, as far as I know, new. The corrected pointwise variant drops from 0.0807 to 0.0488 H@3 on ML-100K, and the corrected ranking puts listwise on top. Second, it ships RecRankerEval, a modular harness for five design dimensions, and contributes a clean Amazon-Music evaluation. The component ablations—DBSCAN over K-means, XSimGCL as initial recommender, Llama3 as backbone—are useful for anyone building on this paradigm.\n\nThe soft spots are real but not fatal. The leakage attribution is confounded: removing the hint removes both ground-truth content and any numeric hint. The paper doesn't run the control that would separate these, e.g., replacing the true rating with the initial recommender's predicted score. Given that the pointwise hint is literally the relevance label, I still think leakage is the most plausible explanation, but the causal claim as written is underdetermined. Also, for a reproducibility paper, Table 3 shows only the reimplementation—there's no side-by-side column of the original RecRanker numbers. The claim 'we reproduce the original results' is asserted, not demonstrated. The BookCrossing failure is blamed on simulated timestamps, but that's an assumption, not a verified cause. And the code isn't linked, which makes the framework hard to adopt right now.\n\nOn balance, I'd send this to peer review. The leakage finding is important enough to warrant a careful look even if the current evidence is short of conclusive. A good referee would ask for the predicted-score control, the side-by-side table, and the code release. The paper's own conclusions—especially that RecRanker's pointwise advantage was an artifact—are probably right, but they need to be argued more carefully.\n\nI'd bring this to our reading group and likely cite the leakage result in my own work once it clears review.","headline":"A useful reproduction study that exposes a real leakage problem in RecRanker's pointwise prompts, but the central attribution is under-supported and the missing side-by-side numbers weaken the reproducibility claim.","tokens_in":20849,"tokens_out":2973,"would_cite":true,"duration_ms":31677,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reproduces RecRanker and finds that its pointwise variant's apparent superiority is an artifact of data leakage in prompts.","keywords":["RecRanker","LLM-based recommendation","top-k recommendation","data leakage","instruction tuning","reproducibility","user sampling","RecRankerEval"],"falsifier":"Run RecRanker in the pointwise setting on ML-100K with the hint field preserved but its value changed from the true user rating to the initial recommender's predicted score; if hit rate at rank 3 stays near the leaked 0.0807 instead of falling to the corrected 0.0488, the leakage attribution is wrong, while a fall to about 0.0488 would confirm that the ground-truth content of the hint, not the hint structure, was carrying the performance.","tokens_in":1682,"feed_emoji":"🎯","tokens_out":4222,"duration_ms":77350,"temperature":0.7,"pith_summary":"This paper asks whether RecRanker's reported top-k recommendation results survive reimplementation and which parts of the design actually matter. It finds that pairwise and listwise variants reproduce, but the pointwise variant's high scores come from data leakage: the prompts embed ground-truth ratings as hint scores. A corrected version with those scores removed drops from 0.0807 to 0.0488 in hit rate at rank 3 on ML-100K, aligning with the other variants. The paper contributes RecRankerEval, a modular evaluation framework spanning user sampling, initial recommender, LLM backbone, dataset, and tuning method, and uses it to show that RecRanker transfers to Amazon-Music but not BookCrossing, and that stronger initial recommenders, DBSCAN sampling, and Llama3 all improve results.","feed_headline":"Pointwise wins in RecRanker were prompt leakage","feed_subtitle":"Removing ground-truth scores from pointwise prompts drops hit@3 from 0.081 to 0.049 on ML-100K.","key_machinery":"The load-bearing mechanism is the pointwise prompt's hint field, which contains a score that is not a neutral recommendation signal but a value derived directly from the user's ground-truth rating. That turns what should be a learned ranking task into a lookup of the correct answer inside the input, and it is what inflates the pointwise numbers. The corrective machinery is RecRankerEval's modular substitution setup, which lets each component of user sampling, initial recommender, LLM backbone, dataset, and instruction tuning method be swapped independently so that the effect of removing the leaking hint, changing the sampler, or upgrading the base model can be measured in isolation.","core_discovery":"The paper's central claim is that RecRanker's pointwise instruction-tuning variant achieves its unusually high performance by data leakage: the pointwise prompts include explicit scores derived from user ratings, so interacted or ground-truth items receive high scores while non-interacted items receive low or zero scores, effectively handing the model the answers inside the prompt. After removing these hint scores, the corrected pointwise variant's hit rate at rank 3 on ML-100K falls from 0.0807 to 0.0488, and listwise and corrected hybrid variants become the strongest instruction-tuning strategies. The paper further proposes RecRankerEval, an extensible framework that modularizes five components of the RecRanker paradigm, and reports that RecRanker-style models generalize to a new domain (Amazon-Music) but fail on BookCrossing when timestamps must be simulated, while swapping in stronger initial recommenders and newer LLM backbones improves overall performance.","pith_inferences":["A direct extension the paper does not run is to keep the pointwise hint structure but replace the ground-truth rating with the initial recommender's predicted score; this control would separate the leakage effect from the effect of losing any hint at all.","If the leakage finding generalizes, other LLM-based recommenders that encode ratings or predicted scores inside pointwise-style prompts may have similarly inflated results, and a simple prompt rewrite would serve as a cheap leakage audit.","The framework's modular swaps imply a testable recipe: compare two configurations that differ only in one component, and treat any large performance jump as a warning that the changed component is carrying information it should not.","Because the reproduction relies on single-GPU LoRA fine-tuning rather than the original full fine-tuning on 16 GPUs, a full-resource replication would be the natural next step to confirm that the corrected ranking of variants is not an artifact of the cheaper training regime."],"forward_implications":["RecRanker's originally reported pointwise results should not be treated as a fair baseline; the corrected pointwise run drops to 0.0488 hit rate at rank 3 on ML-100K.","After the correction, the listwise variant is the strongest instruction-tuning strategy in most configurations, suggesting that ranking-oriented prompts beat score-prediction prompts for LLM-based top-k recommendation.","RecRanker's design transfers to a new domain such as Amazon-Music with real timestamps, but fails on BookCrossing when timestamps must be fabricated, indicating that temporal data quality is essential for this pipeline.","RecRanker's performance improves when the initial recommender is upgraded from MF or LightGCN to XSimGCL, and when the backbone is upgraded from Llama2 to Llama3.","DBSCAN-based user sampling generally outperforms both K-means and random sampling across instruction-tuning variants, pointing to density-based denoised sampling as a better default."],"supporting_citations":[{"why":"Supplies the original RecRanker model, its prompt designs, datasets, and the reported numbers that this paper reproduces and then diagnoses as leaked.","marker":"[15]"},{"why":"Provides the MF baseline and initial recommender used to construct ranking lists and to measure the LLM's gains over a standard collaborative filtering method.","marker":"[10]"},{"why":"Provides the LightGCN initial recommender, which the paper uses as a baseline and as the fixed initial model in the user-sampling experiments.","marker":"[7]"},{"why":"Supplies the Llama2 backbone used for the main instruction-tuning reproduction, so the leakage finding is tied to the exact model family the original work used.","marker":"[21]"},{"why":"Supplies LoRA, the parameter-efficient fine-tuning method that makes the single-GPU reimplementation feasible and is therefore a premise of the resource-light reproduction.","marker":"[8]"},{"why":"Provides XSimGCL, the additional stronger initial recommender used to test whether a better base ranking list amplifies the benefit of LLM instruction tuning.","marker":"[38]"}],"fun_headline_variants":["RecRanker pointwise gains due to prompt leakage","Data leakage inflates RecRanker pointwise performance","RecRanker pointwise results traced to leaky prompts","Prompt leakage explains RecRanker pointwise hike","Ground-truth in prompts skews RecRanker pointwise"],"cache_read_input_tokens":22784,"weakest_assumption_plain":"The leakage conclusion assumes that removing the hint scores isolates the leaked ground-truth signal, but the paper has no control that keeps a hint whose content is not ground truth (such as the initial recommender's own predicted score), so part of the drop from 0.0807 to 0.0488 could come from losing any hint at all; a second load-bearing premise is that single-GPU LoRA fine-tuning faithfully represents RecRanker's original full fine-tuning on 16 A800 GPUs.","fun_headline_variants_meta":{"raw":{"variants":["RecRanker pointwise gains due to prompt leakage","Data leakage inflates RecRanker pointwise performance","RecRanker pointwise results traced to leaky prompts","Prompt leakage explains RecRanker pointwise hike","Ground-truth in prompts skews RecRanker pointwise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1457,"prompt_tokens":1062,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":317}},"tokens_in":678,"tokens_out":395,"duration_ms":4036,"temperature":1.0,"reasoning_tokens":317,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:17:31.197995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RecRanker in the pointwise setting on ML-100K with the hint field preserved but its value changed from the true user rating to the initial recommender's predicted score; if hit rate at rank 3 stays near the leaked 0.0807 instead of falling to the corrected 0.0488, the leakage attribution is wrong, while a fall to about 0.0488 would confirm that the ground-truth content of the hint, not the hint structure, was carrying the performance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MF baseline and initial recommender used to construct ranking lists and to measure the LLM's gains over a standard collaborative filtering method."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LightGCN initial recommender, which the paper uses as a baseline and as the fixed initial model in the user-sampling experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Llama2 backbone used for the main instruction-tuning reproduction, so the leakage finding is tied to the exact model family the original work used."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies LoRA, the parameter-efficient fine-tuning method that makes the single-GPU reimplementation feasible and is therefore a premise of the resource-light reproduction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides XSimGCL, the additional stronger initial recommender used to test whether a better base ranking list amplifies the benefit of LLM instruction tuning."}],"review_version":1}