{"id":"4d4e8abc-b65f-43da-88c8-0e56a8d50aa7","arxiv_id":"2608.11973","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Sci-Surf reports that feedback-derived verbal user profiles improve personalized arXiv recommendation relevance by 10.4 percentage points while LLM-generated multimodal digests remain largely faithful.","lead":"Sci-Surf is a personalized arXiv recommender that builds a verbal profile of each user from likes and dislikes, reranks each day's new papers with that profile, and generates blog-style summaries containing figures. The authors report that adding these profiles raised relevant recommendations from 18.9% to 29.3% for 15 real users.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal split for profile construction vs. evaluation is unspecified; if profiles include feedback from scored days, the 10.4% gain reflects memorization, not prediction.","rationale":"The reader's weakest assumption already names the absence of a temporal separation and the exclusive-paper annotation. My read converges on the same point, and I would make the temporal leakage the primary condition because it is the cheapest way for the reported gain to be spurious: if the profile verbalizer saw likes from the evaluation days, the reranker is effectively being scored on training data. The annotation-selection issue is real but secondary; it biases the magnitude of the difference, whereas leakage would invalidate the direction. The reader's CONDITIONAL verdict is therefore the right level: the system is coherent, the retrieval benchmark is standard, and the reported effect is modest, but the paper must clarify the split and supply uncertainty. I do not move the verdict; I would strengthen the acceptance condition to require a held-out future window and a random-sample annotation. If those checks fail, the claim should be withdrawn.","tokens_in":6801,"tokens_out":6282,"duration_ms":63104,"concrete_test":"Ask the authors to specify and enforce the temporal split: rebuild profiles from likes on days 1..T (or the stated training interval), then re-run the ablation only on recommended papers from days strictly after T (never used for prompt tuning or profile extraction), annotating a random sample of all top-5 ranks, not just exclusive papers. If the +Profile minus baseline relevance gap on this withheld future window remains positive with a bootstrap 95% CI excluding 0, the central claim survives; otherwise the 10.4% gain should be attributed to label leakage or overfitting rather than predictive personalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract; Table 2) is that adding verbalized profiles improves predictive alignment by 10.4 points (18.9% to 29.3% relevant) for 15 users. The load-bearing condition is that profiles are evaluated on days they did not see. Section 2.2.3 defines a chronological 70/30 split and says the validation set is used to improve the verbalization prompt, but Section 3.1.2 does not state whether the ablation rankings come from held-out days after profile construction, from the validation days used for prompt tuning, or from the same interaction logs that generated the profiles. If profiles are extracted from likes on the very days whose top-5 lists are annotated, the +Profile condition has direct access to the labels it is being scored against; the reported improvement then measures in-sample fitting, not predictive alignment. A compounding issue is that Section 3.1.2 annotates only papers appearing exclusively in one list, so percentages are computed on non-random subsets, and no confidence intervals or significance tests are reported. The manuscript's own template placeholders indicate the evaluation section is not final. The digestion evaluation does not bear on this claim, so the recommendation claim rests entirely on this unprotected comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Sci-Surf is an academic discovery system that combines daily arXiv paper recommendations with LLM-based reranking over verbalized user intent profiles, and a multimodal blog-style paper digestion pipeline. The system ingests new papers, precomputes structured digests, retrieves candidates by vector search, reranks them with an LLM that incorporates a verbalized user profile built from interaction feedback, and delivers a daily top-5 list. The paper reports an offline retrieval benchmark on LitSearch, a 15-user longitudinal evaluation of personalized reranking, an automated hallucination evaluation of 1,000 generated digests, and a small human agreement study for the hallucination detector. The central empirical claim is that adding verbalized profiles raises relevant recommendations from 18.9% to 29.3% (a 10.4 percentage point increase), described in the abstract as a '10.4% average improvement in predictive alignment.'","tokens_in":6939,"tokens_out":3232,"duration_ms":32699,"significance":"If the main evaluation claims hold, Sci-Surf would be a useful demonstration of feedback-driven intent personalization for academic discovery, with the notable strengths of a deployed daily pipeline, real user interaction logs, and public code and prompts. The digestion pipeline's structured JSON-based multimodal generation, with a reported 0.22 critical errors per digest, is also a promising direction. However, the headline recommendation result currently rests on an evaluation whose temporal split is unspecified, which annotates only a non-random subset of recommended papers, and which reports no uncertainty quantification; the digestion evaluation has no baseline comparison. These issues are load-bearing because the abstract's central quantitative claim depends on them. The contribution is defensible in principle, but the evidence as presented is not yet sufficient to support the claimed improvement.","major_comments":[{"comment":"The temporal split between profile construction and evaluation is unspecified. Section 2.2.3 defines a chronological 70/30 split into training and validation days, and states that the validation set is used to improve the verbalization prompt, but Section 3.1.2 does not state whether the top-5 lists annotated for the ablation are generated from days held out after profile construction, from the validation days used for prompt tuning, or from the same interaction logs that produced the profiles. If profiles are built from feedback on the very days whose recommendations are annotated, the +Profile condition has direct access to the labels it is scored against, and the 10.4 percentage point gain measures in-sample fitting rather than predictive alignment. The authors must specify and enforce a strict temporal split, and evaluate only on days after the profile was last updated.","section":"§3.1.2 / Table 2"},{"comment":"The ablation annotates only papers that appear exclusively in either the personalized or the baseline top-5 list. This is a non-random subset of the ranked output: a paper that appears in both lists is excluded, and the proportion of relevant papers in the exclusive subsets need not equal the proportion in the full lists. The reported Rel.% values are therefore not unbiased estimates of the relevance of the two conditions' top-5 outputs. The authors should annotate all top-5 papers from both lists (or a random sample), or otherwise correct for the selection, and should report per-user distributions, confidence intervals, and a significance test across the 15 users.","section":"§3.1.2 / Table 2"},{"comment":"The abstract's phrase '10.4% average improvement in predictive alignment' overstates the reported result. Table 2 shows an increase from 18.9% to 29.3% relevant recommendations, which is an absolute increase of 10.4 percentage points, or a relative increase of about 55%. The paper should report the change as percentage points and should accompany it with a confidence interval or significance test, given the small number of users.","section":"Abstract / §3.1.2 / Table 2"},{"comment":"The digestion evaluation does not support the abstract's claim of 'measurable improvements' in digestion quality. Section 3.2 reports absolute hallucination counts for the proposed pipeline and human agreement with the automated detector, but it compares against no baseline summary method (e.g., abstract-only, TLDR, or a text-only digest), so it cannot demonstrate that the proposed multimodal blog-style digests improve over existing alternatives. A baseline comparison is needed for the claimed improvement in digestion quality.","section":"§3.2 / Abstract"}],"minor_comments":[{"comment":"The manuscript contains multiple template placeholders that should be removed before submission: 'Do Not Use This Code' in the CCS Concepts line, 'Conference acronym ’XX', 'Woodstock, NY', the year 2018 in the ACM Reference Format, and the placeholder DOI.","section":"General / ACM template"},{"comment":"As noted in the major comments, the '10.4%' wording should be corrected to '10.4 percentage points' to avoid misreporting the magnitude of the improvement.","section":"Abstract"},{"comment":"The freshness window of 'past 3–5 days' is vague; if this is a system parameter, it should be stated precisely, and the similarity threshold should also be specified or referenced to the released code.","section":"§2.2.2"},{"comment":"The first paragraph of Section 3.2.1 says 'Given the full paper text and the corresponding digest, the model identifies unsupported or inconsistent statements,' but the next paragraph says the comparison is among Gemini and Qwen variants using 'HTML-parsed, JSON-formatted inputs.' Please clarify whether the hallucination detection prompt receives full text or the structured JSON representation, and whether the reported 0.22 rate is for the JSON-based pipeline.","section":"§3.2.1"},{"comment":"The human evaluation for digestion uses only three annotators and reports moderate agreement (Pearson 0.61, Cohen's κ = 0.45). This is acceptable as a sanity check of the detector, but the language 'moderate-to-strong consistency' should be softened, since κ = 0.45 is conventionally moderate agreement.","section":"§3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly a demo/system manuscript with unfinished template placeholders, and the evaluation section appears not to be final. The central recommendation claim is currently unprotected: the temporal split issue and the exclusive-only annotation bias could each invalidate the 10.4 percentage point result, and the digestion claim lacks a baseline. These are fixable with additional analysis and reporting, so major revision rather than rejection seems appropriate, provided the authors can supply the missing held-out evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nIf the headline is the 10.4% improvement, know this: the reported statistics don't support it as stated. The gain is 10.4 percentage points (18.9% to 29.3%) on a small 15-user study with no error bars, no significance test, and an annotation protocol that only labels papers present in exactly one of the two lists. That protocol can bias the comparison, and the abstract's phrasing makes it worse.\n\nWhat is genuinely new here is the integrated system: feedback-derived verbalized user profiles, represented as persona plus negative constraints and ranking heuristics, are reinserted into an LLM reranker over a daily arXiv feed, with precomputed multimodal blog-style digests. The individual pieces are known, but the endpoint is a reasonable system contribution. The offline retrieval benchmark on LitSearch is solid and picks GritLM as a default; that part is fine. The digestion pipeline, with placeholders replaced by figures and evaluated for hallucination by an LLM detector cross-checked against three human raters (kappa = 0.45), is a decent evaluation for a demo paper.\n\nThe soft spot is the central claim. Table 2's evaluation does not state whether the verbalized profiles were built only from interactions prior to the days whose top-5 lists were annotated. The paper defines a chronological 70/30 split for prompt tuning, but the ablation section is silent on how (or whether) that split is applied to the reranking comparison. If the profiles contain feedback from the evaluation days themselves, the +Profile condition has direct access to the labels, and the 10.4 point gain is memorization, not predictive alignment. That is a load-bearing ambiguity, not a minor quibble. Also, annotating only exclusive papers means the percentages are computed on non-random subsets; a paper present in both lists is never judged, so the relative improvement could be inflated by only considering easy-to-separate cases. No confidence intervals, no significance tests. For 15 users, the effect is plausible but unproven.\n\nThe digestion evaluation could also use a baseline summary method; without it, we only know the pipeline doesn't hallucinate egregiously, which is nice but not a performance number.\n\nWho is this for? Researchers building academic discovery systems, especially those wanting to combine LLM profiling with reranking. It deserves a serious referee, but the revisions need to be real: add error bars or a significance test, clarify the temporal split, annotate a random sample of the ranked output, and fix the abstract wording.","headline":"The paper is a solid systems demo, but the headline 10.4% improvement is not yet supported by the reported statistics.","tokens_in":7550,"tokens_out":2598,"would_cite":false,"duration_ms":23096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding LLM-verbalized user intent profiles to a reranking prompt raises relevant daily paper recommendations from 18.9% to 29.3% in a real-user study.","keywords":["scientific literature discovery","personalized recommendation","LLM user profiling","feedback-driven intent modeling","multimodal paper digestion","blog-style summaries","reranking","hallucination detection"],"falsifier":"Have independent evaluators annotate every paper in both the baseline and profile top-5 lists over a larger user cohort and compute the relevance difference with confidence intervals; if the gap disappears when all ranked items are annotated rather than only papers appearing exclusively in one list, the paper's central claim is not supported. Also check whether the month-long evaluation described in the abstract and the November 2025 to May 2026 log window described in the body refer to the same study.","tokens_in":6515,"feed_emoji":"🎯","tokens_out":8274,"duration_ms":75791,"temperature":0.7,"pith_summary":"This paper tries to establish that feeding an LLM-generated summary of a researcher's past likes and dislikes into a reranking prompt improves daily paper recommendations by 10.4 percentage points on average. The evidence is a real-user study with 15 users, where the share of recommendations judged relevant rises from 18.9% without the profile to 29.3% with it. The paper also claims its blog-style digests, which combine text, figures, and tables, stay faithful to the source papers, with an estimated 0.22 critical errors per digest. If these numbers hold, the system offers an inexpensive route to personalization: no model retraining, just a textual profile that keeps pace with shifting interests.","feed_headline":"Verbalized user profiles lift relevant paper picks from 18.9% to 29.3%","feed_subtitle":"Daily feed study: feedback-based intent profiles raise relevant recommendations by 10.4 percentage points.","key_machinery":"The load-bearing unit is the verbalized user intent profile $\\pi_u$, a structured text produced by an LLM verbalizer from the user's historical liked and disliked papers. It has three parts: a persona description, a set of negative constraints, and a set of ranking heuristics. The argument turns on a single controlled comparison: the reranker prompt with and without $\\pi_u$. Everything else in the pipeline, such as vector search over abstracts and precomputed multimodal digests, feeds candidates into that comparison, and the paper attributes the measured relevance gain to the profile's presence.","core_discovery":"The central discovery, on the paper's own terms, is that verbalizing a user's interaction history into a structured intent profile, consisting of a persona description, negative constraints, and ranking heuristics, and then appending that profile to the LLM reranker's prompt improves recommendation relevance from 18.9% to 29.3% over a baseline with vector retrieval and the same reranker but no profile. The very-relevant fraction rises from 0.5% to 3.6%. The same pipeline generates structured digests organized as TL;DR, motivation, contributions, method, figure/table analysis, results, and outlook; an LLM-based judge estimates the JSON-input pipeline produces 0.22 critical errors and 0.76 minor errors per digest, with moderate-to-strong agreement with human judges.","pith_inferences":["If the mechanism generalizes, the same verbalized-profile trick could be tested in other LLM-reranking settings, such as news, job postings, or clinical updates, where user feedback accumulates but the item pool changes daily; the paper does not make this claim.","A natural extension the paper does not run is to feed the profile into the digest generator as well, so that summaries emphasize the aspects a given user cares about rather than a fixed section order.","Most of the reported gain sits in the domain-relevant category rather than in highly relevant items, which suggests the profile may expand coverage of a broad area more than it produces spot-on hits; this is an editorial reading, not the paper's claim."],"forward_implications":["Daily literature feeds can become personalized by appending a feedback-built profile to an LLM ranking prompt, with no fine-tuning of the retriever or reranker.","Because profiles are updated incrementally from logged feedback, recommendations can track a researcher's shifting interests across months.","Precomputed blog-style digests with roughly 0.22 critical errors per digest are presented as usable as a default reading presentation for newly released papers.","The batch-based two-stage design means the personalization cost is paid once per daily batch, not per user query."],"supporting_citations":[{"why":"Supplies the base retriever used to build the candidate pool; its Recall@5 and Recall@20 drive the choice of retriever in the pipeline.","marker":"[9]"},{"why":"Supplies a second embedding-based retriever that the offline benchmark treats as a baseline; including it bounds the retrieval quality claim.","marker":"[2]"},{"why":"Supplies a standard document embedding baseline that the retrieval evaluation compares against; it motivates the switch to the chosen retriever.","marker":"[3]"}],"fun_headline_variants":["Verbalized profiles lift relevant paper picks to 29.3%","Profile prompts boost paper relevance from 18.9% to 29.3%","Verbalized user profiles improve relevant picks by 10.4 points","Sci-Surf: profile-fed reranking lifts relevant picks to 29.3%","Paper recommenders improve 10.4 points with verbalized profiles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim stands on the assumption that the 10.4 percentage-point gain observed across 15 users is caused by the verbalized profile rather than by how the annotated papers were chosen or by chance variation.","fun_headline_variants_meta":{"raw":{"variants":["Verbalized profiles lift relevant paper picks to 29.3%","Profile prompts boost paper relevance from 18.9% to 29.3%","Verbalized user profiles improve relevant picks by 10.4 points","Sci-Surf: profile-fed reranking lifts relevant picks to 29.3%","Paper recommenders improve 10.4 points with verbalized profiles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000676,"raw_usage":{"total_tokens":3029,"prompt_tokens":856,"completion_tokens":2173,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":2072}},"tokens_in":472,"tokens_out":2173,"duration_ms":16198,"temperature":1.0,"reasoning_tokens":2072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:20:14.933594+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent evaluators annotate every paper in both the baseline and profile top-5 lists over a larger user cohort and compute the relevance difference with confidence intervals; if the gap disappears when all ranked items are annotated rather than only papers appearing exclusively in one list, the paper's central claim is not supported. Also check whether the month-long evaluation described in the abstract and the November 2025 to May 2026 log window described in the body refer to the same study.","supporting_citations":[],"review_version":1}