Sci-Surf: Navigating Scientific Literature Discovery through Human Feedback and Intelligent Summarization

Fang Guo; Hui Chen; Qi Zhu; Rongcan Pei; Shuqi He; Yue Zhang

REVIEW 4 major objections 5 minor 15 references

Sci-Surf: Navigating Scientific Literature Discovery through Human Feedback and Intelligent Summarization

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding LLM-verbalized user intent profiles to a reranking prompt raises relevant daily paper recommendations from 18.9% to 29.3% in a real-user study.

desk verdict The paper is a solid systems demo, but the headline 10.4% improvement is not yet supported by the reported statistics. read the letter →

arxiv 2608.11973 v2 pith:Y2BIHKHJ submitted 2026-08-12 cs.IR

Fang Guo , Qi Zhu , Rongcan Pei , Shuqi He , Hui Chen , Yue Zhang This is my paper · ORCID

classification cs.IR

keywords scientificliteraturediscoverypersonalizedrecommendationLLMuserprofilingfeedback-drivenintentmodelingmultimodalpaperdigestionblog-stylesummariesrerankinghallucinationdetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that feeding an LLM-generated summary of a researcher's past likes and dislikes into a reranking prompt improves daily paper recommendations by 10.4 percentage points on average. The evidence is a real-user study with 15 users, where the share of recommendations judged relevant rises from 18.9% without the profile to 29.3% with it. The paper also claims its blog-style digests, which combine text, figures, and tables, stay faithful to the source papers, with an estimated 0.22 critical errors per digest. If these numbers hold, the system offers an inexpensive route to personalization: no model retraining, just a textual profile that keeps pace with shifting interests.

What carries the argument

The load-bearing unit is the verbalized user intent profile $\pi_u$, a structured text produced by an LLM verbalizer from the user's historical liked and disliked papers. It has three parts: a persona description, a set of negative constraints, and a set of ranking heuristics. The argument turns on a single controlled comparison: the reranker prompt with and without $\pi_u$. Everything else in the pipeline, such as vector search over abstracts and precomputed multimodal digests, feeds candidates into that comparison, and the paper attributes the measured relevance gain to the profile's presence.

What would settle it

Have independent evaluators annotate every paper in both the baseline and profile top-5 lists over a larger user cohort and compute the relevance difference with confidence intervals; if the gap disappears when all ranked items are annotated rather than only papers appearing exclusively in one list, the paper's central claim is not supported. Also check whether the month-long evaluation described in the abstract and the November 2025 to May 2026 log window described in the body refer to the same study.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that verbalizing a user's interaction history into a structured intent profile, consisting of a persona description, negative constraints, and ranking heuristics, and then appending that profile to the LLM reranker's prompt improves recommendation relevance from 18.9% to 29.3% over a baseline with vector retrieval and the same reranker but no profile. The very-relevant fraction rises from 0.5% to 3.6%. The same pipeline generates structured digests organized as TL;DR, motivation, contributions, method, figure/table analysis, results, and outlook; an LLM-based judge estimates the JSON-input pipeline produces 0.22 critical errors and 0.76 minor errors per digest, with moderate-to-strong agreement with human judges.

Load-bearing premise

The central claim stands on the assumption that the 10.4 percentage-point gain observed across 15 users is caused by the verbalized profile rather than by how the annotated papers were chosen or by chance variation.

Editorial extensions

If this is right

Daily literature feeds can become personalized by appending a feedback-built profile to an LLM ranking prompt, with no fine-tuning of the retriever or reranker.
Because profiles are updated incrementally from logged feedback, recommendations can track a researcher's shifting interests across months.
Precomputed blog-style digests with roughly 0.22 critical errors per digest are presented as usable as a default reading presentation for newly released papers.
The batch-based two-stage design means the personalization cost is paid once per daily batch, not per user query.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the mechanism generalizes, the same verbalized-profile trick could be tested in other LLM-reranking settings, such as news, job postings, or clinical updates, where user feedback accumulates but the item pool changes daily; the paper does not make this claim.
A natural extension the paper does not run is to feed the profile into the digest generator as well, so that summaries emphasize the aspects a given user cares about rather than a fixed section order.
Most of the reported gain sits in the domain-relevant category rather than in highly relevant items, which suggests the profile may expand coverage of a broad area more than it produces spot-on hits; this is an editorial reading, not the paper's claim.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

If the headline is the 10.4% improvement, know this: the reported statistics don't support it as stated. The gain is 10.4 percentage points (18.9% to 29.3%) on a small 15-user study with no error bars, no significance test, and an annotation protocol that only labels papers present in exactly one of the two lists. That protocol can bias the comparison, and the abstract's phrasing makes it worse.

What is genuinely new here is the integrated system: feedback-derived verbalized user profiles, represented as persona plus negative constraints and ranking heuristics, are reinserted into an LLM reranker over a daily arXiv feed, with precomputed multimodal blog-style digests. The individual pieces are known, but the endpoint is a reasonable system contribution. The offline retrieval benchmark on LitSearch is solid and picks GritLM as a default; that part is fine. The digestion pipeline, with placeholders replaced by figures and evaluated for hallucination by an LLM detector cross-checked against three human raters (kappa = 0.45), is a decent evaluation for a demo paper.

The soft spot is the central claim. Table 2's evaluation does not state whether the verbalized profiles were built only from interactions prior to the days whose top-5 lists were annotated. The paper defines a chronological 70/30 split for prompt tuning, but the ablation section is silent on how (or whether) that split is applied to the reranking comparison. If the profiles contain feedback from the evaluation days themselves, the +Profile condition has direct access to the labels, and the 10.4 point gain is memorization, not predictive alignment. That is a load-bearing ambiguity, not a minor quibble. Also, annotating only exclusive papers means the percentages are computed on non-random subsets; a paper present in both lists is never judged, so the relative improvement could be inflated by only considering easy-to-separate cases. No confidence intervals, no significance tests. For 15 users, the effect is plausible but unproven.

The digestion evaluation could also use a baseline summary method; without it, we only know the pipeline doesn't hallucinate egregiously, which is nice but not a performance number.

Who is this for? Researchers building academic discovery systems, especially those wanting to combine LLM profiling with reranking. It deserves a serious referee, but the revisions need to be real: add error bars or a significance test, clarify the temporal split, annotate a random sample of the ranked output, and fix the abstract wording.

Referee Report

4 major / 5 minor

Summary. Sci-Surf is an academic discovery system that combines daily arXiv paper recommendations with LLM-based reranking over verbalized user intent profiles, and a multimodal blog-style paper digestion pipeline. The system ingests new papers, precomputes structured digests, retrieves candidates by vector search, reranks them with an LLM that incorporates a verbalized user profile built from interaction feedback, and delivers a daily top-5 list. The paper reports an offline retrieval benchmark on LitSearch, a 15-user longitudinal evaluation of personalized reranking, an automated hallucination evaluation of 1,000 generated digests, and a small human agreement study for the hallucination detector. The central empirical claim is that adding verbalized profiles raises relevant recommendations from 18.9% to 29.3% (a 10.4 percentage point increase), described in the abstract as a '10.4% average improvement in predictive alignment.'

Significance. If the main evaluation claims hold, Sci-Surf would be a useful demonstration of feedback-driven intent personalization for academic discovery, with the notable strengths of a deployed daily pipeline, real user interaction logs, and public code and prompts. The digestion pipeline's structured JSON-based multimodal generation, with a reported 0.22 critical errors per digest, is also a promising direction. However, the headline recommendation result currently rests on an evaluation whose temporal split is unspecified, which annotates only a non-random subset of recommended papers, and which reports no uncertainty quantification; the digestion evaluation has no baseline comparison. These issues are load-bearing because the abstract's central quantitative claim depends on them. The contribution is defensible in principle, but the evidence as presented is not yet sufficient to support the claimed improvement.

major comments (4)

[§3.1.2 / Table 2] The temporal split between profile construction and evaluation is unspecified. Section 2.2.3 defines a chronological 70/30 split into training and validation days, and states that the validation set is used to improve the verbalization prompt, but Section 3.1.2 does not state whether the top-5 lists annotated for the ablation are generated from days held out after profile construction, from the validation days used for prompt tuning, or from the same interaction logs that produced the profiles. If profiles are built from feedback on the very days whose recommendations are annotated, the +Profile condition has direct access to the labels it is scored against, and the 10.4 percentage point gain measures in-sample fitting rather than predictive alignment. The authors must specify and enforce a strict temporal split, and evaluate only on days after the profile was last updated.
[§3.1.2 / Table 2] The ablation annotates only papers that appear exclusively in either the personalized or the baseline top-5 list. This is a non-random subset of the ranked output: a paper that appears in both lists is excluded, and the proportion of relevant papers in the exclusive subsets need not equal the proportion in the full lists. The reported Rel.% values are therefore not unbiased estimates of the relevance of the two conditions' top-5 outputs. The authors should annotate all top-5 papers from both lists (or a random sample), or otherwise correct for the selection, and should report per-user distributions, confidence intervals, and a significance test across the 15 users.
[Abstract / §3.1.2 / Table 2] The abstract's phrase '10.4% average improvement in predictive alignment' overstates the reported result. Table 2 shows an increase from 18.9% to 29.3% relevant recommendations, which is an absolute increase of 10.4 percentage points, or a relative increase of about 55%. The paper should report the change as percentage points and should accompany it with a confidence interval or significance test, given the small number of users.
[§3.2 / Abstract] The digestion evaluation does not support the abstract's claim of 'measurable improvements' in digestion quality. Section 3.2 reports absolute hallucination counts for the proposed pipeline and human agreement with the automated detector, but it compares against no baseline summary method (e.g., abstract-only, TLDR, or a text-only digest), so it cannot demonstrate that the proposed multimodal blog-style digests improve over existing alternatives. A baseline comparison is needed for the claimed improvement in digestion quality.

minor comments (5)

[General / ACM template] The manuscript contains multiple template placeholders that should be removed before submission: 'Do Not Use This Code' in the CCS Concepts line, 'Conference acronym ’XX', 'Woodstock, NY', the year 2018 in the ACM Reference Format, and the placeholder DOI.
[Abstract] As noted in the major comments, the '10.4%' wording should be corrected to '10.4 percentage points' to avoid misreporting the magnitude of the improvement.
[§2.2.2] The freshness window of 'past 3–5 days' is vague; if this is a system parameter, it should be stated precisely, and the similarity threshold should also be specified or referenced to the released code.
[§3.2.1] The first paragraph of Section 3.2.1 says 'Given the full paper text and the corresponding digest, the model identifies unsupported or inconsistent statements,' but the next paragraph says the comparison is among Gemini and Qwen variants using 'HTML-parsed, JSON-formatted inputs.' Please clarify whether the hallucination detection prompt receives full text or the structured JSON representation, and whether the reported 0.22 rate is for the JSON-based pipeline.
[§3.2.2] The human evaluation for digestion uses only three annotators and reports moderate agreement (Pearson 0.61, Cohen's κ = 0.45). This is acceptable as a sanity check of the detector, but the language 'moderate-to-strong consistency' should be softened, since κ = 0.45 is conventionally moderate agreement.

Circularity Check

0 steps flagged · score 2.0 of 10

No construction-level circularity; the 10.4% improvement is an empirical ablation, not a definitional identity, with only a minor non-load-bearing self-citation.

full rationale

The paper's derivation chain is empirical rather than formal. The verbalized profile is produced by an LLM function pi_u = V(T, PDFs) (Eq. 1) from labeled user feedback, and the reranking comparison in Table 2 measures annotated relevance proportions; there is no equation in which the predicted relevance is identical to the profile-extraction input by construction. The retrieval component is anchored to an external benchmark (LitSearch) and an external retriever (GritLM [9]); digestion is evaluated against full paper content with an LLM detector plus a human-alignment check (Pearson 0.61, Cohen's kappa 0.45). The only self-citation, [5] (SciMine), appears in a related-work enumeration of human-in-the-loop screening and is not load-bearing. The main caveat is that Section 3.1.2 does not restate the chronological 70/30 split defined in Section 2.2.3; if the annotated top-5 lists came from the same days used to extract profiles, the reported gain would be partly in-sample rather than predictive. That ambiguity is an evaluation-validity concern, not a definitional circularity, because the paper does define a held-out split for profile construction and the evaluated quantity is not algebraically forced by the profile. Template placeholders in the submitted text reinforce the incompleteness of the write-up but do not encode a circular step.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claims rest on software behavior and LLM outputs rather than on mathematical derivation. The only numeric choices that affect the evaluation are operational parameters (k=5, freshness, similarity threshold, split ratio), none of which are varied in sensitivity analyses. The key load-bearing assumptions are about the reliability of LLM profile extraction, of human annotations, and of the LLM hallucination detector. No new physical or formal entities are introduced.

free parameters (4)

daily recommendation size k = 5
Set by hand for the demo (Section 2.2.2); determines what users see and what logs are collected for the evaluation.
freshness window = 3-5 days
Chosen heuristic in the vector search stage (Section 2.2.2) to prioritize recent papers; value not tuned or reported with sensitivity.
similarity threshold = not specified
Used to remove weak semantic matches in vector search (Section 2.2.2); no value or sensitivity analysis is given.
train/validation split ratio = 70/30
Chosen chronological split for profile verbalization (Section 2.2.3); the paper does not report robustness to this split.

assumptions (6)

domain assumption LLM verbalizer and reranker outputs faithfully encode user preferences and paper relevance.
Section 2.2.3 uses an LLM to extract persona, negative constraints, and ranking heuristics from feedback and instantiates them in the reranking prompt; the paper relies on this without formal or mechanistic verification.
domain assumption Manual relevance annotations in the ablation are accurate and unbiased.
Section 3.1.2 depends on human annotators labeling papers as highly relevant, domain-relevant, or irrelevant; no inter-annotator agreement is reported for these labels.
domain assumption The LLM-based hallucination detector is a valid proxy for factual faithfulness.
Section 3.2.2 reports only moderate agreement (Pearson 0.61, Cohen's kappa 0.45) with three human users, then uses the detector to conclude digests maintain satisfactory faithfulness.
domain assumption The arXiv Computer Science daily batch is an adequate and representative corpus for the recommendation evaluation.
Section 2.2.1 collects 300-500 daily CS submissions; the paper does not analyze coverage bias relative to user interests.
domain assumption Feedback logs from the 15 users are representative of real-world researcher preferences over time.
Section 3.1.2 relies on interaction logs from 15 active users from November 2025 to May 2026; no demographic or field coverage details are given.
domain assumption The chronological 70/30 split prevents feedback from validation days leaking into profile construction.
Section 2.2.3 defines the split but Section 3.1.2 does not state that the final ablation profiles were built only from training days; if validation feedback was included, the evaluation is optimistic.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Sci-Surf: Navigating Scientific Literature Discovery through Human Feedback and Intelligent Summarization." pith.science (2026). https://pith.science/paper/Y2BIHKHJ

@misc{pith2026260811973,
  author       = {Pith},
  title        = {Pith review of: Sci-Surf: Navigating Scientific Literature Discovery through Human Feedback and Intelligent Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y2BIHKHJ}},
  note         = {Machine review of arXiv:2608.11973}
}

read the original abstract

The rapid growth of scientific publications makes it increasingly difficult for researchers to identify relevant new studies and effectively comprehend them. Existing academic discovery platforms typically rely on static topic subscriptions or embedding-based similarity and provide only abstracts or short summaries, offering limited support for nuanced intent modeling and in-depth paper summarization. We present Sci-Surf, an intent-centric knowledge discovery system that integrates feedback-driven personalized recommendation with multi-modal blog-style paper digestion. Our approach refines user intent representations through LLM-based user profiling, while generating structured summaries that synthesize textual and visual information from full papers. The demo presents an end-to-end academic discovery pipeline and demonstrates measurable improvements in both recommendation quality and digestion quality through real-user evaluations. Specifically, the integration of verbalized profiles led to a 10.4% average improvement in predictive alignment with real-world user preferences throughout a month-long online evaluation.

Figures

Figures reproduced from arXiv: 2608.11973 by the authors.

**Figure 1.** Overview of Sci-Surf. User intent is initialized during onboarding and continuously refined from interaction feedback, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages

[1]

Fengyu Cai, Xinran Zhao, Tong Chen, Sihao Chen, Hongming Zhang, Iryna Gurevych, and Heinz Koeppl. 2024. MixGR: Enhancing retriever generalization for scientific domain through complementary granularity. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 10369– 10391

work page 2024
[2]

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu
[3]

Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld
[4]

Pádraig Cunningham and Barry Smyth. 2025. An analysis of the impact of gold open access publications in computer science.Commun. ACM68, 8 (2025), 62–69

work page 2025
[5]

Fang Guo, Yun Luo, Linyi Yang, and Yue Zhang. 2023. Scimine: An efficient sys- tematic prioritization model based on richer semantic information. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 205–215

work page 2023
[6]

Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. 2025. Pasa: An llm agent for comprehensive academic paper search. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 11663–11679

2025
[7]

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 5303–5315

2023
[8]

Priyanka Mandikal and Raymond Mooney. 2024. Sparse meets dense: A hybrid approach to enhance scientific document retrieval.arXiv preprint arXiv:2401.04055 (2024)

arXiv 2024

Show all 15 references

[9]

Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Aman- preet Singh, and Douwe Kiela. 2024. Generative Representational Instruction Tuning.ArXivabs/2402.09906 (2024). https://api.semanticscholar.org/CorpusID: 267681873

2024 arXiv
[10]

Xiaofeng Shi, Yuduo Li, Qian Kou, Longbin Yu, Jinxin Xie, and Hua Zhou. 2025. Spar: Scholar paper retrieval with llm-based agents for enhanced academic search. arXiv preprint arXiv:2507.15245(2025)

2025 arXiv
[11]

Rens Van De Schoot, Jonathan De Bruin, Raoul Schram, Parisa Zahedi, Jan De Boer, Felix Weijdema, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, et al. 2021. An open source machine learning framework for efficient and transparent systematic reviews.Natu...

2021
[12]

Yunyi Zhang, Ruozhen Yang, Siqi Jiao, SeongKu Kang, and Jiawei Han. 2025. Scientific paper retrieval with llm-guided semantic-based ranking.arXiv preprint arXiv:2505.21815(2025)

2025
[13]

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. 2025. Large language models for information retrieval: A survey.ACM Transactions on Information Systems44, 1 (2025), 1–54. Received 20 Februar...

2025
[2020]

https://api.semanticscholar

SPECTER: Document-level Representation Learning using Citation- informed Transformers.ArXivabs/2004.07180 (2020). https://api.semanticscholar. org/CorpusID:215768677

2020 arXiv
[2024]

InAnnual Meeting of the Association for Computational Linguistics

BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. InAnnual Meeting of the Association for Computational Linguistics. https://api.semanticscholar.org/ CorpusID:267413218

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Fengyu Cai, Xinran Zhao, Tong Chen, Sihao Chen, Hongming Zhang, Iryna Gurevych, and Heinz Koeppl. 2024. MixGR: Enhancing retriever generalization for scientific domain through complementary granularity. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 10369– 10391

work page 2024

[2] [2]

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu

[3] [3]

Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld

[4] [4]

Pádraig Cunningham and Barry Smyth. 2025. An analysis of the impact of gold open access publications in computer science.Commun. ACM68, 8 (2025), 62–69

work page 2025

[5] [5]

Fang Guo, Yun Luo, Linyi Yang, and Yue Zhang. 2023. Scimine: An efficient sys- tematic prioritization model based on richer semantic information. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 205–215

work page 2023

[6] [6]

Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. 2025. Pasa: An llm agent for comprehensive academic paper search. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 11663–11679

2025

[7] [7]

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 5303–5315

2023

[8] [8]

Priyanka Mandikal and Raymond Mooney. 2024. Sparse meets dense: A hybrid approach to enhance scientific document retrieval.arXiv preprint arXiv:2401.04055 (2024)

arXiv 2024