{"id":"ceb0dabf-3b36-4b82-b9da-bde53bc135e6","arxiv_id":"2505.02418","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A human-in-the-loop RAG system that lets users select relevant document blocks and later uses their interaction logs to improve retrieval, with small user studies showing gains over baseline RAG.","lead":"This paper describes SymbioticRAG, a retrieval-augmented generation system that lets people manually pick relevant parts of documents and then uses those picks to improve future searches. It reports higher relevance and satisfaction in three small user studies than standard RAG baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline D improvement is not interpretable as better retrieval: SymbioticRAG's intention summary is built from the same user-selected blocks (H) that Eq. (1) uses as the gold standard.","rationale":"The paper's contribution includes a real system with a document processing pipeline, an interactive UI, and a useful human-on-the-loop validation component; those parts are described concretely and are not the subject of my objection. The load-bearing issue is confined to the quantitative evaluation that supports the headline claim. The reader's weakest_assumption is essentially correct: H is treated as an unbiased gold standard even though the retriever under test can influence H. I would sharpen this: in the SymbioticRAG condition the dependence is direct, because the retriever's input (the LLM intention summary) is generated from interaction logs that record H. Without a stated temporal split in the evaluation protocol, Table 1's D values for SymbioticRAG are consistent with the retriever returning blocks it was told the user liked, not with a generalizable improvement in retrieval. The small sample (3 evaluators × 5 sessions per strategy), absent significance testing, and the abstract's claim of ChatGPT/Claude comparisons that do not appear in Table 1 are additional weaknesses, but they are secondary. The proposed temporal-holdout rerun is feasible with the existing system and would settle the concern; if the gap persists under that protocol, the conditional acceptance should stand. Accordingly I do not move the verdict: the reader's CONDITIONAL seems right, with the condition made explicit as a temporal and exclusion holdout.","tokens_in":11473,"tokens_out":6230,"duration_ms":77884,"concrete_test":"Run the evaluation again with a strict temporal holdout and an independent gold standard. For each test query, (i) build the SymbioticRAG intention summary only from interaction logs recorded before that query, and explicitly exclude the current query's selected blocks H from the summary; (ii) obtain H either before the user sees any retrieval results or from a pre-annotated gold set created by the same evaluators without the UI; (iii) recompute D from Eq. (1) for the three strategies with the same evaluators and sessions, and report per-session distributions rather than aggregate means. If SymbioticRAG's D remains near 0.52–0.61 under this protocol while NaiveRAG and LabelNaiveRAG remain near 0.78–0.92, the concern is resolved. If D rises to the baseline range, the headline improvement is a leakage artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 describes the SymbioticRAG retriever as augmenting queries with an LLM summary of user interaction logs. Section 4 defines the main outcome D = 1 − |H∩R|/|H∪R| (Eq. 1), where H is the set of layout blocks selected by the evaluator and R is the retriever's top-k set. The central numerical claim (Table 1: D = 0.52–0.61 for SymbioticRAG vs. 0.78–0.92 for NaiveRAG) requires that R is produced without knowledge of the H it is scored against. The paper never specifies a temporal split that guarantees this. During an evaluation session, the interaction logs that feed the intention summary record exactly the behaviors that define H: clicking search results and selecting/deselecting blocks. If the summary can include selections from the same query or conversation being scored, R is partly constructed from the gold set, so the Jaccard overlap is inflated by construction. The paper itself acknowledges the leakage mechanism in Section 3.3 when it rejects direct log concatenation because retrieval 'will converge to specific contents which already exists inside the user logs'; an LLM summary reduces but does not eliminate this dependence. A second related confound is that H is collected after the user has viewed R in the UI, so even for the baselines the gold standard is not independent of the system being scored. Thus the reported gains do not yet establish that SymbioticRAG better captures user intent; they may only show that the retriever echoes selections that define the metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SymbioticRAG, a retrieval-augmented generation framework with a two-level design: Level 1 gives users an interactive interface to inspect source documents and manually select relevant layout blocks, and Level 2 is an experimental step that augments the query with an LLM-generated summary of the user's interaction logs to personalize retrieval. The system includes a document processing pipeline (layout detection, OCR, table/formula/figure extraction), an extensible retriever module, and a human-on-the-loop validation interface. The evaluation compares three retrieval strategies—NaiveRAG, LabelNaiveRAG, and SymbioticRAG—across literature review, geological exploration, and education scenarios, using a human-retriever distance D (Jaccard-based) and user satisfaction scores. The paper reports substantially lower D values and higher satisfaction for SymbioticRAG (D=0.52–0.61, S=3.67–4.13) than the baselines, and interprets this as evidence that interaction summaries better capture user intent. It also presents a case study illustrating a user's progression from 'unconscious incompetence' to 'conscious incompetence'.","tokens_in":11745,"tokens_out":4455,"duration_ms":57666,"significance":"If the reported effect were valid, the paper would make a useful contribution to human-centered RAG: the interface design (source-document verification, staging area, report generation) and the idea of leveraging interaction logs to personalize retrieval are timely and practically relevant. The document processing pipeline is described in concrete detail, and the human-on-the-loop validation interface is a sensible way to improve extraction quality. The paper also explicitly positions its Level 2 implementation as an experiment, which is honest. However, the central quantitative claim—that SymbioticRAG retrieval is closer to human-selected content than the baselines—is not supported by the evaluation as designed, due to the circular relationship between the retriever's inputs and the gold-standard labels. The significance of the paper therefore depends on whether the authors can provide a non-circular, statistically sound evaluation.","major_comments":[{"comment":"The main outcome metric D in Eq. (1) compares the set H of layout blocks selected by the evaluator with the retriever's top-k set R. In the SymbioticRAG condition, R is produced by augmenting the query with an LLM summary of the user's interaction logs, which include the same block selections that define H (Section 3.3). If the summary is generated from the same conversation being scored, R is partly constructed from H itself, and the reported D values (0.52–0.61) are inflated by construction. The paper's own statement that direct log concatenation was excluded because retrieval 'will converge to specific contents which already exists inside the user logs' acknowledges this dependence; an LLM summary reduces but does not eliminate it. The evaluation must impose a temporal split (for example, using only logs from earlier sessions) or compare against independently collected relevance judgments.","section":"Section 3.3 and Eq. (1)"},{"comment":"The gold-standard set H is not collected independently of the system being evaluated. Participants select layout blocks after the retriever has displayed its top-k results in the UI, so H can be influenced by what R shows; this acquiescence effect applies to all strategies but is especially problematic for SymbioticRAG because the same displayed-and-clicked blocks also enter the intention summary. The paper needs to separate the measurement of user relevance from the retrieval display, for instance by asking users to judge relevance in a blinded setting or by using post-hoc relevance assessments that do not depend on the manipulated system.","section":"Section 4, evaluation protocol"},{"comment":"The paper claims 'significant improvements' but reports no variance, confidence intervals, or statistical tests. With only three evaluators per scenario and five sessions per strategy, the aggregate D and S values in Table 1 could easily be driven by individual differences or session-level dependence. The authors should report per-session or per-evaluator results and apply appropriate paired tests or effect-size estimates with uncertainty; alternatively, the language should be softened to 'substantial differences' pending a larger study.","section":"Section 4, Table 1"}],"minor_comments":[{"comment":"The definition of D is underspecified: it is unclear whether H and R are computed per query, per conversation, or as an aggregate over all sessions in a scenario. This should be stated explicitly because it affects the interpretation of the numbers in Table 1.","section":"Section 4, Eq. (1)"},{"comment":"The evaluation does not state the value of k used in the actual user study. The text mentions 'currently k=5' in the system description, but the evaluation section should confirm that the same k was used for all three retrieval strategies and all sessions.","section":"Section 3.3"},{"comment":"'Significant improvements' is used without statistical support. Either add formal tests or replace 'significant' with 'large' or 'substantial' to avoid overclaiming.","section":"Abstract and Section 4"},{"comment":"The figure contains a typo ('Analysze') and the case study is anecdotal; it is fine as an illustrative example but should be labeled as such rather than as validation.","section":"Figure 7"},{"comment":"The sentence beginning 'We also tried to concatenate user logs directly with query...' is grammatically awkward and should be rewritten for clarity.","section":"Section 3.3"},{"comment":"There are minor capitalization inconsistencies for the baseline names (e.g., 'NaïveRAG' vs 'LabelNaïveRAG'), and the term 'LabelNaïveRAG' is sometimes written as 'Label NaiveRAG'. Please standardize.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The system design and the basic idea of learning from user interactions are promising, and the manuscript is clearly written. However, the central quantitative evaluation is circular as designed: the SymbioticRAG retriever uses the same user selections that define the gold-standard metric. I would require a redesigned study with temporal separation of the interaction logs from the evaluation set, an independent relevance gold standard, and proper uncertainty quantification. If such a study cannot be performed, the paper's main claim would not be supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what you should know: this is a real system paper, not a toy. They built a full document-processing pipeline that turns heterogeneous documents into layout-aware blocks, added a human-on-the-loop validation interface, and an interactive chat UI where users can accept or reject retrieved blocks. The novelty is modest—each component exists in prior work—but the integration is coherent and the three evaluation scenarios (literature review, geology, education) are sensible.\\n\\nThe central claim is that their SymbioticRAG retriever, which augments the query with an LLM summary of user interaction logs, achieves much lower human-retriever distance (0.52–0.61) than NaiveRAG (0.85–0.92). The soft spot is that this metric is partly circular. The interaction logs that feed the summary come from the same sessions whose selected blocks define the gold set H in Eq. (1). The paper never specifies a temporal split that prevents the retriever from seeing the answers it is scored against. The authors themselves note that concatenating raw logs causes retrieval to converge to content already in the logs—a leak they excluded. The LLM summary is a softer version of the same leak. That does not make the system useless, but it does make Table 1 uninterpretable as evidence of better retrieval.\\n\\nAlso soft: nine evaluators total, no variance or significance test, and the abstract claims a comparison with ChatGPT and Claude that never appears in the results. The satisfaction scores may still mean something, but they are not convincing on their own.\\n\\nWhat is good: the document processing pipeline is genuinely practical, the report-generation workflow is a nice extension, and the direction of learning from interaction logs is worth pursuing. The paper is clearly written, and the authors are transparent about the raw-log failure mode, which suggests they know where the difficulties lie.\\n\\nMy recommendation: this deserves a serious referee because the system is substantial and the problem is worth airing. But the evaluation needs to be redone with a proper temporal split or a separate gold standard before the headline claims can be believed. If the experimental section is fixed, the paper could be a solid systems contribution. It is probably most useful to people building RAG systems with human feedback and to the document-intelligence community.","headline":"This is a substantial system paper whose headline evaluation is contaminated: the retriever under test is built from the same user selections that define the gold standard, making Table 1 uninterpretable as a measure of retrieval quality.","tokens_in":12299,"tokens_out":2803,"would_cite":false,"duration_ms":32879,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SymbioticRAG: feeding user-interaction summaries into retrieval cuts the human-retriever distance roughly in half.","keywords":["retrieval-augmented generation","human-computer symbiosis","relevance judgment","interaction logs","document intelligence","layout-aware retrieval","user satisfaction","personalized retrieval"],"falsifier":"Run the same three scenarios in a counterbalanced design where half the evaluators mark relevant blocks before seeing any retrieval results and half mark them after; if the human-retriever distance advantage of SymbioticRAG shrinks or vanishes in the before condition, the improvement is driven by users anchoring on displayed blocks rather than by the intention summary better capturing true relevance.","tokens_in":11237,"feed_emoji":"🔎","tokens_out":5041,"duration_ms":51901,"temperature":0.7,"pith_summary":"The paper argues that retrieval-augmented generation should be a two-way loop: humans explore source documents and curate relevant passages, while the retriever learns from those interactions. It reports that appending an LLM-generated summary of a user's interaction history to the query before embedding reduces the human-retriever distance from the 0.85–0.92 range of a naive semantic retriever to 0.52–0.61 in three scenarios: literature review, geological exploration, and education. User satisfaction rises correspondingly, from 1.80–2.47 to 3.67–4.13 on a five-point scale. The claim matters because relevance is ultimately determined by people, and conventional RAG treats query formulation as a one-shot act, which fails when users do not yet know what they need.","feed_headline":"User clicks teach the retriever: retrieval mismatch nearly halves","feed_subtitle":"Adding an LLM summary of user interactions to the query brings retrieved blocks much closer to what people actually select.","key_machinery":"The load-bearing mechanism is the interaction-log intention summary: every click, selection, deselection, page navigation, and like or dislike is logged, an LLM condenses those logs into a natural-language statement of what the user appears to be after, and that statement is concatenated with the original query before the embedding step that drives semantic search. The measured quantity carrying the comparison is the human-retriever distance $D$, a set-similarity score between the layout blocks the user selected and the blocks the retriever returned. The document-processing pipeline supplies fine-grained, position-preserving layout blocks (tables, formulas, figures, text) so that human selection and retrieval operate on the same atomic units. The interface's staging area turns human curation into logged training signal, which is what makes the loop bidirectional in principle.","core_discovery":"The central discovery is that the retriever can be brought markedly closer to human judgment by augmenting the query with an LLM summary of the user's own interaction logs (which blocks were clicked, selected, deselected, and explored) before semantic search. The paper implements this as its Level 2 experiment and measures the gap between human-selected layout blocks $H$ and retriever-returned blocks $R$ by $D = 1 - |H \\cap R| / |H \\cup R|$. Across three scenarios, $D$ falls to 0.52–0.61 for the interaction-augmented retriever, versus 0.78–0.83 for label-diversified retrieval and 0.85–0.92 for naive semantic retrieval, with mean satisfaction rising to 3.67–4.13. The authors read this as evidence that user interaction summaries capture semantic intent better than the raw query alone, and as groundwork for a retrieval model that continuously adapts to an individual user.","pith_inferences":["A natural extension the paper does not run is an ablation varying how much interaction history is summarized; the claim predicts that $D$ should improve with interaction length up to a saturation point and degrade if the summary is randomized.","The $D$ metric likely conflates retrieval quality with interface influence, because users select from blocks the retriever already displayed, so part of the alignment for SymbioticRAG could be anchoring; a clean test would let users mark relevant blocks before any retrieval is shown and compare $D$ against the paper's protocol.","The intention-summary idea could be tested without the custom interface by replaying logged interactions from a standard RAG chat and appending summaries to queries, separating the effect of the summary from the effect of the interface.","If Level 2 is taken to its end point, the interaction logs could fine-tune the embedding model itself rather than being summarized into the prompt; the paper's results suggest the signal is rich enough that this is worth trying."],"forward_implications":["If the reported distances reflect true alignment, interaction-augmented retrieval could replace raw semantic search as the default for multi-turn, exploratory question answering over documents.","The same logs that improve retrieval can be accumulated as training data for a personalized retriever, moving from prompt-level summarization to model-level adaptation.","Document processing that preserves layout-block identity lets users verify and curate at the level of tables, figures, and formulas rather than whole pages, which is a precondition for the measured alignment.","The system's report-generation workflow shows a concrete downstream payoff: curated blocks become citable evidence in generated drafts, reducing the need to re-check sources after generation.","The pattern holds across literature review, geology, and education, suggesting the mechanism transfers across expertise levels and document types rather than being tuned to one domain."],"supporting_citations":[{"why":"Supplies the vision of human-computer symbiosis and early collaborative problem formulation that SymbioticRAG operationalizes.","marker":"(Licklider, 1960)"},{"why":"Establishes user-defined relevance criteria, the basis for treating human-selected blocks as ground truth.","marker":"(Barry, 1994)"},{"why":"Frames relevance as situational and human-centered, motivating the human-arbiter design and the distance metric.","marker":"(Saracevic, 2019)"},{"why":"Documents the persistent gap between retrieval output and human notions of relevance that SymbioticRAG targets.","marker":"(Craswell et al., 2022)"},{"why":"Represents the graph-based attempt to diversify relevance beyond similarity, the comparison point for why human signal is needed.","marker":"(Edge et al., 2024)"},{"why":"Provides the E5 embedding model used by all compared retrieval strategies, carrying the semantic search step.","marker":"(Wang et al., 2024b)"},{"why":"Supplies DocLayout-YOLO for layout detection, producing the atomic layout blocks that both retriever and human select.","marker":"(Zhao et al., 2024b)"},{"why":"Shows prior use of human feedback signals to refine RAG, supporting the Level 2 idea of learning from interactions.","marker":"(Bai et al., 2024)"}],"fun_headline_variants":["User clicks teach retriever: mismatch nearly halves","LLM summaries of user clicks sharpen retrieval","From clicks to query: retriever learns user intent","Interaction logs bridge human and machine retrieval","Clicks as context: retriever aligns with humans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the layout blocks each evaluator selected are an unbiased gold standard for what is truly relevant, uninfluenced by the blocks the retriever happened to display or suggest.","fun_headline_variants_meta":{"raw":{"variants":["User clicks teach retriever: mismatch nearly halves","LLM summaries of user clicks sharpen retrieval","From clicks to query: retriever learns user intent","Interaction logs bridge human and machine retrieval","Clicks as context: retriever aligns with humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000348,"raw_usage":{"total_tokens":1947,"prompt_tokens":1030,"completion_tokens":917,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":844}},"tokens_in":646,"tokens_out":917,"duration_ms":10173,"temperature":1.0,"reasoning_tokens":844,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:51:55.109421+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three scenarios in a counterbalanced design where half the evaluators mark relevant blocks before seeing any retrieval results and half mark them after; if the human-retriever distance advantage of SymbioticRAG shrinks or vanishes in the before condition, the improvement is driven by users anchoring on displayed blocks rather than by the intention summary better capturing true relevance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Frames relevance as situational and human-centered, motivating the human-arbiter design and the distance metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the persistent gap between retrieval output and human notions of relevance that SymbioticRAG targets."}],"review_version":1}