{"id":"2978666c-fc36-4795-baaf-d52d6fee1e11","arxiv_id":"2501.02838","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of user feedback techniques for improving generative information retrieval systems, covering alignment, continual learning, conversational learning, and prompt learning.","lead":"This chapter surveys how generative information retrieval systems can be improved by using user feedback, including feedback types, alignment methods, and learning strategies. It is a reference overview for researchers working on GenIR systems that adapt to user behavior.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RLCF necessity claim in §2.2.1 is the weakest load-bearing link: the survey asserts, rather than demonstrates, that pointwise-input RLHF/RLAIF cannot teach fine-grained discrimination, and its recommended groupwise direction rests on this claim.","rationale":"The paper is a survey chapter, so its value lies in accurate organization and well-grounded pointers to future work. I read the abstract's broad claim about innovative feedback-driven techniques in that light: the claim would hold if the taxonomy is sound and the highlighted directions are genuinely supported. The most load-bearing weakness is the near-impossibility claim about pointwise-input alignment in §2.2.1. It is used to elevate RLCF from one proposal to a necessary IR-specific direction, but the evidence cited is the authors' own work plus one related paper, without neutral replication. Moreover, the claim is stronger than the experiments warrant: a pointwise-input method can receive groupwise information through its reward labels, so the input-paradigm dichotomy does not imply that uniqueness cannot be learned pointwise. This is a substantive correctness risk for the survey's recommendation, but it is localized and fixable, and it does not undermine the rest of the chapter's descriptive content. The reader's CONDITIONAL verdict is therefore appropriate; my stress-test does not move it, so I recommend UNCHANGED. The concrete test I propose would settle whether the concern lands by comparing RLCF against a pointwise-input baseline with groupwise labels under identical conditions. If the authors soften the claim to 'not yet demonstrated' or supply such a comparison, the concern would be resolved.","tokens_in":17985,"tokens_out":6025,"duration_ms":108106,"concrete_test":"Re-run the RLCF evaluation (same base LLM, same corpus, same discriminative metrics such as output distinctiveness and downstream retrieval/ranking quality) with an added baseline: pointwise-input alignment where the reward model is trained on pairwise/groupwise labels of summary uniqueness generated offline for each document, without feeding multiple documents as inputs during training. If this baseline matches or exceeds RLCF, the claim that pointwise input cannot teach fine-grained discrimination is refuted and the §2.2.1 motivation collapses. If it clearly underperforms, the concern is resolved in the authors' favor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper is a survey, so its central claim about the evolution of feedback-driven GenIR depends on the accuracy of its synthesis and the soundness of its highlighted directions. The weakest link is in §2.2.1, where the authors motivate RLCF by stating that pointwise-input RLHF/RLAIF methods make it 'difficult, if not impossible, to teach LLMs to generate outputs unique to an input.' This is presented as a settled limitation and used to conclude that groupwise input/output alignment is necessary for fine-grained discrimination in IR. The cited support is primarily the authors' own SIGIR 2024 paper (ref [41]) plus ref [42]; no independent replication or neutral comparison against a pointwise-input baseline is reported. The inference is not logically forced: a pointwise-input reward model trained on labels that encode groupwise comparisons, such as pairs of unique versus generic summaries labeled according to offline similarity to other documents, could in principle teach uniqueness, because the groupwise information would be carried by the labels rather than by the reward function's input. The chapter does not consider this alternative, so the 'impossible' wording overstates what the evidence shows. Since this claim is the main justification for positioning RLCF as a distinct and necessary IR-specific alignment technique, the survey's otherwise useful taxonomy inherits an unverified assertion. The flaw is correctable, but as written it weakens the reliability of Section 2's central recommendation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This chapter surveys approaches for improving generative information retrieval (GenIR) systems using user feedback. It begins by broadening the notion of 'user' to include humans, LLM agents, and clients, and distinguishes implicit and explicit feedback. Four general strategies are outlined: prompt engineering, fine-tuning, preference/intent integration, and agent-based feedback. Section 2 examines alignment, first identifying objectives shared with general LLMs (preventing harm, user intent, ethics) and objectives specific to information access (personalization, fine-grained discrimination). The authors propose a taxonomy of reward collection methods based on pointwise vs. groupwise input and pointwise vs. groupwise training, and they place RLHF, RLAIF, and RLCF within this taxonomy. Optimization methods, including PPO, DPO, and ranking-based losses, are summarized. Section 3 reviews continual learning, conversational search, and prompt learning, and Section 4 lists challenges such as user intention understanding, limited but rich feedback, user-centric evaluation, and privacy. The central thesis is that innovative feedback-driven techniques, particularly groupwise contrastive feedback alignment (RLCF), are advancing GenIR beyond traditional feedback usage.","tokens_in":18302,"tokens_out":7959,"duration_ms":74734,"significance":"The chapter offers a useful organizational scheme for alignment methods in IR by adapting learning-to-rank terminology, and it correctly identifies fine-grained discrimination as an underappreciated requirement in generative retrieval. The extension of 'user' to agents and clients is timely and relevant to current GenIR practice. However, the chapter's most consequential recommendation—that pointwise-input RLHF/RLAIF cannot teach fine-grained discrimination and therefore groupwise RLCF is necessary—is not established by the evidence cited and is presented too confidently. As a survey, the chapter need not contain new experiments, but it should accurately represent the strength of the evidence. The footnote in Section 2.1.2 acknowledging that the categorization is 'not inclusive' is welcome, but the main text does not always maintain that caution. If the overclaim is softened and the taxonomy is clarified, the survey would be a solid reference for researchers and practitioners.","major_comments":[{"comment":"The claim that RLHF/RLAIF's pointwise-input paradigm makes it 'difficult, if not impossible' to teach LLMs to generate outputs unique to an input is load-bearing for the chapter's recommendation of RLCF, but it is supported mainly by the authors' own SIGIR 2024 paper (ref [41]) and one related work (ref [42]), with no independent replication or neutral comparison. The 'impossible' wording is an overstatement: a pointwise-input reward model trained on labels that encode groupwise comparisons could in principle learn to reward uniqueness, because groupwise information would be carried by the labels rather than by the reward function's input. Please soften the claim to 'current pointwise-input methods typically fail to...' and discuss this alternative, or provide a formal argument why label-based pointwise approaches cannot succeed.","section":"§2.2.1"},{"comment":"The taxonomy's two dimensions (input and training) are not applied consistently enough for readers to place methods unambiguously. RLHF and RLAIF are described as pointwise input and potentially pairwise/groupwise in training, but Table 1's cell entries are not clearly separated by column, and RLCF is presented as the only groupwise input/output method. Section 2.2.2's ranking-based methods (RRHF, RAFT) also operate on groups of responses, so the chapter should explicitly clarify whether 'groupwise' refers to the set of input documents or the set of output responses for a single prompt. Without this distinction, the positioning of RLCF as the IR-specific groupwise alignment technique is confusing and may overstate its novelty.","section":"§2.2.1 and Table 1"}],"minor_comments":[{"comment":"The manuscript contains numerous typos that should be corrected in revision, including 'feedback feedback' (Section 1), 'forus' (Section 2), 'Micorsoft' and 'raciest' (Section 2), 'visal' (Section 1.2), and 'functiosn' (Section 2.2.1).","section":"General"},{"comment":"The statement 'In the first technical report of ChatGPT [18]' cites the GPT-4 Technical Report, not the ChatGPT technical report; please correct the citation to the appropriate ChatGPT/InstructGPT paper or rephrase the sentence.","section":"Section 2, reference [18]"},{"comment":"The entries in Table 1 are not visually separated by column, making it difficult to determine which references correspond to pointwise input, pairwise/groupwise input, pointwise training, or pairwise/groupwise training; reformat the table with clear column and row labels.","section":"Table 1"},{"comment":"Figure 3, as rendered in the text, has overlapping labels such as 'Pointwise Optimization' and 'Pairwise/Groupwise Training' with unclear arrows; redrawing the figure with separate panels for the input and training dimensions would improve readability.","section":"Figure 3"},{"comment":"The sentence 'The massive knowledge about conversation patterns and the world of LLMs also makes it a promising end-to-end foundation to be an end-to-end foundation model for personalized conversational search systems' repeats 'end-to-end' and should be revised.","section":"Section 3.2"},{"comment":"The future-direction bullet on self-feedback loops is intriguing but underdeveloped; since the chapter extends the notion of user to agents and clients, one or two sentences on how to detect or mitigate feedback loops would strengthen the discussion.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The chapter leans heavily on the authors' own RLCF paper (ref [41]) for the central technical claim. This is not circular, but the lack of independent evidence increases the risk that the survey overstates the method's necessity. I recommend that the revision explicitly acknowledge this reliance and broaden the evidence base, perhaps by citing independent evaluations or by adding a clearly labeled caveat. The manuscript also has many typos and citation inaccuracies (e.g., reference [18]), which suggests the chapter needs a careful editorial pass. The scope fits an IR/GenIR venue; the taxonomy, once clarified and hedged appropriately, would be a useful contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on 2501.02838. It's a solid survey of how user feedback feeds into GenIR systems, with the most useful bit being the pointwise/groupwise taxonomy for reward collection and training in §2.2.1. That framing, borrowed from learning-to-rank, gives a clean way to compare RLHF, RLAIF, RLCF, and the optimization methods. The chapter also covers the broadened notion of 'user' (human, agent, client), feedback types, and learning scenarios (continual, conversational, prompt learning) in a compact way. As a survey, it's readable and mostly accurate.\n\nThe soft spot is the load-bearing claim about pointwise-input methods. The text says it's 'difficult, if not impossible' for pointwise RLHF/RLAIF to teach an LLM to produce outputs unique to an input, and uses that to position RLCF as a necessary IR-specific alignment method. The support is essentially the authors' own SIGIR 2024 paper plus one related work. No independent replication or neutral comparison is reported. More importantly, the inference isn't forced: a pointwise reward model trained on labels that encode groupwise information (e.g., pairwise judgments about uniqueness) could in principle carry the discriminative signal through the labels, not the input grouping. The chapter doesn't consider this alternative, so the 'impossible' wording overstates the evidence. Since the RLCF recommendation rests on this claim, the survey's central argument inherits the overstatement. This is correctable in revision: soften the claim, acknowledge the label-carries-information alternative, and flag the lack of independent evidence.\n\nOther issues are minor: the chapter leans on the authors' own prior work in several places (not a problem per se, but worth noting), and there are scattered typos that a copyedit would fix.\n\nBottom line: this is a useful organizational survey that deserves peer review, but the RLCF motivation should be reframed from 'impossible' to 'not yet demonstrated with pointwise inputs.' I'd recommend conditional acceptance after those revisions. For a reading group, it's worth a skim if people are working in this area, but I wouldn't build a position on the RLCF claim without checking the underlying paper.","headline":"A useful survey of feedback-driven GenIR improvement, but the RLCF necessity claim is asserted on the authors' own evidence and needs tempering.","tokens_in":18729,"tokens_out":1964,"would_cite":false,"duration_ms":18773,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey chapter argues that improving generative information retrieval with feedback now hinges on alignment methods that see groups of inputs, because pointwise reward collection cannot make models produce outputs unique to each…","keywords":["generative information retrieval","user feedback","LLM alignment","reinforcement learning from human feedback","reinforcement learning from AI feedback","contrastive feedback alignment","learning to rank","conversational search"],"falsifier":"Run the same base LLM through three alignments on identical reward data: pointwise RLHF, pointwise RLAIF, and groupwise-input RLCF, then test on a corpus of near-duplicate documents where the task is to generate a distinct snippet or query for each. If pointwise-aligned models achieve the same level of output distinctness and retrieval utility as the groupwise model, the chapter's central claim is wrong; if groupwise wins consistently across several base models and corpora, the claim is supported.","tokens_in":17826,"feed_emoji":"🔍","tokens_out":8811,"duration_ms":77008,"temperature":0.7,"pith_summary":"This chapter maps how generative information retrieval (GenIR) systems can be improved from user feedback, and it makes a specific argument: in the GenIR era, feedback goes beyond clicks and ratings from humans, and the most consequential innovations lie in alignment methods that treat user preferences as training signals. It distinguishes two bodies of techniques—aligning model behavior with user factors, and learning from user feedback through continual learning, conversational ranking, and prompting. The chapter's sharpest claim is that standard pointwise-input alignment methods such as RLHF and RLAIF cannot teach an LLM to produce outputs that are unique to a given input, which is exactly what information access needs; methods that compare outputs across inputs, such as RLCF, can. If correct, this means the future of search and recommendation depends on groupwise alignment and on treating agents and client systems as legitimate users.","feed_headline":"Align on groups of inputs to make generative search discriminate","feed_subtitle":"This is why pointwise rewards fail for search: outputs must be unique to each input.","key_machinery":"The organizing device is a two-axis taxonomy borrowed from learning to rank: reward input (pointwise vs groupwise) and reward training (pointwise vs groupwise). On this grid, RLHF and RLAIF are pointwise-input methods, while RLCF (Reinforcement Learning from Contrastive Feedback) is the named mechanism that is groupwise in both input and training. RLCF lets the LLM generate outputs for several similar inputs at once and builds rewards by contrasting outputs across inputs, which is the mechanism claimed to teach fine-grained discrimination among near-duplicate documents.","core_discovery":"On the paper's own terms, the discovery is a re-framing: user feedback for GenIR systems must be understood as coming from humans, LLM agents, and other systems, and the techniques that exploit it divide into alignment (reward collection and parameter optimization) and learning (continual learning, conversational ranking, prompt learning). The survey proposes that the pointwise/groupwise distinction from learning to rank explains why off-the-shelf alignment fails for information access: RLHF and RLAIF score outputs one prompt at a time, so they cannot teach a model to distinguish among highly similar documents, whereas groupwise input/output contrastive feedback (RLCF) can. The chapter concludes that such innovative techniques, beyond traditional feedback use, are driving GenIR evolution and lists open problems around intent understanding, limited but rich feedback, user-centric evaluation, and privacy.","pith_inferences":["One step beyond the paper: if groupwise input is what teaches fine-grained discrimination, then RLCF should also improve multi-document summarization, recommendation explanation, or any generation task where outputs for similar inputs must differ; no such experiment is reported in the chapter.","The chapter's widening of 'user' to agents and clients suggests a failure mode the authors list as open: self-feedback loops in which an agent's generated content becomes training data for the same system, amplifying artificial intentions; one could simulate repeated agent-system-agent rounds and measure representational drift.","Borrowing the learning-to-rank analogy further, groupwise alignment should reduce optimization variance as well as improve discriminative quality; this is a quantitative prediction that could be checked by comparing reward-model variance across pointwise and groupwise training runs.","A neutral benchmark for 'output uniqueness'—for example, pairwise distinctness of generated responses to near-identical inputs—would let the field test the central claim without relying on the authors' own task setups."],"forward_implications":["Alignment for information access should move from pointwise reward input to groupwise input/output paradigms, since fine-grained discrimination among similar items is a core IA need.","Models that generate summaries, snippets, or rewrites for similar documents can be expected to produce more distinct, useful outputs after RLCF-style alignment than after standard RLHF or RLAIF.","User feedback from LLM agents and client systems should be treated as a first-class signal in GenIR, requiring methods that handle bi-directional and multi-party interactions.","The learning-to-rank toolbox (pairwise/listwise losses and groupwise scoring) becomes directly relevant to LLM alignment, so ranking-theoretic insights can guide future alignment methods.","Conversational search systems can leverage LLM-generated query rewrites and session data to improve intent understanding, with continual learning as the mechanism for keeping generative systems current despite catastrophic forgetting."],"supporting_citations":[{"why":"Supplies the RLCF method and the similar-documents example showing off-the-shelf LLMs generate identical summaries; it is the empirical basis for the groupwise-input alignment claim.","marker":"[41]"},{"why":"Cited as evidence that naively adopting RLHF/RLAIF-style alignment does not improve information-access task performance, motivating the need for groupwise contrastive alignment.","marker":"[42]"},{"why":"Defines Reinforcement Learning from Human Feedback, the pointwise-input baseline the chapter categorizes and contrasts with groupwise methods.","marker":"[19, 20]"},{"why":"Defines RLAIF, the AI-feedback pointwise-input method the chapter classifies and compares against groupwise contrastive alignment.","marker":"[44]"},{"why":"RLCD, the contrast-distillation variant of RLAIF, is cited as a pointwise-input example of AI-feedback reward generation.","marker":"[45]"},{"why":"Supplies the learning-to-rank background whose pointwise/pairwise/groupwise terminology the chapter borrows to classify alignment reward collection.","marker":"[46]"},{"why":"Introduces groupwise multivariate scoring functions, the ranking concept the chapter extends to groupwise input/output reward paradigms.","marker":"[47]"},{"why":"DPO is the preference-based optimization method that the chapter contrasts with PPO and connects to pairwise/listwise ranking methods.","marker":"[52]"},{"why":"RRHF is the listwise version of DPO that the chapter uses to argue for ranking-based optimization in model alignment.","marker":"[53]"},{"why":"RAFT ranks multiple response candidates and fine-tunes the model on the highest-reward samples, supporting the chapter's case for listwise optimization.","marker":"[54]"}],"fun_headline_variants":["Groupwise feedback beats pointwise for generative search","Why pointwise rewards fail for generative information access","Align GenIR on groups, not single inputs, for better search","User feedback for GenIR: groupwise contrastive learning advances"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that what the authors see in their own studies—that rewarding one output at a time cannot teach a model to produce output unique to a given input, and that comparing several inputs side by side fixes this—holds for GenIR systems generally.","fun_headline_variants_meta":{"raw":{"variants":["Groupwise feedback beats pointwise for generative search","Why pointwise rewards fail for generative information access","Align GenIR on groups, not single inputs, for better search","User feedback for GenIR: groupwise contrastive learning advances"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000556,"raw_usage":{"total_tokens":2588,"prompt_tokens":829,"completion_tokens":1759,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":1693}},"tokens_in":445,"tokens_out":1759,"duration_ms":14073,"temperature":1.0,"reasoning_tokens":1693,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:00:56.701235+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same base LLM through three alignments on identical reward data: pointwise RLHF, pointwise RLAIF, and groupwise-input RLCF, then test on a corpus of near-duplicate documents where the task is to generate a distinct snippet or query for each. If pointwise-aligned models achieve the same level of output distinctness and retrieval utility as the groupwise model, the chapter's central claim is wrong; if groupwise wins consistently across several base models and corpora, the claim is supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited as evidence that naively adopting RLHF/RLAIF-style alignment does not improve information-access task performance, motivating the need for groupwise contrastive alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines RLAIF, the AI-feedback pointwise-input method the chapter classifies and compares against groupwise contrastive alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RLCD, the contrast-distillation variant of RLAIF, is cited as a pointwise-input example of AI-feedback reward generation."},{"cited_title":"Foundations and Trends® in Information Retrieval 3(3), 225–331 (2009) https://doi.org/10.1561/ 1500000016","cited_arxiv_id":null,"evidence_quote":"Supplies the learning-to-rank background whose pointwise/pairwise/groupwise terminology the chapter borrows to classify alignment reward collection."},{"cited_title":"In: Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S","cited_arxiv_id":null,"evidence_quote":"DPO is the preference-based optimization method that the chapter contrasts with PPO and connects to pairwise/listwise ranking methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RRHF is the listwise version of DPO that the chapter uses to argue for ranking-based optimization in model alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RAFT ranks multiple response candidates and fine-tunes the model on the highest-reward samples, supporting the chapter's case for listwise optimization."}],"review_version":1}