{"id":"dab69e65-e9a9-461d-8f89-59e17ff65303","arxiv_id":"2505.14680","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"NExT-Search is a proposed paradigm to collect process-level user feedback in generative AI search through active user debugging and a simulated 'shadow user' agent.","lead":"This paper proposes NExT-Search, a design for giving AI search engines fine-grained user feedback at every step, from breaking down the query to writing the final answer. It argues that restoring this feedback loop could let AI search improve itself the way traditional Web search did, and it describes modes for both active debugging and AI-simulated feedback from passive users.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central flywheel rests on unvalidated premise that users will debug and that simulated feedback matches real preferences; both are explicitly left to future work.","rationale":"The reader's weakest assumption is precisely that users will provide useful process-level feedback and that Shadow User Mode will faithfully simulate user preferences. My analysis identifies the same load-bearing point and traces its consequences through Section 4.2: user edits are treated as positive/negative supervision, and simulated feedback is treated as training signal. If either source is biased, noisy, or too sparse, the proposed data flywheel does not turn. The paper is transparent about these limitations and explicitly leaves empirical validation to future work (Section 6), so a CONDITIONAL verdict is appropriate rather than REJECT. I found no internal inconsistency in the proposed architecture; the concern is about external validity of the core behavioral and statistical assumptions. The concrete test would settle the concern with modest resources, and the paper itself calls for such user studies. I agree with the reader's verdict and do not see grounds to change it.","tokens_in":14999,"tokens_out":2273,"duration_ms":23366,"concrete_test":"Build a small prototype of NExT-Search for a realistic complex task (e.g., conference travel planning) and recruit 50–100 users. For each session, log whether users perform any debug action, have expert raters judge the quality of each user edit, and compare Shadow User Mode's suggested edits against the user's actual edits in a held-out set. Pre-register thresholds: if fewer than 30% of users debug in more than half of their sessions, if more than 30% of user edits are rated neutral or harmful by experts, or if Shadow User Mode suggestions diverge from actual user edits on more than 30% of items, the flywheel premise fails. This directly tests the load-bearing assumptions in Sections 3.2, 3.3, and 4.2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core claim is that NExT-Search restores a feedback-driven improvement loop for generative AI search. The loop in Section 4 depends on two unvalidated assumptions: (1) enough users will voluntarily perform effortful, stage-level debugging in User Debug Mode, and (2) Shadow User Mode's AI-generated feedback faithfully represents real user preferences. Section 4.2 treats user edits as gold labels, e.g., original sub-queries as negative examples and revised sub-queries as positive examples, but user edits can be idiosyncratic or mistaken, and treating them as ground truth can inject noise or bias into training. Shadow User Mode (Section 3.3) generates simulated feedback from learned user profiles; with sparse interaction data, such simulation may hallucinate preferences or amplify existing model biases, producing feedback that drifts from actual user intent. The paper explicitly acknowledges in Section 5 that Shadow User Mode supplies 'abundant but potentially noisy signals' and that building faithful user simulators is a critical open challenge, but it provides no proof-of-concept, user study, or simulation to show the loop can be closed. Unlike the low-cost implicit signals (clicks, dwell time) that powered traditional web search, the feedback in NExT-Search is explicit, cognitively expensive, and likely subject to strong selection bias. The Feedback Store incentive may also attract gaming rather than high-quality debugging. Since the entire paradigm hinges on the quantity, quality, and faithfulness of these feedback signals, the central claim remains unsupported without empirical validation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper diagnoses a 'feedback loop disconnect' in generative AI search: unlike traditional Web search, where document-level implicit feedback (clicks, dwell time) directly improves ranking models, generative search pipelines (query decomposition, retrieval, answer generation) receive only coarse final-answer feedback. To close this gap, the authors propose NExT-Search, a vision for a feedback-rich ecosystem built on two modes: User Debug Mode, where users intervene at each pipeline stage, and Shadow User Mode, where a personalized LLM agent simulates feedback for less interactive users. The paper further sketches online adaptation, offline model updates, and a Feedback Store incentive mechanism. It is explicitly a perspective piece, with no system, experiments, or data; Section 6 states that empirical validation is left to future work.","tokens_in":15361,"tokens_out":3974,"duration_ms":37618,"significance":"Conditional on its assumptions, the paper identifies a real and timely problem: current generative search engines lack stage-level supervision, which impedes continuous improvement. The proposed paradigm is conceptually coherent and more concrete than most vision papers, offering specific interaction examples, training strategies, and a monetization mechanism. The paper also honestly acknowledges its key open challenges, especially the fidelity of user simulation and the noise in user-provided feedback. As a position paper, it is a useful agenda-setting contribution, though its central effectiveness claims remain unvalidated.","major_comments":[{"comment":"The offline update rationale treats user edits as gold labels: 'treating the latter as positive examples and the former as negative examples' for query decomposition, and similarly for retrieval and generation. This assumes that user corrections are always correct and the original system outputs are always wrong. In practice, user edits can be mistaken, idiosyncratic, or adversarial, and aggregating such signals without quality control could inject systematic bias into the learned models. The paper should discuss mechanisms to detect and downweight unreliable feedback, for example through cross-user agreement, edit verification, or noise-robust training objectives.","section":"Section 4.2"},{"comment":"The entire Shadow User Mode relies on a personalized user agent that 'faithfully reflects user preferences,' yet the paper provides only prompt templates and a list of future research topics. Given that this is the load-bearing assumption for scaling feedback collection, the paper should offer a concrete validation strategy, such as agreement rates between agent-simulated feedback and actual user corrections, or a protocol for A/B testing the agent's impact on downstream model improvement. Without such a proposal, the claim that this mode 'provides AI-assisted feedback for less interactive users' remains an untested assertion.","section":"Section 3.3 and Section 5"},{"comment":"The Feedback Store incentivizes users to 'receive direct financial compensation when others adopt their solutions.' This introduces a potential gaming problem: users may generate low-quality or generic templates to maximize downloads, or may misrepresent their debugging processes to increase perceived value. The paper does not address how the platform would ensure feedback quality, prevent spam, or handle disputes. At a minimum, the discussion should acknowledge these incentive risks and outline possible safeguards (e.g., reputation systems, quality verification, or platform-mediated review).","section":"Section 3.5"}],"minor_comments":[{"comment":"There are several spacing and typographical errors: 'afeedback loop disconnect' in the abstract should be 'a feedback loop disconnect'; 'anactiveUser Debug Mode' in Section 1 should be 'an active User Debug Mode'; 'online adaption' in Section 3.2 should be 'online adaptation' for consistency.","section":"Abstract and Section 1"},{"comment":"The comparison bullet 'Potential for End-to-End Solutions' is presented as a contrast, but the 'risk of hallucinated content' is not unique to generative search; traditional search snippets can also mislead. Consider framing this more carefully or citing evidence on hallucination rates in generative search.","section":"Section 2.3"},{"comment":"Figure 3 is very small and not referenced in the main text beyond the caption. The two mechanisms (online adaptation and offline update) are described in Section 4, but the figure adds little; either expand it to show the feedback flow, or consider removing it.","section":"Figure 3"},{"comment":"The paper does not discuss existing partial feedback mechanisms in commercial generative search engines, such as Perplexity's 'related searches' or 'focus' features, or Bing Copilot's follow-up prompts. A short paragraph on how NExT-Search differs from these would strengthen the novelty claim.","section":"Related Work"},{"comment":"The sentence 'Due to the lack of publicly available datasets, we leave empirical validation and system implementation to future work' is a reasonable limitation, but the lack of datasets alone does not prevent a small-scale user study or simulation-based demonstration. The paper would benefit from acknowledging this choice more explicitly.","section":"Section 6"}],"recommendation":"minor_revision","confidential_remarks":"This is a well-written perspective paper that fits a SIGIR audience, but it is largely an unfalsifiable proposal. The authors are honest about the lack of validation, and the major assumptions are explicitly framed as open challenges. I recommend minor revision rather than rejection because the core diagnosis is convincing and the proposed ecosystem is a useful research agenda. The authors should address the three major comments I raised to strengthen the technical credibility of their framework."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a perspective piece, and it is a good one. It earns its place by naming the feedback loop disconnect clearly and proposing a concrete architecture—User Debug Mode plus Shadow User Mode—that is more specific than most \"future of search\" essays. The examples are useful and the writing is honest about what is missing.\n\nWhat is actually new is the integration, not the ingredients. User simulation, RLHF, RAG pipelines, and click flywheels all appear in the cited literature. The synthesis into a single ecosystem with online adaptation, offline update, and a feedback store is new enough, and it is coherent. The authors also deserve credit for putting the main risks in Section 5: they explicitly say Shadow User Mode produces \"abundant but potentially noisy signals\" and that faithful user simulation is open work. Section 6 says empirical validation is left to future work because of missing datasets. That is the right framing for a position paper.\n\nThe soft spot is exactly the one the paper itself flags, and the stress-test note does not overstate it. The whole loop depends on users actually doing stage-level debugging and on simulated feedback being faithful to real preferences. Those are empirical claims. The offline update section treats user edits as gold labels—original sub-queries as negatives, revised ones as positives—but user edits can be idiosyncratic, and without a study we have no idea whether the noise drowns the signal. The Feedback Store also reads more like a wish than a design; incentive gaming is a real concern. These are not fatal flaws for a paper that explicitly says it is envisioning a paradigm, but they do mean the paper's claim about \"rebuilding\" the ecosystem is a proposal, not a demonstrated result.\n\nWho this is for: IR researchers and industry teams thinking about how to close the feedback loop in generative search. I would not require experiments for acceptance if the venue treats this as a perspective track, but a serious referee should press the authors to be clearer about which assumptions are testable and what evidence would change their minds.\n\nRecommendation: send it to peer review. It is a legitimate contribution that is transparent about its own limits.","headline":"A clear, honest vision paper that diagnoses a real problem and proposes a sensible two-mode feedback architecture, but every central effectiveness claim is explicitly untested.","tokens_in":15805,"tokens_out":1598,"would_cite":true,"duration_ms":14210,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Restore the feedback loop AI search lost","keywords":["generative AI search","user feedback","feedback loop disconnect","query decomposition","user debug mode","shadow user mode","online adaptation","offline update"],"falsifier":"A controlled deployment would settle it: give one group of users debug panels at each search stage and another group only final-answer ratings, then compare per-stage correction quality and downstream model gains. If correction rates are negligible, or if offline updates trained on debug logs do not improve retrieval or generation accuracy beyond the baseline, the central claim fails.","tokens_in":14736,"feed_emoji":"🔍","tokens_out":3592,"duration_ms":33803,"temperature":0.7,"pith_summary":"The paper argues that generative AI search has broken the feedback-driven improvement loop that made traditional Web search successful, because users can usually only rate the final answer rather than the intermediate steps. It proposes NExT-Search, a paradigm that reintroduces fine-grained, stage-level feedback through two modes: an active User Debug Mode and a passive Shadow User Mode. If the paradigm works, search systems could again learn continuously from user interactions, refining query decomposition, retrieval, and answer generation from real corrections. The paper is a vision paper: it lays out the architecture and the learning loops, and leaves empirical validation and system implementation to future work.","feed_headline":"Restore the feedback loop AI search lost","feed_subtitle":"A two-mode design lets users debug each search stage, so AI answers can improve from real corrections.","key_machinery":"The central machinery is a stage-level feedback schema attached to the standard retrieval-augmented generation pipeline: query decomposition, retrieval and ranking, and answer generation. Each stage has a defined set of debug actions—adding, removing, or reordering sub-queries; annotating relevance; re-ranking documents; applying filters; correcting facts; editing content; adjusting style—so that user corrections are attributable to a specific component. Two feedback channels produce these signals: User Debug Mode supplies explicit high-fidelity corrections, and Shadow User Mode supplies AI-generated pseudo-feedback from a learned user profile. The signals then drive online adaptation and offline fine-tuning of each stage's models.","core_discovery":"The central claim is that the 'feedback loop disconnect' is the key obstacle to generative AI search's long-term improvement, and that the disconnect can be repaired by collecting feedback at each pipeline stage instead of only at the final answer. NExT-Search does this with User Debug Mode, where engaged users add or remove sub-queries, re-rank retrieved documents, and correct or restyle generated answers; and Shadow User Mode, where a personalized agent simulates user preferences and proposes edits for confirmation. The collected signals feed two loops: online adaptation, which re-runs downstream stages in the current session, and offline update, which turns corrected sub-queries, relevance annotations, and answer edits into training data for each module. The paper maintains that this restores the data flywheel that powered traditional search.","pith_inferences":["A natural extension is to treat expert users' debug traces as a reusable curriculum for the search pipeline, so that later users inherit validated decompositions and answer edits without redoing the work.","The Shadow User agent's fidelity is the crux of the whole scheme: if simulated feedback systematically reflects the model's own biases rather than real user preferences, the offline update loop could reinforce blind spots instead of correcting them.","One testable extension would be to compare system improvement rates under three conditions—explicit debug only, shadow mode only, and mixed—using edit-acceptance rates as a proxy for feedback quality.","The stage-level feedback schema could transfer to other retrieval-augmented systems, such as customer-support chatbots or code assistants, wherever pipeline-stage attribution is hard."],"forward_implications":["If users debug query decomposition, the corrections become direct training pairs for the decomposition model, so the system learns which sub-queries users actually need.","Relevance annotations and re-ranking actions provide document-level supervision, recovering the kind of fine-grained feedback that powered traditional ranking models.","Factual corrections and style edits in the generated answer can be used to fine-tune the generator, for instance through reinforcement learning from human feedback.","The combined online and offline loops let a single session be repaired in real time while long-term logs accumulate for periodic model updates.","The proposed feedback store could turn debugging templates into a marketplace, giving users a tangible incentive to keep contributing."],"supporting_citations":[{"why":"Shows that user behavior signals can improve web search ranking, grounding the claim that fine-grained feedback drives traditional search improvement.","marker":"[1]"},{"why":"Establishes clickthrough data as a training signal for ranking models, the canonical example of the feedback loop.","marker":"[20]"},{"why":"Explains how to interpret clickthrough data as implicit feedback, supporting the use of behavioral signals as supervision.","marker":"[21]"},{"why":"Surveys implicit feedback types in search, providing the taxonomy of signals that generative search is said to lack.","marker":"[23]"},{"why":"Defines the retrieval-augmented generation pipeline that the paper's query decomposition, retrieval, and generation stages are based on.","marker":"[12]"},{"why":"Characterizes the advance of AI agents in search and the coarse feedback available to generative systems, motivating the disconnect.","marker":"[41]"},{"why":"Surveys generative search and recommendation in the LLM era, grounding the description of the modern search pipeline.","marker":"[26]"},{"why":"Describes the data flywheel in search engines, the feedback-driven improvement cycle the paper wants to restore.","marker":"[24]"}],"fun_headline_variants":["Feedback fix for AI search: debug every stage","AI search gets a feedback loop again, stage by stage","User feedback at every step, not just the final answer","NExT-Search: rebuild the AI search feedback flywheel","Debug mode and shadow users revamp AI search feedback"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole improvement loop depends on users actually giving meaningful stage-level corrections, and on the personalized agent's simulated feedback faithfully matching real user preferences; if either fails, the collected signals are too sparse or too noisy to train on.","fun_headline_variants_meta":{"raw":{"variants":["Feedback fix for AI search: debug every stage","AI search gets a feedback loop again, stage by stage","User feedback at every step, not just the final answer","NExT-Search: rebuild the AI search feedback flywheel","Debug mode and shadow users revamp AI search feedback"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1273,"prompt_tokens":998,"completion_tokens":275,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":195}},"tokens_in":614,"tokens_out":275,"duration_ms":2621,"temperature":1.0,"reasoning_tokens":195,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:29:06.362815+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled deployment would settle it: give one group of users debug panels at each search stage and another group only final-answer ratings, then compare per-stage correction quality and downstream model gains. If correction rates are negligible, or if offline updates trained on debug logs do not improve retrieval or generation accuracy beyond the baseline, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes clickthrough data as a training signal for ranking models, the canonical example of the feedback loop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Surveys implicit feedback types in search, providing the taxonomy of signals that generative search is said to lack."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Characterizes the advance of AI agents in search and the coarse feedback available to generative systems, motivating the disconnect."},{"cited_title":"2011.An introduction to search engines and web navigation","cited_arxiv_id":null,"evidence_quote":"Describes the data flywheel in search engines, the feedback-driven improvement cycle the paper wants to restore."}],"review_version":1}