{"id":"206df0f6-ac6d-4beb-992e-ca3803b56230","arxiv_id":"2501.02635","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"User-selected context plus a short partial intent lets language models generate or retrieve the full information need; partial intent mitigates the noise of larger contexts in adapted Inquisitive and MS MARCO settings.","lead":"Researchers tested whether a search system can guess a user's full question if the user highlights a word, sentence, or paragraph and optionally adds a hint like 'why' or 'applications'. They found that such hints help offset the distraction of larger highlighted text, making the approach promising for search interfaces that reduce typing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ2's 'intent helps' result may be an artifact of deriving partial intents from the target question itself; an independent-intent control is required before the central claim can be accepted.","rationale":"The paper's core empirical contribution is RQ2: adding a partial intent improves prediction and can compensate for larger contexts. The proposed interaction strategy is 'promising' only if that improvement transfers to real users. Section 4 builds the intent field by having an LLM extract it from the very target question, so the intent is not independent of the label. The model is therefore asked to predict a question from a clue that was generated from that same question. This is a genuine leakage path, not just a mismatch with consensus; it makes the direction of the result predictable and its magnitude suspect. The concern is testable and fixable, so it does not warrant rejection, but it does make the current evidence conditional. I agree with the reader's assessment: the same load-bearing assumption is at the center of the conditional verdict. The paper deserves credit for proposing the task, using standard models and metrics, and running both generation and retrieval experiments; those parts are not in question. The missing independent control is what separates a promising framework from an established one.","tokens_in":14665,"tokens_out":3488,"duration_ms":37819,"concrete_test":"Generate new independent intents for the MS MARCO 10k subset by prompting Yi-34B-Chat with only the simulated Source paragraph, instructing it to predict what aspect a reader might want to know, with no access to the target query or the original reformulated intent. Then retrain and evaluate the encoder-decoder and cross-encoder models in the Source vs Source+IndependentIntent and Context vs Context+IndependentIntent conditions, using the same hyperparameters and significance tests. If the intent benefit over Source remains non-significant or shrinks materially compared to Table 4, the 'intent mitigates distraction' claim is not supported for realistic intents. For the generation setting, repeat on a random 500-sample subset of MS MARCO comparing original extracted intents against independent intents and generic intents such as 'how' or 'why' to quantify the leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 constructs the partial-search-intent fields by LLM extraction from the target question: Llama-3-8b-Instruct extracts intents from Inquisitive questions, and Yi-34B-Chat reformulates MS MARCO queries into context and intent. The experiments then treat these extracted intents as user-supplied partial intents, which makes RQ2 partly circular. The intent is a distillation of the very question the model is asked to predict; the qualitative examples show intents like 'hatch time' containing the question's content words. In a real interaction, a user would specify intent before the full question is formulated, so real intents would be noisier and less aligned with the target question. The observed benefit of adding intent, which is the central evidence for the claim that 'minimal specified partial search intent can help mitigate these distractions,' could therefore be inflated by target-question leakage. The paper acknowledges the dataset is a silver standard and notes that the Inquisitive intents were not manually validated, but it never tests the transfer from extracted intents to realistic user intents. Without an independent control intent, the RQ2 result does not establish the paper's main practical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an interactive information need prediction task in which a user selects a pre-search context (ranging from a word to a paragraph) and optionally specifies a partial search intent (e.g., \"how\", \"why\", \"applications\"). The authors adapt two datasets, Inquisitive and MS MARCO, by generating context and intent fields with LLMs, and evaluate both question generation (Flan-T5, Llama-3-8b, Yi-34B) and passage retrieval (bi-encoder and cross-encoder BERT) under four input conditions: Context+Intent, Source+Intent, Context, and Source. Their main findings are that larger pre-search contexts degrade prediction performance (RQ1) and that adding a partial intent improves performance, mitigating the negative effect of large contexts (RQ2). The paper concludes that the proposed framework is promising for real-world applications.","tokens_in":14888,"tokens_out":4084,"duration_ms":39667,"significance":"If the RQ2 result is valid, the paper identifies a genuinely novel and practical interaction paradigm: allowing users to highlight context and type a few intent words could let systems predict full questions or retrieve answers with minimal effort. The experimental coverage is broad, including both generation and retrieval models, and the paper is transparent about dataset construction details and hyperparameters. However, the central claim rests on a dataset construction that is partly circular: the \"intent\" field is automatically extracted from the very target question the models are asked to predict. This makes the intent benefit in RQ2 a measure of how much of a question can be recovered from a lossy summary of itself, rather than a measure of how a real user's hint would help. The promise to release code and datasets is a strength, though the actual link is absent from the manuscript.","major_comments":[{"comment":"The partial intent is not user-supplied; it is extracted from the target question using Llama-3-8b-Instruct for the Inquisitive dataset and Yi-34B-Chat for MS MARCO. The RQ2 comparisons (Context+Intent vs. Context, Source+Intent vs. Source) therefore quantify how much of the full question can be reconstructed from an automatically generated summary of that same question, rather than how a genuine user-specified hint would help. The qualitative example in Table 5 makes this concrete: the intent \"hatch time\" contains the content words of the target question \"when do robin eggs hatch\". The paper's acknowledgment that the dataset is a \"silver standard\" and that Inquisitive intents were not manually validated does not address this leakage. An independent control intent, such as a generic intent (e.g., \"explain\", \"what\", \"why\") or an intent obtained from a different annotation source (e.g., human judges), is required before the claim that \"minimal specified partial search intent can help mitigate these distractions\" is supported.","section":"Section 4 (Building the Datasets) and Section 6.2 (RQ2)"},{"comment":"The MS MARCO \"Source\" is simulated as the top BM25 paragraph retrieved for the query, not an actual pre-search context observed from user behavior. This synthetic source may not reflect the properties of real pre-search contexts (e.g., noise, topic drift, user attention). Since both RQ1 (context size) and RQ2 (intent benefit) are evaluated on this simulated source, the external validity of the conclusions is limited. The paper does state in Section 7 that user studies are future work, but the abstract and conclusion go further by claiming the framework is \"suitable for real-world applications\"; that claim is not supported by the current evidence.","section":"Section 4 (Building the Datasets) and Section 7 (Discussion)"}],"minor_comments":[{"comment":"Statistical significance is assessed with Scipy's independent t-test, but the comparisons are performed on the same test samples across input conditions, so a paired test (e.g., paired t-test or Wilcoxon signed-rank test) would be more appropriate.","section":"Section 5 (Experimental Setup)"},{"comment":"The paper states that the code and datasets will be linked in the final version; the current manuscript contains a placeholder. Providing a public link would improve reproducibility.","section":"Section 4 (Building the Datasets)"},{"comment":"There are several typos and minor wording issues, such as \"singe word\" in the abstract, \"potions\" in Section 1, and \"and and\" in Section 5.1.","section":"Throughout"},{"comment":"The RQ1 conclusion that larger context hurts performance is based on comparing Source vs. Context, but the Source and Context are not independent: both are derived from the same BM25 results or sentence spans. It would be useful to report inter-annotator agreement or a manual check of the constructed Context fields.","section":"Section 6 (Results)"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim, as highlighted by the abstract, is the benefit of partial intent. The dataset construction flaw (intent extracted from the target question) is a fundamental issue that must be addressed with a control experiment. The paper's own limitations paragraph acknowledges the need for user studies, but the conclusions are worded more strongly. A major revision that adds an independent-intent control and tempers the real-world claims would be appropriate. The promise to release code and datasets is positive, but the current absence of a link is a practical concern for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the task framing is genuinely new: letting users pick the granularity of pre-search context and optionally add partial intent, then predicting the full question or retrieving the answer, is not something I've seen in anticipatory search or QG work. The paper formalizes this cleanly, adapts two public datasets, and runs a reasonable sweep of generation and retrieval models. RQ1's finding—larger context hurts—is credible and matches intuition. The qualitative examples are helpful.\n\nThe main soft spot is RQ2. The intents were LLM-extracted from the target questions: Llama-3 on Inquisitive, Yi-34B reformulating MS MARCO queries into context and intent. So adding intent is, in part, feeding a compressed version of the answer back. The observed improvement may measure how much of the target is recoverable from the extracted summary rather than how a real user's hint would behave. Real user intents would be noisier and less aligned. The paper acknowledges the 'silver standard' status and that Inquisitive intents were not manually validated, but it never tests the transfer from extracted to user-supplied intents. Without an independent-intent control, the central claim about intent mitigating large contexts is not established.\n\nThere's also a smaller issue: the MS MARCO 'source' is a BM25 top paragraph, which simulates pre-search context but not real browsing behavior. That's a reasonable first approximation, but it should be flagged as a limitation. And the beneficial effect of intent is not consistent across all models and settings—for the cross-encoder, Source+Intent vs Source is not significant (p=0.6748), so the main takeaway rests on a somewhat mixed result.\n\nThat said, the paper is worth refereeing seriously. The task formulation is a useful contribution to interactive IR and question generation, and the methodological flaw is fixable. I'd ask for a control where intents are generated without access to the target question (e.g., from the context only by a human or a model) and for released code and data. The reader's condition matches my reading. I'd probably cite this for the task framing, not for the RQ2 conclusion.","headline":"New interactive framing for information need prediction, but the 'intent helps' result may be an artifact of extracting intents from the target questions; needs an independent control.","tokens_in":15402,"tokens_out":2173,"would_cite":true,"duration_ms":21252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A user-selected context span plus a short optional intent phrase gives generation and retrieval models enough to predict the user's full information need, and the intent phrase offsets the distraction of larger contexts.","keywords":["information need prediction","pre-search context","search intent","interactive retrieval","question generation","passage retrieval","conversational query formulation"],"falsifier":"A user study where participants highlight text and then type their own intent words, without having seen the target question, would settle the claim: if the retrieval and generation gains over the no-intent condition shrink or disappear when intents are genuinely user-supplied, the paper's conclusion that partial intent mitigates large contexts would be falsified.","tokens_in":14463,"feed_emoji":"🔍","tokens_out":7107,"duration_ms":62097,"temperature":0.7,"pith_summary":"The paper studies whether a user's information need can be predicted before it is typed as a query. The interaction it proposes is lightweight: the user highlights a span of what they are reading (from a single word to a whole paragraph) and, optionally, types a few words about what they want to know about that span, such as 'how' or 'hatch time'. The system then either generates the full natural-language question or retrieves the passage that would answer it. Across two adapted datasets and both generation and retrieval models, the paper finds that larger selected contexts degrade prediction, while adding even a short partial intent reliably recovers much of the lost performance. The authors conclude that this interaction strategy is suitable for real-world search systems.","feed_headline":"One intent word rescues question prediction from context noise","feed_subtitle":"Highlight text, add a short hint, and AI can guess the full question you meant to ask.","key_machinery":"The central object is the (pre-search context, partial search intent) input pair. The context is a user-selected span of a source text, ranging from a single word to a full paragraph, and the intent is an optional short phrase indicating what the user wants to know about that span. The paper operationalizes this pair as five input conditions and measures how well question-generation and retrieval models recover the full need from each. The comparisons that carry the argument are RQ1, which contrasts the selected context with the full source, and RQ2, which contrasts inputs with and without the intent phrase. The authors' mechanism, grounded in the qualitative examples, is that a larger context introduces competing topics, while the intent phrase suppresses that distraction and guides the model toward the intended target.","core_discovery":"The paper's central claim is that information need prediction can be cast as an interactive task in which the user supplies a selected pre-search context and an optional partial search intent, and that current generation and retrieval models can solve the task well enough to be useful. The claim is supported by controlled comparisons across five input conditions: the full question, the context plus intent, the source plus intent, the context alone, and the source alone. In the question-generation experiments, fine-tuned Flan-T5 and prompted Llama-3-8B-Instruct and Yi-34B-Chat produce higher BLEU and ROUGE scores when the input is the selected context rather than the full source, and when intent is present rather than absent. In the retrieval experiments, fine-tuned BERT bi-encoder and cross-encoder models show the same pattern in recall at 10 and MRR, and even in the large-source condition the cross-encoder often keeps the target passage in the top ten. The paper's main explanatory claim is that intent words act as a pointer that focuses prediction on the relevant part of a noisy context.","pith_inferences":["The paper's intents were extracted from the target question by LLMs, so the measured benefit is an upper bound; a user study with freely typed intents would reveal the real-world gap.","An interface could pre-generate a small set of intent suggestions from the highlighted context and let the user click one, reducing effort below typing.","The same context-and-intent formulation could be used to predict not just questions but broader task goals, such as which document a user will open next.","The observed cross-encoder robustness suggests answer retrieval could serve as a fallback when the generated question is uncertain, letting the system ask the user to choose among top passages instead of committing to a question."],"forward_implications":["Search interfaces can let users highlight any span of a page and, if the first prediction is off, add a couple of intent words to regenerate or rerank.","Because intent mitigates larger contexts, users need not carefully minimize their selection; a paragraph-level highlight plus a hint is nearly as good as a precise phrase.","Retrieval-based prediction degrades more gracefully than generation, so an answer-first interface could expose predicted passages before committing to a generated question.","The framework extends to retrieval-augmented generation pipelines as a query-formulation step for individual turns, with no new model architecture required."],"supporting_citations":[{"why":"Supplies the Inquisitive dataset of real reader questions with selected spans, used for the question generation experiments.","marker":"[16]"},{"why":"Supplies the MS MARCO V2 queries, corpus, and relevance judgments used for the retrieval experiments.","marker":"[29]"},{"why":"Provides the Flan-T5-Base pre-trained model that is fine-tuned for the generation experiments.","marker":"[8]"},{"why":"Provides BERT, the base model for both the bi-encoder and cross-encoder retrieval models.","marker":"[11]"},{"why":"Provides the Llama-3-8b-Instruct model used both for extracting intents in the Inquisitive dataset and for prompted question generation.","marker":"[15, 43]"},{"why":"Provides the Yi-34B-Chat model used to reformulate MS MARCO queries into context and intent pairs.","marker":"[1]"},{"why":"Provides BM25, the retrieval function used to find topic-matched source paragraphs for MS MARCO queries.","marker":"[37]"},{"why":"Provides the toolkit used to run the BM25 source retrieval.","marker":"[24]"}],"fun_headline_variants":["A single intent word rescues question prediction from context noise","Select context, add a hint, and AI can guess the full question","Context plus one intent word: better question prediction than full source","Short intent hints sharpen question prediction for large selected contexts","One partial intent word cuts through context noise for question prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiments assume that LLM-extracted partial intents behave the same as intents that real users would type: the intents were derived from the target question itself, so they are perfectly aligned with the desired output, and no independent user-supplied control was tested.","fun_headline_variants_meta":{"raw":{"variants":["A single intent word rescues question prediction from context noise","Select context, add a hint, and AI can guess the full question","Context plus one intent word: better question prediction than full source","Short intent hints sharpen question prediction for large selected contexts","One partial intent word cuts through context noise for question prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00058,"raw_usage":{"total_tokens":2702,"prompt_tokens":884,"completion_tokens":1818,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1734}},"tokens_in":500,"tokens_out":1818,"duration_ms":14977,"temperature":1.0,"reasoning_tokens":1734,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:07:30.850888+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A user study where participants highlight text and then type their own intent words, without having seen the target question, would settle the claim: if the retrieval and generation gains over the no-intent condition shrink or disappear when intents are genuinely user-supplied, the paper's conclusion that partial intent mitigates large contexts would be falsified.","supporting_citations":[{"cited_title":"Inquisitive Question Generation for High Level Text Comprehension","cited_arxiv_id":"2010.01657","evidence_quote":"Supplies the Inquisitive dataset of real reader questions with selected spans, used for the question generation experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MS MARCO V2 queries, corpus, and relevance judgments used for the retrieval experiments."}],"review_version":1}