{"id":"db73d46f-690a-4957-b83a-2b30da45cbb5","arxiv_id":"2608.05876","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"G-STEER uses an Intent Elicitation Graph and evidence-state tracking to train a query refiner that routes between memory retrieval, user clarification, and stopping, improving personalized deep research outcomes.","lead":"G-STEER trains an AI helper to refine a user's research request before a deep research agent starts, deciding whether to retrieve the user's past preferences, ask a clarifying question, or stop and hand the refined request to the agent. In benchmark tests it improves report personalization while asking about a third as many questions as the strongest clarification method it compared against.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"G-STEER's coverage lead over CEP-Clarify+Rewrite exists only at cosine threshold 0.55; at all five other tested thresholds in Table 9 the baseline wins, and the appendix concedes non-uniformity, so the abstract's strongest-coverage claim overstates the evidence.","rationale":"The reader's weakest assumption is the same concern I identify: the threshold-dependence of the coverage ranking, and I concur fully with the CONDITIONAL verdict. What makes this the single most load-bearing issue is that the paper's own Table 9 demonstrates the fragility: G-STEER exceeds CEP-Clarify+Rewrite on WCov, E-WCov, and F1 at exactly one threshold (0.55) and trails at all five others (0.45, 0.50, 0.60, 0.65, 0.70). A headline metric that ranks the proposed method first at one point and second at every neighboring point is not a robust basis for the abstract's unqualified 'strongest overall weighted target coverage.' The appendix's limitation sentence—'The relative performance of the methods varies across metrics and thresholds, and we therefore do not claim uniform superiority over the full range'—is an explicit acknowledgment that the main-text claim requires hedging; per the review instructions I flag it and weigh it as qualifying the abstract. I do not see this as an internal inconsistency or as misconduct; the table is disclosed, which is to the authors' credit. The mitigations that keep this at CONDITIONAL rather than REJECT are: (1) the downstream P-scores in Tables 4 and 8 are independent of the cosine threshold and consistently favor G-STEER on both DRAs, providing separable evidence that input-level refinement helps personalization; (2) the ablations in Table 3 show IEG-ordered trajectory supervision and GRPO shift metrics in directions consistent with the method's design; (3) the code is released, making the required check feasible. The secondary concerns—simulated users, GPT-5.4 generating targets and scoring coverage, no error bars, single seed—are real but bias absolute scores more than they explain the relative flip at 0.55; the threshold dependence is the one issue that demonstrably reverses the relative ranking using only numbers already in the paper. The proposed concrete test (dense sweep plus seed replication) would settle it: a surviving lead across the sweep would let the paper stand, while a single-point lead would force a rewording of the abstract and main claims. Verdict stays CONDITIONAL: require the reworded, threshold-robust claim plus error bars before the paper can move to ACCEPT.","tokens_in":16445,"tokens_out":11094,"duration_ms":92412,"concrete_test":"Take the released code and recompute the Appendix threshold sweep at finer resolution: compute pooled WCov and E-WCov for every method at thresholds 0.50, 0.51, ..., 0.60, and repeat the main Table 2 pipeline with at least three random seeds (e.g., 42, 7, 2026) at thresholds 0.50, 0.55, and 0.60. Then count the thresholds at which G-STEER's WCov exceeds CEP-Clarify+Rewrite's, and check whether the 0.55 lead survives across seeds. If G-STEER leads at fewer than half of the thresholds in the 0.50-0.60 window, or the 0.55 lead does not replicate across seeds, the abstract's 'strongest overall weighted target coverage' claim is a threshold and seed artifact and must be reworded to report the threshold-specific, seed-specific comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"G-STEER's headline claim of 'strongest overall weighted target coverage' rests on a single point in the evaluation: the cosine-similarity threshold 0.55 used to decide whether a GPT-5.4-extracted framing factor covers a reference target. In Table 9, at thresholds 0.45, 0.50, 0.60, 0.65, and 0.70, CEP-Clarify+Rewrite has higher WCov, E-WCov, and F1 than G-STEER; G-STEER leads only at 0.55 (WCov 0.4253 vs 0.4060; E-WCov 0.4639 vs 0.4497). The ranking flips at every adjacent threshold, so the chosen 0.55 is load-bearing for the paper's primary quantitative claim. The paper itself acknowledges this in the appendix: 'The relative performance of the methods varies across metrics and thresholds, and we therefore do not claim uniform superiority over the full range.' Yet the abstract states the strongest-coverage claim without that qualification, and the main text does not flag it. Compounding the fragility, all results come from a single training run (seed 42) with no error bars, so the roughly 4.7 percent relative WCov margin at 0.55 could be within seed noise. The downstream personalization scores (Table 4) are threshold-independent and do favor G-STEER on both DRAs, which is genuine independent support; but they too lack variance estimates and rely on the Qwen3-32B simulator and the same GPT-5.4 evaluator that generated the targets. The concern is not that the method is ineffective: it is that, as written, the abstract's primary coverage claim is a single-threshold result presented as a general superiority claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes G-STEER, a training framework for input-level personalized deep research query refinement. The method models three coupled decisions — which user-specific framing factors are relevant, whether available evidence supports them, and whether to retrieve from memory, ask the user, or stop and rewrite the query. Training uses an Intent Elicitation Graph (IEG) to construct graph-scaffolded trajectories for supervised fine-tuning, followed by target-anchored GRPO that balances weighted target coverage against acquisition and behavioral costs. At inference, the policy induces evidence slots from the query and profile, chooses Retrieve/Ask/Stop, and generates a refined research specification passed to an unchanged downstream DRA. Experiments on PDR-Bench compare G-STEER with prompting baselines, Mistral-Interact, CEP-Clarify+Rewrite, and an IntentRL-style baseline, reporting intrinsic coverage metrics, ablations, question-burden trade-offs, and downstream report personalization/quality with OpenAI Deep Research and OAgents. The headline claims are strongest weighted target coverage, highest downstream personalization, and roughly one-third as many user questions as the strongest clarification baseline.","tokens_in":16773,"tokens_out":4587,"duration_ms":50297,"significance":"If the headline results hold, G-STEER is a useful contribution: it addresses a practical problem (personalizing black-box deep research agents through input refinement), introduces a clean decomposition of relevance, sufficiency, and grounding-control decisions, and provides a training recipe that uses graph-structured trajectories and target-anchored rewards. The paper is unusually transparent in several ways: it releases code, reports a threshold-sensitivity table, describes the full training configuration, and explicitly acknowledges the simulator limitation in the conclusion. The downstream personalization results in Table 4 are threshold-independent and favor G-STEER on both DRAs, which is genuine supporting evidence. However, the intrinsic coverage claim that appears in the abstract is substantially weaker than the text suggests, because it relies on a single cosine threshold and on a single training seed without variance estimates, and because the evaluation pipeline shares a model (GPT-5.4) between target construction, reward anchoring, and outcome measurement.","major_comments":[{"comment":"The claim that G-STEER achieves the strongest overall weighted target coverage is not supported across the reported threshold sensitivity analysis. At the default threshold 0.55, G-STEER's WCov is 0.4253 versus 0.4060 for CEP-Clarify+Rewrite, but Table 9 shows CEP-Clarify+Rewrite has higher WCov at thresholds 0.45, 0.50, 0.60, 0.65, and 0.70; the same pattern holds for E-WCov and for F1 at most non-default thresholds. The appendix itself states that 'the relative performance of the methods varies across metrics and thresholds, and we therefore do not claim uniform superiority over the full range.' The abstract and Section 5.1 nevertheless state the strongest-coverage claim without this qualification. Either the abstract and main-text claims must be restricted to the 0.55 setting, or the authors should report an aggregate measure (e.g., area under the threshold curve) together with a justification for choosing 0.55 rather than an adjacent threshold.","section":"Main Results (Table 2), Appendix Table 9, Abstract"},{"comment":"All trainable variants are evaluated from a single training run with seed 42, and no confidence intervals or significance tests are reported. The margin on the main coverage metric is small: WCov 0.4253 versus 0.4060 at threshold 0.55, a relative difference of about 4.7%. Given the threshold sensitivity documented in Table 9, a single seed cannot establish that G-STEER is the strongest method; the authors should provide multiple seeds with variance estimates or a paired significance test on the central comparisons.","section":"Appendix: Training Configuration; Table 2"},{"comment":"The evaluation has a potential circularity between the training signal and the coverage metric. GPT-5.4 constructs the elicitation targets and IEGs used for training, the target descriptions are used as reward anchors during GRPO, and the same model extracts user-specific framing factors from the refined queries during evaluation. Because G-STEER is explicitly optimized to align its outputs with these GPT-5.4-generated target descriptions, its high coverage scores may partly reflect an ability to imitate the target generator rather than to satisfy real user preferences. All methods share the same evaluation pipeline, which mitigates baseline unfairness, but it does not address the training-evaluation overlap for G-STEER. The authors should validate the coverage metric on a subset with an independent extraction model or human annotation, and should report whether the ranking persists under that alternative.","section":"Evaluation Details and Elicitation Target Construction"},{"comment":"The reported question counts and the interactive aspects of both training and evaluation rely entirely on the Qwen3-32B user simulator, which is also used to generate GRPO rollouts. The paper acknowledges this limitation, but the central utility-burden claim ('asking roughly one third as many user questions') is only as strong as the simulator's fidelity to real user behavior. A human-subject pilot, or at least an analysis of simulator answer rates, non-committal responses, and question interpretability, would be needed before the interaction-efficiency claim can be transferred to real deployments.","section":"User Simulator; Conclusion"}],"minor_comments":[{"comment":"The phrase 'strongest overall weighted target coverage' should be made precise, e.g., 'at the 0.55 cosine threshold used throughout the main evaluation', or replaced with a statement that explicitly acknowledges the threshold sensitivity reported in the appendix.","section":"Abstract"},{"comment":"The ablation labeled 'Graph SFT only' achieves the highest F1 (0.4248) of all rows, while the full model has F1 0.4082. The interpretation that GRPO trades uniform matching for importance-weighted coverage is plausible and is supported by WCov/E-WCov, but the paper should note this trade-off more explicitly in the main text, since a reader may otherwise view the F1 decrease as a regression.","section":"Table 3"},{"comment":"The P and Q scores are reported as averages over three evaluator rounds, but no standard deviations or per-round ranges are given. Reporting variance would help assess whether differences such as 4.35 versus 4.25 for OpenAI Deep Research are meaningful.","section":"Table 4 and Appendix: Downstream Report Setup"},{"comment":"The caption should define the plotted markers and any fitted curves; currently the reader must infer that each point represents one method and that the dashed or solid lines (if any) indicate trade-off frontiers.","section":"Figure 3"},{"comment":"The 'Test-time user state' column says G-STEER uses a frozen profile and episodic memory, but it is not clear from the table how this state is initialized for unseen users; the appendix explains this, but a footnote in the table would improve readability.","section":"Table 7"},{"comment":"The paper says each final refined query is submitted once and every response is successfully parsed. It would be useful to report the exact number of API calls and any retry behavior, since parsing failures could silently bias the extracted factor set.","section":"Appendix: Evaluation Details"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-organized and the training framework is interesting, but the main quantitative claim needs to be recalibrated to match the authors' own threshold-sensitivity results. The single-seed evaluation and the shared use of GPT-5.4 for target construction, reward anchoring, and outcome extraction are the two biggest risks; both are addressable within the paper's scope. If the authors add variance estimates, an independent or human-annotated coverage check, and qualify the abstract, the contribution would be solid for this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, G-STEER is a real architectural idea, not a repackaged baseline: it couples the relevance decision, the evidence-sufficiency decision, and the retrieve/ask/stop decision, and it uses an Intent Elicitation Graph to scaffold training trajectories rather than relying on flat clarification dialogues. That combination is absent from IntentRL, CEP, and Mistral-Interact. Second, the paper's headline claim is shakier than the abstract implies. The 'strongest overall weighted target coverage' result holds only at cosine threshold 0.55; at every other tested threshold in Table 9, CEP-Clarify+Rewrite wins WCov, E-WCov, and F1. The appendix concedes the non-uniformity, but the abstract does not.\n\nWhat it does well. The ablations are the paper's best evidence: randomizing target order while keeping the same target set and action conditions drops WCov sharply, so the IEG dependencies are doing real work. The utility-burden tradeoff is also genuinely good: G-STEER gets higher E-WCov than CEP with one third the questions. Downstream personalization scores favor G-STEER on both DRAs, and those are threshold-independent. Code and data are released. The paper is also honest about its limits: it flags simulated users and the lack of uniform superiority in the appendix.\n\nSoft spots. The threshold dependence is real and load-bearing for the abstract's strongest-coverage sentence. At 0.55 the margin over CEP is about 4.7% relative in WCov, and with a single training run (seed 42) and no error bars, that margin could be noise. There is also a genuine circularity risk: GPT-5.4 builds the elicitation targets, anchors the training reward, extracts the framing factors during evaluation, and scores downstream reports. That shared evaluator inflates any method's absolute numbers, though it affects baselines too. The user simulator is a stand-in for human interaction, and the paper acknowledges that. None of this makes the method look inert; it means the comparative advantage is not as settled as the abstract suggests.\n\nWho this is for. Anyone working on personalized deep research, clarification policies, or evaluation of interactive query refinement will get value from this. It deserves a serious referee. I'd send it out.\n\nRecommendation: accept the paper for peer review, but push for multi-seed variance estimates, a threshold-robust presentation that does not anchor on a single point, and at least a small human or held-out annotation check for coverage.","headline":"G-STEER is a genuinely useful training framework with real evidence for its design, but the abstract's strongest-coverage claim rides a single cosine threshold and a single seed, so read that sentence as provisional.","tokens_in":17388,"tokens_out":2874,"would_cite":true,"duration_ms":29779,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"G-STEER refines a deep research query into a personalized research specification by deciding when to retrieve memory, ask the user, or stop, and reports the strongest weighted target coverage and downstream personalization across two…","keywords":["personalized deep research","query refinement","intent elicitation graph","evidence grounding","clarification policy","retrieve-ask-stop","target-anchored optimization","reinforcement learning"],"falsifier":"Run the same evaluation with the coverage threshold set to 0.50 and 0.60 instead of 0.55; the paper's own threshold table already shows the strongest baseline achieving higher WCov and E-WCov at those thresholds, so if 0.55 is not the correct operating point the headline ranking collapses. Run a real-user interaction study with the same two deep research agents and measure P-scores on the generated reports; if the simulated-user advantage does not appear with real users, the central claim fails.","tokens_in":16143,"feed_emoji":"🔎","tokens_out":10936,"duration_ms":95427,"temperature":0.7,"pith_summary":"This paper is trying to establish that personalized deep research can be achieved by refining the user's initial request before it reaches an otherwise unchanged deep research agent, rather than by modifying the agent's internals. The proposed G-STEER method organizes user-specific framing factors into an Intent Elicitation Graph, uses graph-scaffolded trajectories to teach a policy when to retrieve from memory, ask the user, or stop, and optimizes the policy so that important factors get covered while interaction costs stay low. On the benchmark test set it reports the strongest weighted target coverage and the highest downstream report personalization across two fixed deep research agents, while asking roughly one third as many user questions as the strongest clarification-and-rewrite baseline. A sympathetic reader would care because it offers a way to personalize black-box deep research systems without internal access, improving the trade-off between research quality and user effort.","feed_headline":"Graph-scaffolded refiner cuts deep-research questions by two-thirds","feed_subtitle":"It covers more user-specific targets and improves report personalization across two deep research agents.","key_machinery":"The load-bearing object is the Intent Elicitation Graph (IEG), a directed acyclic graph whose nodes are user-specific elicitation targets and whose edges encode conditional dependencies: resolving one target can activate or change how another should be elicited. The IEG is used only to construct scenario-controlled training trajectories, not at inference. The second mechanism is an explicit evidence state $\\Sigma_t$, a set of target slots recording each factor's grounding status, supported value, and evidence source; the policy conditions on $\\Sigma_t$ to choose Retrieve, Ask, or Stop. The third is target-anchored policy optimization: trajectory-level GRPO with a reward $R(\\tau, E)=R_{\\mathrm{align}}-C_{\\mathrm{acq}}-C_{\\mathrm{con}}$, where $R_{\\mathrm{align}}$ is a one-to-one matching between covered slots and elicitation targets weighted by rubrics-derived importance, and the costs penalize evidence acquisition and protocol violations. Together these let the policy balance important-factor coverage against user burden.","core_discovery":"The central claim is that a query refiner trained with dependency-scaffolded trajectories and explicit evidence-state tracking can decide, for each user-specific factor, whether retrieval, clarification, or stopping is warranted, and that this decision policy transfers to unseen users and tasks. G-STEER induces framing-factor slots from the initial query, user profile, retrieved memory, and dialogue, updates their grounding status as evidence arrives, and only at Stop generates the refined query. The paper reports that this yields the highest weighted coverage of elicitation targets on the benchmark test set, with WCov 0.4253 and E-WCov 0.4639, and the highest personalization scores on both downstream agents, with 4.1 average questions versus 12.0 for the strongest clarification baseline. The paper is careful to note that it does not claim uniform superiority across all coverage thresholds, since relative performance varies with the cosine-similarity threshold used for matching.","pith_inferences":["A testable prediction is that the efficiency advantage will be more robust than the coverage advantage: because the reported threshold sweep shows the strongest baseline overtaking G-STEER on weighted coverage at neighboring similarity thresholds, a real deployment should expect the question-count reduction to persist even where the coverage lead does not.","The same evidence-state machinery could transfer to other pre-execution personalization tasks, such as code-generation or tool-use planning, where the system must decide between asking the user, checking memory, and proceeding with a refined prompt.","Because both the reference targets and the extracted framing factors are produced by the same large language model, the reported coverage may partly reflect that model's internal consistency; using a different model for factor extraction would give a less entangled measurement.","The user simulator's non-committal response behavior makes the training distribution somewhat optimistic; if real users are more verbose or more contradictory, the Retrieve/Ask/Stop policy would need to handle noisier evidence states than the simulator provides."],"forward_implications":["The same refinement layer can be applied to any deep research agent without changing its retrieval or synthesis internals, making personalization feasible for proprietary systems.","Asking fewer, targeted clarification questions does not cost coverage: the policy prefers memory retrieval and stopping when evidence suffices, so user burden drops without sacrificing important factors.","Dependency structure in the elicitation targets is what drives the gains: randomizing target order in trajectory construction sharply reduces all coverage metrics, even when the same target set is used.","Reinforcement learning on top of graph-scaffolded SFT shifts the policy toward high-importance targets, improving weighted coverage even where uniform F1 does not improve."],"supporting_citations":[{"why":"Supplies the PDR-Bench dataset, user profiles, and the PQR evaluation rubric whose importance weights define WCov and E-WCov.","marker":"Liang et al. 2026"},{"why":"Provides the IntentRL approach to proactive clarification over dependency graphs that G-STEER adapts into the IntentRL-style baseline.","marker":"Luo et al. 2026"},{"why":"Provides the CEP clarify-and-rewrite baseline that G-STEER is compared against and that it outperforms on coverage with fewer questions.","marker":"Zhang et al. 2024"},{"why":"Supplies the Mistral-Interact baseline for implicit user intention understanding, used as a prior-work comparison.","marker":"Qian et al. 2024"},{"why":"Supplies the GRPO algorithm used for the target-anchored trajectory-level policy optimization.","marker":"Shao et al. 2024"},{"why":"Supplies the Qwen3-8B backbone for the trained policy and the Qwen3-32B model used as the user simulator in training and evaluation.","marker":"Yang et al. 2025"},{"why":"Provides the Sentence-BERT encoder used to compute semantic similarity for target-slot matching in rewards and coverage evaluation.","marker":"Reimers and Gurevych 2019"},{"why":"Supplies the MPNet encoder inside the all-mpnet-base-v2 model used for semantic matching and retrieval.","marker":"Song et al. 2020"},{"why":"Provides the FAISS index used for retrieval-based user memory in the Retrieve action.","marker":"Douze et al. 2024"},{"why":"Provides the OAgents open-source framework used as one of the two fixed downstream deep research agents in report evaluation.","marker":"Zhu et al. 2025"}],"fun_headline_variants":["Graph-scaffolded refiner cuts deep-research questions by 66%","Ask less, personalize more: G-STEER refines deep-research queries","Deep research refiner asks 66% fewer questions, boosts personalization","G-STEER: graph-trained query refiner trims user questions, lifts report quality","Smart query refiner for deep research: fewer questions, better personalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on two untested equivalences: that a cosine-similarity threshold of 0.55 correctly determines when a refined query covers a user-specific target, and that the simulated user responses faithfully represent real user behavior.","fun_headline_variants_meta":{"raw":{"variants":["Graph-scaffolded refiner cuts deep-research questions by 66%","Ask less, personalize more: G-STEER refines deep-research queries","Deep research refiner asks 66% fewer questions, boosts personalization","G-STEER: graph-trained query refiner trims user questions, lifts report quality","Smart query refiner for deep research: fewer questions, better personalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001078,"raw_usage":{"total_tokens":4505,"prompt_tokens":936,"completion_tokens":3569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":3466}},"tokens_in":552,"tokens_out":3569,"duration_ms":24536,"temperature":1.0,"reasoning_tokens":3466,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:52:30.436112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same evaluation with the coverage threshold set to 0.50 and 0.60 instead of 0.55; the paper's own threshold table already shows the strongest baseline achieving higher WCov and E-WCov at those thresholds, so if 0.55 is not the correct operating point the headline ranking collapses. Run a real-user interaction study with the same two deep research agents and measure P-scores on the generated reports; if the simulated-user advantage does not appear with real users, the central claim fails.","supporting_citations":[{"cited_title":"2025 , publisher =","cited_arxiv_id":null,"evidence_quote":"Provides the OAgents open-source framework used as one of the two fixed downstream deep research agents in report evaluation."}],"review_version":1}