{"id":"fd09c376-d584-41f6-ae0b-d710360b704a","arxiv_id":"2412.10933","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A framework combining population-level user intent analysis with chat-session-level LLM question generation improves judged usefulness and discoverability in an enterprise AI assistant.","lead":"Enterprise AI assistants often leave users unsure what to ask next. This paper describes a two-stage framework that suggests follow-up questions by analyzing all users' intents and the current chat context, and human raters found it more useful and discoverable than a simple baseline in Adobe's assistant.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed improvement over baseline is not statistically established: the enhanced method differs from the baseline on three axes and the reported pairwise evaluation lacks significance testing, so conditional acceptance is the right ceiling.","rationale":"The reader's weakest assumption correctly identifies the evaluation as the load-bearing element: the claimed improvements in usefulness and discoverability are only as strong as the pairwise human comparison, and that comparison is both confounded and under-analyzed. I agree with the reader's assessment rather than escalating to rejection because the paper is a practical systems contribution and the proposed framework is plausible; the defect is in the evidence, not in an internal contradiction. The most damaging single issue is that the enhanced prompt explicitly instructs the model to produce category-tagged, exploration-oriented questions, so the discoverability criterion is partly baked into the method, while the baseline prompt is a combined response-plus-suggestion prompt. This makes the 33.4% vs 23.2% discoverability margin unsurprising and not attributable to the population-level intent analysis. The lack of significance testing compounds the problem, especially since annotator E2 reverses the usefulness preference and P2 shows a tie. The paper's own stated future work—production engagement metrics—acknowledges that the current evaluation does not measure real user behavior. I found no other load-bearing concern: the framework description is coherent, the prompt template is concrete, and the use of GPT-3.5 for both conditions is a reasonable design choice, even if not a full ablation. The most useful next step would be the decomposition experiment described above, paired with proper inferential statistics, before the headline claim can be accepted as demonstrated.","tokens_in":11125,"tokens_out":3144,"duration_ms":30240,"concrete_test":"Run the same human evaluation with three additional arms that decompose the enhanced condition: baseline plus separate generation pass only; baseline plus chat-history context only; baseline plus category instructions only, all using the same 250 sessions and same annotators. Compute per-session paired significance (e.g., Wilcoxon signed-rank or exact McNemar on per-session preferences) and inter-annotator agreement (Cohen's kappa) for each contrast. If the category-instruction arm alone reproduces the discoverability gain, or if no contrast reaches p<0.05 with multiple-comparison correction, then the framework's specific contribution is unsupported; otherwise the conditional acceptance remains justified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that the framework improves usefulness and discoverability over a straightforward LLM baseline—rests entirely on Table 2's pairwise human annotations. That comparison is confounded: the enhanced condition adds (i) a separate generation pass, (ii) chat-history context, and (iii) pre-defined intent categories in the prompt, while the baseline has none. Any of these could drive the observed 35.4% vs 27.8% usefulness and 33.4% vs 23.2% discoverability differences; Table 3 shows one engineer annotator (E2) actually preferred the baseline on usefulness (50.0% vs 40.6%), and no significance test or inter-annotator agreement is reported. The paper also acknowledges selecting four of ten annotators for Table 3 'for conciseness,' so the role-level narrative is post hoc. Without ablations or paired statistics, the aggregate preference cannot be attributed to the framework's population-level intent analysis, and the observed effect may be within annotator noise. This is not an internal inconsistency, but it means the abstract's 'demonstrating' is stronger than the evidence supports.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage framework for generating next-question suggestions in an enterprise conversational AI assistant, evaluated on the Adobe Experience Platform (AEP) AI Assistant. A population-level periodic intent analysis identifies user intent categories (Expansion, Follow-Up, Other), which are then injected into a session-level LLM prompt that also receives the current query, the AI response, chat history, and retrieved documents. The evaluation uses 250 real user interactions and pairwise human annotations by ten annotators on five criteria: relatedness, validity, usefulness, diversity, and discoverability. The aggregate results show raw preference proportions favoring the enhanced method on usefulness (35.4% vs 27.8%) and discoverability (33.4% vs 23.2%), with smaller margins on the other criteria. The paper concludes that the framework improves usefulness and system discoverability.","tokens_in":11296,"tokens_out":5526,"duration_ms":50629,"significance":"If the causal claim were established, this would fill a genuine gap: question suggestion for enterprise assistants with sparse and noisy interaction logs, evaluated on a deployed industrial product. The paper's strengths include the use of real user logs, a clear and practical generation pipeline, pairwise annotation with role-diverse annotators, and the introduction of a discoverability criterion that is well matched to the application. The intent-analysis statistics in Table 1 are a useful descriptive characterization of the data-sparsity problem. However, the current empirical support for the headline claim is incomplete: the comparison is confounded, the reported numbers are raw proportions without significance testing or inter-annotator agreement, and the role-level analysis is post hoc. With additional controlled comparisons and statistical analysis, the contribution could be solid; as it stands, the evidence is directionally suggestive rather than demonstrative.","major_comments":[{"comment":"The headline claim that the framework 'demonstrat[es] improved usefulness and system discoverability' rests on raw pairwise preference proportions over 250 interactions, with no significance tests, confidence intervals, or inter-annotator agreement. The preference margins on the two key criteria are modest (35.4% vs 27.8% for usefulness; 33.4% vs 23.2% for discoverability), and Table 3 shows that one annotator (E2) rated the baseline higher on usefulness (50.0% vs 40.6%). Because annotations are clustered by item and annotator, a paired per-item test with annotator as a random effect (or at least a Wilcoxon signed-rank test on item-level proportions and Cohen's or Fleiss' kappa) is needed before the difference can be attributed to the method rather than to annotator noise.","section":"§4.3, Table 2; Abstract"},{"comment":"The baseline differs from the enhanced method on three simultaneous dimensions: a separate generation pass, use of in-session chat history, and pre-defined intent categories in the prompt. The paper's claimed novelty is the population-level intent analysis, but as designed the aggregate win cannot be attributed to the categories or to any single component. The statement in §4.2 that the same document retrieval was applied to both methods controls only one factor. At minimum, an ablation with the enhanced pipeline minus the pre-defined categories, and ideally a version of the baseline that also receives chat history, is needed to support the paper's attribution of the improvements to its framework.","section":"§4.2; §3.3"},{"comment":"The 'Annotator Role-Based Insights' are presented as findings but are derived from only four of ten annotators, selected 'for conciseness' with no stated selection rule, and the paper asserts that the subset 'effectively captures the key trends' without evidence. No statistical comparison across roles is reported, and E2's reversed usefulness preference shows heterogeneity within a role. This part should be removed from the main results or relabeled as exploratory, with all annotator-level data shown or a transparent selection criterion and appropriate caveats.","section":"§4.3, Table 3, Appendix A.4"},{"comment":"The paper claims improved 'system discoverability,' but the evidence is an offline pairwise annotation of generated suggestions by annotators with AEP familiarity, not a measurement of whether users actually discover or use features. Section 5 explicitly lists production engagement metrics such as click-through rates as future work. The abstract and conclusion should therefore say that the framework improves judged usefulness and judged discoverability of the suggestions in a human evaluation, rather than claiming demonstrated improvement in realized system-level discoverability.","section":"Abstract; §5"}],"minor_comments":[{"comment":"There is a typo: 'Exansion' should be 'Expansion,' and the notation 'H = Sn j=1 Hi' is confusing because the subscript in the union does not match the session index used in the surrounding text.","section":"§3.2"},{"comment":"The criterion name is inconsistent: 'Validity' in the text and definitions but 'Validness' in Table 2 and elsewhere; please standardize the terminology.","section":"§4.2, Table 2, Appendix A.3"},{"comment":"The prompt template says to generate at least one question for each of the 'first four categories' and later for 'four categories,' but only three categories are defined (Expansion, Follow-up, Other); this inconsistency should be reconciled.","section":"Appendix A.1"},{"comment":"The figure contains typos, including 'ST ART EXAMPLES' and 'Y our task,' and the reference list contains a broken LaTeX escape ('V osecky') that should be corrected.","section":"Figure 2"},{"comment":"Table 1 shows that 36% of next questions are unrelated to previous interaction and 23% fall into 'Others,' but the paper does not explain how the prompt's 'Other' category relates to the 'Unrelated' and 'Others' intents; this connection should be clarified.","section":"Table 1; §3.3"}],"recommendation":"major_revision","confidential_remarks":"The empirical gaps identified in the major comments are fixable with additional analysis rather than requiring a fundamentally new system design, so I would not reject on novelty. I would, however, require significance and agreement statistics plus at least one ablation before acceptance, and I would ask the authors to soften the abstract's 'demonstrating' language unless the statistics support it. The paper may fit an applied NLP or industry-track venue better than a general methods venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, well-written applied paper with a coherent framework, but its central claim is not statistically supported. The idea is to combine periodic population-level intent analysis with session-level LLM-based question generation to improve usefulness and discoverability. That combination, applied to Adobe Experience Platform's AI assistant, is genuinely new as far as I know.\n\nWhat it does well: the framework is clearly described (Figure 1, prompt template in the appendix), the intent analysis table (35% unrelated, 30% expansion, 11% follow-up) is a useful reality check, and the human evaluation protocol is careful in some respects: pairwise, blinded, randomized order, five criteria including discoverability. The authors are also honest about data sparsity and the lack of standard benchmarks.\n\nThe main problem is the evaluation. The baseline differs from the enhanced method on three axes at once: a separate generation pass, inclusion of chat history, and predefined categories. Any of these could drive the difference. The aggregate preference for usefulness (35.4% vs 27.8%) and discoverability (33.4% vs 23.2%) is real in the annotations, but no significance test or confidence interval is reported, and there is no inter-annotator agreement. Table 3 shows that engineer E2 actually preferred the baseline on usefulness (50.0% vs 40.6%), so the effect is not uniform. Also, the paper reports only 4 of 10 annotators in that table, selected \"for conciseness\" — that is post hoc selection and weakens the role-level narrative.\n\nThese issues are fixable in principle: report per-item statistics, compute agreement, add an ablation that isolates the categories, or at least re-run with a matched prompt. Absent that, the abstract's \"demonstrating\" is too strong. I would phrase it as \"suggesting\" or \"indicating.\"\n\nThis deserves peer review, not desk rejection. It is an applied paper with a real deployment, and the framework is sensible enough to warrant a serious referee who can push on the evaluation. I would want to see revisions before acceptance, but it is a reasonable submission to an applied NLP or HCI venue. For my own reading group, I would probably give it a maybe; it is not going to change how I think about conversational AI, but the intent analysis and the discoverability angle are worth discussing.","headline":"A clear applied framework for proactive question suggestions in an enterprise assistant, but the evaluation doesn't yet support the 'demonstrating' claim.","tokens_in":11830,"tokens_out":1766,"would_cite":false,"duration_ms":15169,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding population-level intent categories to session-level LLM prompting makes follow-up question suggestions in an enterprise conversational assistant more useful and better at revealing underused features, and…","keywords":["enterprise conversational AI","question suggestion","discoverability","large language models","retrieval-augmented generation","user intent analysis","human evaluation","Adobe Experience Platform"],"falsifier":"A controlled A/B test in the live AEP assistant, where users are randomly assigned to receive either the enhanced suggestions or the baseline suggestions, measuring click-through rate and downstream feature exploration: if the enhanced suggestions are not clicked or acted upon more often than the baseline over a statistically meaningful number of sessions, the paper's claim that they improve usefulness and discoverability would be contradicted.","tokens_in":10928,"feed_emoji":"💡","tokens_out":3768,"duration_ms":32207,"temperature":0.7,"pith_summary":"The paper tries to establish that a simple two-stage framework can materially improve the quality of next-question suggestions in a real enterprise conversational assistant. The first stage periodically analyzes what the user population actually asks after receiving answers; the second stage uses an LLM at the chat-session level to generate suggestions conditioned on the current query, the assistant's response, retrieved documents, and those population-level categories. On 250 real interactions from Adobe Experience Platform's AI Assistant, human annotators preferred the enhanced suggestions over a plain LLM baseline on usefulness (35.4% versus 27.8%) and on discoverability (33.4% versus 23.2%), while most other criteria were rated equally good. If the result transfers to live use, proactive and category-informed suggestions could become a standard way to onboard new users and surface underused capabilities in enterprise AI systems.","feed_headline":"Proactive question suggestions boost enterprise AI discoverability","feed_subtitle":"Two-stage intent-plus-LLM framework beats a plain LLM baseline on usefulness and feature discovery in AEP Assistant.","key_machinery":"The central mechanism is a two-stage framework: population-level user intent analysis that categorizes next-question intents into types such as Expansion and Follow-Up, and chat-session-level question generation that prompts an LLM with the session's query history, the AI response, retrieved documents, and those predefined categories. The categories are the device that steers the LLM away from generic follow-ups toward questions that both continue the user's immediate task and introduce related, possibly unknown platform capabilities.","core_discovery":"The paper's central claim is that combining periodic population-level user intent analysis with chat-session-level LLM generation produces follow-up question suggestions that are both more useful to the user and better at exposing the system's underutilized features than an LLM baseline that only sees the current query and response. The evidence is a pairwise human evaluation in which ten annotators compared suggestion sets across five criteria. The largest advantages appear exactly where the framework is designed to help: usefulness and discoverability, with the enhanced set preferred 35.4% to 27.8% and 33.4% to 23.2% respectively.","pith_inferences":["Beyond the paper: because the enhanced method differs from the baseline in several ways at once (a separate generation pass, inclusion of chat history, and predefined categories), a follow-up ablation varying one factor at a time could isolate which component drives the reported gains.","Beyond the paper: the paper's human-preference results predict, but do not measure, actual behavior; a production A/B test on click-through rate and downstream feature adoption would convert the preference signal into a behavioral outcome.","Beyond the paper: if the role-based differences replicate, a personalized category set per user role may outperform the one-size-fits-all Expansion/Follow-Up split, and could be tested in the same pairwise framework.","Beyond the paper: the 36% 'unrelated' share of next questions found in the intent analysis suggests that non-contextual suggestions, such as periodic feature spotlights, may be worth adding alongside contextual ones to truly improve discoverability."],"forward_implications":["If the central claim is correct, the framework can be deployed in other enterprise assistants where historical interaction data is too sparse to train conventional query-prediction models.","The periodic intent analysis can be rerun as the platform evolves, so the suggestion categories track new capabilities and shifting user behavior.","Even with minimal session history, the LLM-plus-RAG setup can still produce suggestions that are contextually relevant, because the retrieved documents and categories add grounding.","The annotator-role differences suggest that tailoring suggestion categories to user roles, such as engineer versus product manager, could further improve perceived usefulness and discoverability."],"supporting_citations":[{"why":"Supplies the human evaluation criteria and pairwise comparison methodology that the paper adapts for measuring query suggestion quality.","marker":"Baek et al. (2024)"},{"why":"Provides the retrieval-augmented generation technique used to fetch relevant documents for both the baseline and the enhanced question generation.","marker":"Lewis et al. (2020)"},{"why":"Demonstrates that proactive dialogue and timely suggestions can improve user experience, motivating the paper's design of proactive question suggestions.","marker":"Kuang et al. (2024)"},{"why":"Shows that conversational recommendations can improve discoverability of system functionalities, a key criterion the paper evaluates.","marker":"Kalirai et al. (2024)"},{"why":"Provides the standard definition of discoverability that the paper applies to the enterprise AI context.","marker":"Norman (2013)"},{"why":"Highlights the growing challenge of feature discoverability in complex systems, framing the problem the framework addresses.","marker":"Mackamul et al. (2024)"},{"why":"Supports the choice of pairwise comparisons over independent ratings as more reliable relative judgments.","marker":"Stewart et al. (2005)"}],"fun_headline_variants":["Two-stage question suggestions lift enterprise AI discovery","Context-aware prompts reveal hidden AI assistant features","Intent-based follow-ups boost usefulness and discoverability","Hybrid intent and LLM suggestions improve AI assistant use"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the pairwise preferences of ten annotators on 250 selected interactions predict how the broader population of real Adobe Experience Platform users would experience the suggestions in live use, despite strong disagreements among individual annotators and the lack of statistical significance testing.","fun_headline_variants_meta":{"raw":{"variants":["Two-stage question suggestions lift enterprise AI discovery","Context-aware prompts reveal hidden AI assistant features","Intent-based follow-ups boost usefulness and discoverability","Hybrid intent and LLM suggestions improve AI assistant use"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000131,"raw_usage":{"total_tokens":1041,"prompt_tokens":768,"completion_tokens":273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":384,"completion_tokens_details":{"reasoning_tokens":214}},"tokens_in":384,"tokens_out":273,"duration_ms":3067,"temperature":1.0,"reasoning_tokens":214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:27:37.686564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled A/B test in the live AEP assistant, where users are randomly assigned to receive either the enhanced suggestions or the baseline suggestions, measuring click-through rate and downstream feature exploration: if the enhanced suggestions are not clicked or acted upon more often than the baseline over a statistically meaningful number of sessions, the paper's claim that they improve usefulness and discoverability would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the standard definition of discoverability that the paper applies to the enterprise AI context."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the choice of pairwise comparisons over independent ratings as more reliable relative judgments."}],"review_version":1}