{"id":"dda93ebe-b7d3-4a52-a486-88be1a7653f3","arxiv_id":"1908.04832","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A field trial of 10,000 conversations plus 16 lab participants shows that in an open-domain social bot, users prefer storytelling and games over search and feel the bot controls the conversation too much.","lead":"SlugBot, an open-domain Alexa social bot, was deployed to more than 10,000 users and then studied in the lab; users rated storytelling and games more positively than search and chit-chat. The study suggests that designers give conversational agents opinions and personal stories rather than making them talking encyclopedias.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Search-module comparisons in Table 2 may reflect fallback after coverage failures rather than user dislike of factual information, weakening the main design implication.","rationale":"The reader's weakest assumption correctly identifies self-selection as a threat to causal module comparisons. My stress-test points to a related but more specific mechanism: Search is not simply a user-selected module; it is also the system's automatic fallback when understanding or content fails. This makes the low Search ratings in Table 2 especially hard to interpret as evidence against information provision. The paper itself flags this possibility in the Discussion and hedges the overall conclusions in the Limitations, so the concern is acknowledged rather than hidden. Because the field data are observational and the authors already present the module comparisons as suggestive rather than definitive, the appropriate verdict remains conditional rather than acceptance or rejection. The proposed analysis of signature logs would directly test whether the Search effect survives when fallback-triggered conversations are separated from user-initiated question answering.","tokens_in":16329,"tokens_out":5036,"duration_ms":56652,"concrete_test":"Using the per-turn signature logs described in Section 4.1, split Search-labeled turns into (a) user-initiated question answering (a question-answer pair with no immediately preceding failure or fallback signal) and (b) system-initiated fallback (triggered after an ASR/NLU low-confidence event or after no curated content matched). Recompute Table 2 with conversations classified by the dominant Search trigger, and rerun the Mann-Whitney tests for group (a) against Games and Storytelling. If the user-initiated Search rating is not significantly lower, or the gap materially shrinks, the claim that users disprefer factual information is not supported by the field data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative contrast—Search rated worse than Storytelling, Games, and Chit-Chat (Table 2, Section 4.1)—is vulnerable to a trigger confound. Section 3.2 states that search is used 'to fill in gaps in our database of content' and as a 'fall-back strategy' when curated content is exhausted. Thus a conversation labeled 'Combined Search' is not simply one in which the user chose fact-seeking; it may be one in which the system failed to understand or lacked topic content and fell back to Wikipedia or QA. Users who receive fallback responses may rate the whole conversation lower for reasons unrelated to a preference against factual information. The Discussion even acknowledges 'search is technically difficult and it may have been that it returned irrelevant results.' Because ratings are assigned to whole conversations (Section 4.1) and any conversation containing a Search turn is placed in the Search group, the 3.01 mean rating and the reported Mann-Whitney differences can be driven by system failures rather than by module content. The self-selection limitation noted in the Limitations section compounds this: users could choose Games or Storytelling, whereas Search often triggered automatically. If the low Search ratings are largely a failure artifact, the design implication 'move away from conversational systems that simply provide factual information' loses its main quantitative support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the design and large-scale field evaluation of SlugBot, an open-domain social conversational system that participated in the 2018 Alexa Prize. SlugBot combines topic-oriented chit-chat, interactive games, storytelling, and search-based fallback strategies, with content sourced from public corpora and crowdsourcing. The reported deployment collected over 10,000 conversations in August 2018, with user ratings and system logs; an in-lab qualitative study of 16 participants was also conducted. The central claims are that storytelling and games receive higher user ratings and lead to longer conversations than search and general chit-chat, and that users find the system too controlling in general chit-chat. From this, the authors draw design implications favoring conversational systems that express opinions and share personal stories over systems that primarily provide factual information.","tokens_in":16506,"tokens_out":4365,"duration_ms":43330,"significance":"If the module-level comparison is taken at face value, the paper makes a useful empirical contribution to the design of open-domain social chatbots, providing evidence at a scale (over 10,000 conversations) that is rare in this area. The use of nonparametric Mann-Whitney tests for the rating comparisons and the grounding of qualitative claims in participant quotes are strengths. However, the central quantitative comparison is confounded by user self-selection into modules and by the system's use of search as a fallback after coverage failures, which the authors themselves acknowledge in the Limitations section. As a result, the paper's headline design implication---that users do not want factual information provision---is not fully supported by the reported data. The paper is honest about these limitations, but the Abstract and Discussion state the implication more strongly than the evidence warrants.","major_comments":[{"comment":"The comparison of ratings across dialogue modules is confounded by the system's fallback behavior and by user self-selection. Section 3.2 states that search is used 'to fill in gaps in our database of content' and is a 'fall-back strategy,' while Section 4.1 notes that 'users were able to choose which modules they interacted with.' Therefore, a conversation labeled Combined Search may have been a conversation in which the system failed to understand the user or lacked topic content, triggering a search-based response, rather than a conversation in which the user deliberately sought factual information. The lower ratings and shorter durations for Combined Search in Table 2 could be driven by these system failures. The Mann-Whitney tests show that the differences are statistically significant, but they do not establish that the cause is a user preference against factual information. This confound directly affects the central design implication stated in the Abstract and Section 5.","section":"Section 4.1 / Table 2"},{"comment":"The Discussion contains an internal tension. It acknowledges, 'search is technically difficult and it may have been that it returned irrelevant results - a possibility that we intend to evaluate more systematically,' yet it also states, 'users did not seem to want Information Provision, as evidenced by low ratings for Search modules.' If the low search ratings may reflect a failure artifact, they cannot simultaneously serve as evidence that users are averse to factual information. The paper should either re-analyze the log data to separate user-initiated search from system-initiated fallback search (or at least remove conversations with known system errors), or substantially temper the conclusion to say that the current implementation of search underperformed. As written, the key design implication is not supported.","section":"Section 5"},{"comment":"The qualitative study is described as confirming the quantitative results from Table 2, but this is not an independent confirmation. The 16 participants were instructed to engage in general conversation first and then directly interact with specific modules, so the module exposure was not self-selected but still not randomized, and the sample is small and homogeneous (mean age 22.3). The qualitative reactions are valuable as complementary evidence, but they should be presented as illustrative rather than as confirming the deployment comparison. Additionally, the participant demographics sentence states that 'Seven owned an Alexa device while 7 described themselves as having limited or no Alexa experience'; these two groups sum to 14, leaving 2 participants unaccounted for, and the overlap between the two categories is unclear.","section":"Section 4.2"}],"minor_comments":[{"comment":"There is a typo in the sentence 'Overall, this approach is imrpoves the ability of SB to stay on topic'; 'imrpoves' should be 'improves', and the phrase 'is improves' should be corrected.","section":"Section 3.2"},{"comment":"In the sentence reporting the Mann-Whitney tests, 'topic-oriented Chat-Chat' should be 'topic-oriented Chit-Chat'.","section":"Section 4.1"},{"comment":"Table 2 reports mean and median ratings, turns, and time for each module but does not report the number of conversations assigned to each module group. These counts are necessary to assess the Mann-Whitney U values and to understand the composition of each group, especially because many conversations involve multiple modules.","section":"Table 2"},{"comment":"The caption states 'Higher ratings are significantly more likely with longer conversations,' but the reported Pearson correlation is only r = 0.18. 'More likely' is imprecise for a continuous rating variable; consider phrasing such as 'longer conversations tend to receive higher ratings.'","section":"Figure 7 caption"},{"comment":"References [55] and [56] appear to be the same paper (Thorne, Korobov, and Morgan 2007) and are listed with identical bibliographic details. One should be removed or the citations should be merged.","section":"References"},{"comment":"The paper reports 'over 10,000 individual conversations' and 'over 290,000 user turns' but does not provide exact counts or the number of unique users. Providing exact numbers would strengthen the scale claims and facilitate comparison with other deployments.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"This is an honest and useful field report, and the authors explicitly acknowledge the lack of a controlled deployment. The main problem is that the Abstract and Discussion present the search-vs.-playful-modules contrast as evidence against factual information provision, while the method and the authors' own caveats show that the search condition is confounded with system failures and self-selection. I would encourage the editor to solicit a revision that either re-analyzes the data (e.g., distinguishing user-initiated from system-initiated search) or substantially softens the design implication. With that change, the paper could be a valuable contribution to the CUI community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing to know: this is a genuinely large-scale field study of an open-domain social bot, with module-level ratings from over 10,000 Alexa Prize conversations plus a qualitative follow-up. The core finding—that users engage more and rate higher with games and storytelling than with search and general chit-chat—is real, but the causal reading is thinner than the abstract implies, mainly because search also serves as the system's fallback when coverage fails.\n\nWhat's new and useful: the scale itself is a step beyond prior open-domain evaluations (60 or 700 conversations), and the signature logging that lets the authors attribute ratings to dialogue modules is a real methodological contribution. The system design—crowd-sourced games, stories, and opinionated chit-chat—is also a useful integration of techniques that many in the conversational-UX community will want to study. The paper is honest about its observational design in the Limitations section, and the qualitative quotes ground the quantitative patterns.\n\nThe soft spots, in proportion: (1) Self-selection is the biggest one. Users chose which modules to engage, so the higher ratings for storytelling and games could partly reflect the kinds of users attracted to those modules. The authors acknowledge this, but it remains a real limit on any causal claim. (2) The search-fallback confound is valid and partially acknowledged. A conversation labeled 'Combined Search' may be one where the system failed to understand or lacked content and fell back to Wikipedia or QA. That could lower ratings for reasons unrelated to user preference against factual information. The authors do note in the Discussion that search may have returned irrelevant results, so this isn't hidden—but it does weaken the strong design implication in the abstract: 'move away from conversational systems that simply provide factual information.' The qualitative data suggests users liked facts when embedded in a lively conversation, so the paper's own nuance is more credible than the abstract's. (3) The qualitative sample is 16 young users, fine for generating hypotheses, not for strong population claims.\n\nOverall, this deserves a serious referee. The empirical contribution is solid for the conversational-UX subfield, and the limitations are stated rather than buried. I'd recommend peer review with requests to release the data/artifacts where possible and to reframe module comparisons as correlational, or supplement with a controlled follow-up. I'd cite it while working on social dialogue evaluation.","headline":"A valuable large-scale field study of an open-domain social bot whose module-level comparisons are real but observational—worth reviewing, with the search-fallback confound noted.","tokens_in":17078,"tokens_out":1944,"would_cite":true,"duration_ms":22152,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a field trial of over 10,000 conversations, users rated a social bot's storytelling and games higher than its search and general chit-chat, while also finding the system too controlling in open chit-chat.","keywords":["conversational interfaces","open-domain dialogue","Alexa Prize","storytelling","user study","field trial","games","chit-chat"],"falsifier":"Randomly assign users to four otherwise identical bot conditions that differ only in module type, search, chit-chat, games, and storytelling, and compare ratings; if the storytelling advantage over search falls below significance or reverses, the paper's central module-preference conclusion is an artifact of user self-selection.","tokens_in":16116,"feed_emoji":"💬","tokens_out":6339,"duration_ms":59594,"temperature":0.7,"pith_summary":"The paper reports a large-scale field deployment of SlugBot, an open-domain social chatbot built for the 2018 Alexa Prize, and argues that how a conversational system engages users matters more than how much factual information it can retrieve. Across over 10,000 conversations, storytelling and interactive games produced longer sessions and higher user ratings, while search-based question answering produced short, low-rated exchanges. Qualitative follow-up interviews with 16 users add a second finding: in open chit-chat users found the system too controlling, but the predictable structure of stories and games made system-led turns acceptable and even welcome. Users also credited the opinionated, playful persona with intelligence and personality. If correct, the study supports a design shift from assistants that answer facts to social agents with opinions, stories, and structured play.","feed_headline":"Storytelling beats search in a 10,000-conversation social bot trial","feed_subtitle":"Ten thousand logged chats show users rate stories and games above search, and find chit-chat too controlling.","key_machinery":"The central object is SlugBot (SB), an open-domain social dialogue system whose dialogue manager logs a 'signature' for every system turn, labeling which of four high-level modules produced it: Search, Topic-oriented Chit-Chat, Interactive Games, or Storytelling. Those signatures let the authors retrospectively group the 10,000 field conversations by module and compare ratings, turn counts, and durations. The supporting mechanism is content sourcing: crowd-sourced games and hypothetical questions, public fables and personal narratives, and topic-indexed trivia and news retrieved by Elasticsearch. The signature logging is what turns a messy public deployment into a per-module user study, while the playful, opinion-driven content is what the authors claim drives the positive ratings.","core_discovery":"On its own terms, the paper's central discovery is that module design determines perceived conversational success: conversations involving the Storytelling module received the highest average user rating (3.62, versus 3.12 for chit-chat, 3.20 for games, and 3.01 for search), and search-driven interactions were significantly worse across rating, turns, and duration. In the authors' interpretation, users do not want a talking encyclopedia; factual information is welcomed only when it serves another activity, such as supporting an opinion or extending a story. The qualitative data further show that structured activities with clear turn-taking, such as stories told in installments and games with predictable question formats, let users understand their role, reduce coverage ambiguity, and lead users to attribute intelligence and personality to the system. The same structure is experienced as over-control in open chit-chat, where users wanted more influence over the agenda. The paper's prescriptive conclusion is that future social bots should have their own opinions and personal stories to share, with SlugBot as a working example.","pith_inferences":["The per-module rating gaps are correlational because users self-selected modules; a randomized assignment that controls user type could confirm whether the storytelling advantage is causal.","If the mechanism is opinion exchange plus predictable structure, then other structured activities, such as debates, collaborative storytelling, or shared hypotheticals, should produce similar engagement without needing more factual content.","The 'too controlling' complaint suggests quantifiable design targets, such as a user-initiated topic-switch success rate or the fraction of system turns that ask the user to choose a direction, which could be measured and optimized.","The finding that users impute intelligence from opinionated responses implies that perceived competence may be manipulable by style of response rather than by underlying knowledge coverage."],"forward_implications":["Search-style question answering, on its own, shortens conversations and lowers ratings, so open-domain social bots should not be built primarily around factual retrieval.","Stories and games can carry long, highly rated conversations even when the underlying understanding is imperfect, because their predictable structure lowers ambiguity for the user.","Users attribute intelligence and personality to a bot that expresses and defends opinions, so opinion exchange is a usable design lever for perceived humanness.","System-led control is acceptable inside a clearly signaled activity, but in free chit-chat it reads as domination; future designs need to hand agenda control to the user.","Initial system prompts that promise broad coverage raise expectations that hurt later ratings, so expectation-setting language should be conservative."],"supporting_citations":[{"why":"Supplies the Alexa Prize deployment and the 1-5 user rating scale from which all quantitative comparisons are drawn.","marker":"[22]"},{"why":"Provides the winning system's mean rating and turn/duration statistics used as the comparison baseline for SlugBot's overall performance.","marker":"[9]"},{"why":"Describes the SlugBot system architecture and dialogue manager whose per-turn signatures enable module-level analysis.","marker":"[5]"},{"why":"Supplies the corpus of personal narratives used to build SlugBot's Storytelling module.","marker":"[19]"},{"why":"Supplies the PersonaBank narratives and story intention graphs used as storytelling content.","marker":"[29]"},{"why":"Provides the conversational analysis that grounds the design assumption that everyday talk exchanges opinions and activities rather than facts.","marker":"[24]"},{"why":"Documents that retrieved-corpus open-domain systems are judged less habitable, motivating the authored-content approach.","marker":"[17]"}],"fun_headline_variants":["10k chats show storytelling beats search in bots","Users favor storytelling bots over controlling chit-chat","Stories engage, search suffocates in Alexa Prize bot","In open-domain bots, storytelling wins, control loses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparisons assume that users who chose to engage with stories or games are comparable to those who engaged with search or chit-chat, but since users picked their own modules, the rating differences could reflect the types of users each module attracted rather than the module's quality.","fun_headline_variants_meta":{"raw":{"variants":["10k chats show storytelling beats search in bots","Users favor storytelling bots over controlling chit-chat","Stories engage, search suffocates in Alexa Prize bot","In open-domain bots, storytelling wins, control loses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000942,"raw_usage":{"total_tokens":4058,"prompt_tokens":1013,"completion_tokens":3045,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":2981}},"tokens_in":629,"tokens_out":3045,"duration_ms":22937,"temperature":1.0,"reasoning_tokens":2981,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:31:10.971494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly assign users to four otherwise identical bot conditions that differ only in module type, search, chit-chat, games, and storytelling, and compare ratings; if the storytelling advantage over search falls below significance or reverses, the paper's central module-preference conclusion is an artifact of user self-selection.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Alexa Prize deployment and the 1-5 user rating scale from which all quantitative comparisons are drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the winning system's mean rating and turn/duration statistics used as the comparison baseline for SlugBot's overall performance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the SlugBot system architecture and dialogue manager whose per-turn signatures enable module-level analysis."},{"cited_title":"Fox Tree, and Marilyn Walker","cited_arxiv_id":null,"evidence_quote":"Supplies the corpus of personal narratives used to build SlugBot's Storytelling module."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PersonaBank narratives and story intention graphs used as storytelling content."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the conversational analysis that grounds the design assumption that everyday talk exchanges opinions and activities rather than facts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that retrieved-corpus open-domain systems are judged less habitable, motivating the authored-content approach."}],"review_version":1}