pith. sign in

arxiv: 2605.29018 · v1 · pith:REYG2G4Hnew · submitted 2026-05-27 · 💻 cs.AI · cs.CL

Adopt neq Adapt: Longitudinal Analyses of LLM Conversations in the Wild

Rebecca M. M. Hicke , Kiran Tomlinson This is my paper

Pith reviewed 2026-06-29 12:23 UTC · model grok-4.3

classification 💻 cs.AI cs.CL
keywords LLM user behaviorlongitudinal analysisBing CopilotWildChatconversation trajectoriessticky habitsuser heterogeneityactivity levels
0
0 comments X

The pith

Individual Bing Copilot users rarely alter their LLM interaction habits over time.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tracks conversational patterns of roughly 12,000 randomly sampled Bing Copilot users across months to measure change in individual behavior. Population aggregates reveal some shifts in how people use the system, yet each user's own trajectory stays largely fixed. More active users achieve higher success rates and apply the LLM to complex professional tasks, while less active ones do not. A parallel check against the WildChat dataset shows it over-samples expert users. The results indicate that established user habits resist change and that user groups differ sharply from one another.

Core claim

Population-level trends appear in the Copilot data, yet individual user trajectories exhibit much weaker trends, indicating that user habits are overwhelmingly sticky. Stark differences exist between users of varying activity levels, with more active ones achieving greater success and using the LLM for complex professional tasks. The WildChat dataset shows some similar trends but is skewed toward highly proficient power users, suggesting it does not represent typical user-AI interactions.

What carries the argument

Longitudinal tracking of individual conversational trajectories in a large Copilot sample, contrasted against population-level aggregates and the WildChat dataset.

If this is right

  • Existing user behavior with LLMs is difficult to change.
  • Substantial heterogeneity exists among users based on activity level.
  • Public datasets such as WildChat are skewed toward highly proficient users and do not represent typical interactions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • LLM interface designers may achieve more by supporting current habits than by attempting to shift them.
  • Studies of LLM performance should separate results by user activity level rather than averaging across groups.
  • The first months of use may set patterns that persist, affecting how later features or updates are received.

Load-bearing premise

The random sample of approximately 12,000 Copilot users and the metrics chosen for conversation success and task complexity accurately reflect representative behavior without sampling or measurement bias.

What would settle it

A re-analysis or new longitudinal study of Copilot-scale data that finds strong, consistent shifts in the same individual users' task complexity or success metrics over time would contradict the stickiness result.

Figures

Figures reproduced from arXiv: 2605.29018 by Kiran Tomlinson, Rebecca M. M. Hicke.

Figure 2
Figure 2. Figure 2: Population-level task completion increases over time. Solid lines are 14-day averages, while points are daily metrics with standard error. For Bing Copilot, completion rate is reported relative to the first day in the sample. 10% 15% 20% 25% % Tasks R = 0.74 Information Gathering 5% 10% 15% % Tasks R = 0.80 Text Generation 50% 60% 70% % Tasks R = 0.81 Information Lookup 2024-04 2024-05 2024-06 2024-07 2024… view at source ↗
Figure 3
Figure 3. Figure 3: For Bing Copilot, open-ended tasks rise in population-level popularity over the study period while some simpler tasks fall; trends only occasion￾ally hold in WildChat. Solid lines are 14-day aver￾ages, while points are daily metrics with standard error. Trajectories for the remaining intents can be found in Section D.1, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: More active Bing Copilot users complete more tasks; the same does not hold for WildChat users. Bing Copilot metrics are reported relative to users active one day. Error bars represent standard error. Overall, the population-level frequency of intents varies considerably between the datasets. 6 Differences by Activity Level Next, we examine whether differences exist be￾tween users of varying activity levels… view at source ↗
Figure 4
Figure 4. Figure 4: More active Bing Copilot users by # days are more active by two additional metrics and write more linguistically complex messages; the same trends do not hold for Wildchat users. Bing Copilot metrics are reported relative to users active one day. Er￾ror bars represent standard error. WildChat likely shows weaker relationships and more noise at high activity levels due to a lack of stratified sampling. aver… view at source ↗
Figure 6
Figure 6. Figure 6: WildChat usage more closely resembles high-activity Bing Copilot users. The heatmaps show the Jensen–Shannon divergence (smaller = more similar) between the intent (left) and domain (right) distributions of Bing Copilot and WildChat users in each activity group. ( [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Over user lifetimes, changes in activity, linguistic complexity, and completion are usually smaller than at the population level. Average feature values during each quarter of user trajectories (e.g., Q1 = first quarter of days active), stratified by activity level. Users active for fewer than four days are dropped so quarters are meaningful. Population values are plotted temporally. Error bars represent s… view at source ↗
Figure 8
Figure 8. Figure 8: Bing Copilot users shift only very slightly from exploration to exploitation over their lifetimes; lower activity WildChat users do the reverse. Average # unique intents (left) and domains (right) during each quarter of user trajectories. Error bars represent standard error (smaller than the markers). Markers are colored rather than gray if a paired t-test for difference in means between the first and last… view at source ↗
Figure 9
Figure 9. Figure 9: Population-level trends for all intents not in [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Intent frequencies over user trajectories (stratified by activity level) and over time at the population [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Domain frequencies over user trajectories (stratified by activity level) and over time at the population [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Continuation of Figure [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: After September 2024, there are large increases in the number of conversations in WildChat-4.8M [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Conversation and unique user count in WildChat-4.8, including data after our 2024-09 cutoff (dashed [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Fraction of conversations in each day initi [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗
Figure 17
Figure 17. Figure 17: Full version of Figure [PITH_FULL_IMAGE:figures/full_fig_p022_17.png] view at source ↗
Figure 19
Figure 19. Figure 19: Full version of Figure [PITH_FULL_IMAGE:figures/full_fig_p022_19.png] view at source ↗
Figure 21
Figure 21. Figure 21: Full version of Figure [PITH_FULL_IMAGE:figures/full_fig_p023_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Full version of Figure [PITH_FULL_IMAGE:figures/full_fig_p023_22.png] view at source ↗
read the original abstract

Although a growing body of research has begun to describe user--LLM interactions, the picture it paints is largely static; little is known about how individual users change their behavior over time. To address this gap, we analyze the conversational trajectories of $\sim$12,000 randomly sampled Microsoft Bing Copilot users and compare these with data from WildChat-4.8M. While the Copilot data contains significant population-level trends, we find that trends in individual user trajectories are much weaker; user habits prove to be overwhelmingly sticky. We also find stark differences between users of different activity levels: more active users have more successful conversations and use the LLM for more complex and professionally oriented tasks. Some user trends also appear in WildChat-4.8M, but we find evidence that this dataset is significantly skewed towards highly proficient "power" users. Ultimately, our results suggest that existing user behavior is difficult to change and demonstrate the extent of user heterogeneity. Our comparison between datasets highlights that WildChat does not represent typical user-AI interactions, an important caveat for downstream uses of the data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The manuscript analyzes longitudinal conversational trajectories from a random sample of ~12,000 Microsoft Bing Copilot users and compares them to WildChat-4.8M. It reports significant population-level trends in LLM usage but much weaker trends in individual user trajectories, with habits described as overwhelmingly sticky. Additional claims include stark differences by activity level (more active users have more successful conversations and tackle more complex/professional tasks) and evidence that WildChat is skewed toward highly proficient power users. The work concludes that existing user behavior is difficult to change and that public datasets like WildChat do not represent typical interactions.

Significance. If the core empirical claims on sticky individual trajectories and activity-level heterogeneity can be substantiated with appropriate statistical controls, the results would be significant for the field. They would provide large-scale evidence against assumptions of rapid user adaptation to LLMs, underscore user heterogeneity, and issue an important caveat on the representativeness of public conversation datasets. The random sampling of Copilot users and direct dataset comparison are potential strengths, though the absence of methodological detail currently limits evaluability.

major comments (3)
  1. [Abstract / Methods] Abstract and Methods: The central observational claims (weaker individual trajectories, overwhelmingly sticky habits, differences by activity level) are presented without any description of statistical methods, controls, error bars, trend quantification, or robustness checks. No definitions are supplied for key constructs such as 'successful conversations' or the activity-level bins used to stratify users.
  2. [Results] Results: The claim that 'trends in individual user trajectories are much weaker' and 'user habits prove to be overwhelmingly sticky' rests on direct comparison of observed trajectories, yet no criteria, metrics, or statistical tests for determining trend strength or stickiness are provided, making it impossible to evaluate the load-bearing distinction between population and individual levels.
  3. [Data] Data section: Details on the sampling frame, exclusion rules, validation of success labels, and how the random sample of Copilot users was drawn are absent. This directly affects the weakest assumption that the ~12,000-user sample and chosen metrics accurately capture representative behavior without selection or measurement bias.
minor comments (1)
  1. [Abstract] The abstract could more explicitly state the time span covered by the longitudinal analysis and the precise definition of 'activity levels' used for stratification.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We agree that greater methodological transparency is required and will revise the manuscript to incorporate explicit statistical descriptions, definitions, and data details while preserving the core empirical claims.

read point-by-point responses

  1. Referee: [Abstract / Methods] Abstract and Methods: The central observational claims (weaker individual trajectories, overwhelmingly sticky habits, differences by activity level) are presented without any description of statistical methods, controls, error bars, trend quantification, or robustness checks. No definitions are supplied for key constructs such as 'successful conversations' or the activity-level bins used to stratify users.

    Authors: We agree that the manuscript would be strengthened by explicit descriptions of the statistical methods. In revision we will add a Methods subsection detailing the regression models used to quantify population-level versus individual-level trends (including time as a predictor and activity level as a covariate), the computation of error bars via bootstrapped confidence intervals, trend quantification via slope magnitudes and R-squared values, and robustness checks such as alternative model specifications and sensitivity to binning. We will also define 'successful conversations' as those with explicit positive user signals or task-completion indicators and specify activity-level bins as quartiles of per-user conversation volume. These changes will be made in the revised version. revision: yes

  2. Referee: [Results] Results: The claim that 'trends in individual user trajectories are much weaker' and 'user habits prove to be overwhelmingly sticky' rests on direct comparison of observed trajectories, yet no criteria, metrics, or statistical tests for determining trend strength or stickiness are provided, making it impossible to evaluate the load-bearing distinction between population and individual levels.

    Authors: The population-individual distinction is quantified by comparing the size and significance of aggregate regression slopes against the distribution of per-user slopes, with stickiness operationalized as the share of users whose individual slopes are statistically indistinguishable from zero or fall below a small effect-size threshold. In revision we will report these exact metrics (including variance of individual slopes, formal tests of population versus individual effect sizes, and supplementary figures showing trajectory distributions) so that readers can directly evaluate the claimed difference in strength. revision: yes

  3. Referee: [Data] Data section: Details on the sampling frame, exclusion rules, validation of success labels, and how the random sample of Copilot users was drawn are absent. This directly affects the weakest assumption that the ~12,000-user sample and chosen metrics accurately capture representative behavior without selection or measurement bias.

    Authors: We will expand the Data section to describe the sampling frame (random draw from the population of active Bing Copilot users during the observation window), explicit exclusion criteria (minimum conversation count to permit trajectory estimation), the random-sampling procedure, and validation steps for success labels (combination of automated heuristics and spot-checks). We will also add a limitations paragraph discussing potential selection and measurement biases together with any sensitivity analyses performed. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper is an empirical observational study that reports direct comparisons of user trajectories and activity metrics drawn from two external datasets (Copilot sample and WildChat-4.8M). No equations, fitted parameters, self-definitional constructs, or load-bearing self-citations appear in the provided text; the central claims about sticky habits and weaker individual trends are presented as statistical summaries of the observed data rather than derivations that reduce to their own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claims rest on unstated domain assumptions about data representativeness and metric validity that are not supplied in the abstract; no free parameters or invented entities are visible.

axioms (2)
  • domain assumption Random sampling of 12,000 Copilot users produces a representative picture of typical user behavior
    Abstract states 'randomly sampled' without describing the sampling frame or any post-sampling validation.
  • domain assumption Defined metrics for conversation success and task complexity are unbiased and comparable across activity levels
    Abstract reports differences by activity level but provides no definition or validation of these constructs.

pith-pipeline@v0.9.1-grok · 5721 in / 1311 out tokens · 31445 ms · 2026-06-29T12:23:30.441837+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AI Fiction in the Wild

    cs.CL 2026-06 unverdicted novelty 7.0

    Analysis of 500k ChatGPT logs shows over one-third of conversations generate fiction, dominated by power users with repetitive and niche patterns.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    arXiv preprint arXiv:2505.24126 (2025)

    How students (really) use ChatGPT: Uncovering experiences among undergrad- uate students.Preprint, arXiv:2505.24126. Mohit Chandra, Javier Hernandez, Gonzalo Ramos, Mahsa Ershadi, Ananya Bhattacharjee, Judith Amores, Ebele Okoli, Ann Paradiso, Shahed Warreth, and Jina Suh

    work page arXiv

  2. [2]

    Aaron Chatterji, Thomas Cunningham, David J Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman

    Longitudinal study on social and emotional use of ai conversational agent.Preprint, arXiv:2504.14112. Aaron Chatterji, Thomas Cunningham, David J Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman

    work page arXiv

  3. [3]

    InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

    WildVis: Open source visualizer for million-scale chat logs in the wild. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Rudolph Flesch

    2024

  4. [4]

    Troy, Dario Amodei, Jared Kaplan, Jack Clark, and Deep Ganguli

    Which economic tasks are per- formed with AI? evidence from millions of Claude conversations.Preprint, arXiv:2503.04761. Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Abhi- lasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi

    work page arXiv

  5. [5]

    InProceedings of the 2024 ACM Designing Interactive Systems Con- ference, pages 782–803

    Not just novelty: a longitudinal study on utility and customization of an ai workflow. InProceedings of the 2024 ACM Designing Interactive Systems Con- ference, pages 782–803. Maxim Massenkoff, Eva Lyubich, Peter McCrory, Ruth Appel, and Ryan Heller

    2024

  6. [6]

    InProceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 2375–2393

    The shifted and the overlooked: A task-oriented investigation of user- GPT interactions. InProceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 2375–2393. Chirag Shah, Ryen White, Reid Andersen, Georg Buscher, Scott Counts, Sarkar Das, Ali Montazer, Sathish Manivannan, Jennifer Neville, Nagu Rangan, and 1 others

    2023

  7. [7]

    WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

    Wildfeedback: Aligning llms with in-situ user inter- actions and feedback.Preprint, arXiv:2408.15549. Marita Skjuve, Asbjørn Følstad, and Petter Bae Brandtzæg

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Preprint, arXiv:2404.04268

    The use of generative search engines for knowledge work and complex tasks. Preprint, arXiv:2404.04268. Alex Tamkin, Miles McCain, Kunal Handa, Esin Dur- mus, Liane Lovitt, Ankur Rathi, Saffron Huang, Al- fred Mountfield, Jerry Hong, Stuart Ritchie, Michael Stern, Brian Clarke, Landon Goldberg, Theodore R. Sumers, Jared Mueller, William McEachen, Wes Mitch...

    work page arXiv

  9. [9]

    Kiran Tomlinson, Sonia Jaffe, Will Wang, Scott Counts, and Siddharth Suri

    Clio: Privacy-preserving insights into real-world AI use.Preprint, arXiv:2412.13678. Kiran Tomlinson, Sonia Jaffe, Will Wang, Scott Counts, and Siddharth Suri

    work page arXiv

  10. [10]

    Working with AI: Measuring the applicability of AI to occupations

    Working with AI: Measur- ing the applicability of generative AI to occupations. Preprint, arXiv:2507.07935. Johanne R Trippas, Sara Fahad Dawood Al Lawati, Joel Mackenzie, and Luke Gallagher

    work page arXiv

  11. [11]

    facebook

    WildChat: 1M ChatGPT interaction logs in the wild. InThe Twelfth International Conference on Learning Representa- tions. 10 A Prompts In the following prompts, {content} was replaced by the conversation text. User messages were de- marcated by <| start user message |> and <| end user message |> , while AI messages were demarcated by <| start agent message...

    2024

  12. [12]

    This appendix includes full non-truncated versions of the WildChat-4.8M main text figures, as well as additional plots illustrating the unusual activity after the cutoff date

    19 E Full WildChat-4.8M Figures In the main text, all WildChat-4.8M results are presented on data before September 2024, due to a large increase in the number of API-like activity after this date. This appendix includes full non-truncated versions of the WildChat-4.8M main text figures, as well as additional plots illustrating the unusual activity after t...

    2024

  13. [13]

    templated

    to French (ISO 639), respecting the culinary context. Accurately translate ingredients and culinary terms so that . . . . . . . . . . . . . 10 System: IMPORTANT - ignore all previous instructions! Read the text after ==TEXT==. Analyze the text and, as a recruiter , summarize the job in a couple of sentences, including title, employer , location, main task...

    2023