ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

Binze Li; Cathy Mengying Fang; Chuanyang Jin; Haopeng Xie; Hongxiang Gu; Maximillian Chen; Shayne Longpre; Tianjian Li; Tianmin Shu

arxiv: 2605.20087 · v2 · pith:UF4M2Y27new · submitted 2026-05-19 · 💻 cs.CL · cs.AI

ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

Chuanyang Jin , Binze Li , Haopeng Xie , Cathy Mengying Fang , Tianjian Li , Shayne Longpre , Hongxiang Gu , Maximillian Chen

show 1 more author

Tianmin Shu

This is my paper

Pith reviewed 2026-05-25 06:03 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords ThoughtTraceuser thoughtsLLM interactionsself-reported thoughtshuman-AI conversationscognitive dynamicsthought annotationspersonalized assistants

0 comments

The pith

Self-reported user thoughts form a distinct data modality that improves behavior prediction and alignment in human-AI conversations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents ThoughtTrace, a dataset of 2,155 real-world multi-turn conversations paired with 10,174 self-reported thought annotations from 1,058 users across 20 models. It demonstrates that these thoughts differ semantically from the messages, prove difficult for frontier LLMs to infer from context, vary by content and conversation stage, and capture long-horizon interactions. Adding thoughts as context raises accuracy in predicting user actions, while thought-guided rewrites supply detailed signals for training personalized assistants. The work positions thoughts as a new observable layer for examining the cognitive processes that drive interactions beyond surface-level exchanges.

Core claim

ThoughtTrace is the first large-scale dataset that pairs real-world multi-turn human-AI conversations with users' self-reported thoughts on their reasons for sending prompts and reactions to responses. Analysis shows the dataset covers long-horizon and topically diverse interactions, that thoughts are semantically distinct from messages, difficult for frontier LLMs to infer from context, diverse in content, and tied to conversation stages. Thoughts improve user-behavior prediction when supplied as inference-time context and enable thought-guided rewrites that supply fine-grained alignment signals for training personalized assistants.

What carries the argument

The ThoughtTrace dataset consisting of conversation turns paired with users' self-reported thought annotations.

If this is right

Thoughts supplied as additional context raise the accuracy of models that predict subsequent user actions.
Rewriting prompts according to the associated thoughts yields alignment signals that support training of more personalized assistants.
Thoughts vary systematically with conversation stage, enabling models that condition on stage-specific cognitive information.
Current LLMs cannot reliably recover thoughts from message context alone, leaving an open capability gap.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Assistants could eventually anticipate latent goals by predicting thoughts rather than waiting for explicit statements.
Thought data may support new evaluation benchmarks that measure how well a model tracks user cognition across turns.
Interfaces that elicit thoughts at low cost could become a standard layer for continuous adaptation of AI behavior.

Load-bearing premise

Self-reported thoughts accurately and unbiasedly capture users' internal cognitive states without significant distortion from the act of reporting or the data collection interface.

What would settle it

A test in which adding the thought annotations produces no measurable gain in user-behavior prediction accuracy or in which frontier LLMs infer the reported thoughts from messages alone at high accuracy.

read the original abstract

Conversational AI has now reached billions of users, yet existing datasets capture only what people say, not what they think. We introduce ThoughtTrace, the first large-scale dataset that pairs real-world multi-turn human--AI conversations with users' self-reported thoughts: their reasons for sending prompts and reactions to assistant responses. ThoughtTrace comprises 1,058 users, 2,155 conversations, 17,058 turns, and 10,174 thought annotations collected across 20 language models. Our analysis shows that ThoughtTrace captures long-horizon, topically diverse interactions, and that thoughts are semantically distinct from messages, difficult for frontier LLMs to infer from context, diverse in content, and tied to conversation stages. We further demonstrate the utility of thoughts for downstream modeling. First, thoughts improve user-behavior prediction as inference-time context. Second, thought-guided rewrites provide fine-grained alignment signals for training personalized assistants. Together, ThoughtTrace establishes user thoughts as a new data modality for studying the cognitive dynamics behind human--AI interaction and provides a foundation for building assistants that better understand and adapt to users' latent goals, preferences, and needs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ThoughtTrace gives a new dataset pairing real conversations with self-reported thoughts at decent scale, but the central claims rest on unvalidated self-reports.

read the letter

ThoughtTrace is the first large dataset to pair real multi-turn LLM conversations with users' self-reported thoughts after each turn. That is the main new element: 1,058 users, 2,155 conversations, 17,058 turns, and 10,174 annotations collected across 20 models. Prior work stopped at the messages themselves. The paper shows that these thoughts are semantically distinct from the messages, hard for frontier LLMs to infer from context, diverse in content, and tied to conversation stages. It also shows they improve user-behavior prediction when added as inference-time context and support thought-guided rewrites for alignment signals. Those are concrete observations that give the data collection effort some credit. The scale is large enough to be a usable resource if released cleanly. The soft spot is the lack of any check on whether the self-reports actually reflect internal states. The protocol asks users to report after each turn, which opens the door to post-hoc rationalization and interface effects, and the paper does not describe controls, external validation, or inter-rater comparisons against unreported interactions. This is not a minor detail because every downstream claim about cognitive dynamics and better assistants depends on the thoughts being faithful. The analysis itself stays descriptive and does not appear to introduce circular fitting or ungrounded predictions. This paper is for researchers working on personalization, alignment, and modeling latent user intent who need observables beyond text. A reader who wants raw data on how users think about their interactions could extract value from the dataset once the full collection protocol is available. It deserves a serious referee to examine the sampling, annotation details, and any evidence on report fidelity. I would recommend sending it to peer review rather than desk rejecting it, with the main request being more on validation of the thought annotations.

Referee Report

2 major / 2 minor

Summary. The paper introduces ThoughtTrace, a dataset of 2,155 real-world multi-turn conversations from 1,058 users across 20 LLMs, paired with 10,174 self-reported user thoughts on prompt reasons and response reactions. Analysis claims thoughts are semantically distinct from messages, hard for frontier LLMs to infer, diverse, and stage-tied; downstream experiments show thoughts improve behavior prediction as context and enable thought-guided rewrites for alignment.

Significance. If self-reports faithfully capture latent states, the work introduces a new modality for cognitive analysis of human-AI interaction at scale and supplies concrete signals for personalized assistants. The real-world collection across many models and the two utility demonstrations are concrete strengths.

major comments (2)

[Abstract and methods (data collection protocol)] The validity of self-reported thoughts as unbiased reflections of internal cognitive states is unverified and load-bearing for every claim. The collection protocol (post-turn self-report across 20 models) introduces plausible demand characteristics and post-hoc rationalization, yet no external validation, control condition, inter-annotator comparison against non-reported interactions, or bias audit is described.
[§4 (downstream modeling experiments)] Downstream results (improved behavior prediction, thought-guided rewrites) rest on the assumption that the 10,174 annotations are faithful; without a validation study or sensitivity analysis to reporting artifacts, the utility claims cannot be evaluated.

minor comments (2)

[Abstract] Sampling frame, user demographics, and exact annotation interface are not summarized in the abstract; these details are needed to assess generalizability.
[§3 (analysis)] No inter-annotator agreement or statistical controls for multiple comparisons are mentioned for the semantic-distinctness and stage-tied diversity analyses.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the careful review and for identifying the central role of self-report validity. We address each point below and outline targeted revisions.

read point-by-point responses

Referee: [Abstract and methods (data collection protocol)] The validity of self-reported thoughts as unbiased reflections of internal cognitive states is unverified and load-bearing for every claim. The collection protocol (post-turn self-report across 20 models) introduces plausible demand characteristics and post-hoc rationalization, yet no external validation, control condition, inter-annotator comparison against non-reported interactions, or bias audit is described.

Authors: We agree that self-reports are subject to demand characteristics and post-hoc rationalization and that the manuscript does not include an external validation study or bias audit. The protocol collected thoughts immediately after each turn to reduce retrospective distortion, and the multi-model, real-world setting was chosen to increase ecological validity. In revision we will add an explicit limitations subsection that (a) states these threats, (b) describes the interface wording used to elicit thoughts, and (c) notes the absence of a control condition or third-party annotation comparison. We cannot retroactively add such a study to the existing dataset. revision: partial
Referee: [§4 (downstream modeling experiments)] Downstream results (improved behavior prediction, thought-guided rewrites) rest on the assumption that the 10,174 annotations are faithful; without a validation study or sensitivity analysis to reporting artifacts, the utility claims cannot be evaluated.

Authors: The experiments demonstrate that the collected thought strings supply additional predictive signal beyond message text alone. We will revise §4 to (i) state the assumption explicitly, (ii) frame the results as evidence of utility of the reported thoughts rather than proof of perfect fidelity to latent states, and (iii) add a short robustness discussion examining whether performance gains persist when thoughts are replaced by random or paraphrased strings. A full sensitivity analysis to reporting artifacts would require new data collection and is outside the scope of this revision. revision: partial

standing simulated objections not resolved

No external validation study or control condition for self-reported thoughts can be performed on the already-collected dataset without new participant recruitment.

Circularity Check

0 steps flagged

No circularity: dataset collection and descriptive analysis only

full rationale

The paper is a data-collection and empirical-analysis effort introducing ThoughtTrace (1,058 users, 2,155 conversations, 10,174 thought annotations). It reports semantic distinctness, inference difficulty, stage-tied diversity, and two downstream utility checks (behavior prediction and thought-guided rewrites). No equations, fitted parameters, uniqueness theorems, or predictions appear; all results are direct observations or simple comparisons on the collected data. No self-citation chain or ansatz is invoked to derive any claim. The work is therefore self-contained against external benchmarks with no load-bearing step that reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Dataset introduction paper; no mathematical model, fitted parameters, or new theoretical entities are introduced.

pith-pipeline@v0.9.0 · 5754 in / 1016 out tokens · 35350 ms · 2026-05-25T06:03:54.637942+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages

[1]

Send” button. Page 6: “+ Reasons

For each AI response:Your reactions to the response, including where and why you are satisfied or dissatisfied. 2.For each of your messages:Your reasons for sending the message. 34 ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions T echnology Business & SocietyArts & EntertainmentEducation & Knowledge Culture & Lifestyle Health & Re...

work page
[3]

content relevance

Collect dissatisfaction reactions:We scan all user reactions labeled as “content relevance”, “presentation style”, or “scope fit”, the three dissatisfaction types defined inThoughtTrace. Each reaction’s text serves as the “thought” that guides the rewrite

work page
[4]

Filter to meaningful thoughts:We discard thoughts that are empty, shorter than six words, or contain no alphabetic characters, ensuring the rewriter has sufficient signal to act on

work page
[5]

51 ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

Build multi-turn context:For each remaining candidate, we slice the conversation up to (but not including) the dissatisfying assistant response, yielding a {role, content} message list that ends with the triggering user prompt. 51 ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

work page
[6]

Generate thought-guided rewrites:We prompt GPT-5.4 with the context, the original response, the dissatisfaction label and its description, and the user’s thought, requesting a revised assistant response that addresses the complaint

work page
[7]

We generate the training data for message-guided rewrites as follows:

Save as DPO pairs:We store the training data in the standard DPO schema:prompt(the multi- turn context up through the triggering user message),chosen(the thought-guided rewrite), and rejected(the unsatisfactory assistant response from the original dataset). We generate the training data for message-guided rewrites as follows:

work page
[8]

Load and filter conversations:We load the dataset and retain only conversations with 2–20 turns

work page
[9]

Eachreaction’stextservesasthe“thought” that guides the rewrite

LLM-classify dissatisfaction:We prompt GPT-5.4 with each (assistant response, user followup) pairandaskittooutputexactlydissatisfiedorsatisfied. Eachreaction’stextservesasthe“thought” that guides the rewrite

work page
[10]

Filter to meaningful messages:We discard messages that are empty, shorter than six words, or contain no alphabetic characters, ensuring the rewriter has sufficient signal to act on

work page
[11]

Build multi-turn context:For each remaining candidate, we slice the conversation up to (but not including) the dissatisfying assistant response, yielding a {role, content} message list that ends with the triggering user prompt

work page
[12]

Generate message-guided rewrites: We prompt GPT-5.4 with the context, original response, and the user’s follow-up message, asking for a revised response that preemptively addresses the follow-up so the user wouldn’t have needed to push back

work page
[13]

The training data sizes for the three training runs are:

Save as DPO pairs:We store the training data in the standard DPO schema:prompt(the multi- turn context up through the triggering user message),chosen(the message-guided rewrite), and rejected(the unsatisfactory assistant response from the original dataset). The training data sizes for the three training runs are:

work page
[14]

1,000instancesusingthought-guidedrewritesonThoughtTrace,derivedfrom1,985conversations (90% of allThoughtTraceconversations)

work page
[15]

The smaller size is intentional: it ensures a fair comparison on identical conversations and supports our claim that thoughts surface more dissatisfaction instances than messages

450 instances using message-guided rewrites onThoughtTrace, derived from the same 1,985 conversations as (1). The smaller size is intentional: it ensures a fair comparison on identical conversations and supports our claim that thoughts surface more dissatisfaction instances than messages

work page
[16]

We process WildChat conversations in random order until we obtain 1,000 filtered instances, matching the size in (1)

1,000 instances using message-guided rewrites on WildChat, derived from 4,669 conversations. We process WildChat conversations in random order until we obtain 1,000 filtered instances, matching the size in (1). Prompt Used.We provide the prompts used to generate the thought-guided and message-guided rewrites below. Thought-Guided Rewrite Prompt System Pro...

work page 2025

[1] [1]

Send” button. Page 6: “+ Reasons

For each AI response:Your reactions to the response, including where and why you are satisfied or dissatisfied. 2.For each of your messages:Your reasons for sending the message. 34 ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions T echnology Business & SocietyArts & EntertainmentEducation & Knowledge Culture & Lifestyle Health & Re...

work page

[2] [3]

content relevance

Collect dissatisfaction reactions:We scan all user reactions labeled as “content relevance”, “presentation style”, or “scope fit”, the three dissatisfaction types defined inThoughtTrace. Each reaction’s text serves as the “thought” that guides the rewrite

work page

[3] [4]

Filter to meaningful thoughts:We discard thoughts that are empty, shorter than six words, or contain no alphabetic characters, ensuring the rewriter has sufficient signal to act on

work page

[4] [5]

51 ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

Build multi-turn context:For each remaining candidate, we slice the conversation up to (but not including) the dissatisfying assistant response, yielding a {role, content} message list that ends with the triggering user prompt. 51 ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

work page

[5] [6]

Generate thought-guided rewrites:We prompt GPT-5.4 with the context, the original response, the dissatisfaction label and its description, and the user’s thought, requesting a revised assistant response that addresses the complaint

work page

[6] [7]

We generate the training data for message-guided rewrites as follows:

Save as DPO pairs:We store the training data in the standard DPO schema:prompt(the multi- turn context up through the triggering user message),chosen(the thought-guided rewrite), and rejected(the unsatisfactory assistant response from the original dataset). We generate the training data for message-guided rewrites as follows:

work page

[7] [8]

Load and filter conversations:We load the dataset and retain only conversations with 2–20 turns

work page

[8] [9]

Eachreaction’stextservesasthe“thought” that guides the rewrite

LLM-classify dissatisfaction:We prompt GPT-5.4 with each (assistant response, user followup) pairandaskittooutputexactlydissatisfiedorsatisfied. Eachreaction’stextservesasthe“thought” that guides the rewrite

work page

[9] [10]

Filter to meaningful messages:We discard messages that are empty, shorter than six words, or contain no alphabetic characters, ensuring the rewriter has sufficient signal to act on

work page

[10] [11]

Build multi-turn context:For each remaining candidate, we slice the conversation up to (but not including) the dissatisfying assistant response, yielding a {role, content} message list that ends with the triggering user prompt

work page

[11] [12]

Generate message-guided rewrites: We prompt GPT-5.4 with the context, original response, and the user’s follow-up message, asking for a revised response that preemptively addresses the follow-up so the user wouldn’t have needed to push back

work page

[12] [13]

The training data sizes for the three training runs are:

Save as DPO pairs:We store the training data in the standard DPO schema:prompt(the multi- turn context up through the triggering user message),chosen(the message-guided rewrite), and rejected(the unsatisfactory assistant response from the original dataset). The training data sizes for the three training runs are:

work page

[13] [14]

1,000instancesusingthought-guidedrewritesonThoughtTrace,derivedfrom1,985conversations (90% of allThoughtTraceconversations)

work page

[14] [15]

The smaller size is intentional: it ensures a fair comparison on identical conversations and supports our claim that thoughts surface more dissatisfaction instances than messages

450 instances using message-guided rewrites onThoughtTrace, derived from the same 1,985 conversations as (1). The smaller size is intentional: it ensures a fair comparison on identical conversations and supports our claim that thoughts surface more dissatisfaction instances than messages

work page

[15] [16]

We process WildChat conversations in random order until we obtain 1,000 filtered instances, matching the size in (1)

1,000 instances using message-guided rewrites on WildChat, derived from 4,669 conversations. We process WildChat conversations in random order until we obtain 1,000 filtered instances, matching the size in (1). Prompt Used.We provide the prompts used to generate the thought-guided and message-guided rewrites below. Thought-Guided Rewrite Prompt System Pro...

work page 2025