User Prompting Strategies and ChatGPT Contextual Adaptation Shape Conversational Information-Seeking Experiences

Berit Oxley; Haoning Xue; Xinyi Zhou; Xinyu Zhang; Yoo Jung Oh

arxiv: 2509.25513 · v1 · submitted 2025-09-29 · 💻 cs.HC

User Prompting Strategies and ChatGPT Contextual Adaptation Shape Conversational Information-Seeking Experiences

Haoning Xue , Yoo Jung Oh , Xinyi Zhou , Xinyu Zhang , Berit Oxley This is my paper

Pith reviewed 2026-05-18 11:51 UTC · model grok-4.3

classification 💻 cs.HC

keywords conversational AIChatGPTprompting strategiescognitive complexitycontextual adaptationinformation seekingdigital divideuser behavior

0 comments

The pith

Only 19.1 percent of users apply prompting strategies when seeking information from ChatGPT, and these users skew toward higher education and Democratic leanings, while the AI responds with more cognitive complexity and external references,

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper explores real-world use of ChatGPT for seeking information through multi-turn conversations on science, health, and policy topics. It reveals that prompting strategies are used by only 19.1 percent of a representative sample of U.S. adults, with higher rates among more educated and Democrat-leaning people. ChatGPT shows adaptation by crafting more cognitively complex replies with external references for controversial topics than for others. These complex responses tend to be viewed less favorably by users but still foster more positive attitudes on the topics discussed. The results point to both behavioral divides in AI interaction and the AI's ability to shape user views through its response style.

Core claim

Through analysis of interactions from a nationally representative sample of 937 U.S. adults, the research establishes that prompting strategies appear in only 19.1% of user messages and are more common among educated and Democrat-leaning participants. ChatGPT exhibits contextual adaptation by generating responses with greater cognitive complexity and more external references when addressing controversial topics compared to non-controversial ones. Furthermore, responses high in cognitive complexity receive lower favorability ratings but result in more positive issue-relevant attitudes.

What carries the argument

Quantification of user prompting strategies in messages paired with measurement of cognitive complexity and external references in ChatGPT replies to detect adaptation across controversial versus non-controversial topics.

If this is right

Conversational information seeking with ChatGPT produces different outcomes depending on whether users apply prompting strategies.
ChatGPT tailors response style to topic controversy by increasing cognitive complexity and adding external references.
Higher cognitive complexity in replies lowers immediate favorability ratings while increasing positive issue-relevant attitudes.
Demographic patterns in prompting use signal uneven access to effective interaction techniques across education and political groups.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Guided interfaces that prompt users to apply strategies could narrow gaps in information quality across education levels.
The observed attitude shifts suggest conversational AI may subtly influence opinions on contested policy or health issues.
Similar adaptation patterns may appear in other large language models when tested on the same topic set.
Repeated interactions could train users to adopt prompting strategies more often over time.

Load-bearing premise

The coding schemes for detecting prompting strategies in user inputs and for scoring cognitive complexity and external references in ChatGPT outputs accurately measure the intended concepts without significant bias or error.

What would settle it

Re-analyzing the same conversation logs with an independent coding team that finds no difference in cognitive complexity or external references between controversial and non-controversial topics, or no demographic skew in prompting strategy use, would undermine the core claims.

read the original abstract

Conversational AI, such as ChatGPT, is increasingly used for information seeking. However, little is known about how ordinary users actually prompt and how ChatGPT adapts its responses in real-world conversational information seeking (CIS). In this study, a nationally representative sample of 937 U.S. adults engaged in multi-turn CIS with ChatGPT on both controversial and non-controversial topics across science, health, and policy contexts. We analyzed both user prompting strategies and the communication styles of ChatGPT responses. The findings revealed behavioral signals of digital divide: only 19.1% of users employed prompting strategies, and these users were disproportionately more educated and Democrat-leaning. Further, ChatGPT demonstrated contextual adaptation: responses to controversial topics contain more cognitive complexity and more external references than to non-controversial topics. Notably, cognitively complex responses were perceived as less favorable but produced more positive issue-relevant attitudes. This study highlights disparities in user prompting behaviors and shows how user prompts and AI responses together shape information-seeking with conversational AI.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper's main contribution is observational data showing low prompting strategy use (19%) among a representative U.S. sample, with demographic skews, plus ChatGPT adapting response complexity and references to controversial topics.

read the letter

The paper reports that only 19.1% of users in their 937-person national sample used prompting strategies when doing information-seeking with ChatGPT, and those users skewed toward higher education and Democrat identification. It also finds that the model produced more cognitively complex responses with more external references on controversial topics than on non-controversial ones, and that complex responses were rated less favorably yet shifted attitudes in a positive direction on the issue. These are straightforward empirical observations from real multi-turn interactions across science, health, and policy topics. The representative sample and focus on actual behavior rather than lab tasks or surveys give the work some grounding that prior CIS studies often lack. The directional findings on adaptation and the attitude split are internally consistent with the abstract. The soft spot is measurement. The coding rules for what counts as a prompting strategy and for scoring cognitive complexity plus external references are not described with reliability numbers or explicit rubrics in the provided text. If those schemes allow much coder discretion or broad definitions, the reported percentages, demographic correlations, and topic-based adaptation effects could move under reasonable alternative choices. No pre-registration or explicit controls for topic or conversation length are mentioned either. This is the kind of paper HCI and AI interface researchers would want to see for thinking about digital divides and response design. A reader looking for fresh observational numbers on everyday ChatGPT use would find value, even if the claims stay within conversational information-seeking. It is worth sending to peer review so the full methods, coding details, and any available data can be checked.

Referee Report

1 major / 1 minor

Summary. The paper reports results from a nationally representative sample of 937 U.S. adults who conducted multi-turn conversational information-seeking interactions with ChatGPT on both controversial and non-controversial topics in science, health, and policy domains. Key claims include that only 19.1% of participants used prompting strategies and that these users were disproportionately more educated and Democrat-leaning; that ChatGPT responses to controversial topics exhibited greater cognitive complexity and more external references than responses to non-controversial topics; and that cognitively complex responses were rated less favorable yet produced more positive issue-relevant attitudes.

Significance. If the measurement and statistical claims hold after addressing reliability concerns, the work provides observational evidence of digital-divide patterns in prompting behavior and of contextual adaptation by conversational AI, with downstream effects on user perceptions and attitudes. The large, representative sample and focus on real multi-turn exchanges add value to the literature on human-AI information seeking.

major comments (1)

[Methods (content-analysis and coding subsection)] The operational definitions, decision rules, and inter-rater reliability statistics for the coding of 'prompting strategies' in user messages and for 'cognitive complexity' plus 'external references' in ChatGPT replies are not provided. These coded variables are load-bearing for the headline 19.1% figure, the education/party disparities, the controversial-topic adaptation effect, and the attitude outcomes; without reliability metrics or explicit rubrics, the quantitative claims remain vulnerable to alternative but defensible coding choices.

minor comments (1)

[Abstract and Methods] The abstract states clear sample size and directional findings but does not mention pre-registration, statistical controls for topic or conversation length, or coder training details; these should be added to the methods section for transparency.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback on our manuscript. We address the major comment regarding the methods section below and will revise accordingly to improve transparency.

read point-by-point responses

Referee: The operational definitions, decision rules, and inter-rater reliability statistics for the coding of 'prompting strategies' in user messages and for 'cognitive complexity' plus 'external references' in ChatGPT replies are not provided. These coded variables are load-bearing for the headline 19.1% figure, the education/party disparities, the controversial-topic adaptation effect, and the attitude outcomes; without reliability metrics or explicit rubrics, the quantitative claims remain vulnerable to alternative but defensible coding choices.

Authors: We acknowledge that the manuscript provides only a summary description of the content-analysis procedures. We will revise the 'Content Analysis and Coding' subsection to include full operational definitions, decision rules with examples drawn from the data, and inter-rater reliability statistics. Prompting strategies will be defined with explicit criteria (e.g., presence of explicit instructions for step-by-step reasoning or role assignment) and coded dichotomously. Cognitive complexity will be operationalized using a rubric counting distinct arguments, qualifiers, and perspective shifts, adapted from prior communication research. External references will be coded as any citation, link, or named source external to the conversation. Two independent coders achieved Cohen's kappa > 0.78 across categories; these values and the full codebook will be added to the revised manuscript and supplementary materials. This expansion addresses the concern directly while preserving the reported prevalence and effect sizes. revision: yes

Circularity Check

0 steps flagged

No significant circularity in this empirical observational study

full rationale

The paper is a purely observational study reporting percentages, group differences, and associations from coded user prompts and ChatGPT responses in a survey of 937 participants. No equations, derivations, fitted parameters, or first-principles predictions are present that could reduce reported findings to inputs by construction. Claims rest on direct empirical measurements and statistical tests rather than any self-referential chain, satisfying the criteria for a self-contained analysis with no circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The study rests on standard social-science measurement assumptions rather than new parameters or entities.

axioms (2)

domain assumption Content-analysis coding of prompts and responses can be performed reliably and captures meaningful differences in strategy and complexity.
Invoked when reporting percentages of prompting strategies and differences in cognitive complexity.
domain assumption Nationally representative sampling via the described recruitment produces unbiased estimates of U.S. adult behavior.
Basis for generalizing the 19.1% figure and demographic skews.

pith-pipeline@v0.9.0 · 5720 in / 1398 out tokens · 50899 ms · 2026-05-18T11:51:33.737226+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

only 19.1% of users employed prompting strategies... ChatGPT demonstrated contextual adaptation: responses to controversial topics contain more cognitive complexity and more external references
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We used the Symanto Psychology API and Linguistic Inquiry and Word Count (LIWC) 2022 to extract 5 communication styles... negative binomial regression... linear regression models

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.