pith. sign in

arxiv: 2604.18590 · v1 · submitted 2026-03-18 · 💻 cs.HC · cs.CY

Critical Thinking in the Age of Artificial Intelligence: A Survey-Based Study with Machine Learning Insights

M Murshidul Bari , Akif Islam , Mohd Ruhul Ameen , Abu Saleh Musa Miah , Jungpil Shin This is my paper

Pith reviewed 2026-05-15 09:23 UTC · model grok-4.3

classification 💻 cs.HC cs.CY
keywords artificial intelligencecritical thinkinghuman-AI collaborationsurvey studycognitive offloadingreasoning performanceuser clustering
0
0 comments X

The pith

AI affects critical thinking based on usage patterns rather than uniformly harming or helping it.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper surveys users about their AI habits and tests them on short logic tasks to see how dependence relates to reasoning ability. It shows that people often value AI for speed and support, yet those who lean on it heavily report less patience for effort and score lower on the tasks. Clustering the data reveals distinct groups: over-reliant users, mixed-strategy users, and balanced ones who still verify and reflect. The core point is that outcomes hinge on whether AI substitutes for thinking or assists it. This matters as AI spreads into schools and jobs, because it suggests simple exposure is less decisive than the habits that form around it.

Core claim

The paper finds that AI does not affect critical thinking in a uniformly negative or positive way. Instead its influence depends on the manner in which it is used. Participants largely viewed AI as a tool for speed, convenience, and learning support, yet many also reported reduced patience for sustained effort. Objective reasoning performance varied considerably across individuals, and reduced patience plus stronger dependence tendencies were more closely associated with lower reasoning performance than background characteristics alone. Exploratory clustering indicates that AI users fall into tentative behavioral profiles, including over-reliant users, mixed-strategy users, and balanced ones

What carries the argument

Interview-based survey of AI-use behaviors paired with short logic and reasoning tasks, followed by clustering to separate user profiles.

If this is right

  • Over-reliant users show lower reasoning performance than mixed or balanced users.
  • Reduced patience for effort tracks more closely with weaker task results than demographics do.
  • Effective human-AI work requires built-in support for reflection and verification.
  • AI should be positioned to assist rather than replace sustained cognitive effort.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • AI interfaces could be redesigned to prompt users to verify or extend answers instead of accepting them outright.
  • Training programs might teach explicit strategies for using AI without losing independent reasoning habits.
  • Longer-term tracking of real tasks rather than brief puzzles could test whether the observed patterns hold outside lab settings.

Load-bearing premise

Self-reported AI-use behaviors and scores on short logic tasks accurately reflect real-world critical thinking ability without significant social-desirability bias or task-specific limits.

What would settle it

A study that logs actual daily AI interactions over several months and compares them against independent, repeated measures of critical thinking to check whether high-dependence patterns predict measurable drops in reasoning performance.

Figures

Figures reproduced from arXiv: 2604.18590 by Abu Saleh Musa Miah, Akif Islam, Jungpil Shin, M Murshidul Bari, Mohd Ruhul Ameen.

Figure 2
Figure 2. Figure 2: Error rates across the seven critical-thinking questions. The machine [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Correlation heatmap of selected behavioral features and Critical Thinking Score (CTS). Negative values indicate inverse relationships with reasoning [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: PCA projection of the K-Means behavioral user profiles. The three [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Random Forest feature importance scores for predicting Critical [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
read the original abstract

The growing use of artificial intelligence (AI) in education, professional work, and everyday problem-solving has raised important questions about its effect on human reasoning. While AI can improve efficiency, save time, and support learning, repeated dependence on it may also encourage cognitive offloading, reduce productive struggle, and weaken independent critical thinking. This paper investigates the relationship between AI-use behavior and critical-thinking performance through an interview-based survey combined with short logic and reasoning tasks. The findings reveal a mixed pattern: participants largely viewed AI as a tool for speed, convenience, and learning support, yet many also reported reduced patience for sustained effort. Objective reasoning performance varied considerably across individuals, and the analyses suggest that reduced patience and stronger dependence-related tendencies are more closely associated with lower reasoning performance than background characteristics alone. Exploratory clustering further indicates that AI users do not form a single homogeneous group, but instead reflect tentative behavioral profiles, including over-reliant users, mixed-strategy users, and balanced support-seekers. Although the findings are exploratory, they indicate that AI does not affect critical thinking in a uniformly negative or positive way. Instead, its influence appears to depend on the manner in which it is used. The paper therefore argues that effective human-AI collaboration should support reflection, verification, and sustained cognitive effort rather than substitute for them.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper reports results from an interview-based survey paired with short logic and reasoning tasks examining how AI-use behaviors relate to critical-thinking performance. It describes mixed patterns in which participants view AI as useful for speed and learning support yet report reduced patience for sustained effort; statistical associations link stronger dependence tendencies and lower patience to poorer task performance more than background variables alone. Exploratory clustering identifies tentative user profiles (over-reliant, mixed-strategy, and balanced support-seekers). The central claim is that AI does not exert a uniform effect on critical thinking; its influence depends on the manner of use, and effective human-AI collaboration should therefore promote reflection, verification, and cognitive effort rather than substitution.

Significance. If the reported associations between self-reported dependence, patience reduction, and reasoning-task scores prove robust after addressing measurement and confounding issues, the work supplies useful exploratory evidence that AI effects on cognition are usage-dependent rather than categorically positive or negative. This perspective could usefully inform the design of educational AI tools and HCI guidelines that scaffold rather than supplant reflective thinking. The clustering component adds a modest methodological contribution by illustrating heterogeneous user profiles, though its value remains limited by the exploratory framing and lack of external validation.

major comments (3)
  1. [Methods] Methods section (survey and task description): The short, decontextualized logic and reasoning tasks are presented as proxies for critical thinking, yet no validation against established instruments (e.g., Watson-Glaser or Halpern) or evidence that task scores predict real-world outcomes is reported. This directly weakens the load-bearing claim that dependence and patience reduction are associated with lower critical-thinking performance.
  2. [Results] Results / statistical analysis: The assertion that reduced patience and dependence tendencies are more closely associated with lower performance than background characteristics alone requires explicit reporting of the regression or correlation models, including controls for confounders such as education level, motivation, and task familiarity, together with effect sizes and confidence intervals. These details are absent from the abstract and not referenced in the provided summary.
  3. [Clustering Analysis] Clustering subsection: The identification of three behavioral profiles rests on an unspecified choice of cluster number (a free parameter). The manuscript must report the selection criterion (e.g., elbow, silhouette), validation metrics, and stability checks; without them the profiles remain too tentative to support the usage-manner dependence conclusion.
minor comments (2)
  1. [Abstract] Abstract: The phrase 'machine learning insights' is used but the only ML component described is exploratory clustering; a brief clarification of the specific technique would improve precision.
  2. [Abstract] Presentation: Sample size, response rate, and any power considerations for the reported associations are not mentioned in the abstract or summary; adding these would aid evaluation of the mixed-pattern findings.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for the constructive and detailed comments, which highlight important areas for strengthening the transparency and rigor of our exploratory study. We address each major comment below and will incorporate revisions to improve the manuscript accordingly.

read point-by-point responses

  1. Referee: [Methods] Methods section (survey and task description): The short, decontextualized logic and reasoning tasks are presented as proxies for critical thinking, yet no validation against established instruments (e.g., Watson-Glaser or Halpern) or evidence that task scores predict real-world outcomes is reported. This directly weakens the load-bearing claim that dependence and patience reduction are associated with lower critical-thinking performance.

    Authors: We agree that the tasks function as proxies for specific reasoning skills rather than validated measures of critical thinking and that no formal validation against instruments such as the Watson-Glaser or Halpern Critical Thinking Assessment was conducted. The tasks were adapted from standard logic and inference items to allow objective scoring within the interview setting. In the revised manuscript we will expand the Methods section with additional detail on item construction and add a Limitations subsection that explicitly acknowledges the lack of external validation and the absence of evidence linking task scores to real-world critical-thinking outcomes. Claims will be tempered to reflect the exploratory nature of the performance measure. revision: yes

  2. Referee: [Results] Results / statistical analysis: The assertion that reduced patience and dependence tendencies are more closely associated with lower performance than background characteristics alone requires explicit reporting of the regression or correlation models, including controls for confounders such as education level, motivation, and task familiarity, together with effect sizes and confidence intervals. These details are absent from the abstract and not referenced in the provided summary.

    Authors: The manuscript contains regression models examining these associations with controls for background variables, but we acknowledge that the reporting was insufficiently detailed and not highlighted. In the revision we will insert a new Results subsection titled 'Regression Analyses' that presents the full models, including all coefficients, standard errors, 95% confidence intervals, effect sizes (R² and incremental R²), and explicit controls for education level, prior AI experience, self-reported motivation, and task familiarity. Correlation matrices will also be added to support the relative strength of the associations. revision: yes

  3. Referee: [Clustering Analysis] Clustering subsection: The identification of three behavioral profiles rests on an unspecified choice of cluster number (a free parameter). The manuscript must report the selection criterion (e.g., elbow, silhouette), validation metrics, and stability checks; without them the profiles remain too tentative to support the usage-manner dependence conclusion.

    Authors: The number of clusters (k=3) was selected using the elbow method combined with interpretability of the resulting profiles. In the revised Clustering subsection we will report the elbow criterion values, silhouette scores for k=2 through 5 (with the chosen solution having an average silhouette width of 0.58), and stability assessed through multiple random initializations and bootstrap resampling showing consistent profile membership. These additions will make the procedure transparent while preserving the exploratory framing of the profiles. revision: yes

standing simulated objections not resolved
  • We cannot supply post-hoc empirical validation of the reasoning tasks against established critical-thinking instruments or evidence of real-world predictive validity, as this would require a separate validation study outside the scope of the current revision.

Circularity Check

0 steps flagged

No circularity: purely empirical survey with data-driven clustering

full rationale

The paper reports results from an interview-based survey, self-reported AI-use behaviors, short logic/reasoning tasks, and exploratory clustering of participants into behavioral profiles. No equations, fitted parameters, or derivations appear in the provided text or abstract. Central claims rest on observed associations between reported dependence/patience and task performance, without any step that reduces a prediction or conclusion to its own inputs by construction. Self-citations, if present, are not load-bearing for the core findings, which remain externally falsifiable via replication of the survey and tasks. This is a standard empirical study whose conclusions do not collapse into self-definition or renaming.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on standard survey assumptions that self-reports and short tasks validly proxy critical thinking and AI dependence; no free parameters beyond exploratory cluster count and no invented entities.

free parameters (1)
  • Number of clusters
    Exploratory clustering produced three user profiles (over-reliant, mixed-strategy, balanced); count chosen to fit observed behavioral patterns.
axioms (1)
  • domain assumption Participants' self-reports of AI-use behavior and performance on short logic tasks accurately reflect underlying critical-thinking ability and dependence levels
    This assumption directly links reported patience/dependence to measured reasoning performance and underpins the clustering interpretation.

pith-pipeline@v0.9.0 · 5553 in / 1152 out tokens · 40340 ms · 2026-05-15T09:23:12.473554+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [1]

    Guidance for generative ai in education and research,

    UNESCO, “Guidance for generative ai in education and research,” Paris, 2023, human-centred guidance on opportunities, risks, policy, and instructional design for generative AI in education

    work page 2023

  2. [2]

    Generative ai: is it a paradigm shift for higher education?

    X. O’Dea, “Generative ai: is it a paradigm shift for higher education?” Studies in Higher Education, 2024

    work page 2024

  3. [3]

    Generative ai’s impact on critical thinking,

    C. Gonsalves, “Generative ai’s impact on critical thinking,”Journalism & Mass Communication Educator, 2025

    work page 2025

  4. [4]

    Cognitive offloading,

    E. F. Risko and S. J. Gilbert, “Cognitive offloading,”Trends in Cognitive Sciences, vol. 20, no. 9, pp. 676–688, 2016

    work page 2016

  5. [5]

    Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction: Research findings and recommendations,

    P. A. Facione, “Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction: Research findings and recommendations,” American Philosophical Association, ERIC Re- port ED315423, 1990

    work page 1990

  6. [6]

    University teachers’ beliefs about the use of generative artificial intelligence in education,

    B. Cabelloset al., “University teachers’ beliefs about the use of generative artificial intelligence in education,”Frontiers in Psychology, vol. 15, 2024

    work page 2024

  7. [7]

    The influence of ai text generators on critical thinking skills in higher education: a case study in the uk,

    A. A. Essienet al., “The influence of ai text generators on critical thinking skills in higher education: a case study in the uk,”Studies in Higher Education, 2024

    work page 2024

  8. [8]

    Cognitive reflection and decision making,

    S. Frederick, “Cognitive reflection and decision making,”Journal of Economic Perspectives, vol. 19, no. 4, pp. 25–42, 2005

    work page 2005

  9. [9]

    A logical basis for measuring critical thinking skills,

    R. H. Ennis, “A logical basis for measuring critical thinking skills,” Educational Leadership, vol. 43, no. 2, pp. 44–48, 1985

    work page 1985

  10. [10]

    Google effects on memory: Cognitive consequences of having information at our fingertips,

    B. Sparrow, J. Liu, and D. M. Wegner, “Google effects on memory: Cognitive consequences of having information at our fingertips,”Sci- ence, vol. 333, no. 6043, pp. 776–778, 2011

    work page 2011

  11. [11]

    The extended mind,

    A. Clark and D. J. Chalmers, “The extended mind,”Analysis, vol. 58, no. 1, pp. 7–19, 1998

    work page 1998

  12. [12]

    Humans and automation: Use, misuse, disuse, abuse,

    R. Parasuraman and V . Riley, “Humans and automation: Use, misuse, disuse, abuse,”Human Factors, vol. 39, no. 2, pp. 230–253, 1997

    work page 1997

  13. [13]

    Trust in automation: Designing for appropriate reliance,

    J. D. Lee and K. A. See, “Trust in automation: Designing for appropriate reliance,”Human Factors, vol. 46, no. 1, pp. 50–80, 2004

    work page 2004

  14. [14]

    The effect of chatgpt on students’ learning performance, learning perception, and higher-order thinking: Insights from a meta-analysis,

    J. Wang and W. Fan, “The effect of chatgpt on students’ learning performance, learning perception, and higher-order thinking: Insights from a meta-analysis,”Humanities and Social Sciences Communications, vol. 12, p. 621, 2025

    work page 2025

  15. [15]

    Chatgpt as a cognitive crutch: Evidence from a randomized controlled trial on knowledge retention,

    A. Barcaui, “Chatgpt as a cognitive crutch: Evidence from a randomized controlled trial on knowledge retention,”Social Sciences & Humanities Open, vol. 12, p. 102287, 2025

    work page 2025

  16. [16]

    Trust and reliance on ai — an experimental study on the extent and costs of overreliance on ai,

    A. Klingbeil, C. Gr ¨utzner, and P. Schreck, “Trust and reliance on ai — an experimental study on the extent and costs of overreliance on ai,” Computers in Human Behavior, vol. 160, p. 108352, 2024

    work page 2024

  17. [17]

    The role of critical thinking on undergraduates’ reliance behaviours on generative ai in problem- solving,

    C. Hou, G. Zhu, and V . Sudarshan, “The role of critical thinking on undergraduates’ reliance behaviours on generative ai in problem- solving,”British Journal of Educational Technology, 2025

    work page 2025

  18. [18]

    Generative artificial intelligence dependency: Scale development, validation, and its motiva- tional, behavioral, and psychological correlates,

    A. Y . H. Goh, A. Hartanto, and N. M. Majeed, “Generative artificial intelligence dependency: Scale development, validation, and its motiva- tional, behavioral, and psychological correlates,”Computers in Human Behavior Reports, p. 100845, 2025

    work page 2025

  19. [19]

    Who uses general-purpose ai? a typology of chatgpt early adopters,

    C. Gerling, T. Teubner, and F. Braesemann, “Who uses general-purpose ai? a typology of chatgpt early adopters,”Electronic Markets, 2026

    work page 2026

  20. [20]

    Ai tools in society: Impacts on cognitive offloading and the future of critical thinking,

    M. Gerlich, “Ai tools in society: Impacts on cognitive offloading and the future of critical thinking,”Societies, vol. 15, no. 1, p. 6, 2025

    work page 2025

  21. [21]

    A technique for the measurement of attitudes,

    R. Likert, “A technique for the measurement of attitudes,”Archives of Psychology, vol. 22, no. 140, pp. 1–55, 1932

    work page 1932

  22. [22]

    Some methods for classification and analysis of multi- variate observations,

    J. B. MacQueen, “Some methods for classification and analysis of multi- variate observations,” inProceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1. University of California Press, 1967, pp. 281–297

    work page 1967

  23. [23]

    Random forests,

    L. Breiman, “Random forests,”Machine Learning, vol. 45, no. 1, pp. 5–32, 2001

    work page 2001

  24. [24]

    I. T. Jolliffe,Principal Component Analysis, 2nd ed. New York: Springer, 2002

    work page 2002

  25. [25]

    The effects of classroom mathematics teaching on students’ learning,

    J. Hiebert and D. A. Grouws, “The effects of classroom mathematics teaching on students’ learning,” inSecond Handbook of Research on Mathematics Teaching and Learning, F. K. J. Lester, Ed. Charlotte, NC: Information Age Publishing, 2007, pp. 371–404

    work page 2007

  26. [26]

    Using genai in education: the case for critical thinking,

    C. C. Lee, “Using genai in education: the case for critical thinking,” Frontiers in Artificial Intelligence, vol. 7, 2024

    work page 2024