Pith. sign in

REVIEW 3 major objections 5 minor 60 references

Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Principal Trait Analysis, an unsupervised LLM pipeline that mines conversation logs for interpretable behavioral dimensions, derives traits that significantly explain human–AI collaboration outcomes beyond prior performance.

desk verdict A genuinely new unsupervised pipeline for deriving behavioral traits from human-AI logs, with an honest but under-powered evaluation whose predictive claims need a nested cross-validation before they can be trusted. read the letter →

arxiv 2608.11460 v1 pith:LLQZFERC submitted 2026-08-11 cs.CL

Hunter McNichols , Kai Du , Andrew Lan This is my paper · ORCID
classification cs.CL
keywords human-AIcollaborationprincipaltraitanalysisLLM-as-a-judgebehavioraltraitscodingagentsAItutoringoutcomepredictionlearningcurves
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Principal Trait Analysis (PTA), an unsupervised, PCA-inspired pipeline that mines human–AI collaborative conversation logs for interpretable behavioral dimensions it calls traits. PTA extracts behavior observations with LLM prompts, clusters them into candidate traits, scores each session on every trait with an LLM judge, and greedily selects the traits that vary most across collaborators. Evaluated on student–AI tutor dialogues (StudyChat) and developer–AI coding-agent sessions (SWE-Chat), the selected traits add significant explanatory power over a prior-outcome baseline: up to +0.103 in $R^2$ (p=0.028) on Fall 2024 StudyChat exams and +0.048 in $R^2$ (p<0.001) on SWE-Chat session success. The authors are careful to state that these traits are not yet demonstrably skills, because they generalize poorly across semesters and show mostly flat temporal trends.

What carries the argument

Principal Trait Analysis is the central object: a four-stage, PCA-inspired pipeline that converts raw conversation traces into a small set of interpretable behavioral dimensions and a score for every session. Stage 1 prompts an LLM to emit up to five behavior observations per session, either generically or through a theory lens. Stage 2 embeds and clusters these observations into micro-clusters and then candidate traits. Stage 3 uses an LLM-as-a-judge to rate each session on each candidate trait on a 1–5 scale, producing a trait-score matrix. Stage 4 greedily selects $K=10$ principal traits by maximizing relevance, measured by item-total correlation plus factor-analysis communality, while penalizing redundancy in both score space and text-embedding space, mirroring PCA's variance-maximizing logic without requiring orthogonality.

What would settle it

Derive PTA traits on one cohort or cross-validation fold, freeze the trait definitions and scoring prompts, and apply them to held-out sessions or a later cohort while refitting the same regressions: if the incremental $R^2$ over the prior-outcome baseline drops to zero or becomes negative, the reported gains are an artifact of in-sample trait derivation.

Watch

Extended reading notes

Core claim

The central claim is that PTA-derived traits are significant in explaining collaborator behavior across both an educational and a professional coding setting and can help predict task outcomes. On the paper's own numbers, ten selected traits improve outcome regressions over a univariate prior-performance baseline: in StudyChat Fall 2024 the lens-ensemble traits add +0.103 in $R^2$ (p=0.028), and in SWE-Chat they add +0.048 in $R^2$ (p<0.001). The traits are textually interpretable, with positively signed traits such as conceptual_understanding_orientation on StudyChat and workflow_control_delegation on SWE-Chat alongside negatively signed traits such as task_context_specificity and evidence_driven_quality_oversight. The paper also finds that theory-informed extraction lenses yield borderline significant gains in one semester where generic traits do not. It stops short of calling these traits skills, since skill status would require generalization across tasks and evidence of learning curves, which the current results do not provide.

Load-bearing premise

The load-bearing premise is that deriving the traits from the entire dataset and then explaining outcomes in that same dataset does not materially inflate the apparent trait-outcome association; the StudyChat $R^2$ gains are reported from in-sample fits because the sample is too small for held-out partitions.

Editorial extensions

If this is right

  • In educational settings, PTA can surface current, data-driven descriptions of how students use AI tutors, complementing hand-built rubrics that may become stale as LLM capabilities improve.
  • In professional settings, PTA-derived trait scores can serve as interpretable features for predicting whether a coding-agent session will succeed, with held-out predictive gains on SWE-Chat.
  • The lens-ensemble variant shows that domain theory can be injected during unsupervised trait discovery, yielding borderline-significant gains in a semester where generic traits did not.
  • Because trait definitions are derived from the data, they can be re-derived as interaction patterns evolve, giving a mechanism for keeping human-AI collaboration guidelines current.
  • The absence of consistent generalization across semesters and flat temporal trajectories is a direct warning that these traits should not yet be treated as teachable skills.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether traits derived on one semester transfer to another; a direct experiment would freeze trait definitions and scoring on the first cohort and apply them to the second, and the Spring 2025 results suggest the held-out gain may be smaller.
  • The negative regression coefficients attached to intuitively positive traits may reflect redundancy among selected traits rather than true harm; ablating or decorrelating traits before refitting would clarify each behavior's actual direction of association.
  • Applying the same pipeline to non-coding human-AI collaboration traces, such as voice assistants or design tools, would test whether any traits generalize as true collaboration skills or whether all are setting-specific.
  • Scoring traits per turn or per task phase rather than per session would give a finer-grained measure of when behaviors emerge and a more direct test of learning-curve dynamics than the session-level scores plotted here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Principal Trait Analysis (PTA), an unsupervised, PCA-inspired pipeline that derives interpretable behavioral traits from human-AI collaboration transcripts. PTA proceeds in four stages: LLM-based extraction of behavior observations; clustering and refinement into candidate traits; LLM-as-judge scoring of sessions on each trait; and a greedy selection of K principal traits balancing relevance, score-space redundancy, and textual diversity. The method is evaluated on two corpora: StudyChat (student-AI tutor dialogues, with exam scores as outcomes) and SWE-Chat (developer-agent coding sessions, with per-session success as outcomes). The central empirical claim is that PTA-derived traits add significant explanatory or predictive variance beyond a prior-outcome baseline; the strongest reported result is a held-out R2 gain of 0.048 (p<0.001) on SWE-Chat. The paper also reports individual significant traits, a temporal analysis of trait scores, and a careful discussion of why the traits should not yet be called skills.

Significance. If the evaluation concerns were resolved, PTA would be a useful methodological contribution: it offers a scalable, data-driven alternative to fixed theory-based rubrics, produces interpretable textual traits, allows domain expertise to be injected through lenses, and separates trait derivation from outcome modeling. The appendix reproduces all prompts verbatim, which is a concrete reproducibility asset, and the authors' explicit caution against interpreting the traits as skills is scientifically honest. However, the current evidence does not yet establish the headline claim that PTA traits 'can help predict task outcomes.' The StudyChat results are in-sample and explanatory, the generic-pool result fails to replicate in Spring 2025, and the SWE-Chat held-out result is threatened by feature-construction leakage because trait derivation is not nested inside the cross-validation folds. The unvalidated LLM scoring stage also leaves open the possibility that the observed associations reflect judge artifacts rather than latent behavioral constructs.

major comments (3)
  1. [§4.2 (SWE-Chat) and Algorithm 1] The SWE-Chat held-out evaluation is not fully nested. The paper states that cross-validation creates train-test splits and that the train split is used to train the IRT outcome model, but it does not state that Stages 1-4 of PTA are rerun within each training fold. Under the natural reading of Section 4.1, where trait derivation and principal-trait selection use the full corpus, test sessions contribute to micro-cluster formation (Stage 2), candidate-trait naming, and principal-trait selection (Stage 4, via the trait-score matrix Z). This is feature-construction leakage rather than outcome leakage, and it can inflate the reported held-out ΔR2=0.048 and p<0.001. The authors should rerun the full PTA pipeline inside each training fold, or otherwise demonstrate with a stability analysis (e.g., bootstrap or leave-one-out resampling) that the trait definitions and selected trait sets are not shaped by the sessions whose outcomes are being predicted.
  2. [§4.1-§4.2 (StudyChat) and Table 1] The StudyChat evaluation is explicitly in-sample: principal traits are derived from the entire dataset, and the reported R2 improvements come from in-sample fits on the 342 student-exam rows. Because the same sessions determine both the trait definitions and the outcome regression, the p-values (e.g., p=0.028 for Fall 2024 lens-ensemble) do not support predictive or generalizable value. The paper's own cross-semester evidence is also inconclusive: the generic pool is not significant in Spring 2025 (p=0.132) and the lens-ensemble is only borderline (p=0.059). The abstract's claim that PTA traits 'can help predict task outcomes' is therefore supported only by the SWE-Chat result, which is subject to the leakage concern raised above.
  3. [Appendix C.3 and §4.3 (Language Models)] All trait scores are produced by a single unvalidated LLM judge (Qwen3-4B) with no human agreement study, no comparison against an alternative scoring model, and no analysis of potential judge biases such as sensitivity to session length, verbosity, or lexical surface features. Since every regression in Table 1 uses these scores as features, any systematic scoring artifact could generate the observed associations independently of the semantic content of the traits. The authors should report at least a small human-annotation validation sample or a convergent-validity check against an independent scoring method.
minor comments (5)
  1. [§3, Stage 3] The sentence 'we from a trait-score matrix' should read 'we form a trait-score matrix.'
  2. [§5.1] There are typos in this section: 'PTA-derived traits are are powerful' and 'an we observe consistently significant predictive improvements' should be corrected.
  3. [Footnote 1] The footnote labeling the paper as 'Early stage, preliminary work' with a 'principled version forthcoming' is unorthodox for a journal submission and should be removed or substantially revised.
  4. [Table 3] In the definition for change_discipline_and_maintainability, the low pole begins with a stray comma ('...well-documented changes, the user delegates broad outcomes...'); the formatting should be fixed.
  5. [Figure 2 caption] The caption uses 'k=100 micro-clusters' while the text uses K1=100; the notation should be made consistent.

Circularity Check

2 steps flagged · score 5.0 of 10

PTA trait derivation is unsupervised, but the headline R2 gains are not fully out-of-sample: StudyChat is explicitly in-sample, and SWE-Chat's held-out split covers only the outcome model, not trait construction.

  1. fitted input called prediction [Section 4.1 (StudyChat dataset) and Section 4.2 (StudyChat explanatory regression)]
    "The two semesters involve different students, assignments, and exams, so we analyze outcomes on each semester separately but derive principal traits from the entire dataset. ... Since the effective sample size is too small to support a held-out partition, we therefore report explanatory improvement R2 from in-sample fits."

    The principal traits and the trait-score matrix Z are produced from all 1,540 StudyChat sessions, and then the same 342 student-exam rows are regressed on trait scores computed from those same sessions. The regression is in-sample, and Stage 4 principal-trait selection uses variance and correlations in Z computed over the full corpus. Thus the reported +0.103 R2 gain is an in-sample quantity with features constructed from the very sessions whose outcomes are being explained; it cannot serve as out-of-sample evidence for the abstract's predictive claim. Outcomes do not enter trait derivation, but the feature representation is fitted to the same corpus used for evaluation, so the apparent trait-outcome association can be inflated.

  2. fitted input called prediction [Section 4.2 (SWE-Chat predictive regression)]
    "Because each session carries its own outcome, this setup yields one regression target per session, and the sample is large enough that we report predictive R2 on held-out sessions. Specifically, we perform cross validation to create 4 sets of train-test splits of our data, divided across collaborators. We use the train split to train the IRT outcome model, and report the average held-out R2 improvement across all folds."

    The paper states cross-validation only for the IRT outcome model, not for PTA Stages 1-4. No sentence says trait extraction, micro-clustering, candidate-trait formation, or principal-trait selection is re-run inside each training fold; the natural reading is that one PTA trait set was derived from all 2,774 sessions. Test sessions therefore contribute to the micro-clusters, candidate traits, and the 10 selected principal traits that are later scored for those same test sessions. The reported +0.048 R2 gain is consequently not a fully held-out feature prediction: the supervised outcome model is honest, but the feature representation has already seen the test sessions, which can inflate the apparent predictive gain.

full rationale

PTA's derivation stages are genuinely unsupervised with respect to the outcome: Stages 1-4 use conversation text, embeddings, clustering, and score variance rather than exam scores or session-success labels, so the core algorithm is not self-definitional, and there is no load-bearing self-citation chain. The circularity lies in the evaluation protocol rather than in the trait-construction equations. StudyChat is explicitly in-sample: traits are derived from the entire dataset and the R2 gain is reported from in-sample fits, so the Table 1 gain is not out-of-sample support for the predictive claim. SWE-Chat applies cross-validation only to the IRT outcome model; the paper does not state that PTA Stages 1-4 are rerun per training fold, so test sessions plausibly influence the micro-clusters, candidate traits, and principal-trait selection before their outcomes are 'predicted.' This is feature-construction leakage, which can inflate the apparent trait-outcome association even though outcomes never enter trait derivation. The paper's own negative results—non-significant Spring 2025 result, flat temporal trait trends, and the disclaimer that whether traits qualify as skills remains to be seen—show the findings are not artificially forced. The only self-citation, the StudyChat dataset paper, provides data and human-coded dialogue-act corroboration, but it is not used as an authority to justify the main PTA result. On balance, the headline predictive claim is partially compromised by in-sample feature construction, but the method retains independent content, so the circularity score is moderate rather than severe.

Assumptions & free parameters 7 free parameters · 5 assumptions · 1 invented entities

The method introduces seven hand-chosen hyperparameters and relies on unvalidated LLM observation and scoring as measurement. The most consequential assumption is that traits derived from the full corpus can be evaluated on the same corpus without leakage, which weakens the explanatory claims.

free parameters (7)
  • K (number of principal traits) = 10
    Number of final traits chosen by hand for both datasets.
  • K1 (micro-cluster count) = 100
    k-means cluster count for observation embeddings.
  • K2 (nearest neighbors for naming) = 10
    Number of centroid-nearest observations passed to LLM for cluster naming.
  • K3 (agglomerative aggregation target) = 45 (StudyChat), 12 (SWE-Chat)
    Target number of candidate traits after bottom-up clustering.
  • tau (dedup threshold) = 0.85
    Cosine similarity threshold for merging near-duplicate trait definitions.
  • epsilon (selection gate threshold) = 0.25
    Tolerance for the gated set in principal trait selection.
  • factor-analysis factor count m = unspecified
    Stage 4 computes communality from a factor-analysis model but the number of shared factors is not stated.
assumptions (5)
  • domain assumption LLM-generated behavior observations faithfully reflect actual collaborator behavior
    Stage 1 uses GPT-5.5 to extract up to five observations per session, but no human-coded validation or agreement statistics are reported.
  • domain assumption LLM-as-a-judge trait scores are valid measurements on a 1-5 scale
    Stage 3 uses Qwen3-4B to score every session-trait pair; no inter-rater reliability, calibration, or comparison to human annotations is provided.
  • domain assumption Embedding cosine similarity captures semantic similarity for clustering and deduplication
    Stage 2 relies on ModernBERT and MPNet embeddings with thresholds tau=0.85 and epsilon=0.25, with no sensitivity analysis.
  • domain assumption High variance across collaborators identifies the most informative traits
    Stage 4 relevance uses item-total correlation and factor-analysis communality, which measure shared variance rather than outcome alignment, so the selected traits may not be the most outcome-relevant.
  • ad hoc to paper Traits derived from the full corpus can be evaluated on the same corpus without leakage
    PTA derives traits from both semesters of StudyChat and then explains exam outcomes in the same semesters; this in-sample design is acknowledged but not corrected.
invented entities (1)
  • Principal traits
    purpose: Textual descriptions of recurring collaborator behaviors, used as interpretable regression features and behavioral summaries.
    Traits are derived from the corpora with no external validation as stable skills; the paper itself concludes they do not yet qualify as skills.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration." pith.science (2026). https://pith.science/paper/LLQZFERC

@misc{pith2026260811460,
  author       = {Pith},
  title        = {Pith review of: Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LLQZFERC}},
  note         = {Machine review of arXiv:2608.11460}
}
read the original abstract

Large Language Model-powered agents are increasingly used in the workplace via human-artificial intelligence (AI) collaboration. In this new era of work, it is important to understand the kinds of prompting traits that contribute to task success. Moreover, we need to uncover key skills required for modern professionals and inform educators on how to foster these skills among students. Existing guidelines for human-AI collaboration are built from either top-down theory or context-specific observations of human-AI interactions. However, since LLM capabilities are rapidly improving, theory may not be able to explain emerging interaction patterns, and empirical guidelines may become obsolete quickly. In this work, we explore an automated, data-driven approach to uncover patterns, which we term traits, of effective human-AI interaction that are aligned with task outcomes. We propose Principal Trait Analysis, a Principal Component Analysis-inspired algorithm for deriving common traits from patterns in LLM conversations. Our algorithm uses LLM-based processing stages to analyze corpora of human-AI collaborative session traces, deriving common traits across the dataset and scoring each human collaborator's usage style by each trait. The approach also allows domain expertise to be injected during trait discovery and selects the most distinguishing traits to be those that exhibit the highest variance across collaborators. We evaluate PTA on two human-AI collaborative coding datasets, an educational setting (students working with an AI tutor) and a professional setting (developers working with an AI coding agent). We find that PTA-derived traits are significant in explaining collaborator behavior across both settings and can help predict task outcomes. However, whether traits qualify as skills remains to be seen, due to inconclusive results on generalizability and how user traits change over time.

Figures

Figures reproduced from arXiv: 2608.11460 by the authors.

Figure 1
Figure 1. PTA visualized stage-by-stage. Our approach first derives a large set of observations from a conversation corpus which are iteratively refined into a smaller set of candidate traits. Each session of the corpus is scored by the traits which are then used to derive a smaller set of distinct, principal traits. the rapid changes in this field, more automatic and data-driven analyses should be explored, so that new trait… view at source ↗
Figure 2
Figure 2. PTA clustering stages visualized. Panels show one shared t-SNE projection of all observation text embeddings from the StudyChat generic pool. Raw observations (a) are clustered into k = 100 micro-clusters (b), • indicates cluster centroids. The micro-clusters are agglomeratively merged to remove duplicates and form candidate traits (c), ◦ indicates candidate centroids and lines indicate which micro-clusters are merg… view at source ↗
Figure 3
Figure 3. Trait scores over time. Each panel overlays that pool’s significant traits ( [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 42 canonical work pages

  1. [1]

    The StudyChat dataset: Analyzing student dialogues with ChatGPT in an artificial intelligence course

    Hunter McNichols, Fareya Ikram, and Andrew Lan. The StudyChat dataset: Analyzing student dialogues with ChatGPT in an artificial intelligence course. InProceedings of the 16th International Learning Analytics and Knowledge Conference (LAK ’26), pages 53–63, 2026

    work page 2026

  2. [2]

    SWE-chat: Coding agent interactions from real users in the wild.arXiv preprint arXiv:2604.20779, 2026

    SALT-NLP. SWE-chat: Coding agent interactions from real users in the wild.arXiv preprint arXiv:2604.20779, 2026

    arXiv 2026

  3. [3]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    arXiv 2025

  4. [4]

    IEEE Computer Society, 2024

    Hironori Washizaki, editor.Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), Version 4.0. IEEE Computer Society, 2024

    work page 2024

  5. [5]

    Christoph Treude and Marco A. Gerosa. How developers interact with AI: A taxonomy of human-AI collaboration in software engineering.arXiv preprint arXiv:2501.08774, 2025

    arXiv 2025

  6. [6]

    Professional software developers don’t vibe, they control: AI agent use for coding in 2025.arXiv preprint arXiv:2512.14012, 2025

    Ruanqianqian Huang, Avery Reyna, Sorin Lerner, Haijun Xia, and Brian Hempel. Professional software developers don’t vibe, they control: AI agent use for coding in 2025.arXiv preprint arXiv:2512.14012, 2025

    arXiv 2025

  7. [7]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference.arXiv preprint arXiv:2412.13663, 2024

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference.arXiv preprint arXiv:2412.13663, 2024

    arXiv 2024

  8. [8]

    What is AI literacy? Competencies and design considerations

    Duri Long and Brian Magerko. What is AI literacy? Competencies and design considerations. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–16, 2020. doi:10.1145/3313831.3376727

    arXiv 2020

Show all 60 references
  1. [9]

    Zimmerman

    Barry J. Zimmerman. Becoming a self-regulated learner: An overview.Theory Into Practice, 41(2):64–70, 2002. doi:10.1207/s15430421tip4102_2

    2002 doi

  2. [10]

    Michelene T. H. Chi and Ruth Wylie. The ICAP framework: Linking cognitive engagement to active learning outcomes.Educational Psychologist, 49(4):219–243, 2014. doi:10.1080/00461520.2014.965823

    2014

  3. [11]

    Framework for AI fluency (practical summary document), version 1.1

    Rick Dakan and Joseph Feller. Framework for AI fluency (practical summary document), version 1.1. Rin- gling.edu/ai/, 2025.https://ringling.libguides.com/ai/framework. Retrieved 31 July 2026

    2025

  4. [12]

    WildChat: 1M ChatGPT interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat: 1M ChatGPT interaction logs in the wild. InICLR, 2024.https://arxiv.org/abs/2405.01470

    2024 arXiv

  5. [13]

    Botelho, Guanze Liao, Thomas K

    Shan Zhang, Ruiwei Xiao, Anthony F. Botelho, Guanze Liao, Thomas K. F. Chiu, John Stamper, and Kenneth R. Koedinger. How to assess AI literacy: Misalignment between self-reported and objective-based measures. InLAK, 2026.https://arxiv.org/abs/2601.06101

    2026

  6. [14]

    Krathwohl

    David R. Krathwohl. A revision of Bloom’s taxonomy: An overview.Theory Into Practice, 41(4):212–218, 2002. doi:10.1207/s15430421tip4104_2

    2002 doi

  7. [15]

    The impact of AI on developer productivity: Evidence from GitHub Copilot.arXiv preprint arXiv:2302.06590, 2023

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of AI on developer productivity: Evidence from GitHub Copilot.arXiv preprint arXiv:2302.06590, 2023

    2023 arXiv

  8. [16]

    How novices use LLM-based code generators to solve CS1 coding tasks in a self-paced learning environment

    Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara Jane Ericson, David Weintrop, and Tovi Grossman. How novices use LLM-based code generators to solve CS1 coding tasks in a self-paced learning environment. In Koli Calling International Conference on Computing Education R...

    2023

  9. [17]

    Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S

    James Prather, Brent N. Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S. Randrianasolo, Brett A. Becker, Bailey Kimmel, Jared Wright, and Ben Briggs. The widening gap: The benefits and harms of generative AI for novice programmers. InACM Conference on International Computing ...

    2024

  10. [18]

    Koedinger, Albert T

    Kenneth R. Koedinger, Albert T. Corbett, and Charles Perfetti. The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning.Cognitive Science, 36(5):757–798, 2012. doi:10.1111/j.1551-6709.2012.01245.x

    2012

  11. [19]

    Learning factors analysis — a general method for cognitive model evaluation and improvement

    Hao Cen, Kenneth Koedinger, and Brian Junker. Learning factors analysis — a general method for cognitive model evaluation and improvement. InInternational Conference on Intelligent Tutoring Systems (ITS), pages 164–175, 2006. doi:10.1007/11774303_17

    2006 doi

  12. [20]

    The AI literacy heptagon: A structured approach to AI literacy in higher education.Computers and Education: Artificial Intelligence, 10:100540, 2026

    Veronika Hackl, Alexandra Mueller, and Maximilian Sailer. The AI literacy heptagon: A structured approach to AI literacy in higher education.Computers and Education: Artificial Intelligence, 10:100540, 2026

    2026

  13. [21]

    McLaren, and Kenneth R

    Vincent Aleven, Ido Roll, Bruce M. McLaren, and Kenneth R. Koedinger. Help helps, but only so much: Research on help seeking with intelligent tutoring systems.International Journal of Artificial Intelligence in Education, 26(1):205–223, 2016. 12 APREPRINT- AUGUST13, 2026

    2016

  14. [22]

    Large language models offer an alternative to the traditional approach of topic modelling

    Yida Mu, Chun Dong, Kalina Bontcheva, and Xingyi Song. Large language models offer an alternative to the traditional approach of topic modelling. InLREC-COLING, 2024. https://aclanthology.org/2024. lrec-main.887/

    2024

  15. [23]

    ClusterLLM: Large language models as a guide for text clustering

    Yuwei Zhang, Zihan Wang, and Jingbo Shang. ClusterLLM: Large language models as a guide for text clustering. InEMNLP, 2023.https://aclanthology.org/2023.emnlp-main.858/

    2023

  16. [24]

    HICode: Hierarchical inductive coding with LLMs

    Mian Zhong, Pristina Wang, and Anjalie Field. HICode: Hierarchical inductive coding with LLMs. InEMNLP, 2025.https://aclanthology.org/2025.emnlp-main.1580/

    2025

  17. [25]

    TopicGPT: A prompt-based topic modeling framework

    Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit Iyyer. TopicGPT: A prompt-based topic modeling framework. InNAACL, 2024

    2024

  18. [26]

    Goal-driven explainable clustering via language descriptions

    Zihan Wang, Jingbo Shang, and Ruiqi Zhong. Goal-driven explainable clustering via language descriptions. In EMNLP, 2023

    2023

  19. [27]

    Goal driven discovery of distributional differences via language descriptions

    Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal driven discovery of distributional differences via language descriptions. InNeurIPS, 2023

    2023

  20. [28]

    Lam, Janice Teoh, James A

    Michelle S. Lam, Janice Teoh, James A. Landay, Jeffrey Heer, and Michael S. Bernstein. Concept induction: Analyzing unstructured text with high-level concepts using LLooM. InCHI, 2024

    2024

  21. [29]

    Latent factor models meet instructions: Goal-conditioned latent factor discovery without task supervision

    Zhouhang Xie, Tushar Khot, Bhavana Dalvi Mishra, Harshit Surana, Julian McAuley, Peter Clark, and Bod- hisattwa Prasad Majumder. Latent factor models meet instructions: Goal-conditioned latent factor discovery without task supervision. InNAACL, 2025

    2025

  22. [30]

    TnT-LLM: Text mining at scale with large language models

    Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, et al. TnT-LLM: Text mining at scale with large language models. InKDD, 2024

    2024

  23. [31]

    Clio: Privacy-preserving insights into real-world AI use.arXiv preprint arXiv:2412.13678, 2024

    Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, et al. Clio: Privacy-preserving insights into real-world AI use.arXiv preprint arXiv:2412.13678, 2024

    2024 arXiv

  24. [32]

    Stokes, et al

    Ying-Chun Lin, Jennifer Neville, Jack W. Stokes, et al. Interpretable user satisfaction estimation for conversational systems with large language models.arXiv preprint arXiv:2403.12388, 2024

    2024 arXiv

  25. [33]

    Denis Jered McInerney, Geoffrey Young, Jan-Willem van de Meent, and Byron C. Wallace. CHiLL: Zero-shot custom interpretable feature extraction from clinical notes with large language models. InFindings of EMNLP, 2023

    2023

  26. [34]

    Morris, et al

    Vinamra Benara, Chandan Singh, John X. Morris, et al. Crafting interpretable embeddings by asking LLMs questions. InNeurIPS, 2024

    2024

  27. [35]

    Interpretable- by-design text understanding with iteratively generated concept bottleneck.arXiv preprint arXiv:2310.19660, 2023

    Josh Magnus Ludan, Qing Lyu, Yue Yang, Liam Dugan, Mark Yatskar, and Chris Callison-Burch. Interpretable- by-design text understanding with iteratively generated concept bottleneck.arXiv preprint arXiv:2310.19660, 2023

    2023 arXiv

  28. [36]

    Bayesian concept bottleneck models with LLM priors

    Jean Feng, Avni Kothari, Luke Zier, Chandan Singh, and Yan Shuo Tan. Bayesian concept bottleneck models with LLM priors. InNeurIPS, 2025

    2025

  29. [37]

    LLM-based feature generation from text for interpretable machine learning.Machine Learning, 114:241, 2025

    Václav Balek, Lukáš Sýkora, Vilém Sklenák, and Tomáš Kliegr. LLM-based feature generation from text for interpretable machine learning.Machine Learning, 114:241, 2025

    2025

  30. [38]

    Single-agent vs

    Gen Li, Li Chen, Cheng Tang, et al. Single-agent vs. multi-agent LLM strategies for automated student reflection assessment. InPAKDD, 2025

    2025

  31. [39]

    Ober, Ying Zhang, Diego Zapata-Rivera, et al

    Teresa M. Ober, Ying Zhang, Diego Zapata-Rivera, et al. Using LLMs to identify indicators of persistence from students’ dialogues with a pedagogical agent.Journal of Educational Data Mining, 18(1), 2026

    2026

  32. [40]

    Tim Althoff, Kevin Clark, and Jure Leskovec. Large-scale analysis of counseling conversations: An application of natural language processing to mental health.Transactions of the Association for Computational Linguistics, 4:463–476, 2016

    2016

  33. [41]

    Correlations between dialogue acts and learning in spoken tutoring dialogues

    Diane Litman and Kate Forbes-Riley. Correlations between dialogue acts and learning in spoken tutoring dialogues. Natural Language Engineering, 12(2):161–176, 2006

    2006

  34. [42]

    Kristy Elizabeth Boyer, Robert Phillips, Amy Ingram, Eun Young Ha, Michael Wallis, Mladen V ouk, and James Lester. Investigating the relationship between dialogue structure and tutoring effectiveness: A hidden Markov modeling approach.International Journal of Artificial Intell...

    2011

  35. [43]

    Graesser

    Arthur C. Graesser. Conversations with AutoTutor help students learn.International Journal of Artificial Intelligence in Education, 26:124–132, 2016

    2016

  36. [44]

    Natural language processing in mixed-methods text analysis: A workflow approach

    Louisa Parks and Wim Peters. Natural language processing in mixed-methods text analysis: A workflow approach. International Journal of Social Research Methodology, 26(4), 2023. 13 APREPRINT- AUGUST13, 2026

    2023

  37. [45]

    Mining the gold: Student–AI chat logs as rich sources for automated knowledge gap detection.arXiv preprint arXiv:2512.22404, 2025

    Yuheng Fu, Boyang Wu, and Joseph Jay Williams. Mining the gold: Student–AI chat logs as rich sources for automated knowledge gap detection.arXiv preprint arXiv:2512.22404, 2025

    2025

  38. [46]

    Mises, RV and Pollaczek-Geiringer, Hilda Praktische verfahren der gleichungsauflösung.ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik, 9(1):58–77, 1929

    1929

  39. [47]

    MPNet: Masked and permuted pre-training for language understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MPNet: Masked and permuted pre-training for language understanding. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  40. [48]

    Thek-means algorithm: A comprehensive survey and performance evaluation.Electronics, 9(8):1295, 2020

    Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. Thek-means algorithm: A comprehensive survey and performance evaluation.Electronics, 9(8):1295, 2020

    2020

  41. [49]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Processi...

    2023

  42. [50]

    Correction of item-total correlations in item analysis.Psychometrika, 28(2):211–218, 1963

    Sten Henrysson. Correction of item-total correlations in item analysis.Psychometrika, 28(2):211–218, 1963

    1963

  43. [51]

    Harman.Modern Factor Analysis

    Harry H. Harman.Modern Factor Analysis. University of Chicago Press, Chicago, 3rd edition, 1976

    1976

  44. [52]

    Rossi.Mathematical Statistics: An Introduction to Likelihood Based Inference

    Richard J. Rossi.Mathematical Statistics: An Introduction to Likelihood Based Inference. John Wiley & Sons, New York, 2018

    2018

  45. [53]

    Cook, Peter M

    Kelly Hallberg, Thomas D. Cook, Peter M. Steiner, and M. H. Clark. Pretest measures of the study outcome and the elimination of selection bias: Evidence from three within study comparisons.Prevention Science, 19(3):274–283, 2018

    2018

  46. [54]

    Springer, New York, 2004

    Paul De Boeck and Mark Wilson, editors.Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. Springer, New York, 2004

    2004

  47. [55]

    Unpacking vibe coding: Help-seeking processes in student–AI interactions while programming

    Daiana Rinja, Eduardo Araujo Oliveira, Sonsoles López-Pernas, Mohammed Saqr, Marcus Specht, and Kamila Misiejuk. Unpacking vibe coding: Help-seeking processes in student–AI interactions while programming. In Proceedings of the 27th International Conference on Artificial Intell...

    2026

  48. [56]

    what is the function for X

    OpenAI. GPT-5.5 system card. Technical report, OpenAI, April 2026. https://deploymentsafety.openai. com/gpt-5-5/gpt-5-5.pdf. 14 APREPRINT- AUGUST13, 2026 A Significant Traits with Definitions Table 3:Significant traits with definitions.The traits of Table 2, with each trait’s ...

    2026

  49. [57]

    do oneself

    **Delegation** — deciding what to hand to the AI vs. do oneself. Tells: knows what they want and why before asking; delegates the routine while keeping the thinking — vs. hands over the whole problem indiscriminately or offloads work they should own

  50. [58]

    Tells: specifies the output (format, constraints, what they actually need), gives context, guides how the AI should approach it — vs

    **Description** — communicating requirements well. Tells: specifies the output (format, constraints, what they actually need), gives context, guides how the AI should approach it — vs. vague, under-specified, one-line requests (”vague input produces vague output”)

  51. [59]

    Tells: checks the AI’s output for correctness, questions its reasoning, tests or pushes back before using it — vs

    **Discernment** — critically evaluating what comes back. Tells: checks the AI’s output for correctness, questions its reasoning, tests or pushes back before using it — vs. accepts and copies output uncritically

  52. [60]

    Tells: adapts/verifies output before using it as their own, owns the final product, signals they understand what they’re submitting — vs

    **Diligence** — taking responsibility for the result. Tells: adapts/verifies output before using it as their own, owns the final product, signals they understand what they’re submitting — vs. passes AI output straight through as-is without ownership. Read the whole session and...

    2026

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.