Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Refining a theory-derived coding framework boosts agreement between human annotators but barely changes what in-context-learning LLMs produce, and self-reflective agents remain below human-developed criteria.

desk verdict Useful benchmark of human vs LLM coding on a 7-point ICAP scale, but the headline claim that human refinement improves agreement is confounded by the same annotators writing the criteria and discussing disagreements. read the letter →

arxiv 2607.28651 v1 pith:VUWADZLX submitted 2026-06-07 cs.HC cs.CLcs.CY

Lan Anh Do , Hanling Jiang , Shuchin Aeron , Ayanna K. Thomas This is my paper
classification cs.HCcs.CLcs.CY
keywords ICAPframework7-pointengagementscalecognitivecollaborativediscourseinterraterreliabilityin-contextlearningself-reflectiveLLMagentshuman-machineagreement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to measure cognitive engagement in collaborative discourse and asks whether a refined coding framework transfers from human annotators to large language models. The authors built a 7-point engagement scale from the four ICAP modes, had two trained annotators apply it while they progressively sharpened the criteria, and compared the results with in-context learning, zero-shot prompting, and self-reflective LLM agents. The central finding is asymmetric transfer: the human-refined criteria raised human-human agreement by a kappa of about 0.10, whereas the same criteria moved LLM-based annotation by less than 0.04. Agent-driven refinement of the criteria produced better cross-model agreement than human-machine agreement but stayed below the level achieved when both models used human-developed criteria. The significance is practical: scalable automated annotation of engagement depends not only on clearer instructions but on the human process that produces them.

What carries the argument

The central object is an extended 7-point ICAP scale: an ordinal coding scheme that inserts intermediate levels between the four canonical engagement modes (Interactive, Constructive, Active, Passive) to capture gradations such as brief acknowledgements below active agreement or 'how/why' questions at the edge of constructive engagement. The argument runs through a refinement loop around this scale. Humans refine the criteria after disagreements; the authors give LLMs a parallel loop in which an agent labels a sample, reflects on its own uncertainty, and modifies, holds, or merges scale levels before proceeding. The scale gives both camps a common measurement language; the refinement loop is

What would settle it

Code the same Stage 3 transcripts with fresh annotators who never participated in criteria development, using Version 1 and Version 2 in counterbalanced order; if their kappa gain is close to zero, the reported 0.10 gain is an artifact of shared training, not the framework.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the benefit of refining a coding framework depends on who does the refining and who applies it. Human annotators improved from quadratic weighted kappa (QWK) 0.906 at the first coding stage to 0.998 at the third after criteria were revised; the two large language models tested under in-context learning showed human-machine agreement that stayed near QWK 0.59–0.66 across criteria versions, with the refined version adding only 0.01–0.04. When two LLM agents were each allowed to reflect on ambiguous cases and update the criteria, agreement between the two models (QWK 0.655–0.790) was substantial but below the QWK 0.851–0.894 range that the same tw

Load-bearing premise

The load-bearing assumption is that the human consensus labels are the correct yardstick and that the rise in human-human agreement is caused by the refined criteria rather than by the two annotators having co-developed those criteria and repeatedly discussed disagreements.

Editorial extensions

If this is right

  • If the divergence is real, codebook refinement should be treated as a human-annotation activity; gains from clearer criteria will not automatically appear in LLM outputs when only prompt instructions are updated.
  • Human-consensus examples used in in-context learning do not reliably improve LLM-human agreement over zero-shot prompting, so few-shot example selection should not be assumed beneficial for theory-based coding tasks.
  • LLM agents can generate their own internally consistent criteria, but those criteria generalize less well across different foundation models than human-developed criteria do.
  • Mid-scale levels (3–5) are the main source of coding disagreement for both humans and agents, so future refinements should target decision rules at those boundaries.
  • Achieving scalable automated engagement coding may require fine-tuning on human-refined labels or multimodal input rather than better prompting alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper leaves implicit: because the same two annotators co-developed Criteria Version 2 and met regularly to resolve disagreements, part of the human kappa gain may be a shared-training effect; fresh annotators who never saw Version 1 would isolate the framework's contribution.
  • The agent chose only to modify existing levels, never to add, merge, or delete a level; this conservative behaviour may be an artifact of the reflection prompt, and a version that rewards structural changes might produce more human-like refinements.
  • If zero-shot prompting keeps matching or beating in-context learning as models improve, the marginal value of curated human exemplars may continue to fall, shifting the cost-benefit balance toward prompt design and evaluation rather than example curation.
  • A concrete way to test the multimodal gap is to give the LLM transcripts plus turn-level timing or audio cues from the same videos and check whether human-machine agreement moves toward the human-human level.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper extends the ICAP framework to a 7-point ordinal scale for coding cognitive engagement in collaborative problem-solving discourse, and compares three annotation routes: trained human annotators, ICL/zero-shot LLM labeling with GPT-4o and GPT-5.2, and self-reflective LLM agents that iteratively refine the coding criteria. The authors report high human interrater reliability across three coding stages (QWK 0.906–0.998), moderate human–machine agreement (QWK ~0.59–0.66), and machine–machine agreement that is higher than human–machine but below human–human (QWK ~0.85–0.89 under human criteria, 0.66–0.79 under agent-refined criteria). The central claims are that the human-refined criteria improved human–human agreement by Δκ ≈ 0.10 but produced only small gains for ICL-based LLMs (Δκ < 0.04), and that agent-refined criteria improved cross-model agreement relative to the original 4-point ICAP but not to the level of human-refined criteria.

Significance. If the causal claims were secure, the paper would make a useful contribution to computational social science and educational data mining: it would show that theory-driven codebook refinement helps human annotators but does not transfer to black-box LLM prompting, and that self-reflective agents, while promising, do not yet match human-developed criteria for cross-model consistency. The paper has notable strengths: it makes code available, provides complete prompts and criteria versions in appendices, compares multiple chance-corrected agreement coefficients, and is unusually candid about procedural limitations. Those strengths are real. However, the headline causal claim about the human-refinement effect is confounded by shared annotator development and discussion, and the comparisons lack uncertainty quantification. The significance would be much higher if the human-refinement effect were demonstrated with independent annotators and with confidence intervals on all key kappa values.

major comments (4)
  1. [Method: Human Annotation; Table 2] The Δκ ≈ 0.10 improvement in human–human agreement (QWK rising from 0.906 in Stage 1 to 0.998 in Stage 3) cannot be causally attributed to the written criteria because the same two annotators co-developed Criteria Version 2, met regularly to resolve disagreements, and were mediated by the first author in Stage 1. Shared learning, memory of discussed cases, and co-evolved interpretations are plausible alternative explanations. The Limitations section acknowledges this ('may partly reflect these opportunities for discussion'), but the abstract presents Δκ = 0.10 as a headline result without this qualification. To make the claim load-bearing, the authors should either recruit independent annotators who did not participate in criteria development, or explicitly reframe the human-refinement result as exploratory and subject to this confound.
  2. [Results: Interrater Reliability; Table 2] All reported QWK, Krippendorff's α, and Randolph's κ values are point estimates without confidence intervals, significance tests, or multilevel/unit-clustered modeling. Many of the key comparisons—e.g., Δκ≈0.10 vs. Δκ<0.04, ICL vs. zero-shot differences of 0.01–0.04, and Stage 2 vs. Stage 3 differences—are of a magnitude that may fall within sampling variability, especially with N=42 group conversations and unequal stage sample sizes. The authors should report bootstrap or analytic CIs, and ideally model agreement at the group or trial level, before drawing conclusions about differential transfer of criteria to humans versus LLMs.
  3. [Abstract; Table 2] The abstract's statement that 'Agent-refined frameworks improved cross-model agreement' is ambiguous. Table 2 shows agent-refined QWK values of 0.655–0.790, which are clearly below human-refined QWK values of 0.874–0.894. If 'improved' refers to improvement over the original 4-point ICAP (QWK=0.613 in Table 1), that baseline should be stated explicitly. Without specifying the comparison baseline, the abstract's phrasing may mislead readers into thinking agent refinement outperformed human refinement on cross-model consistency.
  4. [Method: In-context Learning; Results: Human–Machine Reliability] Human consensus labels serve both as the ground truth for constructing ICL examples and as the benchmark for evaluating LLM labels. While example transcripts are disjoint from test transcripts, the evaluation standard is the same consensus process that produced the examples. This creates a partial circularity, and it also means that any biases or idiosyncrasies in the human consensus labels are baked into both sides of the human–machine agreement comparison. The authors should at least acknowledge this explicitly and, ideally, evaluate on a held-out set of labels produced by independent annotators who were not part of the consensus discussions.
minor comments (5)
  1. [Abstract] The abstract uses 'kappa' generically, but Table 2 reports QWK, Randolph's κ, and Krippendorff's α. The Δκ = 0.10 value should specify which coefficient (appears to be QWK) and should be qualified as Stage 1 vs. Stage 3 of the same annotators.
  2. [Table 2] The table formatting is inconsistent: some cells contain semicolons, decimal alignment is uneven, and the notation 'H⋆M' is defined only in the note as 'agreement between the human consensus annotations and an LLM,' but the symbol '⋆' is not used consistently in the row labels. Also, the 'Cri-A1' and 'Cri-A2' rows under 'Zero-shot' appear to belong to M1M2 but their placement is easy to misread; consider merging or separating the sections more clearly.
  3. [Figure 3] The right panel plots (1 − CosSim) between Criteria Version 1 and Version 2 for each engagement level, but the text does not define what CosSim measures (cosine similarity between which representations of the criteria?). A brief definition is needed for interpretability.
  4. [Method: In-context Learning] The paper does not state the number of ICL examples used per prompt, only that they were randomly sampled from a disjoint transcript. Since example count is a key hyperparameter that affects ICL behavior, please report it and, ideally, show a sensitivity analysis.
  5. [Appendix D] Criteria Version 2 labels Level 4 as 'Between Active and Constructive' while the main text refers to Level 4 as 'Initiating Constructive.' Please use consistent terminology across the main text and appendix.

Circularity Check

1 steps flagged · score 3.0 of 10

Human-refinement gain is measured on the same annotators who co-authored the codebook; LLM comparisons remain independent.

  1. other [Abstract; Method: Human Annotation (Stage 2-3); Results: Human-Human Reliability; Limitations]
    "The human-refined framework improved agreement among human annotators (Delta kappa = 0.10) ... Following Stage 2, the first author and two coders collaboratively developed Criteria Version 2 ... The high interrater reliability observed for human coders may partly reflect these opportunities for discussion."

    The reported framework improvement is not an independent test of the written codebook: the same two annotators whose agreement is measured co-authored Criteria Version 2, and the codebook was refined from their own Stage 1-2 disagreements. The Stage 3 H1H2 gain (QWK 0.906 to 0.998) therefore conflates the criteria with the annotators' shared memory and discussion, a confound the paper explicitly concedes. This makes the abstract's Delta kappa = 0.10 a partly self-validating result rather than a prediction from the framework. The LLM-based claims are not circular: ICL examples come from a disjoint transcript, and the human-machine agreement metric is not fitted from the model outputs.

full rationale

The paper is an empirical measurement study rather than a derivation, so most claimed 'results' are simple reliability statistics. The LLM comparisons are not circular: ICL examples are drawn from a transcript strictly disjoint from the test transcripts, and the evaluation against human consensus labels is a standard external benchmark, not a fitted parameter. The one step approaching circularity is the abstract's claim that the human-refined framework improved human annotator agreement by Delta kappa = 0.10. The same two annotators developed Criteria Version 2 and regularly discussed disagreements, so their near-perfect Stage 3 agreement (QWK = 0.998) may reflect co-authorship and shared discussion rather than the framework alone. The paper's Limitations section explicitly states this ('may partly reflect these opportunities for discussion'), which is why the concern is weighed in the score. However, this is a methodological confound, not a mathematical reduction: the kappa values are not computed from the codebook by construction, and the central human-vs-LLM gap (human agreement high, LLM-human agreement moderate) is not forced by the design. Self-citations (Do & Thomas, 2025) are limited to task provenance and are not load-bearing. Overall circularity is partial and localized to the human-refinement headline, hence a score of 3 rather than 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper is empirical; the main 'axioms' are the validity of the ICAP ordering, the use of human consensus as ground truth, the adequacy of point-estimate reliability metrics, and the sufficiency of text-only inputs for LLMs. Free parameters are the hand-built 7-point scale, the agent memory size K=3, and the unreported ICL example count.

free parameters (3)
  • 7-point engagement scale level definitions (Criteria V1/V2) = Levels 1-7 definitions in Appendix D
    The central measurement instrument; hand-built by the authors as an extension of ICAP, refined between stages without an external criterion.
  • Agent memory buffer size K = K=3 samples and 3 criteria versions
    Chosen by hand in the agent design; no ablation or sensitivity analysis is reported.
  • Number of ICL examples = Not reported
    The paper says 'a few examples' in the prompt but does not state the count; this affects ICL performance and is an unreported design choice.
assumptions (4)
  • domain assumption The ICAP modes form an ordered hierarchy of cognitive engagement and the 7-point scale preserves that ordering.
    The entire coding scheme is anchored to this ordering; if the ordering is wrong, the labels lose theoretical meaning.
  • domain assumption Human consensus labels are accurate ground truth for cognitive engagement.
    These labels are used to build ICL examples and to evaluate LLM outputs; no external validation (e.g., relation to learning outcomes) is provided.
  • domain assumption Interrater reliability metrics (QWK, Krippendorff's alpha, Randolph's kappa) capture annotation quality.
    All conclusions are drawn from agreement coefficients; there is no task-performance or outcome-based validity check.
  • domain assumption Text-only transcripts are sufficient input for LLM annotation of engagement.
    LLMs received only text while humans saw video; the authors note in Limitations that this may have reduced LLM sensitivity to multimodal cues.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents." pith.science (2026). https://pith.science/paper/VUWADZLX

@misc{pith2026260728651,
  author       = {Pith},
  title        = {Pith review of: Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VUWADZLX}},
  note         = {Machine review of arXiv:2607.28651}
}
read the original abstract

Collaboration supports learning and problem-solving, but its effectiveness depends on cognitive engagement during discourse. This study applies an extended 7-point ICAP framework based on the Interactive, Constructive, Active, and Passive modes to characterize variation in cognitive engagement during collaborative dialogue. Engagement was coded by trained human annotators and compared with large language model (LLM)-based labeling approaches, including in-context learning (ICL), zero-shot prompting, and self-reflective agents. Interrater reliability among human annotators was robust across framework refinement stages (kappa = 0.906-0.998), higher than the moderate agreement observed for ICL-based annotation (kappa = 0.541-0.609). The human-refined framework improved agreement among human annotators (Delta kappa = 0.10), but produced only modest gains for ICL-based LLMs (Delta kappa less than 0.04). Agent-refined frameworks improved cross-model agreement but remained below the human-refined framework. These findings highlight the promise of agent-based approaches and the importance of continued interaction between theory-guided human annotation and LLM-based methods in future work.

Figures

Figures reproduced from arXiv: 2607.28651 by the authors.

Figure 1
Figure 1. Reflective LLM Agent System With Memory Workflow. Results Interrater Reliability We used three agreement coefficients: quadratic weighted kappa (QWK) (Fleiss & Cohen, 1973), Krippendorff’s α (Hayes & Krippendorff, 2007), and Randolph’s κ (Randolph, 2005). QWK captures chance-corrected agreement on the ex￾tended 7-point scale. Krippendorff’s α is a standard reliabil￾ity index in content analysis. Randolph’s κ is a fr… view at source ↗
Figure 2
Figure 2. Confusion Matrix Between Human Annotators. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Agent Automated Refinement and Human Criteria [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 2 canonical work pages

  1. [1]

    Topics in cognitive science , volume=

    Active-constructive-interactive: A conceptual framework for differentiating learning activities , author=. Topics in cognitive science , volume=. 2009 , publisher=

    2009

  2. [2]

    2006 , publisher=

    Computer-supported collaborative learning , author=. 2006 , publisher=

    2006

  3. [3]

    2023 , eprint=

    The Rise and Potential of Large Language Model Based Agents: A Survey , author=. 2023 , eprint=

    2023

  4. [4]

    2021 , eprint=

    What Makes Good In-Context Examples for GPT- 3 ? , author=. 2021 , eprint=

    2021

  5. [5]

    Teaching and Teacher Education , volume=

    Using an extended ICAP-based coding guide as a framework for the analysis of classroom observations , author=. Teaching and Teacher Education , volume=. 2023 , publisher=

    2023

  6. [6]

    Chemistry Education Research and Practice , volume=

    Investigating small-group cognitive engagement in general chemistry learning activities using qualitative content analysis and the ICAP framework , author=. Chemistry Education Research and Practice , volume=. 2022 , publisher=

    2022

  7. [7]

    Journal of Engineering Education , volume=

    Differentiated overt learning activities for effective instruction in engineering classrooms , author=. Journal of Engineering Education , volume=. 2013 , publisher=

    2013

  8. [8]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Large language models for data annotation and synthesis: A survey , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

Show all 45 references
  1. [9]

    Proceedings of the National Academy of Sciences , volume=

    ChatGPT outperforms crowd workers for text-annotation tasks , author=. Proceedings of the National Academy of Sciences , volume=. 2023 , publisher=

    2023

  2. [10]

    The journal of the learning sciences , volume=

    Quantifying qualitative analyses of verbal data: A practical guide , author=. The journal of the learning sciences , volume=. 1997 , publisher=

    1997

  3. [11]

    Personnel psychology , volume=

    Content analysis: An introduction to its methodology , author=. Personnel psychology , volume=. 2004 , publisher=

    2004

  4. [12]

    The journal of the learning sciences , volume=

    When smart groups fail , author=. The journal of the learning sciences , volume=. 2003 , publisher=

    2003

  5. [13]

    Analyzing discourse: Text and talk , volume=

    Discourse as an interactional achievement: Some uses of ‘uh huh’and other things that come between sentences , author=. Analyzing discourse: Text and talk , volume=

  6. [14]

    American educational research journal , volume=

    Question asking during tutoring , author=. American educational research journal , volume=. 1994 , publisher=

    1994

  7. [15]

    Educational psychologist , volume=

    Facilitating elaborative learning through guided student-generated questioning , author=. Educational psychologist , volume=. 1992 , publisher=

    1992

  8. [16]

    Educational Psychology Review , volume=

    When Does Learning by Non-interactive Teaching Work? A Large-Scale Analysis of Learner Characteristics in a Classroom Setting , author=. Educational Psychology Review , volume=. 2025 , publisher=

    2025

  9. [17]

    advances in learning and instruction series

    Collaborative learning: Cognitive and computational approaches. advances in learning and instruction series. , author=. 1999 , publisher=

    1999

  10. [18]

    Educational psychologist , volume=

    The ICAP framework: Linking cognitive engagement to active learning outcomes , author=. Educational psychologist , volume=. 2014 , publisher=

    2014

  11. [19]

    2025 , eprint=

    OpenAI GPT-5 System Card , author=. 2025 , eprint=

    2025

  12. [20]

    Mind, Brain, and Education , volume=

    In-Person Collaboration, but Not Virtual, Enhances Problem-Solving Efficiency and Knowledge Transfer from Groups to Individuals , author=. Mind, Brain, and Education , volume=. 2025 , publisher=

    2025

  13. [21]

    , author=

    GI transfer in multicue judgment tasks: Discussion improves group members’ knowledge about target relations. , author=. Journal of Experimental Psychology: Learning, Memory, and Cognition , volume=. 2021 , publisher=

    2021

  14. [22]

    Group Processes & Intergroup Relations , volume=

    Group-to-individual problem-solving transfer , author=. Group Processes & Intergroup Relations , volume=. 2008 , publisher=

    2008

  15. [23]

    Group-to-Individual Problem-Solving Transfer , volume =

    Laughlin, Patrick and Carey, Harold and Kerr, Norbert , year =. Group-to-Individual Problem-Solving Transfer , volume =. Group Processes & Intergroup Relations , doi =

  16. [24]

    , author=

    Collaborative learning: The benefits and costs. , author=. 2019 , publisher=

    2019

  17. [25]

    Tutorials in quantitative methods for psychology , volume=

    Computing inter-rater reliability for observational data: an overview and tutorial , author=. Tutorials in quantitative methods for psychology , volume=

  18. [26]

    2018 , publisher=

    Content analysis: An introduction to its methodology , author=. 2018 , publisher=

    2018

  19. [27]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  20. [28]

    arXiv preprint arXiv:2403.17661 , year=

    Language models for text classification: is in-context learning enough?(2024) , author=. arXiv preprint arXiv:2403.17661 , year=

    2024 arXiv

  21. [29]

    Companion of the 2025 ACM International Joint Conference on Pervasive and Ubiquitous Computing , pages =

    Le, Ha and Choube, Akshat and Das Swain, Vedant and Mishra, Varun and Intille, Stephen , title =. Companion of the 2025 ACM International Joint Conference on Pervasive and Ubiquitous Computing , pages =. 2026 , isbn =. doi:10.1145/3714394.3756185 , abstract =

    2025

  22. [30]

    Political Analysis , author=

    Automated Text Classification of News Articles: A Practical Guide , volume=. Political Analysis , author=. 2021 , pages=. doi:10.1017/pan.2020.8 , number=

    2021 doi

  23. [31]

    Proceedings of the National Academy of Sciences , volume =

    GPT is an effective tool for multilingual psychological text analysis , author =. Proceedings of the National Academy of Sciences , volume =. 2024 , doi =

    2024

  24. [32]

    2025 , eprint=

    Aparna Elangovan and Lei Xu and Jongwoo Ko and Mahsa Elyasi and Ling Liu and Sravan Bodapati and Dan Roth , title=. 2025 , eprint=

    2025

  25. [33]

    Continuous Judgment Tasks , author=

    Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Tasks , author=. 2025 , eprint=

    2025

  26. [34]

    Generative Pre-trained Transformers for Coding Text Data? An Analysis with Classroom Orchestration Data , booktitle =

    Amarasinghe, Ishari and Marques, Francielle and Ortiz-Beltr. Generative Pre-trained Transformers for Coding Text Data? An Analysis with Classroom Orchestration Data , booktitle =. 2023 , doi =

    2023

  27. [35]

    2020 , eprint=

    Language Models are Few-Shot Learners , author=. 2020 , eprint=

    2020

  28. [36]

    2024 , eprint=

    Language Models for Text Classification: Is In-Context Learning Enough? , author=. 2024 , eprint=

    2024

  29. [37]

    2024 , eprint=

    MEGAnno+: A Human-LLM Collaborative Annotation System , author=. 2024 , eprint=

    2024

  30. [38]

    C rowd A gent: Multi-Agent Managed Multi-Source Annotation System

    Qin, Maosheng and Zhu, Renyu and Xia, Mingxuan and Chenchenkai and Zhu, Zhen and Lin, Minmin and Zhao, Junbo and Xu, Lu and Fan, Changjie and Wu, Runze and Wang, Haobo. C rowd A gent: Multi-Agent Managed Multi-Source Annotation System. Proceedings of the 2025 Conference on Emp...

    2025 doi

  31. [39]

    2024 , eprint=

    GPT-4o System Card , author=. 2024 , eprint=

    2024

  32. [40]

    Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning

    Yin, Qingyu and He, Xuzheng and Leong, Chak Tou and Wang, Fan and Yan, Yanzhao and Shen, Xiaoyu and Zhang, Qiang. Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuning. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. do...

    2024 doi

  33. [41]

    Educational and psychological measurement , volume=

    The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability , author=. Educational and psychological measurement , volume=. 1973 , publisher=

    1973

  34. [42]

    2020 , eprint=

    MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers , author=. 2020 , eprint=

    2020

  35. [43]

    Scientific Reports , volume =

    LLMs outperform outsourced human coders on complex textual analysis , author =. Scientific Reports , volume =. 2025 , doi =

    2025

  36. [44]

    , author=

    Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa. , author=. Online submission , year=

  37. [45]

    Communication methods and measures , volume=

    Answering the call for a standard reliability measure for coding data , author=. Communication methods and measures , volume=. 2007 , publisher=

    2007

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.