Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

THiNK: Can Large Language Models Think-aloud?

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that iterative feedback-driven revision of math word problems reveals and improves higher-order thinking in LLMs, while a persistent gap remains at applying knowledge to realistic contexts.

desk verdict Interesting framework, but the composite quality score as defined cannot produce the reported numbers, so the quantitative results are not reproducible. read the letter →

arxiv 2505.20184 v1 pith:OTFG2SJS submitted 2025-05-26 cs.CL cs.AI

Yongan Yu , Mengqian Wu , Yiran Lin , Nikki G. Lobczowski This is my paper
classification cs.CLcs.AI
keywords higher-orderthinkingBloom'sTaxonomyLLMevaluationmulti-agentframeworkmathwordproblemsfeedback-drivenrefinementthink-aloudprotocolchain-of-thoughtprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces THiNK, a multi-agent evaluation framework that tests whether large language models exhibit higher-order thinking by having them iteratively revise flawed math word problems under structured feedback. The central claim is that this feedback-driven 'think-aloud' process exposes a reliable cognitive profile: models pass remembering and understanding tasks comfortably, but their scores drop sharply at applying knowledge to realistic situations. The paper further claims that iterative critique from Bloom's Taxonomy-aligned agents significantly raises performance in higher-order categories—analyzing, evaluating, and creating—while human-expert qualitative checks suggest the revised problems are more faithful to domain logic. If true, the framework offers a scalable way to both diagnose and improve deep reasoning in LLMs rather than just measuring surface accuracy.

What carries the argument

The load-bearing mechanism is a parallel multi-agent evaluation stage in which six agents (A1–A6) each score problems against one Bloom's Taxonomy level—remembering, understanding, applying, analyzing, evaluating, creating—using chain-of-thought prompting, while a seventh holistic agent (A7) scores lexical and syntactic quality and emits improvement suggestions. All judgments are anchored to a 'Five Keys' rubric: math concepts and domains, prerequisite skills, mathematical representations, alternative values, and narrative stories. Each agent returns a performance score and a confidence score, which feed a composite quality score $Q(p_i) = 0.5 \cdot \mathrm{PR} + 0.3 \cdot \mathrm{AA} + 0.2 \cdot \mathrm{AC}$; problems below the $Q > 85$ threshold cycle back to the model with the A7 suggestions, and the loop runs up to $R$ iterations. The iteration count and the score trajectory across rounds are themselves treated as measurements of the model's revision ability and reasoning depth.

What would settle it

Recruit a panel of mathematics educators to blind-score the original, zero-shot, and THiNK-revised problems using the same Five Keys rubric, and compare their rankings to the GPT-4O agents' rankings. If human experts do not systematically rank the THiNK-guided outputs higher—or if their scores diverge sharply from the agent scores—the claim that structured feedback improves higher-order thinking is not supported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that LLM reasoning is systematically uneven across Bloom's Taxonomy levels, and that this unevenness only becomes visible when evaluation is iterative rather than single-turn. Across seven models, average scores are high for Remembering (88.48) and Understanding (76.02) but fall to 71.15 for Applying, and that drop appears in nearly every model tested. When the same models revise their problems in response to structured feedback, higher-order scores rise—GPT-4O gains 4.21 points in Analyzing and 2.92 in Evaluating—while smaller open models like Mistral-8B keep a wide gap between surface fluency (91.62 in Remembering, 66.92 in Applying). The paper reads this as evidence that feedback-driven refinement exercises cognitive abilities that single-turn benchmarks never reach, and that instruction-tuned closed-source models benefit most from structured revision.

Load-bearing premise

The whole evaluation depends on the assumption that the 'Five Keys' rubric, as applied by GPT-4O judge agents, measures genuine higher-order thinking; if the judge's scores merely reward outputs that resemble the judge's own style, the reported improvements say more about self-consistency than about reasoning ability.

Editorial extensions

If this is right

  • LLM evaluation should shift from single-turn accuracy benchmarks to iterative feedback loops that expose how a model responds to critique and revision.
  • The consistent drop in the Applying category identifies a concrete target: transferring learned procedures into realistic contexts remains unsolved by current instruction tuning.
  • The same loop that diagnoses cognitive levels also produces improved problems, so evaluation doubles as a refinement pipeline for generating higher-quality educational content.
  • Instruction-tuned closed-source models show the narrowest performance band across all six levels, making them the strongest candidates for tutor-like applications, while small open models converge faster but cap at lower final quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A competing explanation for the measured gains is rubric conformity: GPT-4O both scores the revisions and produces them, so 'improvement' may track how well the model imitates its own judge's preferences rather than genuine cognitive growth; the paper's Limitations section concedes that no external experts verified outputs.
  • The critique-revise loop is domain-general in principle: the same think-aloud protocol could probe abstraction and transfer in code generation, scientific explanation, or legal reasoning, where applying knowledge to realistic contexts is a documented bottleneck.
  • A decisive test would decouple judge from examinee—for instance, scoring with a different model family or with human raters—to see whether the reported improvement survives the separation; if it does not, the framework measures self-consistency rather than higher-order thinking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper introduces THiNK, a multi-agent evaluation framework that asks LLMs to iteratively revise flawed math word problems under feedback from six Bloom's-taxonomy-aligned agents plus a holistic agent. The composite quality score Q, defined as a weighted sum of pass rate, inter-agent agreement, and average confidence, determines success and drives the refinement loop. The authors apply THiNK to seven LLMs, reporting that models are strong at lower-order levels, weak at Applying, and improve in higher-order levels with feedback; a qualitative example from Qwen2.5-14B is used to illustrate domain-appropriate reasoning.

Significance. If the framework were valid, it would offer a theory-grounded, automated alternative to accuracy-only benchmarks for probing reasoning, and the released code would be a useful community resource. The paper targets a real evaluation gap, and the educational grounding in Bloom's Taxonomy is a reasonable starting point. However, the quantitative claims rest on a scoring protocol whose printed definition cannot produce the reported numbers, and the evaluation loop has GPT-4O serving as judge, generator of synthetic inputs, and evaluated subject without external validation. These issues are not presentation-level; they undermine the central empirical claim as written.

major comments (5)
  1. [§3.3, Q definition and success rule] Q as defined is arithmetically inconsistent with the success rule. PR is defined as the average of indicators 1(PS_j(pi)>tau), hence lies in [0,1]; AA is Cohen's Kappa, in [-1,1]; AC lies in [0,100]. With alpha=0.5, beta=0.3, gamma=0.2, the maximum possible Q is 0.5*1 + 0.3*1 + 0.2*100 = 20.8, so the condition Q>85 can never be satisfied. Yet Table 2 reports Qavg values of 71-82 and Rpass approximately 2, which requires problems to pass. The manuscript must state an explicit normalization for PR and AA (e.g., percentages) or correct the threshold; without this, Tables 1-2 are not reproducible from the paper as written.
  2. [§3.3, §4.1, Algorithm 1] The symbol tau is used both as the per-agent passing threshold inside PR and as the quality threshold in Algorithm 1's while loop, and no value is reported for either. A separate condition 'Success(pi)=1(Q(pi)>85)' is introduced without connecting it to tau. In addition, Rpass(pi) is defined as min{r | A(p_i^(r)) > tau}, although A denotes the set of agents rather than a scalar quality score; this should presumably be Q(p_i^(r)). The ambiguity makes the pass/fail loops and the reported rounds-to-pass values impossible to interpret.
  3. [§3.2.2, §4.2, Limitation] The evaluation is self-referential for GPT-4O: GPT-4O generates the synthetic bad questions, instantiates all six Bloom agents and the holistic agent A7, and is itself one of the seven evaluated models. For GPT-4O, the examiner and the examinee share the same inductive biases, so reported improvements may reflect conformity to the judge's own criteria rather than independent gains in higher-order thinking. The Limitation section acknowledges the absence of external expert verification and the risk of rubric overfitting but does not address this judge-overlap. A minimal remedy would be a human-scored sample or a second independent judge model with agreement statistics.
  4. [§3.3, AA definition] Cohen's Kappa is defined for two raters, but AA is computed over seven agents. The manuscript does not specify whether pairwise kappa values are averaged, whether Fleiss' kappa is used, or how negative kappa values enter a weighted sum that also includes PR in [0,1]. This makes the AA component ill-defined and further blocks reproduction of Q.
  5. [§4.3, Table 4, Limitation] The qualitative validation is under-reported and internally inconsistent with the Limitation statement. The contributions claim validation 'by a human expert,' and Table 4 is labeled 'Human Expert Quality Evaluation,' but the Limitation section says no external experts were recruited; only one example (Qwen2.5-14B) is shown, with no annotation protocol, rater count, or inter-rater agreement. The qualitative claims in the abstract and conclusion are therefore not substantiated.
minor comments (5)
  1. [Table 1] The arrow notation (e.g., '86.92↑26.92') is not defined in the text; it appears to report a final score and a change from the previous round, but the reader must infer this.
  2. [Figure 2] The figure contains several typos and unclear labels ('No able to sovle it.', 'Great Sugeestions!', 'THINK' vs 'THiNK'); please correct and harmonize.
  3. [§4.1] The Rpass formula uses A(p_i^(r)) > tau where A is the agent set; use the composite quality score Q(p_i^(r)) or define a new symbol.
  4. [§3.4] The claimed mapping from LLM text generation to a think-aloud protocol should be justified more carefully; generating a revised problem after reading feedback is not the same as real-time articulation of thought.
  5. [§3.2.1, Appendix B.1] Dataset construction details are incomplete: the total number of problems in D is not explicitly stated, and the generation process for the 100 synthetic questions is only summarized; include full prompts and counts if space permits.

Circularity Check

1 steps flagged · score 6.0 of 10

The central 'structured feedback improves higher-order thinking' result is partly self-confirming: the GPT-4O agents that write the feedback also define the score through one unvalidated rubric, and the paper itself concedes that no external experts validated the outputs.

  1. self definitional [Section 3.2.2, Section 3.3, Algorithm 1, Section 4.3, Limitation]
    "the holistic evaluation agent A7 further outputs an improvement suggestion: A7(pi) = (P S7(pi), CS7(pi), IS(pi)) ... This feedback assesses whether the problem satisfies the 'Five Keys' components ... Q(pi) = α · P R(pi) + β · AA(pi) + γ · AC(pi) ... Success(pi) = 1(Q(pi) > 85) ... pi ← LLM(pi, IS)"

    The criterion being measured ('higher-order thinking performance') is operationalized by the same multi-agent rubric that produces the improvement suggestions. Algorithm 1 feeds rubric-derived suggestions back into the LLM and then recomputes Q with the same rubric, so the reported improvement is partly convergence to the judge's own criteria rather than an external measure of reasoning.

full rationale

The main empirical claim is that structured feedback loops significantly improve reasoning performance, especially higher-order thinking. In this framework, the feedback is generated by GPT-4O agents using the paper's 'Five Keys' rubric, and the quality score Q is computed by the same rubric. Thus the intervention and the outcome share the same unvalidated measurement construct; a model that follows the suggestions is rewarded by the very criteria embedded in those suggestions. The paper's own Limitation explicitly acknowledges the lack of external expert verification and the risk of rubric overfitting, which supports this partial-circularity finding. This is not a case of a maliciously fitted parameter or a self-citation chain; the rubric is an authored input rather than a fitted output, and the task retains independent mathematical constraints. Separately, the printed Q formula is arithmetically inconsistent with the reported numbers: with PR in [0,1], AA as Cohen's Kappa in [-1,1], and AC in [0,100], the maximum possible Q is about 20.8, far below the required threshold of 85. That is a reproducibility blocker rather than a circularity, but it compounds the difficulty of verifying the claimed improvements. The self-citation to Wolcott and Lobczowski (2021) for the think-aloud protocol is not load-bearing, since this is a standard method with independent support. On balance, the central improvement claim is partially circular by construction, but the framework also contains independent empirical content, so a score of 6 is appropriate.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The framework's central claim rests on scoring rubrics invented in this paper and on LLM judges that share the identity of an evaluated model. The free parameters are few but sensitive, and the validation is non-external.

free parameters (3)
  • Composite quality weights (alpha, beta, gamma) = 0.5, 0.3, 0.2
    Chosen by 'expert tuning' to combine pass rate, agreement, and confidence; they directly determine which revisions pass the success threshold (Section 3.3).
  • Success threshold = 85 (composite Q)
    The threshold for declaring a problem 'successful' is set to Q > 85 without reported sensitivity analysis (Section 3.3).
  • Agent-level passing threshold tau = Not reported
    The pass rate definition depends on tau, but its value is never specified, so the binary indicators used for PR and AA are underdetermined (Section 3.3).
assumptions (4)
  • domain assumption Bloom's Taxonomy is a valid model for assessing LLM cognitive skills.
    The whole scoring architecture maps six agents to Bloom's levels without empirical justification that this mapping measures the same constructs in machines (Section 3.1).
  • ad hoc to paper The 'Five Keys' components capture the relevant dimensions of math word problem quality.
    This rubric is introduced in Appendix A.1 and used in every agent prompt; no validation against established question-quality frameworks or human ratings is provided.
  • ad hoc to paper GPT-4O-based agents produce reliable performance and confidence scores without human calibration.
    The framework treats LLM judge scores as ground truth; the paper's Limitation section acknowledges the absence of external expert verification.
  • ad hoc to paper Iterative text generation instantiates a think-aloud protocol.
    The think-aloud protocol from cognitive psychology is invoked, but the LLM is not asked to verbalize concurrent thoughts; it merely generates revised problems from prompts (Section 3.4).
invented entities (2)
  • The 'Five Keys' rubric
    purpose: A new scoring schema for evaluating math word problem quality and cognitive rigor.
    Introduced in Appendix A.1 and embedded in all agent prompts; no external validation or prior literature establishes its validity.
  • Synthetic flawed problems generated by GPT-4O
    purpose: The synthetic subset D_syn_bad used as evaluation material.
    These are produced by the same model that serves as judge, so they inherit its notion of what counts as a flaw (Section 3.2.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of THiNK: Can Large Language Models Think-aloud?." pith.science (2026). https://pith.science/paper/OTFG2SJS

@misc{pith2026250520184,
  author       = {Pith},
  title        = {Pith review of: THiNK: Can Large Language Models Think-aloud?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTFG2SJS}},
  note         = {Machine review of arXiv:2505.20184}
}
read the original abstract

Assessing higher-order thinking skills in large language models (LLMs) remains a fundamental challenge, especially in tasks that go beyond surface-level accuracy. In this work, we propose THiNK (Testing Higher-order Notion of Knowledge), a multi-agent, feedback-driven evaluation framework grounded in Bloom's Taxonomy. THiNK frames reasoning assessment as an iterative task of problem generation, critique, and revision, encouraging LLMs to think-aloud through step-by-step reflection and refinement. This enables a systematic evaluation of both lower-order (e.g., remember, understand) and higher-order (e.g., evaluate, create) thinking skills. We apply THiNK to seven state-of-the-art LLMs and perform a detailed cognitive analysis of their outputs. Results reveal that while models reliably perform lower-order categories well, they struggle with applying knowledge in realistic contexts and exhibit limited abstraction. Structured feedback loops significantly improve reasoning performance, particularly in higher-order thinking. Qualitative evaluations further confirm that THiNK-guided outputs better align with domain logic and problem structure. The code of our framework provides a scalable methodology for probing and enhancing LLM reasoning, offering new directions for evaluation grounded in learning science, which is available at our GitHub repository.

Figures

Figures reproduced from arXiv: 2505.20184 by the authors.

Figure 1
Figure 1. A figure shows the "think-aloud" process [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the THINK. The pipeline begins with flawed math problems that are iteratively refined. The core multi-agent evaluation stage uses six Bloom-aligned agents and one heuristic agent to assess quality, providing scores and targeted feedback. Guided by the "Five Keys" and prior suggestions, LLMs revise or generate new problems via a think-aloud process. A quality threshold determines success or triggers furth… view at source ↗
Figure 3
Figure 3. Comparison between HOT and LOT. The scale is the sum of scores across corresponding levels. lower average Rpass values, indicating faster con￾vergence during iterative revision. However, this efficiency often comes at the cost of lower final quality scores, reflecting a trade-off between revi￾sion speed and output quality. These findings sug￾gest that while smaller models may adapt feedback quicker, larger models ex… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's Taxonomy

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Linear probes on LLM residual streams classify Bloom's Taxonomy levels with high accuracy, but the result may reflect prompt lexico-semantic cues rather than a general cognitive-complexity representation.

Reference graph

Works this paper leans on

34 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    By analyzing which concepts are invoked, we assess the knowledge dimension activated during problem-solving and whether the LLM navigates these domains coherently

    Math Concepts and Domains: This dimension identifies the core mathematical ideas underlying a task, such as algebraic structures, number theory, or geometry. By analyzing which concepts are invoked, we assess the knowledge dimension activated during problem-solving and whether the LLM navigates these domains coherently

    work page

  2. [2]

    Thilo Hagendorff, Sarah Fabi, and Michal Kosinski

    Has the creativity of large-language models peaked? an analysis of inter- and intra-llm variability. Thilo Hagendorff, Sarah Fabi, and Michal Kosinski

    work page

  3. [3]

    Representations are critical for logical coherence and traceability in reasoning

    Mathematical Representations: These include formal expressions (e.g., symbolic notation, equa- tions), diagrams, or stepwise procedures. Representations are critical for logical coherence and traceability in reasoning. Evaluating this component helps identify whether an LLM applies opera- tions in a structured and intelligible manner

    work page

  4. [4]

    A model’s ability to adapt its reasoning across such variants reflects generalization ability—an essential attribute of HOT

    Alternative Values: This refers to variations in the input parameters of a problem that preserve its underlying structure. A model’s ability to adapt its reasoning across such variants reflects generalization ability—an essential attribute of HOT

    work page

  5. [5]

    Five Keys

    Narrative Stories: Embedding problems in real-world or socio-cultural contexts situates abstract mathematical reasoning within meaningful scenarios. This component supports engagement and contextual transfer, and allows us to probe whether the LLM can maintain reasoning integrity when the task is couched in diverse narrative frames. By formally integratin...

    work page

  6. [6]

    In 2023 IEEE International Con- ference On Artificial Intelligence Testing (AITest) , pages 169–171

    Leveraging cognitive science for testing large language models. In 2023 IEEE International Con- ference On Artificial Intelligence Testing (AITest) , pages 169–171. IEEE. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, and 1 others. 2022. Beyond...

    arXiv 2023

  7. [8]

    ArXiv, abs/2401.12491

    Assessing and understanding creativity in large language models. ArXiv, abs/2401.12491. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others

    arXiv

  8. [9]

    Five Keys

    Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Pro- cessing Systems, 36:46595–46623. Qingyu Zhou and Danqing Huang. 2019. Towards gen- erating math word problems from equations and top- ics. In Proceedings of the 12th international confer- ence on natural language generation, pages 494–503. Anat Zohar and Yehudit J ...

    work page 2019

Show all 34 references
  1. [11]

    These skills serve as proxies for prior knowledge and inform whether the LLM draws upon relevant background competence

    Prerequisite Skills: This component captures the foundational knowledge—both conceptual and procedural—that a learner or model must possess to attempt a solution. These skills serve as proxies for prior knowledge and inform whether the LLM draws upon relevant background competence

  2. [15]

    You should try to understand and retrieve the specific mathematical information in it such as facts, patterns, objects, or contextual information, and decipher these meanings

    Analyze the generated problem of the last round. You should try to understand and retrieve the specific mathematical information in it such as facts, patterns, objects, or contextual information, and decipher these meanings

  3. [16]

    It includes understanding and organizing information, analyzing relationships, drawing conclusions, and distinguishing nuances

    Use cognitive skills essential for processing and applying information effectively. It includes understanding and organizing information, analyzing relationships, drawing conclusions, and distinguishing nuances. Additionally, you should evaluate ideas critically

  4. [17]

    These new expressions should have the same form as the given expressions in the previous generated math problem

    Generate mathematical expressions for the new problems. These new expressions should have the same form as the given expressions in the previous generated math problem. They must have the same complexity as well. Choose values to substitute into the expression, and calculate t...

  5. [18]

    The generated stories must be a mathematical word problem with the corresponding expressions

    Generate stories for these mathematical expressions with the appropriate questions based on the chosen values. The generated stories must be a mathematical word problem with the corresponding expressions. The story must be creative and unique

  6. [19]

    question

    Following and combining the previous steps, and you will generate a new creative version of the given math problem. Review the generated new version math problem, ensuring all the criteria are satisfied and double check it. Provide your evaluation in JSON format with these exa...

  7. [20]

    Ambiguous wording or missing critical information

  8. [21]

    Unrealistic assumptions or scenarios

  9. [22]

    Multiple possible interpretations

  10. [23]

    Contradictory information

  11. [24]

    ID": null,

    Unclear requirements or expectations The question should follow this format: {{ "ID": null, "question": "The question text", "LaTeX question": "The question text with LaTeX formatting", "solution": "Explanation of why the question is flawed and what information is missing or a...

  12. [25]

    Math concepts and domains

  13. [26]

    Required skills to solve the problem

  14. [27]

    Math expressions as sequence of operations

  15. [28]

    Values that substitute into expressions

  16. [29]

    The narrative story based on real-life socio-cultural experiences **Step 2: Lexical and Syntactic Complexity Analysis** - Type-Token Ratio (TTR) - Yngve Score - Frazier Score - Frazier–Roark Score - Developmental Level - Syntactic Frequency - Mean Dependency Distance (MDD) - S...

  17. [30]

    Lexical and Syntactic Complexity

  18. [31]

    Error Count and Severity

  19. [32]

    Clarity and Solvability

  20. [33]

    Answerability Penalty

  21. [34]

    performance_score

    Structural Consistency and Creativity **Scoring Guidance:** - 90–100: Clear, simple, and error-free problem. - 70–89: Minor complexity or errors that slightly impact clarity. - 50–69: Moderate complexity and multiple identifiable issues. - 0–49: Significant errors, ambiguity, ...

  22. [2010]

    English language teaching, 3(4):237– 248

    Vygotsky’s zone of proximal development: Instructional implications and teachers’ professional development. English language teaching, 3(4):237– 248. Ramya Srinivasan, Hiroya Inakoshi, and Kanji Uchino

  23. [2018]

    Inter- national journal of social sciences and humanities, 2(1):24–32

    Higher order thinking skills assessment to- wards critical thinking on mathematics lesson. Inter- national journal of social sciences and humanities, 2(1):24–32. Michael D Wolcott and Nikki G Lobczowski. 2021. Us- ing cognitive interviews and think-aloud protocols to understan...

    2021 arXiv

  24. [2023]

    Nature Computational Science, 3(10):833–838

    Human-like intuitive behavior and reasoning biases emerged in large language models but disap- peared in chatgpt. Nature Computational Science, 3(10):833–838. Yinghui He, Yufan Wu, Yilin Jia, Rada Mihalcea, Yu- long Chen, and Naihao Deng. 2023. Hi-tom: A benchmark for evaluati...

    2023 arXiv

  25. [2024]

    TOFEDU: The Future of Education Journal, 3(5):1488–1499

    Enhancing higher order thinking skills (hots) in education: Strategies and outcomes. TOFEDU: The Future of Education Journal, 3(5):1488–1499. Steffen Saifer. 2018. HOT skills: Developing higher- order thinking in young learners. Redleaf Press. Nicy Scaria, Suma Chenna, and Dee...

    2018

  26. [2025]

    In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–23

    Think together and work better: Combining humans’ and llms’ think-aloud outcomes for effective text evaluation. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–23. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educat...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.