Pith. sign in

REVIEW 3 major objections 6 minor 58 references

The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that state-of-the-art LLMs pass natural-language tests on physical concepts while failing abstract grid transformations of the same concepts, a paired gap it presents as quantitative evidence of the stochastic parrot…

desk verdict PhysiCo is a useful new benchmark with a real human-LLM gap, but the stochastic parrot conclusion rests on a validity assumption about grid labels that the paper doesn't fully establish. read the letter →

arxiv 2502.08946 v1 pith:QC4FNFRY submitted 2025-02-13 cs.CL cs.AIcs.CVcs.LG

Mo Yu , Lemao Liu , Junjie Wu , Tsz Ting Chung , Shunchi Zhang , Jiangnan Li , Dit-Yan Yeung , Jie Zhou This is my paper · ORCID
classification cs.CLcs.AIcs.CVcs.LG
keywords stochasticparrotphysicalconceptunderstandingsummativeassessmentgridreasoningLLMevaluationabstractPhysiCo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to make the stochastic parrot critique measurable. It introduces PhysiCo, a set of four-choice tasks on 52 high-school physical concepts, where each concept appears twice: once as ordinary text or images the model must recognize or describe, and once as input-output grid pairs that visually encode the same physical idea. State-of-the-art models score above 95 percent on the text and image side, while on the grid side they land roughly 40 percentage points below human accuracy, with humans near 90 percent and the best models in the mid-40s to mid-50s. The authors read this paired gap as quantitative evidence that LLMs can recall and restate concept knowledge without being able to apply it to abstract representations, the pattern the stochastic parrot metaphor names.

What carries the argument

The load-bearing mechanism is a paired summative assessment built on two cognitive levels. Low-level tasks, such as selecting a concept from a masked Wikipedia definition, recognizing it in photos, and generating a free-text description, target memory and paraphrase. High-level tasks present three input-output grid pairs as a four-choice concept-identification problem, where the grids abstract physical phenomena into colored-cell transformations so that solving them requires mapping a transformation rule to a concept rather than retrieving text. The same four-choice format and concept pool keep the two levels comparable, making the low-level versus high-level gap the quantitative signature of parroting. Auxiliary controls use in-context prompting, chain-of-thought, and fine-tuning on synthetic grid data to test whether grid unfamiliarity, rather than missing conceptual understanding, explains the failure.

What would settle it

A concrete check: take a random sample of PhysiCo-ASSOCIATIVE grid pairs, ask a fresh panel of annotators to supply their own concept labels without seeing the official choices, and measure agreement with the ground-truth label. If independent label agreement is close to the model error rate, then the benchmark's labels depend on individual association and part of the reported human-model gap reflects label subjectivity rather than a deficit in physical understanding; if humans overwhelmingly reproduce the assigned labels while all models still fail, the authors' interpretation survives.

Watch

Extended reading notes

Core claim

The paper's central claim is that the stochastic parrot phenomenon is real and measurable for physical concepts: LLMs retrieve and restate concept knowledge fluently but cannot apply it to abstract grid transformations of the same concepts. On PhysiCo, text-only and multimodal models including GPT-4o, o1, and Gemini 2.0 Flash Thinking score above 95 percent on low-level concept selection while remaining around 40 percentage points below humans on high-level tasks, with human accuracy at 92.0, 89.5, and 77.8 across the three sets and the best models roughly 46-55. The authors further claim that this failure is intrinsic rather than format-driven, because in-context learning, chain-of-thought prompting, and fine-tuning on grid data or on labeled examples of the same concepts produce little or no improvement.

Load-bearing premise

The diagnosis rests on the assumption that each grid pair has one correct physical-concept label, chosen by the annotators, so a model picking a different option has necessarily failed to understand the concept.

Editorial extensions

If this is right

  • Concept knowledge measured by natural-language question answering is not equivalent to concept knowledge measured by abstract relational tasks, so language-only benchmark scores can overstate physical understanding.
  • Current prompting and fine-tuning recipes, including few-shot demonstrations, chain-of-thought, synthetic matrix data, and training on labeled examples of the same concepts, are unlikely to close the gap because they address format familiarity rather than the underlying abstraction.
  • PhysiCo can serve as a summative test for future models: a model approaching human-level on the CORE and ASSOCIATIVE sets would be evidence that deep physical-concept understanding is improving rather than just language mimicry.
  • Reasoning-oriented models improve somewhat over base LLMs but remain far below humans, so test-time computation alone does not remove the deficit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paired-gap result generalizes beyond physics, then claims that LLMs understand any rich domain should be backed by at least one transfer task that cannot be solved by recalling training text; language-only evaluations are a weak upper bound.
  • A testable extension of PhysiCo would vary the abstraction level of the grid encodings per concept, from iconic to arbitrary, to estimate how much of the human-model gap is due to the distance between the grid transformation and the real-world phenomenon.
  • The ASSOCIATIVE set's 10.4 percent annotator disagreement suggests that concept labels there are partly conventional; future versions could report per-instance label agreement and model accuracy conditional on agreement, separating ambiguous-label errors from failed-understanding errors.
  • The authors leave open whether the gap is inherent to LLMs or a training-data artifact; a strong test would train a model from scratch on corpora that pair grid transformations with physical-concept language and measure whether PhysiCo accuracy rises.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces PhysiCo, a benchmark for assessing whether LLMs understand physical concepts, built around Bloom's taxonomy as a two-level summative assessment. Low-level subtasks ask models to recognize physical concepts from masked Wikipedia-style definitions and from natural images, and to generate concept descriptions; high-level subtasks ask models to classify three input-output grid-pair demonstrations into one of four physical concepts, using hand-authored CORE grids and ARC-derived ASSOCIATIVE grids. Across six research questions, the authors report that most LLMs score above 90-95% on the low-level tasks, while state-of-the-art models such as GPT-4o, o1, o3-mini-high, Gemini 2.0 Flash Thinking, and DeepSeek-R1 score roughly 42-52% on the high-level CORE-Test grid tasks, compared with 89.5% for humans. The authors interpret this discrepancy as quantitative evidence of the 'stochastic parrot' phenomenon and argue, via in-context learning and fine-tuning experiments, that the gap reflects intrinsic difficulty of deep understanding rather than unfamiliarity with the grid format.

Significance. If the validity assumptions hold, this is one of the first controlled, paired demonstrations that LLMs can verbalize and recognize concepts while failing to apply them in a non-linguistic abstract task, lending quantitative support to the stochastic-parrot debate. The benchmark construction is thoughtful: it pairs low-level and high-level tasks over the same concepts, includes human baselines, and probes format familiarity and training robustness. The reported central gap is large and robust in magnitude, and the public benchmark release is a useful community resource. The main risk to the central claim is the unvalidated assumption that the assigned grid labels are recoverable from the grid content alone, rather than from annotation conventions shared by the designers and human judges; this needs to be addressed before the stochastic-parrot conclusion can be taken at face value.

major comments (3)
  1. [§3.2.2, §6 (RQ2)] The gold labels are not shown to be recoverable from grid content alone, and this is load-bearing for the stochastic-parrot claim. Section 3.2.2 states that each grid pair is 'guaranteed' to illustrate the assigned concept, but this is an annotator assertion, and three input-output pairs underdetermine the transformation rule; the intended physical interpretation is what fixes the label. The RQ2 human validation uses the same four-choice format and judges who share the authors' design conventions, so the 89.5% accuracy shows recoverability given those conventions, not recoverability from the grid alone. The paper's own ASSOCIATIVE data illustrate the problem: 10.4% of instances drew disagreement, and human accuracy on those was 33.3%. I recommend adding a control study in which naive annotators provide free-form concept labels for sampled grid pairs without candidate options, and reporting inter-annotator agreement on label assignment. Without such evidence, the observed low-level/high-level gap could partly reflect difficulty in inferring the annotation convention rather than absence of physical concept understanding.
  2. [§6 (RQ5)] The format-familiarity checks do not rule out the convention problem. The 86.7% object-recognition result shows that GPT-4o can parse colors, shapes, and positions in the grids, but recognizing glyphs is not the same as knowing which physical concept a particular transformation is intended to denote. The ICL and fine-tuning experiments in Table 4 increase familiarity with matrix formats, but they do not expose the model to the specific concept-label mapping on held-out concepts; RQ6 supplies same-concept demonstrations, but only on an 80-example subset, and the results overlap within error bars (Table 5). A stronger test would provide explicit concept-label demonstrations for a few instances per concept and then test whether models can apply that mapping to new instances; if they still fail, the convention explanation would be substantially weakened.
  3. [Table 3, §6 (RQ3)] The thinking-model results (o1, o3-mini-high, DeepSeek-R1) are reported as single numbers without standard deviations or the number of runs, despite the protocol stated in Section 5 that 'we run each experiment 3 times and compute the average and standard derivation.' Because these models are part of the state-of-the-art comparison, the paper should either run them under the same protocol or clearly mark single-run estimates as preliminary; the current presentation makes the comparison between these rows and the others potentially misleading.
minor comments (6)
  1. [§3.2.2, Table 3 caption] The italic-font notation for the thinking models is not defined in the table caption; please add a footnote specifying the exact model snapshots for o1, o3-mini-high, DeepSeek-R1, and Gemini 2.0 Flash Thinking Experimental.
  2. [Abstract] The abstract contains a subject-verb agreement error: 'The grids represents varying levels' should be 'The grids represent varying levels'.
  3. [Figure 5 prompt, Figure 8 prompt] There is a typo in Figure 5 ('definiton' should be 'definition'), and the final line of the user prompt in Figure 8 is duplicated ('Your response should end with your choice of answer.') twice; please fix both.
  4. [§6 (RQ3), Appendix D] The model identifier is inconsistent: the main text refers to 'o1-2024-12-17' while Appendix D uses 'o1-preview'; please adopt a single naming convention throughout.
  5. [§5.2 (Remark)] The recall evaluation of generated descriptions uses the same annotators who created the CORE gold properties; since the judges know which properties they intended, this may inflate the recall numbers. Please state whether any independent judges were used or acknowledge this as a limitation.
  6. [§7] The claim that 'there is a lack of quantitative experiments to precisely verify this viewpoint' is too strong, given the substantial prior work on LLM understanding benchmarks; consider softening it to 'no controlled paired comparison' to avoid overstating the novelty.

Circularity Check

1 steps flagged · score 2.0 of 10

The main stochastic-parrot claim rests on a measured performance gap, not on a fitted parameter or on a self-citation chain. The only self-referential element is the secondary self-play metric for concept generation, which the paper does not use to drive its central conclusion.

  1. other [Appendix B.3, 'Additional Results on the Self-Play Game']
    "For each generated description of a concept, we mask the synonyms of the concept in it as in the previous selection subtask, and ask the same LLM to identify the concept being described from four options."

    This metric is explicitly self-referential: the same model both produces the description and is then tested on recognizing the concept from that description. High self-play accuracy partly reflects self-consistency and the model's access to lexical cues it generated itself, rather than independent recoverability of the concept from the text. However, the paper does not rely on this metric for the main claim: Section 5.2 also reports human evaluation with 100% for GPT-3.5 and GPT-4, and the low-level conclusion is separately supported by near-ceiling concept-selection accuracy. Thus this self-play step is a minor self-referential evaluation, not a load-bearing circular derivation of the stochastic-parrot conclusion.

full rationale

The paper's central derivation is an empirical comparison: LLMs score above 95% on natural-language concept selection and generation, while scoring far lower on the grid-based PhysiCo tasks. That gap is interpreted as evidence of the stochastic-parrot phenomenon, but the interpretation is not equivalent to the inputs by construction. There are no fitted parameters renamed as predictions, and no equation reduces the high-level task outcome to the low-level task outcome. The authors' use of human accuracy in RQ2 as validity evidence rests on the assumption that the grid pairs unambiguously encode the annotated physical concept; this is a construct-validity concern rather than a circular derivation, especially because the paper itself discloses the 10.4% disagreement and 33.3% human accuracy on disagreed ASSOCIATIVE instances. RQ5's format-familiarity checks (86.7% object recognition, ICL/FT on grid-format data) address an alternative explanation and are not circular. The concurrent self-citation (Wu et al., 2025) is corroborative and not load-bearing, since the paper's own human study independently shows grid familiarity. The only self-referential element is the secondary self-play metric in Appendix B.3, which does not carry the main argument. Overall, the central claim has independent empirical content, so the circularity score is low.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric parameters are fitted; the central claim is an empirical benchmark result. The load-bearing assumptions are about task validity and ground-truth labeling, listed below. No new physical entities are posited.

assumptions (4)
  • domain assumption Success on the grid-format high-level subtasks is a valid operationalization of deep physical concept understanding.
    Section 2 and Section 3.1 map Bloom's taxonomy to the two subtask levels; this mapping is assumed, not derived, and underlies the stochastic parrot conclusion.
  • domain assumption The annotated labels for CORE and ASSOCIATIVE grid instances are treated as ground truth.
    Section 3.2.2; human annotators' associations define the correct answers. For ASSOCIATIVE, annotators themselves disagree on 10.4% of instances, so the ground truth is partly subjective.
  • domain assumption Grid-format inputs reduce memorization because LLMs' pretraining data is less likely to correlate matrix patterns with physical concepts.
    Section 3.2.2; this is plausible but unverified. RQ5 tests format familiarity only indirectly for visual grids, not for memorized associations.
  • domain assumption LoRA fine-tuning and ICL on the provided data are representative of what could improve performance.
    Section 6 RQ6 and Appendix F; the negative result is used to conclude intrinsic difficulty, but only light-weight training was tried on small data, including 80 examples for the ICL subset and LoRA for 3 epochs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding." pith.science (2026). https://pith.science/paper/QC4FNFRY

@misc{pith2026250208946,
  author       = {Pith},
  title        = {Pith review of: The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QC4FNFRY}},
  note         = {Machine review of arXiv:2502.08946}
}
read the original abstract

In a systematic way, we investigate a widely asked question: Do LLMs really understand what they say?, which relates to the more familiar term Stochastic Parrot. To this end, we propose a summative assessment over a carefully designed physical concept understanding task, PhysiCo. Our task alleviates the memorization issue via the usage of grid-format inputs that abstractly describe physical phenomena. The grids represents varying levels of understanding, from the core phenomenon, application examples to analogies to other abstract patterns in the grid world. A comprehensive study on our task demonstrates: (1) state-of-the-art LLMs, including GPT-4o, o1 and Gemini 2.0 flash thinking, lag behind humans by ~40%; (2) the stochastic parrot phenomenon is present in LLMs, as they fail on our grid task but can describe and recognize the same concepts well in natural language; (3) our task challenges the LLMs due to intrinsic difficulties rather than the unfamiliar grid format, as in-context learning and fine-tuning on same formatted data added little to their performance.

Figures

Figures reproduced from arXiv: 2502.08946 by the authors.

Figure 1
Figure 1. Illustration of a “Stochastic Parrot” by our [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Examples of input-output grids labeled as [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of the research questions answered [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The prompt template used for generating de [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: The prompt template used for guessing the [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: An example of our masked description for the concept [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The prompt template used in RQ 3. The pair of an INPUT GRID and an OUTPUT GRID consists of one example of a physical phenomenon in matrix format. {{ UPLOADED IMAGE }} [USER] In the given image, there are two columns of matrices with elements represented by different co…
Figure 8
Figure 8. Figure 8: The prompt template used in RQ 4. UPLOADED IMAGE is an image consists of three or more examples like in [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 21 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Sam Acquaviva, Yewen Pu, Marta Kryven, Theodoros Sechopoulos, Catherine Wong, Gabrielle Ecanow, Maxwell Nye, Michael Tessler, and Josh Tenenbaum. 2022. Communicating natural programs to humans and machines. Advances in Neural Information Processing Systems, 35:3731--3743

    work page 2022

  5. [5]

    Patricia Armstrong. 2010. Bloom’s taxonomy. Vanderbilt University Center for Teaching, pages 1--3

    work page 2010

  6. [6]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

    2005

  7. [7]

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610--623

    2021

  8. [8]

    Emily M Bender and Alexander Koller. 2020. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5185--5198

    work page 2020

Show all 58 references
  1. [9]

    a is b" fail to learn

    Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023. The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv preprint arXiv:2309.12288

    2023 arXiv

  2. [10]

    Paul Black and Dylan Wiliam. 1998 a . Assessment and classroom learning. Assessment in Education: principles, policy & practice, 5(1):7--74

    1998

  3. [11]

    Paul Black and Dylan Wiliam. 1998 b . Inside the black box: Raising standards through classroom assessment. Granada Learning

    1998

  4. [12]

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258

    2021 arXiv

  5. [13]

    Ali Borji. 2023. Stochastic parrots or intelligent systems? a perspective on true depth of understanding in llms. A Perspective on True Depth of Understanding in LLMs (July 11, 2023)

    2023

  6. [14]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  7. [15]

    Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.481 FLUTE : Figurative language understanding through textual explanations . In Proceedings of the 2022 Conference on Empirical Methods in Natural Langua...

    2022 doi

  8. [16]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  9. [17]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821

    2024 arXiv

  10. [18]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv ...

    2023 arXiv

  11. [19]

    Fran c ois Chollet. 2019. On the measure of intelligence. arXiv preprint arXiv:1911.01547

    2019 arXiv

  12. [20]

    Dorottya Demszky, Diyi Yang, David S Yeager, Christopher J Bryan, Margarett Clapper, Susannah Chandhok, Johannes C Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, et al. 2023. Using large language models in psychology. Nature Reviews Psychology, 2(11):688--701

    2023

  13. [21]

    Denis Donadel, Francesco Marchiori, Luca Pajola, and Mauro Conti. 2024. Can llms understand computer networks? towards a virtual system administrator. arXiv preprint arXiv:2404.12689

    2024 arXiv

  14. [22]

    Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. 2024. Flocks of stochastic parrots: Differentially private prompt learning for large language models. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. 2022. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230--244

    2022

  16. [24]

    Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580--587

    2014

  17. [25]

    Wynne Harlen and Mary James. 1997. Assessment and learning: differences and relationships between formative and summative assessment. Assessment in education: Principles, policy & practice, 4(3):365--379

    1997

  18. [26]

    Kaiming He, Georgia Gkioxari, Piotr Doll \'a r, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961--2969

    2017

  19. [27]

    Da Silva Gameiro Henrique, Andrei Kucharavy, and Rachid Guerraoui. 2023. Stochastic parrots looking for stochastic parrots: Llms are easy to fine-tune and hard to detect with other llms. arXiv preprint arXiv:2304.08968

    2023 arXiv

  20. [28]

    Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi

    Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2023. https://doi.org/10.18653/v1/2023.acl-long.41 Do androids laugh at electric sheep? humor `` understanding '' benchmarks from the new yorker caption contest . In...

    2023 doi

  21. [29]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

    2021

  22. [30]

    Di Huang, Ziyuan Nan, Xing Hu, Pengwei Jin, Shaohui Peng, Yuanbo Wen, Rui Zhang, Zidong Du, Qi Guo, Yewen Pu, et al. 2024. Anpl: Towards natural programming with interactive decomposition. Advances in Neural Information Processing Systems, 36

    2024

  23. [31]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  24. [32]

    David R Krathwohl. 2002. A revision of bloom's taxonomy: An overview. Theory into practice, 41(4):212--218

    2002

  25. [33]

    Jiangnan Li, Qiujing Wang, Liyan Xu, Wenjie Pang, Mo Yu, Zheng Lin, Weiping Wang, and Jie Zhou. 2024. Previously on the stories: Recap snippet identification for story reading. arXiv preprint arXiv:2402.07271

    2024 arXiv

  26. [34]

    Zihao Li. 2023. The dark side of chatgpt: legal and ethical challenges from stochastic parrots and hallucination. arXiv preprint arXiv:2304.14347

    2023

  27. [35]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023 a . Improved baselines with visual instruction tuning

    2023

  28. [36]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 b . Visual instruction tuning. In NeurIPS

    2023

  29. [37]

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431--3440

    2015

  30. [38]

    MetaAI. 2024. https://ai.meta.com/blog/meta-llama-3/ Introducing meta llama 3: The most capable openly available llm to date

    2024

  31. [39]

    Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1--40

    2023

  32. [40]

    Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. 2023. https://arxiv.org/abs/2307.04721 Large language models as general pattern machines . ArXiv preprint, abs/2307.04721

    2023 arXiv

  33. [41]

    Melanie Mitchell and David C Krakauer. 2023. The debate over understanding in ai’s large language models. Proceedings of the National Academy of Sciences, 120(13):e2215907120

    2023

  34. [42]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  35. [43]

    Di Peng, Liubin Zheng, Dan Liu, Cheng Han, Xin Wang, Yan Yang, Li Song, Miaoying Zhao, Yanfeng Wei, Jiayi Li, et al. 2024. Large-language models facilitate discovery of the molecular signatures regulating sleep and activity. Nature Communications, 15(1):3685

    2024

  36. [44]

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. 2019. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vi...

    2019

  37. [45]

    Natalie Shapira, Guy Zwirn, and Yoav Goldberg. 2023. https://doi.org/10.18653/v1/2023.findings-acl.663 How well do large language models perform on faux pas tests? In Findings of the Association for Computational Linguistics: ACL 2023, pages 10438--10451, Toronto, Canada. Asso...

    2023 doi

  38. [46]

    John Chong Min Tan and Mehul Motani. 2023. https://arxiv.org/abs/2310.05146 Large language model (llm) as a system of multiple expert agents: An approach to solve the abstraction and reasoning corpus (arc) challenge . ArXiv preprint, abs/2310.05146

    2023 arXiv

  39. [47]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  40. [48]

    Xiaoyu Tong, Rochelle Choenni, Martha Lewis, and Ekaterina Shutova. 2024. Metaphor understanding challenge dataset for llms. arXiv preprint arXiv:2403.11810

    2024 arXiv

  41. [49]

    Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D Goodman. 2023. https://arxiv.org/abs/2309.05660 Hypothesis search: Inductive reasoning with language models . ArXiv preprint, abs/2309.05660

    2023 arXiv

  42. [50]

    Yile Wang, Sijie Cheng, Zixin Sun, Peng Li, and Yang Liu. 2024. https://arxiv.org/abs/2401.11725 Speak it out: Solving symbol-related problems with symbol-to-language conversion for language models . ArXiv preprint, abs/2401.11725

    2024 arXiv

  43. [51]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  44. [52]

    Junjie Wu, Mo Yu, Lemao Liu, Dit-Yan Yeung, and Jie Zhou. 2025. https://wujunjie1998.github.io/araoc-benchmark.github.io/ Understanding llms’ fluid intelligence deficiency: An analysis of the arc task . In Proceedings of the 2025 Conference of the Nations of the Americas Chapt...

    2025

  45. [53]

    Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. 2023. Language models meet world models: Embodied experiences enhance language models. Advances in neural information processing systems, 36

    2023

  46. [54]

    Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias B Khalil. 2023. https://arxiv.org/abs/2305.18354 Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations . ArXiv preprint, abs/2305.18354

    2023 arXiv

  47. [55]

    Mo Yu, Jiangnan Li, Shunyu Yao, Wenjie Pang, Xiaochen Zhou, Zhou Xiao, Fandong Meng, and Jie Zhou. 2023. Personality understanding of fictional characters during book reading. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2023

  48. [56]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  49. [57]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  50. [58]

    [[[CONCEPT]]]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.