Pith. sign in

REVIEW 3 major objections 6 minor 30 references

Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs score more than 35 points below humans on Korean sound-change questions, and supplying pronunciations narrows that gap.

desk verdict KoGEM is a solid, reusable Korean grammar benchmark, but the 'experiential knowledge' explanation is confounded by answer leakage in the §6.2 intervention. read the letter →

arxiv 2506.01237 v1 pith:BYLS2SC2 submitted 2025-06-02 cs.CL cs.AI

SungHo Kim , Nayeon Kim , Taehee Jeon , SangKeun Lee This is my paper
classification cs.CLcs.AI
keywords Koreangrammarlinguisticcompetencephonologicalalternationlargelanguagemodelszero-shotevaluationexperientialknowledgeKoGEMbenchmarkmorphology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KoGEM is a new 1,524-question Korean grammar benchmark drawn from official Korean exams, organized into 16 subcategories under phonology, morphology, syntax, semantics, and norms. Zero-shot testing of 27 LLMs against human scores shows the models do surprisingly well on definitional and rule-based tasks but fall far behind on tasks that require real-world linguistic experience, especially phonological alternation, where the average LLM trails humans by more than 35 percentage points. The paper argues this gap is not general weakness but a specific lack of spoken-language experience: appending machine-generated pronunciation text to the answer choices improves accuracy by 3.1 to 26.3 percent, and appending morpheme decompositions improves morphological questions by 7.1 to 20.0 percent. A sympathetic reader would take away that current LLMs have latent linguistic knowledge that can be activated when experiential cues are supplied, and that fine-grained grammar benchmarks expose abilities coarse accuracy hides.

What carries the argument

The load-bearing machinery is the benchmark itself: 1,524 multiple-choice items from official Korean exams, organized into five main categories and 16 subcategories so that tasks requiring experiential knowledge (Phonological Alternation, Morpheme, Pragmatics) are separated from rote tasks (Vocabulary, Orthography). On top of this taxonomy, the paper's diagnostic tool is a controlled prompt augmentation: it appends pronunciation text from the g2pK grapheme-to-phoneme converter to each choice for phonological questions, and appends morpheme segmentation from the Kiwi analyzer to each choice for morphological questions. These augmentations operationalize human subvocalization and morphological decomposition, letting the authors test whether performance rises when the missing experiential knowledge is supplied in text.

What would settle it

Re-run the Phonological Alternation items with choices whose g2pK transcriptions are phonetically plausible but do not spell out the alternation under test; if accuracy still rises by the same margin, the intervention is supplying usable textual redundancy rather than the missing phonological experience, and the experiential-knowledge diagnosis would need revision.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs' Korean linguistic competence is uneven in a specific way: they handle definitional knowledge and syntactic rule application almost at human level, but they lag sharply wherever answering requires grounding text in real-world linguistic experience. The largest measured failure is Phonological Alternation, where the average of 27 LLMs scores 29.06 percent versus 66.93 percent for humans, a gap of more than 35 points, and the model that spends the most time thinking (s1-32B) deliberates longest exactly on the subcategories where the human–model gap is largest. When prompts are augmented with the pronunciations that humans subvocalize, generated by the g2pK grapheme-to-phoneme converter, accuracy rises by 3.1 to 26.3 percent on Phonological Alternation; when morpheme decompositions from the Kiwi analyzer are appended, accuracy rises by 7.1 to 20.0 percent on Morpheme. The paper concludes that incorporating such experiential knowledge could enhance the linguistic competence of LLMs, and that KoGEM's fine-grained subcategories reveal hidden facets that aggregate scores conceal.

Load-bearing premise

The load-bearing premise is that the 35-point gap on Korean sound-change questions reflects missing real-world spoken-language experience rather than harder questions, tokenization quirks, or the pronunciation text revealing the answer.

Editorial extensions

If this is right

  • If the diagnosis is right, LLMs' poor Korean phonology is not solved by scaling alone: o1-preview, the only model to beat humans overall (81.04 vs 63.04), still posts its smallest margin over humans in Phonology.
  • The 3.1–26.3 percent gains from pronunciation text and 7.1–20.0 percent gains from morpheme text imply that prompt-side knowledge injection can partly compensate for missing experiential grounding.
  • Thinking time becomes a diagnostic: s1-32B deliberates longest on exactly the subcategories where the human–model gap is largest, suggesting LLMs treat experiential tasks as effortful rather than impossible.
  • Across 27 models the category-level pattern is stable, which makes it unlikely that simple memorization of the exam questions explains the variation.
  • Subcategory scores, not overall averages, are the right yardstick for linguistic competence, because an overall leader can still hide a 35-point phonology gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would replace g2pK outputs with transcriptions that leave the target alternation unresolved; if gains persist, the models are exploiting extra textual redundancy, and if gains vanish, the experiential-knowledge story is supported.
  • The same pipeline could be applied to other agglutinative or phonologically rich languages, since the five main categories are language-general; Korean-specific subcategories would need local norms and tools.
  • For model builders, adding explicit phoneme-level and morpheme-level annotations to training or prompts may buy more linguistic competence per unit of compute than scaling parameters or test-time thinking alone.
  • A cautionary reading: because the pronunciation text is generated deterministically from the written forms, part of the improvement could come from answer leakage; the paper itself leaves this ambiguity open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces KoGEM, a 1,524-question multiple-choice Korean grammar benchmark drawn from four types of official exams, organized into five main categories and 16 subcategories. The authors evaluate 27 LLMs in a zero-shot setting and obtain human performance from public answer-rate statistics and crowdsourcing. They report that the average LLM lags humans most on Phonological Alternation, Morpheme, and Pragmatics, and they attribute this gap to a lack of real-world experiential knowledge. To support this interpretation, they append g2pK-generated pronunciation text and Kiwi-based morpheme segmentations to the choices and observe accuracy gains of 3.1–26.3 and 7.1–20.0 percentage points respectively, concluding that incorporating experiential knowledge can enhance LLM linguistic competence.

Significance. The benchmark itself is a valuable contribution: the construction is careful and transparent, with OCR plus manual review, three Korean-language-major annotators with majority-vote categorization, and a public code/data release. Evaluating 27 models of varied provenance and comparing against human performance on official exam questions is a useful and reproducible setup that fills a real gap for Korean linguistic competence evaluation. The raw accuracy results and the fine-grained subcategory breakdown are solid descriptive findings. However, the central causal claim, that the human–LLM gap on phonological alternation stems from missing experiential knowledge and that appending such knowledge improves competence, is not uniquely supported by the reported experiments. The intervention can be explained by answer leakage, and the gap analysis lacks statistical or difficulty controls. The paper would be publishable after substantially qualifying the interpretive claim or adding the missing controls.

major comments (3)
  1. [Abstract; Sections 1 and 9; Section 6.2, Figure 4(a)] The abstract and Conclusion assert that LLMs struggle with tasks requiring real-world experiential knowledge and that supplying such knowledge can enhance linguistic competence. The Section 6.2 intervention does not uniquely license this causal reading. For Phonological Alternation items, appending g2pK pronunciation text to each choice provides the phonologically realized form that the question asks the model to identify, so the gains in Figure 4(a) are compatible with surface-level matching between the appended transcription and the orthographic choices, without any latent phonological experience. The Kiwi morpheme intervention in Figure 4(b) has the same structure, since the segmentation decomposes exactly the morphemes under test. The caveat in the final paragraph of Section 6.2 is more accurate than the abstract and conclusion, which drop it. I request that the claims be reworded to state that providing explicit phonological and morphological information improves accuracy, and that the experiential-knowledge interpretation be either supported by a control condition or explicitly labeled as one plausible account among others.
  2. [Section 5, Phonology; Table 8] The claim that the 35-percentage-point human–LLM gap on Phonological Alternation stems from LLMs' lack of multimodal, experiential knowledge is not supported by the analysis as reported. No significance test is provided, and no control is made for item difficulty, passage length, number of choices, or the interaction with Korean tokenization. Table 8 shows that this subcategory has a low absolute accuracy for most models, so the gap may reflect general item difficulty rather than a specifically phonological deficit. The authors should add per-item error analyses or an item-level regression that uses the available human per-question accuracies as a difficulty control, and they should discuss whether orthographic-tokenization schemes systematically disadvantage phonological alternation items. Without such controls, the 'grounded multimodal knowledge' explanation is only one of several plausible accounts of the observed gap.
  3. [Table 1; Section 4.3; Section 6.1, Figure 5] The headline comparison is built on the unweighted average of 27 highly heterogeneous models, and the paper reports no confidence intervals or significance tests for the difference between this average and human performance. The variance is large: Phonology accuracy ranges from 21.13% (Claude-3-haiku) to 71.83% (o1-preview) in Table 1, so statements about what 'LLMs' collectively cannot do should be qualified by this spread. Similarly, the thinking-time analysis in Figure 5 is presented as evidence that slow subcategories are those requiring experiential knowledge, but longer elapsed time may also be explained by input length or prompt complexity, since question-level covariates are not examined. These analyses should be reported as descriptive rather than as evidence for the causal deficit narrative.
minor comments (6)
  1. [Figure 4 caption] The caption states that the values between bars indicate the 'rate of increase', but the numbers (5.1, 26.3, 3.1) are arithmetic differences in accuracy percentage points, not relative rates of increase; please reword the caption to say 'percentage-point increase'.
  2. [Table 1 and Appendix A] There are systematic typographical spacing errors in the model name 'HyperCLOV A' (for example, in Table 1 and in Appendix A); these should be corrected to 'HyperCLOVA'.
  3. [Section 5, Syntax] The Korean example sentence '나는밥을먹는다*I rice eat' appears garbled and is repeated awkwardly; the Korean text, romanization, and English gloss should be formatted as separate, clearly separated lines.
  4. [Section 6.2] The g2pK example includes the English gloss 'To-day, the skai iz kli-er', which is not a literal gloss and reads as a phonological joke; please replace it with a standard interlinear gloss so the example serves its explanatory purpose.
  5. [Appendix B.2] The hyperparameter section says output tokens are capped at 200 but also says the prompt sets a 100-character maximum for the explanation; please clarify which constraint is active and whether this affected any responses.
  6. [Table 5 and Appendix G] The comparison of predicted class distributions to the gold label distribution is informal; adding a metric such as chi-square distance or KL divergence would make the claim of 'no severe class imbalance' more objective.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is externally grounded and the main results are measured, not derived from the paper's own labels or fitted parameters.

full rationale

KoGEM's measurements are externally grounded: items are extracted from official Korean exams (CSAT, NUAT, HSQE, CSE), and human scores come from published response statistics and crowdsourcing rather than from the models being evaluated. The zero-shot accuracies in Table 1 are raw empirical measurements under a fixed prompt; no parameter is fitted to a target and then renamed as a prediction. The taxonomy in Section 2 is defined before evaluation from theoretical linguistics and the Korean high-school curriculum, so the conclusion that LLMs lag on 'experiential knowledge' subcategories is an interpretation of measured gaps, not an identity. The Section 6.2 interventions append g2pK pronunciation or Kiwi morpheme text to choices; these are empirical ablations. There is a real confound: for Phonological Alternation items, the appended pronunciation text can encode the alternation the item probes, so the 3.1%-26.3% gains may reflect answer leakage rather than latent experiential knowledge, and the paper itself hedges in Section 6.2: 'this improvement does not conclusively determine whether LLMs possess latent pronunciation knowledge that remains underutilized or simply lack such knowledge.' A confounded explanation is a validity concern, not circularity, because the reported improvement is measured rather than derived from the conclusion. Self-citations (e.g., Kim et al., 2022, 2024d) are background descriptions of Korean morphology and are not load-bearing. No uniqueness theorem or ansatz is imported from the authors' prior work, and no prediction is equivalent to its inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical benchmark, not a derivation, so no fitted numerical parameters appear. The main cognitive load is carried by domain assumptions about the taxonomy, the proxy nature of exam questions, the reliability of human baselines, and the correctness of external linguistic tools used in the intervention experiments.

assumptions (4)
  • domain assumption The five-category, 16-subcategory taxonomy, based on Lyons (1968) and aligned with the Korean high school curriculum, partitions Korean grammar into meaningful components.
    The entire benchmark's interpretability rests on this taxonomy; if the categories do not cleanly separate distinct linguistic competences, the 'hidden facets' analysis could be an artifact of labeling.
  • domain assumption Prescriptive grammar knowledge, as tested in official Korean exams, is a valid proxy for 'linguistic competence'.
    The paper explicitly adopts prescriptive grammar and equality with exam performance; the connection between exam questions and underlying competence is presupposed.
  • domain assumption Human performance estimated from public exam statistics and 10+ crowdworker responses per question accurately reflects human linguistic competence in these categories.
    Section 4.2 and Appendix C; small sample sizes for HSQE/CSE and reliance on third-party statistics for CSAT/NUAT could bias the human baseline.
  • ad hoc to paper The g2pK and Kiwi tools produce correct pronunciations and morpheme segmentations for the test items.
    The intervention experiment in Section 6.2 depends on the accuracy of these external tools; any errors could either mask or inflate the performance gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean." pith.science (2026). https://pith.science/paper/BYLS2SC2

@misc{pith2026250601237,
  author       = {Pith},
  title        = {Pith review of: Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYLS2SC2}},
  note         = {Machine review of arXiv:2506.01237}
}
abstract

We introduce the $\underline{Ko}rean \underline{G}rammar \underline{E}valuation Bench\underline{M}ark (KoGEM)$, designed to assess the linguistic competence of LLMs and humans in Korean. KoGEM consists of 1.5k multiple-choice QA pairs covering five main categories and 16 subcategories. The zero-shot evaluation of 27 LLMs of various sizes and types reveals that while LLMs perform remarkably well on straightforward tasks requiring primarily definitional knowledge, they struggle with tasks that demand the integration of real-world experiential knowledge, such as phonological rules and pronunciation. Furthermore, our in-depth analysis suggests that incorporating such experiential knowledge could enhance the linguistic competence of LLMs. With KoGEM, we not only highlight the limitations of current LLMs in linguistic competence but also uncover hidden facets of LLMs in linguistic competence, paving the way for enhancing comprehensive language understanding. Our code and dataset are available at: https://github.com/SungHo3268/KoGEM.

Figures

Figures reproduced from arXiv: 2506.01237 by the authors.

Figure 1
Figure 1. Zero-shot accuracy of the top three LLMs and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Data distribution of KoGEM, categorized into [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Comparison of LLMs and humans across 16 subcategories. The distributions of all LLMs are depicted [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Comparison of performances in (a) Phonological Alternation and (b) Morpheme subcategories, with and without the additional knowledge, such as pronunciation text and morpheme text. The values between each bar graph pair indicate the rate of increase(▲) compared to the o…
Figure 5
Figure 5. Figure 5: Comparison of the average thinking time re￾quired to solve each question and the corresponding av￾erage score for each subcategory by the s1-32B model. Dashed lines indicate the three longest test times. erage time per subcategory. As shown in [PITH_FULL_IMAGE:figures…
Figure 7
Figure 7. Figure 7: Age distribution and accuracy of participants [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The specific numbers and ratios of linguistic categories and subcategories for each source exam. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Accuracy scores across subcategories for different Top-k average settings and overall LLM performance. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: The instruction provided to crowdworkers for human evaluation outlines the evaluation process, task [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: An example presented to participants during the human evaluation process illustrates the format and [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Examples in the Phonology Category [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Examples in the Morphology Category [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: Examples in the Syntax Category [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: Examples in the Semantics Category [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: Examples in the Norms Category [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 29 canonical work pages

  1. [1]

    It en- hances the connection of knowledge between Korean and English

    Bllossom-8B (Choi et al., 2024): This is a Korean-English bilingual language model based on the open-source LLama3. It en- hances the connection of knowledge between Korean and English

    work page 2024

  2. [2]

    It is trained by utilizing instruction fine-tuning methods, including su- pervised fine-tuning (SFT) and direct pref- erence optimization (DPO) (Rafailov et al., 2023)

    SOLAR-v1.0-10.7B-Instruct (Kim et al., 2024b): This is an advanced LLM with 10.7 billion parameters. It is trained by utilizing instruction fine-tuning methods, including su- pervised fine-tuning (SFT) and direct pref- erence optimization (DPO) (Rafailov et al., 2023)

    work page

  3. [3]

    This model is trained using the data, such as Peng et al

    KULLM-3-10.7B (Lee et al., 2024): This model is instruction-tuned from the upstage/SOLAR-10.7B-v1.0 (Kim et al., 2024b) model. This model is trained using the data, such as Peng et al. (2023) and mixed Korean instruction data (gpt-generated, hand-crafted, etc)

    work page 2023

  4. [4]

    Specifically, this model is trained by utilizing DPO.8

    EEVE-v1.0-10.8B-Instruct (Kim et al., 2024c): This model is a fine-tuned version of yanolja/EEVE-Korean-10.8B-v1.0,7 which is a Korean vocabulary-extended ver- sion of upstage/SOLAR-10.7B-v1.0 (Kim et al., 2024b). Specifically, this model is trained by utilizing DPO.8

    work page

  5. [5]

    This model is trained to utilize the system prompt.9

    EXAONE-3.5-7.8B-Instruct (LG AI Re- search, 2024): This model is an instruction- tuned bilingual (English and Korean) genera- tive model, developed and released by LG AI Research. This model is trained to utilize the system prompt.9

    work page 2024

  6. [6]

    7https://huggingface.co/yanolja/ EEVE-Korean-10.8B-v1.0 8https://github.com/axolotl-ai-cloud/ axolotl 9https://huggingface.co/LGAI-EXAONE/ EXAONE-3.5-7.8B-Instruct

    EXAONE-3.5-32B-Instruct (LG AI Research, 2024): This model shares the same architec- ture as above EXAONE-3.5-7.8B-Instruct, differing only in size. 7https://huggingface.co/yanolja/ EEVE-Korean-10.8B-v1.0 8https://github.com/axolotl-ai-cloud/ axolotl 9https://huggingface.co/LGAI-EXAONE/ EXAONE-3.5-7.8B-Instruct

    work page 2024

  7. [7]

    HyperCLOV A-HCX-DASH-001 (Hyper- CLOV A X Team, 2024): This is an optimized version of HyperCLOVA-HCX-003 that offers faster response times and cost efficiency, making it suitable for simpler tasks while maintaining robust performance

    work page 2024

  8. [8]

    • English-centric LLMs

    HyperCLOV A-HCX-003 (HyperCLOV A X Team, 2024): This is a foundational model in NA VER’s HyperCLOV A X suite, designed for complex and sophisticated tasks, deliver- ing high-quality responses. • English-centric LLMs

    work page 2024

Show all 30 references
  1. [9]

    It has been instruction-tuned to enhance its performance in various natural language understanding and generation tasks

    Llama-3.1-8B-Instruct (Llama Team, 2024): Developed by Meta AI, this model is part of the Llama-3.1 series and contains 8 billion parameters. It has been instruction-tuned to enhance its performance in various natural language understanding and generation tasks

    2024

  2. [10]

    It has been pre-trained on approximately 15 trillion tokens from publicly available sources

    Llama-3-70B (Llama Team, 2024): Devel- oped by Meta AI, Llama-3-70B is a large lan- guage model with 70 billion parameters. It has been pre-trained on approximately 15 trillion tokens from publicly available sources. The model is designed to be multilingual and mul- timodal, w...

    2024

  3. [11]

    It demonstrates superior performance in general knowledge and reasoning tasks, achieving high scores on benchmarks such as MMLU-Pro and MMLU- redux

    Llama-3.1-405B (Llama Team, 2024): This is an expanded version of the Llama series, fea- turing 405 billion parameters. It demonstrates superior performance in general knowledge and reasoning tasks, achieving high scores on benchmarks such as MMLU-Pro and MMLU- redux

    2024

  4. [12]

    It is a text-to-text, decoder-only large language model, with open weights for both pre-trained and instruction-tuned vari- ants

    Gemma-2-9B-Instruct (Team et al., 2024): Developed by Google, this model is part of the Gemma series and contains 9 billion pa- rameters. It is a text-to-text, decoder-only large language model, with open weights for both pre-trained and instruction-tuned vari- ants. Gemma mod...

    2024

  5. [13]

    It is de- signed to deliver high performance across var- ious natural language processing tasks, bene- fiting from its extensive parameter count and advanced training methodologies

    Gemma-2-27B-Instruct (Team et al., 2024): This model is the largest in Google’s Gemma series, featuring 27 billion parameters. It is de- signed to deliver high performance across var- ious natural language processing tasks, bene- fiting from its extensive parameter count and a...

    2024

  6. [14]

    It is designed to handle various nat- ural language understanding and generation tasks, supporting multiple languages, includ- ing English and Chinese

    Qwen2.5-7B-Instruct (Qwen Team, 2024): Developed by Alibaba, this model is part of the Qwen2.5 series and contains 7 billion pa- rameters. It is designed to handle various nat- ural language understanding and generation tasks, supporting multiple languages, includ- ing English...

    2024

  7. [15]

    It offers enhanced performance in lan- guage understanding and generation tasks, with support for multiple languages and a con- text length of up to 128,000 tokens

    Qwen2.5-14B-Instruct (Qwen Team, 2024): This model is a mid-sized variant in Alibaba’s Qwen2.5 series, featuring 14 billion parame- ters. It offers enhanced performance in lan- guage understanding and generation tasks, with support for multiple languages and a con- text length...

    2024

  8. [16]

    It sup- ports a context length of up to 128,000 to- kens and is designed to handle complex tasks across multiple languages, including English and Chinese

    Qwen2.5-32B-Instruct (Qwen Team, 2024): This model is part of Alibaba’s Qwen2.5 se- ries, featuring 32 billion parameters. It sup- ports a context length of up to 128,000 to- kens and is designed to handle complex tasks across multiple languages, including English and Chinese....

    2024

  9. [17]

    It has been fine-tuned using reasoning data generated by DeepSeek-R1, resulting in enhanced performance in reasoning tasks

    DeepSeek-R1-Distill-Qwen-14B (Liu et al., 2024): Developed by DeepSeek, this model is a distilled version of their R1 model, based on Qwen2.5-14B, containing 14 billion parame- ters. It has been fine-tuned using reasoning data generated by DeepSeek-R1, resulting in enhanced pe...

    2024

  10. [18]

    It has been fine- tuned with reasoning data from DeepSeek-R1, achieving state-of-the-art results in various benchmarks

    DeepSeek-R1-Distill-Qwen-32B (Liu et al., 2024): This model is another distilled variant from DeepSeek, based on Qwen2.5-32B, fea- turing 32 billion parameters. It has been fine- tuned with reasoning data from DeepSeek-R1, achieving state-of-the-art results in various benchmarks

    2024

  11. [19]

    budget forcing

    s1-32B (Muennighoff et al., 2025): Devel- oped by Stanford and the University of Wash- ington, s1-32B is a fine-tuned version of Qwen2.5-32B-Instruct. It was optimized for reasoning tasks using 1,000 high-quality sam- ples. The model employs a novel "budget forcing" technique ...

    2025

  12. [23]

    It is designed for real-time interactions and has been integrated into various Google prod- ucts, including Bard and Pixel smartphones

    Gemini-1.5-flash (Gemini Team, 2024): De- veloped by Google DeepMind, Gemini-1.5- flash is a multimodal language model capable of processing text, images, audio, and video. It is designed for real-time interactions and has been integrated into various Google prod- ucts, includ...

    2024

  13. [24]

    It introduces fea- tures such as a Multimodal Live API for real- time audio and video interactions, enhanced spatial understanding, and integrated tool use, including Google Search

    Gemini-2.0-flash-exp (Gemini Team, 2024): An experimental update to the Gemini series, this model offers improved speed and perfor- mance over its predecessors. It introduces fea- tures such as a Multimodal Live API for real- time audio and video interactions, enhanced spatial...

    2024

  14. [25]

    It features a context window of up to 200,000 tokens, allowing it to process extensive text sequences effectively

    Claude-3-haiku (Anthropic, 2024a): Devel- oped by Anthropic, Claude-3-haiku is a large language model designed for complex conver- sational tasks. It features a context window of up to 200,000 tokens, allowing it to process extensive text sequences effectively

  15. [26]

    It maintains a large context window and has been fine-tuned for better alignment with hu- man preferences

    Claude-3.5-Sonnet (Anthropic, 2024b): An enhanced version of the Claude series, this model offers improved performance in lan- guage understanding and generation tasks. It maintains a large context window and has been fine-tuned for better alignment with hu- man preferences

  16. [27]

    It has been widely used in applications requiring natural language understanding and generation

    GPT-3.5-turbo (OpenAI, 2023): Developed by OpenAI, GPT-3.5-turbo is an improve- ment over the original GPT-3.5 model, offer- ing better accuracy in responses. It has been widely used in applications requiring natural language understanding and generation

    2023

  17. [28]

    It offers rapid response times and has been integrated into various applications for real-time interactions

    GPT-4o-mini (OpenAI, 2024a): A smaller and more cost-effective version of OpenAI’s GPT- 4o, this model is capable of processing text, images, and audio. It offers rapid response times and has been integrated into various applications for real-time interactions

  18. [29]

    It ex- hibits rapid response times comparable to hu- man reactions and has enhanced performance in non-English languages

    GPT-4o (OpenAI, 2024a): OpenAI’s GPT-4o is a multimodal model capable of analyzing and generating text, images, and sound. It ex- hibits rapid response times comparable to hu- man reactions and has enhanced performance in non-English languages

  19. [30]

    think- ing

    o1-preview (OpenAI, 2024b): Introduced by OpenAI, o1-preview is designed to solve com- plex problems by spending more time “think- ing” before responding. It outperforms previ- ous models in areas like competitive program- ming, mathematics, and scientific reasoning. B Details...

    2023

  20. [2020]

    In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics , pages 4609–4622, Online

    Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics , pages 4609–4622, Online. Association for Computa- tional Linguistics. Qwen Team. 2024. Qwen2.5: A party of foundation models. ...

    2024

  21. [2023]

    In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741

    Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc. Jaehyung Seo, Hyeonseok Moon, Jaewook Lee, Sug- yeong Eo, Chanjun Park, and Heuiseok Lim....

    2023 arXiv

  22. [2024]

    In Findings of the As- sociation for Computational Linguistics: ACL 2024 , pages 12159–12173, Bangkok, Thailand

    Probing the emergence of cross-lingual align- ment during LLM training. In Findings of the As- sociation for Computational Linguistics: ACL 2024 , pages 12159–12173, Bangkok, Thailand. Association for Computational Linguistics. Lydia White. 1989. Universal Grammar and Sec- ond...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.