Pith. sign in

REVIEW 4 major objections 6 minor 61 references

Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Dataset

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that LLMs have limited understanding of Korean sentence endings, and that adding an 'absent ending' option improves all 11 evaluated models.

desk verdict KoSEnd is a genuinely new benchmark for Korean sentence-ending naturalness, but the headline SE-absent finding rides on LLM-generated gold labels whose hardest subset shows near-chance agreement with humans, so the empirical claim is not yet established. read the letter →

arxiv 2507.03378 v1 pith:5F4GBEUF submitted 2025-07-04 cs.CL

Seunguk Yu , Kyeonghyun Kim , Jungmin Yun , Youngbin Kim This is my paper
classification cs.CL
keywords KoreansentenceendingsLLMevaluationagglutinativelanguageKoSEnddatasetsentence-endingnaturalnessmultiple-choicebenchmarkpromptdesignlow-resourcelanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Korean verbs carry meaning in their sentence endings, so selecting the right ending is a linguistic skill that LLMs are rarely tested on. This paper introduces KoSEnd, a benchmark of 3,000 sentences, each expanded with 15 declarative and imperative ending forms and labeled for how naturally each ending fits the context. Across 11 LLMs, accuracy was low, especially for imperative endings, and model size made little difference. The paper's main finding is that every model improved when the multiple-choice options included the possibility that no natural ending exists, so the difficulty is not simply lack of Korean data but a failure to consider absence as an option. If the benchmark is sound, it offers both a cheap prompt-level intervention and a new probe for measuring grammatical sensitivity in agglutinative languages.

What carries the argument

The load-bearing object is the KoSEnd dataset itself: 1,000 sentences from each of three difficulty corpora (language-learner texts, newspaper articles, and academic paper summaries), each sentence expanded into 15 endings from the declarative and imperative paradigms, where a sentence ending is the final verb suffix carrying statement, question, command, or emotional nuance. A two-stage annotation marks each expanded sentence with zero, one, or several usages, and the gold labels come from majority-voted human annotation on a 20-sentence subset plus LLM-based annotation on the rest. The evaluation turns on two multiple-choice variants: SE-always, where a natural ending is guaranteed, and SE-absent, where "no usages" is an added option; cyclic permutation of option order is used to measure consistency. That absent-ending option is the mechanism behind the paper's headline finding.

What would settle it

Re-annotate the full 3,000-sentence dataset with native Korean speakers under the same multiple-usage instructions, then recompute the 11 model accuracies and the SE-always-to-SE-absent gap; if the gap shrinks or the ranking changes, the reported results were an artifact of the LLM-generated labels.

Watch

Extended reading notes

Core claim

The central discovery, as the paper states it, is that LLMs have only a limited grasp of Korean sentence-ending naturalness, and that explicitly telling a model that a sentence may have no suitable ending produces consistent gains. In the SE-always task the best average accuracy was below 19 percent, with imperative forms harder than declarative; in the SE-absent task every one of the 11 models improved, and the gains were largest for models instruction-tuned on Korean. A secondary finding is that parameter count does not reliably predict performance, and most models' answers shift when the option order is cyclically permuted, so robustness to option order is also part of what 'understanding' means here. The paper concludes that model performance should be read as dependent on what the prompt allows, not only on what the model knows.

Load-bearing premise

The whole evaluation leans on the labels being correct, and about 98 percent of them were produced by gpt-4-turbo rather than by native speakers, with near-chance human agreement on some categories.

Editorial extensions

If this is right

  • Any of the 11 models is likely to answer Korean sentence-ending questions better if the prompt explicitly includes the possibility that no ending is natural, making this intervention a direct candidate for downstream Korean NLU pipelines.
  • Model scale should not be the first lever chosen for improving Korean grammatical understanding; a small Korean-instruction-tuned model can match or beat much larger multilingual ones.
  • Imperative endings, whose usage categories overlap heavily, are the harder class and the natural focus for future training data.
  • Because option-order permutation changes most models' answers, reported accuracies on multiple-choice Korean benchmarks should be averaged over option orders, and consistency should be reported alongside accuracy.
  • Korean instruction-tuning appears to produce larger gains when absence is allowed, suggesting that such tuning improves awareness of optional grammatical categories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: if human annotators disagree on 'naturalness', the single-gold-label evaluation may understate models that choose a plausible but unpopular ending; a distribution-over-options metric would separate that from genuine misunderstanding.
  • The absent-ending gain suggests similar 'none of the above' interventions could be tried for other optional morphology, such as Japanese or Turkish sentence-final particles, and for tasks with optional categories like honorifics.
  • Because the grader was itself an LLM, part of what KoSEnd measures may be agreement between one LLM and another rather than ability to satisfy a human; a human-judged subset would resolve which.
  • If the benchmark is used to guide training, one could test whether fine-tuning on the SE-absent format improves performance on general Korean generation tasks, a connection the paper names as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces KoSEnd, a Korean sentence-ending naturalness benchmark consisting of 3,000 sentences across three difficulty levels (language-learner, newspaper, and academic-summary corpora), each expanded with 15 sentence-ending forms classified into Declarative and Imperative usages. Gold labels are produced by a two-stage annotation process: three native-Korean annotators label 20 sentences per difficulty, and gpt-4-turbo labels the remaining sentences using few-shot examples and cyclic permutation. The authors evaluate 11 open-weight LLMs on two multiple-choice tasks: SE-always, where a natural ending is assumed present, and SE-absent, where a 'no usages' option is added. They report that all models improve under SE-absent, that Imperative endings are harder than Declarative, that parameter count has little effect, and that Korean-instruction-tuned models are more consistent under cyclic permutation. A pilot with two larger models (Llama3.1 70B and Qwen2.5 72B) is included in the appendix.

Significance. If the results hold, KoSEnd addresses a real and under-tested aspect of Korean linguistic competence in LLMs. The paper's strengths include a linguistically motivated taxonomy of 15 ending forms with usage groupings, corpus diversity, a transparent two-stage annotation protocol, evaluation of 11 models with greedy decoding and three cyclic permutations, explicit post-processing with hallucination rates, and an appendix pilot with larger models. The SE-absent intervention is a falsifiable and practically useful finding. However, the benchmark's value depends on the reliability of its gold labels; because the final LLM annotation pipeline is not validated against held-out human judgments, the central quantitative claims remain conditional.

major comments (4)
  1. [§3.3, Table 3, Table 9] The final gold labels for the roughly 98% of KoSEnd that were not human-annotated are produced by gpt-4-turbo with few-shot prompting and cyclic permutation, but the only reliability check reported is the zero-shot pilot on the 300 human-annotated instances per difficulty. That pilot shows weak agreement on the category most relevant to the central result: Hard Imperative usage accuracy is 47.49%, rising to 74.44% when gold-None items are excluded (Table 3). The disagreement is therefore concentrated on exactly the None decisions that the SE-absent intervention adds. Because the few-shot/cyclic-permutation pipeline is never validated on held-out human annotations, the dataset's gold labels are not established as reliable. The authors should validate the final annotation pipeline on a fresh human-annotated sample and report per-category agreement, including None decisions and broken down by difficulty. The Limitations section acknowledges this risk but does not resolve it.
  2. [§5.3, Tables 6, 7] The headline claim that informing models about the possible absence of sentence endings improves performance across all 11 models (+3.30 to +7.33 points) is compared against gold labels that are mostly produced by gpt-4-turbo. If the evaluated models share the annotation model's bias toward None labels for Imperative items, where Table 9 shows None is the gold label roughly half the time, the improvement could reflect agreement with the annotator rather than with human judgments. The authors should recompute the SE-always versus SE-absent comparison on the human-annotated subset only, and ideally condition on whether the gold label comes from humans or from the LLM, to demonstrate that the intervention helps against human ground truth.
  3. [§4 and Appendix B.1] The task definition is internally inconsistent. Section 4 states that 'we provided the model with four usage options for each sentence in a multiple-choice format,' while Appendix B.1 says that in evaluation 'we combined options from all the forms and required the model to select the most natural sentence endings,' implying more than four options and no prior usage knowledge. Since the number and composition of options directly determines the accuracy numbers in Tables 4 and 6, the authors must specify exactly what the options were: were they the 15 ending forms, the usage categories from Table 1, or something else? The reported task setup should be reproducible from the description alone.
  4. [Table 2] Human inter-annotator reliability is low for the most difficult data: Krippendorff's α for Hard usages is 0.300 (Declarative) and 0.417 (Imperative). With majority-vote gold from three annotators at this reliability level, the human subset itself is a noisy yardstick, and the paper does not report how majority-vote instability propagates into the LLM-annotated labels or the evaluation. Reporting per-item annotator agreement, or at least the distribution of 2-vs-3 agreement, would help the reader calibrate the reliability of the human gold labels.
minor comments (6)
  1. [§5.3] The phrase 'SE- always' in Section 5.3 contains a stray space; please fix the typo.
  2. [Figure 3] The caption says scores are 'averaged over all six difficulty levels for both Declarative and Imperative forms'; since there are three difficulty levels and two forms, the phrasing is ambiguous and should be clarified.
  3. [Appendix A.3] The Korean annotation prompt contains '자 연스러운' with an erroneous space; it should be '자연스러운'.
  4. [§3.3] The sentence 'It will require a total of 980 × 15 × 3 = 44,100 sentence ending cases for each' is ambiguous; state explicitly that this is per difficulty level.
  5. [Appendix B.2] The references for OpenChat and Synatra are identified only by a September 2024 leaderboard snapshot; provide a stable URL and access date.
  6. [Table 5] The header 'Numeral differences' should be 'Numerical differences'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; LLM-generated labels are a data-quality limitation, not a circular step.

full rationale

The paper's core claims are that KoSEnd measures LLM understanding of Korean sentence endings and that adding a 'no usages' option improves all 11 evaluated models. Neither claim is derived from its own inputs by construction. The only potentially self-referential element is that 98% of KoSEnd's gold labels were produced by gpt-4-turbo in a two-stage LLM-based annotation (Section 3.3), while the evaluated models are 11 other open-weight LLMs; gpt-4-turbo is not among them, and the evaluated models' outputs are not used to generate the gold labels. The evaluation therefore measures agreement with an external model-based annotator, which is a construct-validity and label-quality concern rather than a circular derivation. The paper's own Limitations section flags this risk, and Table 3 shows near-chance validation accuracy for Hard Imperative usage (47.49%), which is a substantive data-quality limitation; however, the human-annotated subset and Krippendorff's alpha provide partial independent grounding, and the final few-shot/cyclic annotation is not the same prompt configuration as the validated zero-shot run. The SE-absent improvement is an empirical comparison between two multiple-choice tasks with different scored sample sets (SE-always excludes gold-None samples; SE-absent includes them), so raw accuracy gains are not forced by construction; they depend on model choices. No load-bearing self-citations, ansatz-smuggling, or uniqueness-importing steps were found. Hence the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the Lee (2005) taxonomy, the expansion/annotation pipeline, and especially the reliability of LLM-generated labels. The paper acknowledges the label-risk in its Limitations section, but the validation data suggest the risk is substantial.

assumptions (4)
  • domain assumption The 15 sentence-ending forms and their usage categories (Table 1) from Lee (2005) are a valid and complete taxonomy for Korean sentence endings.
    The dataset construction and all evaluation tasks depend on this taxonomy; if it is incomplete or incorrect, the benchmark measures the wrong thing. Invoked in Section 3.2 and Table 1.
  • domain assumption Expanding every sentence with all 15 endings and filtering by naturalness yields a representative sample of sentence-ending usage.
    Section 3.2 says all sentences were expanded with all endings; the filter is annotation, which may miss acceptable endings or introduce awkward combinations.
  • ad hoc to paper gpt-4-turbo's annotations, after few-shot prompting and cyclic permutation, are a reliable proxy for human judgments on the un-annotated 98% of sentences.
    Section 3.3 uses LLM-based annotation for all but 20 sentences per difficulty; validation accuracies in Table 3 fall to 47-55% for Imperative categories, near chance.
  • domain assumption The four usage options presented to models (or the combined options from all forms, per Appendix B.1) make the multiple-choice task well-posed.
    The paper is inconsistent about the option set; if the task is ill-posed, accuracy numbers are not interpretable. Section 4 vs Appendix B.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Dataset." pith.science (2026). https://pith.science/paper/5F4GBEUF

@misc{pith2026250703378,
  author       = {Pith},
  title        = {Pith review of: Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5F4GBEUF}},
  note         = {Machine review of arXiv:2507.03378}
}
read the original abstract

Although LLMs have made significant progress in various languages, there are still concerns about their effectiveness with low-resource agglutinative languages compared to languages such as English. In this study, we focused on Korean, a language known for its complex sentence endings, and evaluated LLMs on this challenging aspect. We introduce the Korean Sentence Endings (KoSEnd) dataset, which includes 3,000 sentences, each annotated for the naturalness of 15 sentence ending forms. These were collected from diverse sources to cover a range of contexts. We evaluated 11 LLMs to assess their understanding of Korean sentence endings, analyzing them based on parameter count and prediction consistency. Notably, we found that informing models about the possibility of missing sentence endings improved performance, highlighting the impact of explicitly considering certain linguistic features.

Figures

Figures reproduced from arXiv: 2507.03378 by the authors.

Figure 1
Figure 1. Impact of the Korean sentence endings on the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Process of constructing the Korean Sentence Endings ( [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Comparison across LLMs based on parameter [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Difference between the accuracy of each cycle and the average accuracy across all cycles after applying [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Average scores for each sentence ending form of the two models, [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 31 canonical work pages

  1. [1]

    Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2024. https://doi.org/10.18653/v1/2024.naacl-long.100 BUFFET : Benchmarking large language models for few-shot cross-lingual transfer . In Proceedings of the 2024 Conference of the North American Chapter of the Associat...

    doi:10.18653/v1/2024.naacl-long.100 2024

  2. [2]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://aclanthology.org/2024.acl-long.172 L ong B ench: A bilingual, multitask benchmark for long context understanding . In Proceedings of the 62nd Annual Meeting of the Association for Com...

    work page 2024

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  4. [4]

    Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Muhammad Satrio Wicaksono, Ivan Parmonangan, Ika Alfina, Ilham Firdausi Putra, Samsul Rahmadani, Yulianti Oenang, Ali Septiandri, ...

    2023

  5. [5]

    Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. 2024. https://doi.org/10.18653/v1/2024.naacl-long.24 LLM s are few-shot in-context low-resource language learners . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 405--433, Me...

    doi:10.18653/v1/2024.naacl-long.24 2024

  6. [6]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  7. [7]

    Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. https://doi.org/10.18653/v1/2023.acl-long.626 Is GPT -3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173--11195, Toronto, Canada. Association for Computat...

    doi:10.18653/v1/2023.acl-long.626 2023

  8. [8]

    Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.39 K or NLI and K or STS : New benchmark datasets for K orean natural language understanding . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 422--430, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.39 2020

Show all 61 references
  1. [9]

    Andrew F Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication methods and measures, 1(1):77--89

    2007

  2. [10]

    Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. https://doi.org/10.18653/v1/2024.naacl-industry.15 A nno LLM : Making large language models to be better crowdsourced annotators . In Proceedings o...

    2024 doi

  3. [11]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  4. [12]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.826 Not all languages are created equal in LLM s: Improving multilingual capability by cross-lingual-thought prompting . In Findings of...

    2023 doi

  5. [13]

    Yerin Hwang, Yongil Kim, Hyunkyung Bae, Jeesoo Bang, Hwanhee Lee, and Kyomin Jung. 2024. Kosmic: Korean text similarity metric reflecting honorific distinctions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Eval...

    2024

  6. [14]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1--43

    2023

  7. [15]

    Myeongjun Jang, Dohyung Kim, Deuk Sin Kwon, and Eric Davis. 2022. https://aclanthology.org/2022.coling-1.325 K o BEST : K orean balanced evaluation of significant tasks . In Proceedings of the 29th International Conference on Computational Linguistics, pages 3697--3708, Gyeong...

    2022

  8. [16]

    Seongbo Jang, Seonghyeon Lee, and Hwanjo Yu. 2024. https://aclanthology.org/2024.lrec-main.865 K o D ialog B ench: Evaluating conversational understanding of language models with K orean dialogue benchmark . In Proceedings of the 2024 Joint International Conference on Computat...

    2024

  9. [17]

    Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee. 2024. Kobbq: Korean bias benchmark for question answering. Transactions of the Association for Computational Linguistics, 12:507--524

    2024

  10. [18]

    Yi g it Bekir Kaya and A C \"u neyd Tantu g . 2024. Effect of tokenization granularity for turkish large language models. Intelligent Systems with Applications, 21:200335

    2024

  11. [19]

    Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.567 S enti LARE : Sentiment-aware language representation learning with linguistic knowledge . In Proceedings of the 2020 Conference on Empirical Methods in Natural ...

    2020 doi

  12. [20]

    Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. 2024 a . https://aclanthology.org/2024.lrec-main.296 CLI c K : A benchmark dataset of cultural and linguistic intelligence in K orean . In Proceedings of the 2024 Joint International Conference on Com...

    2024

  13. [21]

    Jong Myoung Kim, Young-Jun Lee, Yong-Jin Han, Ho-Jin Choi, and Sangkeun Jung. 2024 b . https://openreview.net/forum?id=yfyHxvVzZT Does incomplete syntax influence korean language model? focusing on word order and case markers . In First Conference on Language Modeling

    2024

  14. [22]

    Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. Jglue: Japanese general language understanding evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2957--2966

    2022

  15. [23]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  16. [24]

    NLP & AI Lab and Human-Inspired AI research. 2023. Kullm: Korea university large language model project. https://github.com/nlpai-lab/kullm

    2023

  17. [25]

    Iksop Lee. 2005. Korean Grammar, volume 33. Seoul National University Press

    2005

  18. [26]

    Jiyoung Lee, Minwoo Kim, Seungho Kim, Junghwan Kim, Seunghyun Won, Hwaran Lee, and Edward Choi. 2024. https://aclanthology.org/2024.findings-acl.666 K or NAT : LLM alignment benchmark for K orean social values and common knowledge . In Findings of the Association for Computati...

    2024

  19. [27]

    Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, and William Chandra Tjhi. 2023. Bhasa: A holistic southeast asian linguistic and cultural evaluation suite for large language models. arXiv preprint arXiv:2309.06085

    2023 arXiv

  20. [28]

    Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ninghao Liu, and Mengnan Du. 2024. Quantifying multilingual performance of large language models across languages. arXiv preprint arXiv:2404.11553

    2024 arXiv

  21. [29]

    Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019. Korquad1. 0: Korean qa dataset for machine reading comprehension. arXiv preprint arXiv:1909.07005

    2019 arXiv

  22. [30]

    Tomasz Limisiewicz, Ji r \' Balhar, and David Mare c ek. 2023. https://doi.org/10.18653/v1/2023.findings-acl.350 Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages . In Findings of the Association for Computational...

    2023 doi

  23. [31]

    LiteLLM. 2025. https://docs.litellm.ai/docs/ Litellm documentation . Accessed on February 1, 2025

    2025

  24. [32]

    Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2024. Is translation all you need? a study on solving multilingual tasks with large language models. arXiv preprint arXiv:2403.10258

    2024 arXiv

  25. [33]

    Akylbek Maxutov, Ayan Myrzakhmet, and Pavel Braslavski. 2024. Do llms speak kazakh? a pilot evaluation of seven models. In Proceedings of the First Workshop on Natural Language Processing for Turkic Languages (SIGTURK 2024), pages 81--91

    2024

  26. [34]

    Meta. 2024 a . https://ai.meta.com/blog/meta-llama-3-1/ Introducing llama 3.1: Our most capable models to date . Accessed on February 1, 2025

    2024

  27. [35]

    Meta. 2024 b . https://ai.meta.com/blog/meta-llama-3/ Introducing meta llama 3: The most capable openly available llm to date . Accessed on February 1, 2025

    2024

  28. [36]

    Alessio Miaschi, Dominique Brunato, Felice Dell ' Orletta, and Giulia Venturi. 2020. https://doi.org/10.18653/v1/2020.coling-main.65 Linguistic profiling of a neural language model . In Proceedings of the 28th International Conference on Computational Linguistics, pages 745--7...

    2020 doi

  29. [37]

    Arbi Haza Nasution and Aytug Onan. 2024. Chatgpt label: Comparing the quality of human-generated and llm-generated annotations in low-resource language nlp tasks. IEEE Access

    2024

  30. [38]

    OpenRouter. 2025. https://openrouter.ai/docs/quickstart Openrouter documentation . Accessed on February 1, 2025

    2025

  31. [39]

    Chanjun Park, Hyeonwoo Kim, Dahyun Kim, SeongHwan Cho, Sanghoon Kim, Sukyung Lee, Yungi Kim, and Hwalsuk Lee. 2024 a . https://aclanthology.org/2024.acl-long.177 Open K o- LLM leaderboard: Evaluating large language models in K orean with K o-h5 benchmark . In Proceedings of th...

    2024

  32. [40]

    Dojun Park, Jiwoo Lee, Hyeyun Jeong, Seohyun Park, and Sungeun Lee. 2024 b . Pragmatic competence evaluation of large language models for korean. arXiv preprint arXiv:2403.12675

    2024 arXiv

  33. [41]

    Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Junseong Kim, Youngsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seung...

    2021

  34. [42]

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2024. Language model tokenizers introduce unfairness between languages. Advances in Neural Information Processing Systems, 36

    2024

  35. [43]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don ' t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages ...

    2018 doi

  36. [44]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  37. [45]

    LG Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Stanley Jungkyu Choi, Yemuk Choi, Seokhee Hong, Yeonjung Hong, Junwon Hwang, Hyojin Jeon, et al. 2024. Exaone 3.0 7.8 b instruction tuned language model. arXiv preprint arXiv:2408.03541

    2024

  38. [46]

    Jaehyung Seo, Jaewook Lee, Chanjun Park, SeongTae Hong, Seungjun Lee, and Heui-Seok Lim. 2024. Kocommongen v2: A benchmark for navigating korean commonsense reasoning challenges in large language models. In Findings of the Association for Computational Linguistics ACL 2024, pa...

    2024

  39. [47]

    Ho-Min Sohn. 2001. The korean language. Cambridge University Press

    2001

  40. [48]

    Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2024 a . Kmmlu: Measuring massive multitask language understanding in korean. arXiv preprint arXiv:2402.11548

    2024 arXiv

  41. [49]

    Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jae cheol Lee, Je Won Yeom, Jihyu Jung, Jung woo Kim, and Songseong Kim. 2024 b . https://aclanthology.org/2024.lrec-main.704 HAE - RAE bench: Evaluation of K orean knowledge in language models . In Proceedings of the 2024 Joint ...

    2024

  42. [50]

    Jiayang Song, Yuheng Huang, Zhehua Zhou, and Lei Ma. 2024. Multilingual blending: Llm safety alignment evaluation with language mixture. arXiv preprint arXiv:2407.07342

    2024 arXiv

  43. [51]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  44. [52]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  45. [53]

    Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schuetze, Kemal Oflazer, and David Mortensen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.401 Counting th...

    2023 doi

  46. [54]

    Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. https://aclanthology.org/2020.aacl-main.85 I ndo NLU : Benchmark and resources for evaluati...

    2020

  47. [55]

    Chuhan Wu and Ruiming Tang. 2024. Performance law of large language models. arXiv preprint arXiv:2408.09895

    2024 arXiv

  48. [56]

    Jiannan Xiang, Huayang Li, Defu Lian, Guoping Huang, Taro Watanabe, and Lemao Liu. 2022. https://doi.org/10.18653/v1/2022.findings-acl.35 Visualizing the relationship between encoded linguistic information and task performance . In Findings of the Association for Computational...

    2022 doi

  49. [57]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  50. [58]

    Soyoung Yoon, Sungjoon Park, Gyuwan Kim, Junhee Cho, Kihyo Park, Gyu Tae Kim, Minjoon Seo, and Alice Oh. 2023. https://doi.org/10.18653/v1/2023.acl-long.371 Towards standardizing K orean grammatical error correction: Datasets and annotation . In Proceedings of the 61st Annual ...

    2023 doi

  51. [59]

    Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Advances in Neural Information Processing Systems, 36:5484--5505

    2023

  52. [60]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  53. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.