REVIEW 2 cited by
Large Language Models Lack Understanding of Character Composition of Words
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have demonstrated remarkable performances on a wide range of natural language tasks. Yet, LLMs' successes have been largely restricted to tasks concerning words, sentences, or documents, and it remains questionable how much they understand the minimal units of text, namely characters. In this paper, we examine contemporary LLMs regarding their ability to understand character composition of words, and show that most of them fail to reliably carry out even the simple tasks that can be handled by humans with perfection. We analyze their behaviors with comparison to token level performances, and discuss the potential directions for future research.
Forward citations
Cited by 2 Pith papers
-
ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbots
This paper defines the task of autocompleting user turns in LLM chatbot conversations and releases ChaI-TeA, a benchmark with datasets and a saved-effort metric showing that better ranking of generated completions is ...
-
Why Do Large Language Models (LLMs) Struggle to Count Letters?
LLMs' letter-counting errors are driven mainly by repeated letters in a word, not by how common the word is.
Discussion (0). Continue with ORCID to comment.