Disentangling Codemixing in Chats: The NUS ABC Codemixed Corpus

Akshat Gupta; Insyirah Mujtahid; Kokil Jaidka; Svetlana Churina

REVIEW 4 major objections 5 minor 4 references

Disentangling Codemixing in Chats: The NUS ABC Codemixed Corpus

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The ABC Codemixed Corpus offers 355,641 real private chat messages with token-level language tags and intimacy metadata, as the first labeled general-purpose resource for studying code-mixing in context.

desk verdict The corpus concept is genuinely new, but the headline statistics come from unvalidated LLM tags and the paper contradicts itself; this is not usable as a dataset contribution until the labels are human-checked and the data is released. read the letter →

arxiv 2506.00332 v2 pith:VVUA2OKW submitted 2025-05-31 cs.CL cs.SI

Svetlana Churina , Akshat Gupta , Insyirah Mujtahid , Kokil Jaidka This is my paper

classification cs.CLcs.SI

keywords code-mixingcode-switchinginstantmessaginglanguageidentificationSinglishrelationshipintimacymultilingualNLPprivacy-preservingcorpus

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces the ABC Codemixed Corpus, a live, publicly released dataset of 355,641 real private chat messages voluntarily donated by Singaporean university students. Each message is annotated token-by-token for language, paired with an English translation, and accompanied by self-reported metadata on how intimate the two chat partners are. The authors aim to establish that code-mixing can be studied in its natural habitat—private instant messaging—and that its frequency and function vary systematically with relational closeness. If the corpus and its annotations hold up, it gives computational linguists and sociolinguists a reusable foundation for modeling how multilingual people actually mix languages in everyday conversation, and a template for collecting such data ethically.

What carries the argument

The load-bearing object is the corpus itself, assembled by a donation protocol in which consenting participants self-anonymize and upload selected chat excerpts with relationship metadata. The analytical engine is a token-level language-tagging pipeline built on the Qwen2.5-14B-Instruct large language model, prompted to label each word with one of the Southeast Asian languages or a special tag such as PROPERNOUN, EMOJI, NUMERIC, or PUNCTUATION, followed by post-processing that realigns tags to original tokens and normalizes language labels. A second pass through the same model translates whole conversations into English. These tagged and translated outputs feed the paper's three-level analysis: code-mixing rates across intimacy and message length, LIWC-22 feature correlations, and Systemic Functional Linguistics-based interpretation of conversational strategies.

What would settle it

Have several human annotators fluent in the relevant languages token-tag a stratified sample of, say, 1,000 messages from the released corpus, then compare those human labels with the Qwen2.5 tags. The central claim fails if human–Qwen agreement is no better than the 0.65–0.71 tag-agreement the paper reports against fastText and XLM-RoBERTa, or if the code-mixing-by-intimacy gradient (7.42% → 12.26% → 19.25%) shrinks or disappears when code-mixing is measured from the human labels.

Watch

Extended reading notes

Core claim

The central claim is that the ABC Codemixed Corpus is the first labeled, general-purpose corpus for understanding code-mixing in context, built from authentic private chats rather than social media or scripts. The paper further claims that code-mixing in this corpus is not random noise: it is more common in close relationships (rising from 7.42% of messages among acquaintances to 19.25% in daily interactions) and in short messages (up to 36.7% in intimate conversations), and it correlates positively with pronouns, function words, punctuation, and social references. Read through Systemic Functional Linguistics, these patterns support the paper's interpretation that speakers deploy code-mixed particles and expressions as affective, interpersonal, and discourse-stylistic resources—managing closeness, softening tone, and signaling in-group identity. The corpus is released in JSON format with demographics, intimacy scores, relationship category, token-level language tags, and translations, under a research-only license.

Load-bearing premise

The entire descriptive analysis assumes that the Qwen2.5-14B token-level language tags are accurate enough to measure code-mixing, yet the paper validates those tags only against other automatic taggers—reporting Cohen's $\kappa$ of 0.25–0.26 and tag agreement of 0.65–0.71—and illustrates superiority with selected examples rather than a human-annotated gold standard.

Editorial extensions

If this is right

Researchers can download the JSON corpus and directly test how code-mixing rates, transliteration, and message length co-vary with relationship intimacy, without gathering private chat data themselves.
The token-level tags and English translations give language-identification and translation systems a training and evaluation resource for informal Southeast Asian chat text, where standard taggers are shown to struggle.
The intimacy and relationship-category metadata open the door to content-based relationship classification, an intended use listed in the dataset's own datasheet.
The donation-and-anonymization protocol—voluntary self-anonymized uploads, PII screening, voucher compensation, and a research-only license—offers a replicable template for future ethically collected private-message corpora.
The LIWC correlations suggest code-mixing should be treated by NLP models as a marker of informal, relationally engaged discourse rather than as noise to be stripped out.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My inference: because the released language tags are validated only against other automatic taggers rather than against human annotation, they should be treated as noisy labels; any model trained on them inherits that noise, so a small public human-annotated benchmark would materially raise the corpus's value.
My inference: the intimacy–code-mixing gradient found in Singaporean students is a testable hypothesis about multilingual messaging more generally; re-running the same donation protocol with working adults, older speakers, or diaspora communities would show whether the pattern is universal or specific to this age group and language ecology.
My inference: the Systemic Functional Linguistics analysis suggests discourse particles such as lah, leh, hor, and liao may be sufficient cues to predict conversational intimacy, a quantitative test the paper motivates but does not run.
My inference: the 'general-purpose' label should be read narrowly—the corpus is drawn from a narrow demographic and self-selected excerpts, so its generalizability beyond Singaporean university students is not established by the paper itself.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things before reading: the dataset idea is genuinely new, and the paper's core numbers cannot currently be trusted. It combines private chat donation with relationship-intimacy metadata and token-level language tags for Singaporean code-mixing—nobody has done that in one corpus, and the collection framework with self-anonymization and participant control is a real plus. The SFL analysis in Section 7 is also a thoughtful qualitative pass, with good examples (sayang, paiseh, walao) and a limitations section that honestly flags generalizability and classifier issues.

But the load-bearing evidence is weak. Every code-mixing rate in Table 1, Figure 4, and the LIWC correlations is computed from Qwen2.5-14B token-level tags. Section 6.1 never validates those tags against human annotation. Comparing the LLM to fastText and XLM-R on 43 conversations and getting kappa 0.25–0.26 tells you the taggers disagree with each other, not which one is right. The paper claims superiority from selected examples, and the limitations section itself admits Singlish and dialectal insertions may be misclassified. That makes the descriptive claims unverifiable.

There are also internal contradictions that should not survive even a light edit: Table 1 says 22% of messages are code-mixed, but Section 7 says “Over 30% of messages contained code-mixing.” Table 1 lists 154 unique contributors, while Section 5 says “Of the 166 contributors.” These may be definitional (e.g., 166 recruits who contributed versus 154 after deduplication), but the paper does not say so. And the data is not accessible: anonymous links, no shared code, so nobody can check.

My verdict: reject in current form. The corpus could become a useful resource if the authors release it and add a human-annotated gold standard—even a few thousand tokens would let readers judge tagger accuracy. Until then, the central contribution is unverifiable. I would not cite it. It's worth a second look if resubmitted with real validation, but not referee time as is.

Referee Report

4 major / 5 minor

Summary. The paper introduces the ABC Codemixed Corpus, a collection of 355,641 private chat messages donated by Singaporean university students, with token-level language tags produced by Qwen2.5-14B-Instruct, English translations, self-reported relationship intimacy metadata, and demographic/psychometric survey responses. The authors report descriptive statistics on code-mixing rates by intimacy level and message length, LIWC-based correlations, and an SFL-inspired qualitative analysis of conversational strategies. The central claim is that this is the first labeled, general-purpose, publicly available code-mixed corpus for studying code-mixing in context under rigorous privacy and ethical standards.

Significance. If the labels and statistics were reliable, the corpus would be a valuable resource for sociolinguistic and NLP research on code-mixing in private messaging, with a relatively rare combination of scale, relationship metadata, and ethical data-donation methodology. The paper deserves credit for a clear privacy-preserving collection framework, the inclusion of psychometric intimacy measures, and the ambition to release a live dataset. However, the significance cannot be assessed from the manuscript alone because the corpus is not actually available (all links are anonymized), the language tags that define every code-mixing statistic are not validated against human judgment, and the headline numbers are internally inconsistent. These issues affect the core contribution and are not merely presentational.

major comments (4)

[Section 6.1, Table 2] The token-level language tags from Qwen2.5-14B-Instruct are the sole basis for the code-mixing rates in Table 1, Figure 4, and the Section 7 LIWC correlations, but the paper provides no human-annotated gold standard for these tags. The only evaluation compares Qwen2.5 to fastText and XLM-R on a reduced sample of 43 conversations, with Cohen's kappa 0.25-0.26, which is generally considered poor agreement; the claim of superiority rests on selected examples in Tables 3 and 5 rather than systematic measurement. Without human validation, the corpus cannot be described as 'labeled' in a way that supports the reported quantitative findings.
[Section 5 versus Table 1] The paper reports inconsistent headline statistics. Table 1 lists 154 unique contributors, while Section 5 begins 'Of the 166 contributors'; Section 7 states 'Over 30% of messages contained code-mixing,' whereas Table 1 reports 22% overall. In addition, the Part B message counts (14,166 + 44,590 + 280,745 = 339,501) do not sum to the stated total of 355,641 messages. These discrepancies affect the central descriptive claims and cannot be resolved without access to the data.
[Section 6.2, Table 4] The translation evaluation compares Qwen2.5 outputs only to Google Translate and NLLB-200 using automatic metrics (BLEU 4.47-4.55, BERT-F1 0.81-0.87). No human evaluation is reported, and the low BLEU scores are attributed to 'paraphrasing' without independent evidence. The claim that the LLM-based approach is 'much better' at translating transliterated words is supported only by one illustrative example in Table 5, so the translation quality of the released corpus is unestablished.
[Abstract, Section 8, Appendix A.6] The paper's central promise is a publicly available, live corpus, but all distribution links are given as 'Anonymous' placeholders, and Appendix A.4 states that the annotation and analysis code is 'documented but not publicly shared.' The dataset itself is not accessible to reviewers or readers, so the core artifact cannot be verified, downloaded, or used; this makes the 'publicly available corpus' claim unsubstantiated in the submitted manuscript.

minor comments (5)

[Section 6.1] Section 6.1 refers to 'Table 6.1' but the table is labeled Table 2; the cross-reference should be corrected.
[Appendix B.1.3] Appendix B.1.3 contains the typo 'fune-tuned' for 'fine-tuned.'
[Section 7] Section 7 repeats a comma splice: 'code-mixing is both, a structural feature ... and a stylistic resource' should be 'both a structural feature ... and a stylistic resource.'
[Abstract and Table 1] The abstract states 'over 355,641 messages' but Table 1 and the main text report exactly 355,641; the wording 'over' is imprecise.
[Figures 4 and 5] The captions for Figures 4 and 5 omit the y-axis units and the definition of 'valid token count'; please specify how tokens were counted after preprocessing.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the corpus statistics are direct outputs of the Qwen2.5 tagging pipeline, not derivations equivalent to their own inputs; the absence of a human gold standard is a validity concern, not a circularity concern.

full rationale

The paper is a corpus-and-descriptive-analysis contribution rather than a formal derivation. Its central quantities—message counts, code-mixing rates, intimacy-level breakdowns, and LIWC correlations—are computed directly from messages and from Qwen2.5-14B-Instruct token-level language tags. Those tags are the measurement instrument, not a fitted parameter later relabeled as a prediction, so no step reduces to its own input by construction. Section 6.1 compares the primary tagger against fastText and XLM-R without a human-annotated gold standard; this is a real evaluation weakness, and the paper's own Limitations section concedes that 'complex or low-resource language varieties, such as Singlish or dialectal insertions, may still be misclassified.' That concession, along with the internal inconsistencies between Table 1 (22% code-mixed messages; 154 contributors) and Section 7 ('Over 30% of messages contained code-mixing'; 166 contributors), are accuracy and reporting problems, not circularity. The paper does not rely on a load-bearing self-citation chain: the Qwen2.5 technical report is cited for the model's capabilities, not to establish the corpus's correctness. No uniqueness theorem or prior author-derived ansatz is invoked to force the paper's choices. Because no quoted equation or fitted parameter is shown to be equivalent to its input, the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper relies on several unstated operational decisions and unvalidated labeling assumptions. No fitted numerical parameters appear in a model, but the definition of code-mixing and the LLM labels are load-bearing choices.

free parameters (3)

code-mixed message threshold = not specified
The paper does not define what counts as a code-mixed message (e.g., at least one non-English token, or a minimum percentage). All percentages in Table 1 and Figure 4 depend on this unstated rule.
message length bucket boundaries = not fully specified
Figure 4 uses length buckets (e.g., 0-5 tokens, 51-100, 100+), but the exact boundaries are not listed, affecting the reported pattern.
evaluation sample size after token matching = 43 conversations
Tagging performance is evaluated only on 43 conversations where output token counts matched baselines, a subset of the 100-conversation stratified sample, which may introduce selection bias.

assumptions (4)

domain assumption Qwen2.5 language tags are accurate enough for corpus-level statistics.
All quantitative claims rely on LLM tags; no human annotation is provided. Section 6.1.
domain assumption Participants accurately self-reported relationship intimacy and contributed representative chat excerpts.
The intimacy-level comparisons assume the three relationship categories are meaningful and that selected excerpts are not biased. Section 4 and Limitations.
domain assumption LIWC-22, built for standard English, is valid for code-mixed and Singlish text.
The correlations in Section 7 use LIWC categories on messages that include non-English tokens; unmatched words may be missing, distorting feature proportions.
domain assumption The PCA grouping of LIWC features into five latent categories is a valid representation.
The five categories are used to interpret correlations; the PCA is described only briefly and full loadings are not given in the main text.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Disentangling Codemixing in Chats: The NUS ABC Codemixed Corpus." pith.science (2026). https://pith.science/paper/VVUA2OKW

@misc{pith2026250600332,
  author       = {Pith},
  title        = {Pith review of: Disentangling Codemixing in Chats: The NUS ABC Codemixed Corpus},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VVUA2OKW}},
  note         = {Machine review of arXiv:2506.00332}
}

read the original abstract

Code-mixing involves the seamless integration of linguistic elements from multiple languages within a single discourse, reflecting natural multilingual communication patterns. Despite its prominence in informal interactions such as social media, chat messages and instant-messaging exchanges, there has been a lack of publicly available corpora that are author-labeled and suitable for modeling human conversations and relationships. This study introduces the first labeled and general-purpose corpus for understanding code-mixing in context while maintaining rigorous privacy and ethical standards. Our live project will continuously gather, verify, and integrate code-mixed messages into a structured dataset released in JSON format, accompanied by detailed metadata and linguistic statistics. To date, it includes over 355,641 messages spanning various code-mixing patterns, with a primary focus on English, Mandarin, and other languages. We expect the Codemix Corpus to serve as a foundational dataset for research in computational linguistics, sociolinguistics, and NLP applications.

Figures

Figures reproduced from arXiv: 2506.00332 by the authors.

**Figure 3.** , tagging within Asian languages still has some issues, which are discussed in the Limitations section. 6.2 Translation Performance For evaluating translations, we used the same stratified sample employed for assessing language tags. We compared our translations to two baselines: one generated using the Google Translate API and the other using Facebook’s NLLB-200 MT model. ) (e) Korean (f) Malay [PITH_FULL_IMAGE:f… view at source ↗

**Figure 4.** Average percentage of code-mixing across message length buckets (based on valid token count), grouped by relationship type. n indicates the number of messages in each bucket per dataset. charged expressions (e.g., sayang, jiayou, wah, sian) to convey care, dramatize feelings, or lighten the mood, especially in close relationships where emotional disclosure is common. • Interpersonal-Social Tenor: Speakers maintai… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

4 extracted references · 4 canonical work pages

[1]

and its association with linguistic features (Level

work page
[2]

Wah I really cannot than school already

in how these patterns function socially to manage relationships and actions in everyday conversation (Level 3). While field and mode are useful for capturing the activity type and medium-specific features of digi- tal chat, tenor offers the most analytically rich lens for unpacking the social meaning of code -mixed expression in our data. This is particul...

work page 2016
[2019]

In Proceedings of Interspeech 2019, pages 321–325

Building the singapore english national speech corpus. In Proceedings of Interspeech 2019, pages 321–325. ISCA. Jaeseong Lee, YeonJoon Jung, and Seung-won Hwang

work page 2019
[2024]

role": "system

COMMIT: Code-mixing English-centric large language model for multilingual instruction tuning. In Findings of the Association for Computational Lin- guistics: NAACL 2024, pages 3130–3137, Mexico City, Mexico. Association for Computational Lin- guistics. Jakob RE Leimgruber. 2013. The management of mul- tilingualism in a city -state: Language policy in sin-...

work page 2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

and its association with linguistic features (Level

work page

[2] [2]

Wah I really cannot than school already

in how these patterns function socially to manage relationships and actions in everyday conversation (Level 3). While field and mode are useful for capturing the activity type and medium-specific features of digi- tal chat, tenor offers the most analytically rich lens for unpacking the social meaning of code -mixed expression in our data. This is particul...

work page 2016

[3] [2019]

In Proceedings of Interspeech 2019, pages 321–325

Building the singapore english national speech corpus. In Proceedings of Interspeech 2019, pages 321–325. ISCA. Jaeseong Lee, YeonJoon Jung, and Seung-won Hwang

work page 2019

[4] [2024]

role": "system

COMMIT: Code-mixing English-centric large language model for multilingual instruction tuning. In Findings of the Association for Computational Lin- guistics: NAACL 2024, pages 3130–3137, Mexico City, Mexico. Association for Computational Lin- guistics. Jakob RE Leimgruber. 2013. The management of mul- tilingualism in a city -state: Language policy in sin-...

work page 2024