Language Identification via Compositional Data Analysis: A Linear-Time Classifier Based on Log-Ratio Geometry

Paul-Andrei Pog\u{a}cean; Sanda-Maria Avram

REVIEW 3 major objections 5 minor 28 references

Language Identification via Compositional Data Analysis: A Linear-Time Classifier Based on Log-Ratio Geometry

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Treating letter and bigram frequencies as compositional data and mapping them with a centered log-ratio transform yields a training-free, linear-time language identifier that matches or slightly exceeds tuned benchmarks on six Latin-script

desk verdict Clean, honest application of CLR to language ID — the geometry is right, but the headline comparison to FastText/langid.py is not actually measured on the same test set, so the main empirical claim is untested. read the letter →

arxiv 2607.15238 v1 pith:6KQKFN3T submitted 2026-07-16 cs.CL

Paul-Andrei Pogăcean , Sanda-Maria Avram This is my paper · ORCID

classification cs.CL MSC 68T5068T1062H30

keywords languageidentificationcompositionaldataanalysiscenteredlog-ratiotransformgeometryn-gramfrequencieslinear-timealgorithmdeterministicclassifierzero-sumsubspace

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that raw frequency vectors, because they sum to one, live on a simplex where ordinary Euclidean or Manhattan distances distort true relationships. It applies the centered log-ratio transform, which maps the simplex isometrically to a zero-sum Euclidean hyperplane, so that Euclidean distance becomes a valid compositional distance. On this geometry the authors build a classifier that compares CLR-transformed unigram and bigram profiles of a text against reference profiles, adds a length-decaying diacritic bonus, and predicts the closest language. On 260 held-out texts in six Latin-script languages, it reports 84.0% accuracy under 50 characters, 95.6% for 50–150 characters, and 100% above 150 characters, matching or slightly exceeding established off-the-shelf identifiers. The method requires no training, runs in linear time, and yields explicitly interpretable decisions.

What carries the argument

The centered log-ratio (CLR) transform: given a positive frequency vector x of dimension D, CLR(x)_i = ln(x_i) - (1/D) sum_j ln(x_j). This bijectively maps the D-simplex onto the (D-1)-dimensional zero-sum hyperplane, and the Euclidean norm on that hyperplane is exactly the compositional distance on the original simplex. The classifier uses this isometry twice: once for character unigram frequencies and once for bigram frequencies. The final score for a language is the sum of the two CLR-Euclidean distances to that language's reference profile, minus a term β·m/(1+α·L) that rewards diacritic matches on short texts.

What would settle it

Use the paper's held-out test set of 260 texts, run the standard baseline identifiers cited in the paper (or faithful reimplementations of them) on those exact texts, and compare their length-stratified accuracies to the reported 84/95.6/100. If any baseline meets or exceeds those numbers, the paper's 'matches and slightly exceeds' claim would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that the centered log-ratio (CLR) transformation provides the right metric for language identification. For any positive frequency vector, CLR replaces each coordinate with its log minus the mean log, giving a vector that sums to zero; Euclidean distance in this zero-sum subspace equals the compositional distance. The paper constructs reference profiles for six languages from external corpora, then classifies a new text by summing the CLR-Euclidean distances of its unigram and bigram profiles to each reference profile, with a small length-dependent diacritic adjustment. On the held-out set it reports 84.0% accuracy for short texts, 95.6% for medium texts, and 100% for te

Load-bearing premise

The comparison anchoring the paper's claim of parity with benchmarks relies on accuracy ranges taken from other studies on different test sets, not on running those baselines on the same 260 texts.

Editorial extensions

If this is right

If the reported accuracy is reproducible, language identification for Latin-script texts can be performed with no training, using only precomputed frequency profiles.
The linear-time complexity (a single text pass plus fixed-dimension distance operations) makes the method suitable for real-time and edge deployment.
Because the feature space is fixed, adding a new language requires only constructing its reference profile, not retraining a model.
The interpretable geometry allows decisions to be traced to the distance contributions of unigram vs. bigram features.
The perfect long-text accuracy on the tested languages suggests that compositional separability improves with sequence length, pointing toward theoretical guarantees for sufficiently long inputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension beyond the paper's claims: run the same baselines on the identical held-out set to confirm whether the 'matches and slightly exceeds' wording holds under controlled comparison.
The length-decaying diacritic term is an engineered heuristic; a more principled version might emerge from Bayesian treatments of compositional observations.
The method's reliance on a fixed feature dimension suggests it may naturally extend to other alphabetic scripts (Cyrillic, Greek) with an aligned character set, but not to scripts whose symbol inventory varies.
For code-switched texts, a sliding-window variant of this classifier could localize language boundaries, since the CLR geometry projects mixed texts to intermediate points.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is applying Aitchison's CLR transform to character unigram+bigram frequencies for language identification, with a length-dependent diacritic adjustment on top. The math is standard CoDA and the paper states it correctly, including the isometry theorem. The pipeline is deterministic, linear-time, and the reproducibility pledge plus the disclosed hyperparameter tuning on a separate validation subset are real credits. This is a legitimate domain transfer, not a subfield reshuffling, and the authors do not oversell it as one.

What the paper does well: a clear formal setup, a sensible ablation showing bigrams matter, and an honest limitations section covering code-switching, script coverage, and the short-sequence smoothing bias. The absolute accuracies on their own LDDS data are plausible for six Latin-alphabet languages, and the OPUS experiment at ~250k sequences is a useful sanity check even without baselines.

Where it gets soft: Table 5 is load-bearing and it compares CoDA's held-out LDDS numbers against "consensus accuracy ranges" quoted from other studies — FastText, langid.py, Manhattan, Cosine are never run on LDDS. Those quoted numbers come from different corpora, language sets, and length definitions, so the "matches and slightly exceeds" claim in §5.3 is not actually supported by a controlled comparison. LDDS is a 260-text self-created test set with no reported per-bin counts, so sampling noise is a genuine concern; the difference between 95.6% and a baseline quoted at 93–95% could easily vanish or reverse with a proper head-to-head. This is a measurement-comparability gap rather than a flaw in the CLR geometry itself, and the stress-test note is right: run the baselines on the same texts and we will know.

The central mathematical argument holds up. The soft spot is empirical, not logical, and it is addressable. I would not cite this for the comparative claim, but the method description and the convergence-with-length story are worth knowing for anyone building low-resource or edge LID tools.

For peer review: yes, this deserves a serious referee. It is a coherent, reproducible application of established mathematics to a mature task, and the right editor would ask the authors to rerun their baselines on the same test set with identical preprocessing, add confidence intervals or exact counts, and preferably evaluate on an independent, larger corpus. Desk rejection would be a waste; it just needs the empirical work tightened.

Referee Report

3 major / 5 minor

Summary. This paper proposes a deterministic, training-free language identification method based on compositional data analysis. Character unigram and bigram frequency vectors are treated as compositions on the probability simplex; additive Laplace smoothing is applied, and the centered log-ratio (CLR) transformation maps them into the (D−1)-dimensional zero-sum subspace, where Euclidean distance equals Aitchison distance. Classification is performed by computing the weighted sum of CLR-space Euclidean distances to fixed reference profiles for each language, with a length-dependent diacritic bonus term. The method is evaluated on six Latin-alphabet languages (English, German, Turkish, Romanian, Hungarian, Dutch). On the authors' held-out LDDS set (260 texts), it reports 84.0% accuracy for texts under 50 characters, 95.6% for 50–150, and 100% for over 150. A larger OPUS experiment (~250,000 sentences) reports similar aggregate numbers. The paper claims the method 'matches and slightly exceeds' the robustness of FastText and langid.py while remaining linear-time and fully deterministic.

Significance. The mathematical core is sound: Theorem 3.1 correctly states the CLR isometry between the simplex with Aitchison geometry and the zero-sum hyperplane with Euclidean norm, and the pipeline is genuinely linear-time and deterministic. The reproducibility pledge—public code, fixed hyperparameters, external profile corpora—is a notable strength. If the empirical comparison were made on a common testbed with direct baseline runs, the paper would offer a useful, interpretable, low-resource alternative to neural and embedding-based LID for alphabetic languages. However, as it stands, the central comparative claim is not yet established, because the baseline numbers are borrowed from other experiments rather than measured on the same held-out set.

major comments (3)

[§5.3, Table 5] The central claim 'matches and slightly exceeds the robustness of heavily optimized standard benchmarks' rests on Table 5, where the CoDA row is measured on the authors' LDDS test set but the baseline rows (Raw Euclidean, Manhattan, Cosine, FastText/langid.py) are 'consensus accuracy ranges' quoted from other studies. Those studies use different languages, corpora, class counts, preprocessing, and length definitions, so the columns are not comparable. On a 260-text dataset, the 84.0% vs. ~80–82% difference for short texts is also within plausible sampling error. To support the claim, the authors should run the baselines on LDDS with identical preprocessing and report per-bin counts and confidence intervals.
[§5.4, §5.6, §5.7] The empirical reporting is not sufficiently granular. LDDS contains 260 texts, but Table 6 gives per-language accuracy with one-decimal precision (e.g., 79.5%, 82.5%) without stating how many texts fall into each length bin; a few misclassifications can shift these numbers by several points. The ROC/AUC values are described as 'approximate' with no computational details. The OPUS experiment in §5.7 reports no baselines and no per-bin counts, so it corroborates scaling behavior but cannot substitute for a controlled comparison. Please report exact counts and, at minimum, 95% confidence intervals for the headline accuracies.
[§5.2–5.3] The claim that CLR geometry is responsible for the improvement would be strengthened by an internal ablation that keeps the feature set and classifier fixed and only replaces the CLR distance with the same Euclidean/Manhattan/Cosine distance on the smoothed simplex vectors, evaluated on LDDS. The current Table 5 compares against quoted ranges, not the authors' own features/preprocessing, so it does not isolate the geometric contribution.

minor comments (5)

[Abstract, §3.3] The phrase 'where Euclidean distances correspond to Aitchison distances' is repeated verbatim many times; please use it once and refer back.
[Algorithm 1, line 5] The diacritic match count m is not formally defined. Specify how diacritic characters are identified per language and how m is computed.
[Tables 3 and 4] Column headers list only L<50 and L>150, omitting the 50–150 interval used elsewhere. Include all three bins for consistency.
[References] References [9], [10], [17], and [25] do not appear to be cited in the text; either cite them or remove.
[Figure 1] The ROC figure is described in §5.6 but no plot appears in the manuscript text; ensure the figure is included and that AUC values are tied to actual bin counts.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CLR is an external standard transform; the classifier is deterministic and evaluated on a held-out set; the only flagged issue (quoted baselines in Table 5) is a comparability gap, not a circular reduction.

full rationale

The claimed derivation is: normalize counts (Eq. 1), smooth, CLR-transform (Eq. 3), compute Aitchison/Euclidean distances (Eqs. 6-8), subtract a diacritic adjustment (Eq. 9), and take argmin (Eq. 10). The CLR isometry and uniqueness are attributed to Aitchison [2], an external source, not to this paper or its authors; thus no load-bearing step reduces to a self-citation. Hyperparameters δ, α, β, λ were tuned on a separate 60-text validation set (§5.1) and evaluated on the held-out LDDS, so the reported accuracies are not fitted values renamed as predictions. The authors' own repository [21] is used only as the test set, not as profile input or as a mathematical premise; this is a data-governance caveat, not circularity. Table 5 itself states that 'the baseline values reflect the consensus accuracy ranges reported in rigorous short-text language identification studies (e.g., Vatanen et al. [27])' — an explicit admission that baselines were not measured on LDDS. I flag this as a measurement-comparability limitation (a correctness risk), but it does not make the CoDA numbers equivalent to the baselines by construction. No equation in the paper reduces the prediction to its training input. Therefore no circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper adds no invented entities. Its central claim rests on standard Aitchison geometry, the domain assumption that text frequencies are compositional, a fixed-feature-space assumption, and four hyperparameters tuned on a small validation set. The validation tuning is disclosed and the test set is held out; the main empirical weakness is indirect baseline comparison rather than circular fitting.

free parameters (4)

Laplace smoothing δ = 0.5
Set by grid search on a 60-sample validation subset (§5.1); controls zero-count handling and therefore all log-ratio values, especially for short texts.
Diacritic decay α = 0.05
Grid-searched on validation (Table 3, §4.2); controls how quickly the diacritic bonus B(L) decays with text length.
Diacritic scale β = 2.0
Grid-searched on validation (Table 3, §4.2); scales the diacritic bonus.
Feature weights λ_c, λ_b = 1, 1
Chosen via empirical profiling on the validation set (Table 4, §4.3); equal weighting of unigram and bigram Aitchison distances.

assumptions (3)

standard math CLR is an isometry from the simplex with the Aitchison metric to the zero-sum hyperplane with Euclidean ℓ2 norm, and the Aitchison metric is the unique distance with scale invariance, permutation invariance, and subcompositional coherence.
Invoked in Theorem 3.1 (§3.3) and imported from Aitchison [2]; not re-derived in the paper.
domain assumption Character and bigram frequency vectors are compositional data whose comparisons should respect Aitchison geometry rather than raw Euclidean/Manhattan geometry.
This motivates the whole pipeline (§2.3, §3.2). If relative text frequencies are not appropriately treated as compositions, the CLR mapping is just a nonlinear feature transform with no geometric justification.
domain assumption The feature space is a fixed, shared alphabet/bigram set across all documents and languages.
Definition 3.1 (§3.1) fixes D_c and D_b; the paper explicitly excludes scripts where D would vary because distances would become non-comparable (§1, §6).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Language Identification via Compositional Data Analysis: A Linear-Time Classifier Based on Log-Ratio Geometry." pith.science (2026). https://pith.science/paper/6KQKFN3T

@misc{pith2026260715238,
  author       = {Pith},
  title        = {Pith review of: Language Identification via Compositional Data Analysis: A Linear-Time Classifier Based on Log-Ratio Geometry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KQKFN3T}},
  note         = {Machine review of arXiv:2607.15238}
}

abstract

Language identification is commonly addressed using either neural architectures or statistical n-gram models. Neural approaches typically require substantial computational resources, whereas classical frequency-based methods offer efficient linear-time performance, but rely on distance metrics that are not always appropriate for compositional data. This work models character and bigram frequency distributions as compositional vectors constrained to the simplex and mapped via the centered log-ratio (CLR) transformation bijectively onto the $(D-1)$-dimensional zero-sum subspace of $\mathbb{R}^D$, where Euclidean distances correspond to Aitchison distances. A pipeline is proposed, combining CLR-transformed unigram and bigram features with Laplace smoothing to address sparsity. The method is evaluated on six languages. Experimental results show that the proposed approach achieves robust accuracy across different text lengths, with strong performance for longer sequences. These findings indicate that compositional representations provide a deterministic and computationally efficient alternative for language identification, particularly in settings where interpretability and low resource consumption are essential.

Figures

Figures reproduced from arXiv: 2607.15238 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 3 canonical work pages

[1]

H., & Kareem, F

Abbas, R. H., & Kareem, F. A. E. A. (2019). Text language identification using letters (frequency, self- information, and entropy) analysis for English, French, and German languages.Journal of Southwest Jiaotong University,54(4).https://doi.org/10.35741/issn.0258-2724.54.4.22

work page doi:10.35741/issn.0258-2724.54.4.22 2019
[2]

(1986).The statistical analysis of compositional data

Aitchison, J. (1986).The statistical analysis of compositional data. Chapman and Hall

1986
[3]

Aksan, Y ., Aksan, M., Mersinli, ˘U., & Demirhan, U. U. (2016).A frequency dictionary of Turkish. Routledge

2016
[4]

(2022).ROST (ROmanian stories and other texts) dataset[Data set]

Avram, S.-M. (2022).ROST (ROmanian stories and other texts) dataset[Data set]. Kaggle.https://doi.org/ 10.34740/KAGGLE/DS/2545712

arXiv 2022
[5]

(2021).Deutsches Referenzkorpus: Buchstaben- und Bigrammhäufigkeiten in der deutschen Sprache

Belica, C., & Perkuhn, R. (2021).Deutsches Referenzkorpus: Buchstaben- und Bigrammhäufigkeiten in der deutschen Sprache. Leibniz-Institut für Deutsche Sprache.https://www.ids-mannheim.de/fileadmin/kl/ derewo/derechar-v-XXX-YYY-2021-10-31-1.0.pdf

2021
[6]

Bouma, G. (2015). N-gram frequencies for Dutch twitter data.Computational Linguistics in the Netherlands Journal,5, 25–36.https://www.clinjournal.org/clinj/article/view/45

2015
[7]

Brown, T., Mann, B., Ryder, N., et al. (2020). Language models are few-shot learners.Advances in Neural Information Processing Systems,33, 1877–1901.https://proceedings.neurips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

2020
[8]

B., & Trenkle, J

Cavnar, W. B., & Trenkle, J. M. (1994). N-gram-based text categorization. InProceedings of the Third Annual Symposium on Document Analysis and Information Retrieval(pp. 161–175)

1994

Show all 28 references

[9]

(2017).DeepL translation service.https://www.deepl.com/translator

DeepL SE. (2017).DeepL translation service.https://www.deepl.com/translator

2017
[10]

Evans, O. (2024).Response to: Is there a frequency ranking of letters based on their recurrence in a specific lan- guage?ResearchGate.https://www.researchgate.net/post/Is_there_a_frequency_ranking_of_ letters_based_on_their_recurrence_in_a_specific_language

2024
[11]

Huang, F., & Lee, D. L. (2003). A new language identification method based on PPM. InProceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval(pp. 289–290).https://doi.org/10.1145/860435.860493

2003
[12]

Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T. (2016). FastText.zip: Compressing text classification models.arXiv preprint arXiv:1612.03651.https://arxiv.org/abs/1612.03651

2016 arXiv
[13]

Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (pp. 427–431).https://doi.org/10.18653/v1/E17-2068

2017 doi
[14]

Keuleers, E., Brysbaert, M., & New, B. (2010). SUBTLEX-NL: A new measure for Dutch word frequency based on film subtitles.Behavior Research Methods,42(3), 643–650.https://doi.org/10.3758/BRM.42.3.643

2010 doi
[15]

Lui, M., & Baldwin, T. (2012). langid.py: An off-the-shelf language identification tool. InProceedings of the ACL 2012 System Demonstrations(pp. 25–30).https://aclanthology.org/P12-3005/ Deterministic Language Identification via Compositional-Data Analysis13

2012
[16]

McNamee, P. (2005). Language identification: A solved problem suitable for undergraduate instruction.Journal of Computing Sciences in Colleges,20(3), 94–101

2005
[17]

(2010).Spontán magyar nyelv˝ u beszéd gépi felismerése nyelvspecifikus szabályok nélkül[Doctoral dissertation, BME-VIK].https://www.magyarbeszed.hu/download/PhD_MP.pdf

Mihajlik, P. (2010).Spontán magyar nyelv˝ u beszéd gépi felismerése nyelvspecifikus szabályok nélkül[Doctoral dissertation, BME-VIK].https://www.magyarbeszed.hu/download/PhD_MP.pdf

2010
[18]

Mitrea, A. M. (2012). Statistical study on a literary Romanian corpus for the beginning and ending of the words. InProceedings of the 9th International Conference on Communications (COMM)(pp. 255–258).https://doi. org/10.1109/ICComm.2012.6256108

2012
[19]

(2012).English letter frequency counts: Google Books corpus analysis.http://norvig.com/ mayzner.html

Norvig, P. (2012).English letter frequency counts: Google Books corpus analysis.http://norvig.com/ mayzner.html

2012
[20]

Oravecz, C. (2014). A Magyar nemzeti szövegtár. InA Korpusznyelvészett˝ ol a Neurális Hálókig(pp. 35–45). Tinta Könyvkiadó

2014
[21]

(2025).Language detection and computational profiling dataset[Data set]

Pog ˘acean, P.-A. (2025).Language detection and computational profiling dataset[Data set]. GitHub.https: //github.com/PogaceanPaul/DataSetLanguage

2025
[22]

(2024).Romanian fairy tales and children’s stories dataset[Data set].https://www

Pove¸ sti pentru copii. (2024).Romanian fairy tales and children’s stories dataset[Data set].https://www. povesti-pentru-copii.com

2024
[23]

Tiedemann, J. (2012). Parallel data, tools and interfaces in OPUS. InProceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)(pp. 2214–2218).https://opus.nlpl.eu/

2012
[24]

Tufi¸ s, D., Barbu Mititelu, V ., Irimia, E., Boro¸ s, T., & Coman, A. (2014). The reference corpus of the con- temporary Romanian language (CoRoLa). InProceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)(pp. 1164–1169).https://aclan...

2014
[25]

Zehra, F., Javed, M., Khan, D., & Pasha, M. (2020). Comparative analysis of C++ and Python in terms of memory and time.Preprints.https://doi.org/10.20944/preprints202012.0516.v1

2020
[26]

Truic ˘a, C.-O., Velcin, J., & Boicea, A. (2015). Automatic language identification for Romance languages us- ing stop words and diacritics. In2015 17th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing(pp. 243–246). IEEE.https://doi.org/10.11...

2015 doi
[27]

J., & Honkela, T

Vatanen, T., Väyrynen, J. J., & Honkela, T. (2010). Language identification of short text segments with n- gram models. InProceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)(pp. 3423–3430).http://www.lrec-conf.org/proceedings/lrec...

2010
[28]

N., Kaiser, Ł., & Polosukhin, I

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. InAdvances in Neural Information Processing Systems 30 (NIPS 2017)(pp. 5998–6008).https://arxiv.org/abs/1706.03762

2017 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

H., & Kareem, F

Abbas, R. H., & Kareem, F. A. E. A. (2019). Text language identification using letters (frequency, self- information, and entropy) analysis for English, French, and German languages.Journal of Southwest Jiaotong University,54(4).https://doi.org/10.35741/issn.0258-2724.54.4.22

work page doi:10.35741/issn.0258-2724.54.4.22 2019

[2] [2]

(1986).The statistical analysis of compositional data

Aitchison, J. (1986).The statistical analysis of compositional data. Chapman and Hall

1986

[3] [3]

Aksan, Y ., Aksan, M., Mersinli, ˘U., & Demirhan, U. U. (2016).A frequency dictionary of Turkish. Routledge

2016

[4] [4]

(2022).ROST (ROmanian stories and other texts) dataset[Data set]

Avram, S.-M. (2022).ROST (ROmanian stories and other texts) dataset[Data set]. Kaggle.https://doi.org/ 10.34740/KAGGLE/DS/2545712

arXiv 2022

[5] [5]

(2021).Deutsches Referenzkorpus: Buchstaben- und Bigrammhäufigkeiten in der deutschen Sprache

Belica, C., & Perkuhn, R. (2021).Deutsches Referenzkorpus: Buchstaben- und Bigrammhäufigkeiten in der deutschen Sprache. Leibniz-Institut für Deutsche Sprache.https://www.ids-mannheim.de/fileadmin/kl/ derewo/derechar-v-XXX-YYY-2021-10-31-1.0.pdf

2021

[6] [6]

Bouma, G. (2015). N-gram frequencies for Dutch twitter data.Computational Linguistics in the Netherlands Journal,5, 25–36.https://www.clinjournal.org/clinj/article/view/45

2015

[7] [7]

Brown, T., Mann, B., Ryder, N., et al. (2020). Language models are few-shot learners.Advances in Neural Information Processing Systems,33, 1877–1901.https://proceedings.neurips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

2020

[8] [8]

B., & Trenkle, J

Cavnar, W. B., & Trenkle, J. M. (1994). N-gram-based text categorization. InProceedings of the Third Annual Symposium on Document Analysis and Information Retrieval(pp. 161–175)

1994