REVIEW 3 major objections 7 minor 31 references
QQ: A Language Metadata Toolkit for Multilingual NLP
T0 review · 3 major / 7 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read QQ builds a single traversable graph that merges scattered, conflicting language metadata sources, so researchers can reliably normalize and convert language identifiers across ISO 639, BCP-47, Glottocode, and Wikidata.
desk verdict QQ is a genuinely useful metadata toolkit, but the paper needs to reconcile its motivating malformed-identifier examples with QQ's explicit refusal to parse standards. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The graph data model: Languoid, Script, and Region nodes connected by relational edges, with a canonical-node mechanism that maps all incoming identifiers (ISO 639-1/2/3, BCP-47, Glottocode, Wikidata, Wikipedia) to one node. The graph is the single abstraction that makes identifier conversion, name lookup, and traversal by family, script, or region possible in a few function calls.
What would settle it
Take a random sample of 200 languoids from QQ, manually resolve each identifier to its official registry entry (ISO 639-3, Glottolog, Wikidata), and compare with QQ's canonical resolution. Any mismatch in a hand-checked sample would show that the claimed reliable normalization does not hold universally. Alternatively, compare every ISO 639-3-to-Glottocode mapping in QQ against the official ISO registration authority list; a single unexplained discrepancy is a falsifying observation.
Extended reading notes
Core claim
The paper introduces QQ (QwanQwa), a Python library and browser explorer that unifies language metadata as a graph of languoids, scripts, and regions. Its central claim is that converting between identifier systems — de, deu, ger, stan1295, Q188 all refer to German — becomes reliable and scalable when every identifier resolves to a single canonical node in that graph. QQ merges sources with explicit conflict-resolution rules (source priority plus manual overrides) and ships a pre-compiled database, so users never need to touch the raw, conflicting resources. The three case studies purport to show that the toolkit makes audits, reporting, and dataset-linking workflows faster and less error-pr
Load-bearing premise
QQ's reliability rests on the accuracy and currency of its upstream sources and on the merge rules used when those sources conflict; the paper's own Limitations section concedes that merging and normalization could introduce ambiguities or inaccuracies.
Editorial extensions
If this is right
- Datasets that currently use incompatible identifier schemes (e.g., ISO 639-1 vs. BCP-47) can be joined automatically through the canonical graph.
- Deprecated and 'unknown' codes — the audit found 128 dataset usages of deprecated ISO 639-3 codes — can be flagged and replaced, cleaning up large multilingual corpora.
- Generating language-reporting tables for papers becomes scriptable, supporting reproducibility and accountable language naming.
- Cross-lingual analyses that require merging resources like BabelNet, Concepticon, and NoRaRe can be performed without manual code-mapping, as demonstrated in the colexification case study.
- Because QQ reports conflicts found during database construction, it can act as a quality-control tool that surfaces data issues in upstream language resources.
Reading between the lines
- If adopted widely, a canonical language graph could become a standard layer in the NLP stack, but its long-term reliability depends on ongoing curation and transparent conflict-resolution policies, not just the initial merge.
- The graph's traversability invites new research queries — for example, testing whether languages that share a script or region show predictable patterns in psycholinguistic norms — that would be tedious to run without a unified graph.
- A testable extension would be to benchmark QQ's identifier-resolution accuracy against a hand-verified gold standard sample, giving users a quantitative confidence measure.
- The neutral 'languoid' model sidesteps politically loaded classifications, but downstream tasks may need to recover those distinctions; a practical next step is a compatibility layer for user-defined language/dialect boundaries.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents QQ (QwanQwa), a Python toolkit and graph-based database for language metadata. QQ merges several external resources—Glottolog, LinguaMeta, GlotScript, pycountry, SIL/IANA deprecation lists, Wikidata, and Wikipedia—into a unified graph of Languoid, Script, and Region nodes, with 27,256 languoids, 227 scripts, 5,361 regions, and 97,288 edges. The toolkit exposes a Python API, a CLI, and a browser explorer for identifier conversion (ISO 639-1/2/3, BCP-47, Glottocode, Wikidata), metadata lookup, graph traversal, and queries. The authors demonstrate QQ on three case studies: an audit of language identifiers on the HuggingFace Hub, language reporting workflows, and a psycholinguistic colexification analysis that links BabelNet, Concepticon, and NoRaRe. The paper claims that QQ 'unifies scattered and potentially conflicting metadata' into a reliable normalization and mapping interface.
Significance. If the toolkit works as claimed, it would be a practical contribution to multilingual NLP infrastructure, where identifier mismatch and metadata lookup are recurring bottlenecks. The paper's strengths include a public code repository, a rebuildable database with explicit conflict reporting, open data licensing, and case studies that go beyond a toy example. The graph model is a reasonable design choice, and the HuggingFace audit provides a concrete sense of the identifier heterogeneity in the wild. However, the central reliability claim is not yet supported by a validation of QQ's identifier-resolution correctness, and the paper's own motivating example exposes a gap between the stated problem and the implemented functionality. These issues are addressable but need work before the claims can be accepted as stated.
major comments (3)
- [§1 and §3] The paper introduces NLLB and FineWeb-2 codes such as 'deu_Latn' as a central motivation, but §3 states that 'QQ expects language codes to be well-formed and existing in some source. QQ does not do any parsing of standards.' Since 'deu_Latn' is not a well-formed BCP-47 tag (the language subtag must be 'de', not 'deu'), and LinguaMeta is said to follow the spec, QQ's shipped database very likely contains no entry for the exact motivating string. The advertised normalization of real-world dataset identifiers therefore depends on an external parser (e.g., langcodes) that is only mentioned as 'complementary' and is not integrated or evaluated. The HuggingFace audit (§4) reports 8,189 unique codes and 8,122 valid, but does not state how the 67 remaining malformed or partial codes were handled; if a preprocessing step canonicalized them, that step is part of the core claim and must be specifie
- [§2.1] Conflict resolution in QQ is performed by 'manual resolution or by source priority,' with no independent gold-standard validation. The example of the ISO 3166-2 code AN (LinguaMeta vs. pycountry) is resolved by source priority, and the Limitations section concedes that 'our merging and normalization could introduce ambiguities or inaccuracies.' For a toolkit whose central promise is reliable identifier mapping, the absence of any quantitative evaluation of resolution accuracy is a serious gap. I ask the authors to provide at least: (a) statistics on how many conflicts were resolved by source priority vs. manual overrides, (b) a random-sample validation of resolved identifiers against a manually curated gold standard or against authoritative mappings (e.g., CLDF/Glottolog ISO–Glottocode mappings), and (c) an error analysis of any mismatches. Without this, the claim that QQ 'unifies' and '
- [§4.3] The colexification case study is presented as a demonstration of QQ's utility, but its substantive conclusions ('valence dissimilarity is consistently lower for own-language colexifications') are also used as evidence that the identifier linking works. Since QQ is the only linking mechanism, an identifier-resolution error could in principle create or destroy colexification edges and thereby influence the Rayleigh quotients and Mann-Whitney results. No sensitivity analysis, manual verification of linked concepts, or comparison against a non-QQ baseline (e.g., manual mapping or langcodes) is provided. I do not ask the authors to re-run the entire study, but they should either downgrade the case-study conclusions to 'illustrative' or add a small-scale validation that the QQ-linking step does not drive the reported effects.
minor comments (7)
- [§1] Typo: 'ISO 693-2' should be 'ISO 639-2'. Also, the description of 'deu_Latn' as 'partial BCP-47' is imprecise; it is a non-canonical or malformed BCP-47 tag, and this distinction is relevant to the major comment above.
- [§2.1] The claim that pycountry is 'correctly' listing AN as historical is footnoted to a Wikipedia page; an official ISO or standards-body citation would be more appropriate and would strengthen the conflict-resolution example.
- [Table 1] The table header 'Lang Id Standards' and the legend '1/2/...' are unclear; a reader cannot tell what the numbers mean. Please spell out the standards or add a legend. Also, 'Phylogentic' is a typo for 'Phylogenetic'.
- [§2.2] The example 'db.guess("dutc1256")' returns 'dutc1256'; this is a valid Glottocode, but the example would be clearer if the code were distinguishable from the returned value (e.g., using a different word). Minor.
- [§4.1 and Table 4] The HF audit reports counts of 'Country code', 'Unknown', and 'Wikipedia' codes, but does not define how these categories were identified. Please specify the classification procedure so the audit is reproducible.
- [A.1 / Rayleigh quotient] The equation for the normalized Laplacian is written as 'L=I−D 1/2AD1/2'; the exponents should be '-1/2' to be mathematically correct. The surrounding text also states 'The eigenvalues of L is bounded by [0,2]', which should be 'are bounded'.
- [General] The paper does not provide a pinned version or commit hash for the QQ repository, despite emphasizing versioning and reproducibility. Please include the exact version/commit used for the reported database and case studies.
Circularity Check
No significant circularity: QQ is an external-source integration tool and the case studies are evaluated on independent external data; minor self-citations are not load-bearing.
full rationale
The paper's central deliverable is a compiled metadata graph, not a derived prediction. Identifier mappings are imported from external sources (LinguaMeta, Glottolog, Glotscript, pycountry, SIL, IANA, Wikipedia/Wikidata) rather than fitted to QQ's own outputs, and the conflict-resolution rule in §2.1 is explicitly a curation choice (e.g., LinguaMeta vs pycountry on ISO 3166-2 AN). The Limitations section concedes that 'our merging and normalization could introduce ambiguities or inaccuracies,' which is a data-quality caveat, not a circular step. The three case studies use external datasets (HuggingFace Hub metadata, BabelNet/Concepticon/NoRaRe) and standard statistical procedures (permutation tests, Mann-Whitney U, Cohen's d), so their results are not entailed by construction from QQ's parameters. The only self-referential elements are support citations in §4 ('QQ has already been used in research (e.g., Lent, 2025; Poelman et al., 2025; Accou and Poelman, 2026)') and consistency checks against prior colexification findings by co-authors; these are not load-bearing because the paper independently demonstrates the workflows and the smoothness results are permutation-tested. The §3 statement that 'QQ expects language codes to be well-formed and existing in some source. QQ does not do any parsing of standards' is a scope limitation for malformed BCP-47-style tags, but that is a correctness gap, not an input-output equivalence. No step in the paper reduces to its own inputs by definition, so the circularity score is low.
Assumptions & free parameters
free parameters (2)
- source_priority_order_and_manual_resolutions =
e.g., pycountry over LinguaMeta for AN historical status; repository-specific manual fixes
- HF audit minimum-languages cutoff =
10 languages
assumptions (7)
- domain assumption The external metadata sources (Glottolog, LinguaMeta, pycountry, SIL/IANA, Wikipedia/Wikidata) are sufficiently accurate for the graph's purpose.
- ad hoc to paper QQ's source-priority/manual conflict resolution yields correct canonical facts when sources disagree.
- ad hoc to paper Modeling language, dialect, and family all as one 'Languoid' node does not corrupt identity resolution.
- domain assumption After per-dataset z-scoring, psycholinguistic ratings from different NoRaRe datasets are comparable and can be averaged per concept/language.
- domain assumption Colexification edges from BabelNet/Concepticon correctly represent cross-lingual concept colexification.
- domain assumption Permutation and Mann-Whitney tests treat colexification edge pairs as exchangeable/independent, yielding valid p-values.
- standard math Normalized graph Laplacian L=I-D^{-1/2}AD^{-1/2} has eigenvalues in [0,2].
Cite this review
Pith. "Pith review of QQ: A Language Metadata Toolkit for Multilingual NLP." pith.science (2026). https://pith.science/paper/EWMXXUMZ
@misc{pith2026260300620,
author = {Pith},
title = {Pith review of: QQ: A Language Metadata Toolkit for Multilingual NLP},
year = {2026},
howpublished = {\url{https://pith.science/paper/EWMXXUMZ}},
note = {Machine review of arXiv:2603.00620}
}
read the original abstract
Multilingual NLP research increasingly involves hundreds or thousands of languages across different datasets. Managing, discovering, and reporting language metadata becomes a common hurdle at these scales. We present QQ, a metadata toolkit and browser explorer. QQ compiles language metadata sources into a graph of language varieties, scripts, regions, identifiers, names, and relations, and exposes it through a Python API, a command-line interface, and a browser-based explorer. Users can normalize identifiers, retrieve metadata, traverse relations, and discover which external resources contain a language. We demonstrate QQ on three workflows: an audit of the HuggingFace Hub, linking resources that use different identifier systems, and generating reproducible language-reporting tables. QQ supports FAIR-oriented metadata practices through versioning, open formats, and reusable interfaces.
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Stef Accou and Wessel Poelman. 2026. https://arxiv.org/abs/2601.16629 Typologically Informed Parameter Aggregation . In Findings of the Association for Computational Linguistics : EACL 2026 . Association for Computational Linguistics
arXiv 2026
-
[4]
Emily Bender. 2019. https://thegradient.pub/the-benderrule-on-naming-the-languages-we-study-and-why-it-matters/ The \# BenderRule : On Naming the Languages We Study and Why It Matters . The Gradient
2019
-
[5]
Yiyi Chen, Russa Biswas, and Johannes Bjerva. 2023. https://aclanthology.org/2023.nodalida-1.67/ C olex2 L ang: Language embeddings from semantic typology . In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 673--684, T \'o rshavn, Faroe Islands. University of Tartu Library
2023
-
[6]
Yiyi Chen and Johannes Bjerva. 2023 a . https://doi.org/10.18653/v1/2023.sigmorphon-1.11 Colexifications for bootstrapping cross-lingual datasets: The case of phonology, concreteness, and affectiveness . In Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 98--109, Toronto, Canada. Associa...
-
[7]
Yiyi Chen and Johannes Bjerva. 2023 b . https://arxiv.org/abs/2312.11069 Patterns of Closeness and Abstractness in Colexifications : The Case of Indigenous Languages in the Americas . Preprint, arXiv:2312.11069
arXiv 2023
-
[8]
Marta R. Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, and 1 others. 2024. https://www.nature.com/articles/s41586-024-07335-x Scaling neural machine translation to 200 languages . Natur...
2024
Show all 31 references
-
[9]
Anna Di Natale, Max Pellert, and David Garcia. 2021. Colexification networks encode affective meaning. Affective Science, 2(2):99--111
2021
-
[10]
Robert Forkel and Harald Hammarstr \"o m. 2022. https://journals.sagepub.com/action/showAbstract Glottocodes: Identifiers linking families, languages and dialects to comprehensive reference information . Semantic Web, 13(6):917--924
2022
-
[11]
Greenhill, Christoph Rzymski, Sebastian Bank, Michael Cysouw, Harald Hammarstr \"o m, Martin Haspelmath, Gereon A
Robert Forkel, Johann-Mattis List, Simon J. Greenhill, Christoph Rzymski, Sebastian Bank, Michael Cysouw, Harald Hammarstr \"o m, Martin Haspelmath, Gereon A. Kaiping, and Russell D. Gray. 2018. https://www.nature.com/articles/sdata2018205 Cross- Linguistic Data Formats , adva...
2018
-
[12]
Alexandre Fran c ois. 2008. https://www.degruyterbrill.com/document/doi/10.1075/slcs.106.09fra/html?lang=en Semantic maps and the typology of colexification: Intertwining polysemous networks across languages . In Martine Vanhove, editor, From Polysemy to Semantic Change : Towa...
2008 doi
-
[13]
Jeff Good and Calvin Hendryx-Parker . 2006. https://pure.mpg.de/pubman/faces/ViewItemOverviewPage.jsp?itemId=item_1555146 Modeling contested categorization in linguistic databases . In EMELD Workshop 2006: Digital Language Documentation ; Tools and Standards ; The State of Art
2006
-
[14]
Harald Hammarstr \"o m, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2024. https://glottolog.org/ accessed 2024-09-20 Glottolog 5.0 . Max Planck Institute for Evolutionary Anthropology
2024
-
[15]
Amir Hossein Kargaran, Fran c ois Yvon, and Hinrich Sch \"u tze. 2024. https://aclanthology.org/2024.lrec-main.687/ GlotScript : A Resource and Tool for Low Resource Writing System Identification . In Proceedings of the 2024 Joint International Conference on Computational Ling...
2024
-
[16]
Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. https://openreview.net/forum?id=Y45ZCxslFx MADLAD-400 : A Multilingual And Document-Level Large Audited Dataset . In Thirty-Seventh...
2023
-
[17]
Joshua Lambert and Chris Hartgerink. 2024. https://epiverse-trace.github.io/posts/data-licensing.html Dual licensing R packages with code and data
2024
-
[18]
Heather Lent. 2025. https://arxiv.org/abs/2507.03473 Beyond Weaponization : NLP Security for Medium and Lower-Resourced Languages in Their Own Right . Preprint, arXiv:2507.03473
2025 arXiv
-
[19]
Quentin Lhoest, Albert Villanova del Moral , Yacine Jernite, Abhishek Thakur, Patrick von Platen , Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario S a s ko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, and 1 others. 2021. https...
2021
-
[20]
Johann-Mattis List, Michael Cysouw, and Robert Forkel. 2016. https://aclanthology.org/L16-1379 Concepticon: A Resource for the Linking of Concept Lists . In Proceedings of the Tenth International Conference on Language Resources and Evaluation ( LREC '16) , pages 2393--2400. E...
2016
-
[21]
Johann Mattis List, Annika Tjuka, Frederic Blum, Alžběta Kučerová, Carlos Barrientos Ugarte, Christoph Rzymski, Simon Greenhill, and Robert Forkel, editors. 2025. https://concepticon.clld.org/ CLLD Concepticon 3.4.0 . Max Planck Institute for Evolutionary Anthropology, Leipzig
2025
-
[22]
Roberto Navigli and Simone Paolo Ponzetto. 2010. https://aclanthology.org/P10-1023/ BabelNet : Building a Very Large Multilingual Semantic Network . In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics , pages 216--225. Association for Com...
2010
-
[23]
Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, Daniel Zeman, and 1 others
Joakim Nivre, Marie-Catherine de Marneffe , Filip Ginter, Jan Haji c , Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, Daniel Zeman, and 1 others. 2020. https://aclanthology.org/2020.lrec-1.497/ Universal Dependencies v2: An Evergrowing Multilingual T...
2020
-
[24]
Guilherme Penedo, Hynek Kydl \'i c ek, Vinko Sabol c ec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. https://openreview.net/forum?id=jnRBe6zatP#discussion FineWeb2 : One Pipeline to Scale Them Al...
2025
-
[25]
Wessel Poelman, Thomas Bauwens, and Miryam de Lhoneux . 2025. https://aclanthology.org/2025.emnlp-main.369/ Confounding Factors in Relating Model Performance to Morphology . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages 7273-...
2025
-
[26]
Sandy Ritchie, Daan van Esch , Uche Okonkwo, Shikhar Vashishth, and Emily Drummond. 2024. https://aclanthology.org/2024.lrec-main.921 LinguaMeta : Unified Metadata for Thousands of Languages . In Proceedings of the 2024 Joint International Conference on Computational Linguisti...
2024
-
[27]
Kushal Tatariya, Artur Kulmizev, Wessel Poelman, Esther Ploeger, Marcel Bollmann, Johannes Bjerva, Jiaming Luo, Heather Lent, and Miryam de Lhoneux. 2025. https://arxiv.org/abs/2411.05527 How Good is Your Wikipedia ? Auditing Data Quality for Low-resource and Multilingual NLP ...
2025 arXiv
-
[28]
J \"o rg Tiedemann. 2012. https://aclanthology.org/L12-1246/ Parallel Data , Tools and Interfaces in OPUS . In Proceedings of the Eighth International Conference on Language Resources and Evaluation ( LREC `12) , pages 2214--2218. European Language Resources Association (ELRA)
2012
-
[29]
Annika Tjuka, Robert Forkel, and Johann-Mattis List. 2022. https://doi.org/10.3758/s13428-021-01650-1 Linking norms, ratings, and relations of words and concepts across multiple language varieties . Behavior Research Methods, 54(2):864--884
2022 doi
-
[30]
Rob van der Goot , Esther Ploeger, Verena Blaschke, and Tanja Samardzic. 2025. https://aclanthology.org/2025.emnlp-demos.23/ DistaLs : A Comprehensive Collection of Language Distance Measures . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Proc...
2025
-
[31]
Daan van Esch , Tamar Lucassen, Sebastian Ruder, Isaac Caswell, Clara Rivera, and 1 others. 2022. https://aclanthology.org/2022.lrec-1.538/ Writing System and Speaker Metadata for 2,800+ Language Varieties . In Proceedings of the Thirteenth Language Resources and Evaluation Co...
2022
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.