Pith. sign in

REVIEW 3 major objections 4 minor 16 references

A 303,581-row Korean instruction corpus, generated without seed questions, shows one exact duplicate and zero near-duplicate clusters in audit probes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:11 UTC pith:XVEZYALX

load-bearing objection A genuinely useful 303k Korean instruction corpus with an honest write-up; the contamination 'zeros' are sample-bound and the interpretation overreaches, but the resource itself is the contribution. the 3 major comments →

arxiv 2607.20443 v1 pith:XVEZYALX submitted 2026-05-12 cs.CL cs.LG

GLAN-QnA-KR: A Seedless Taxonomy-Driven Korean Instruction Corpus

Daekeun Kim This is my paper
classification cs.CL cs.LG
keywords Korean instruction corpussynthetic instruction dataseedless taxonomydata contamination auditinstruction tuningduplicate detectionopen datasetKorean NLP
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

GLAN-QnA-KR is a Korean instruction-QA corpus of 303,581 question/answer pairs produced without any seed instruction set: a flat taxonomy of 1,084 disciplines, a difficulty scale, and a single producer model generate every pair. The paper's central claim is that this seedless protocol yields exceptionally clean synthetic data at scale: exactly one exact duplicate question in the full corpus, zero near-duplicate trigram clusters in a 5,000-item probe, and a maximum test-vs-corpus trigram Jaccard of 0.163 against KMMLU, KoBEST, and HAE-RAE-Bench, with zero items at the 0.7-0.8 near-duplicate thresholds. A complementary embedding-level check finds one boundary item at cosine 0.901 and none above 0.95. The paper argues that downstream Korean SFT can therefore use this corpus without a contamination adjustment on those seven evaluation splits. A sympathetic reader would care because it is the first public demonstration that seedless taxonomy-driven synthesis, previously applied to English, transfers to a morphologically distant language at the 300k scale with measurable cleanliness.

Core claim

The central discovery is that a seedless taxonomy-driven synthesis pipeline can produce a large Korean instruction corpus whose redundancy and benchmark overlap are both near zero. Across 303,581 rows, exact duplicate questions number one; a 5,000-question probe finds no character-trigram clusters at Jaccard >= 0.9; and the largest per-test-question trigram Jaccard against any of the seven audited Korean evaluation splits is 0.163, with zero test items at >= 0.7 or >= 0.8. The embedding-based paraphrase check finds one KoBEST/boolq item at cosine 0.901, none at 0.95 or above, against a 20,000-question sample. The paper attributes this cleanliness to the absence of seed instructions: textual

What carries the argument

The load-bearing mechanism is the seedless taxonomy-driven synthesis loop: a flat taxonomy of 1,084 English-labelled disciplines, each sampled into a class session, then a (question, answer) pair generated at a difficulty drawn from {100,...,900} by a single instruction-tuned mixture-of-experts producer model, with no input instruction corpus anywhere in the loop. The companion machinery is the two-layer contamination audit: character-trigram Jaccard over whitespace-normalised text for lexical near-duplicates, and L2-normalised multilingual embedding cosine for semantic paraphrase, both computed per test question against a 20,000-question random sample. The audit thresholds (Jaccard ≥ 0.8, c

Load-bearing premise

The audit probes only 20,000 of the 303,581 rows against benchmark questions (and 5,000 rows for dedup), so the claim that the corpus is substantially contamination-free and near-duplicate-free rests on those samples being representative of the remaining rows.

What would settle it

Run the same character-trigram Jaccard comparison over all 303,581 questions against the full KMMLU test set, or draw a fresh 20,000-row sample; finding any test question with Jaccard ≥ 0.7, or any cluster of corpus questions at trigram Jaccard ≥ 0.9 beyond the known single duplicate, would falsify the cleanliness claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Downstream SFT on this corpus can report KMMLU, KoBEST, and HAE-RAE-Bench scores without a contamination-adjustment step, on the audited splits.
  • Korean instruction data at ≥300k rows no longer requires translation of an English corpus or aggregation of heterogeneous sources; a single seedless pipeline suffices.
  • The release provides a reusable audit protocol: character-trigram Jaccard plus multilingual embedding cosine, with thresholds, can be applied to any future synthetic Korean corpus.
  • The long-tailed answer field (max 177,014 characters) means naive length-packed samplers will be dominated by a few essay-length rows; downstream users should clip or bucket answers.
  • The flat 1,084-way discipline distribution, with top subject at 0.27% of rows, gives balanced coverage across STEM, applied health, business, and education topics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the cleanliness transfers to the unsampled 283,581 rows, the result implies that seedless taxonomy synthesis may be a general recipe for low-contamination synthetic instruction data in other low-resource or morphologically rich languages, not just Korean.
  • The audit's answer-level blind spot is the most likely route to undetected contamination: a producer that reproduces only named entities or numeric answers in differently worded questions would pass both checks, so an answer-level or entity-level audit is a natural next test.
  • The English-tagged taxonomy suggests a practical extension: mapping the 1,084 disciplines to a Korean macro-domain grouping would let downstream users do stratified difficulty- or domain-aware sampling without retraining the producer.
  • Benchmark designers could treat the single cosine 0.901 boundary item as a calibration data point for what unrelated Korean prose looks like in embedding space, since the median max-cosine already sits at 0.81-0.84.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper releases GLAN-QnA-KR, a 303,581-row Korean instruction-QA corpus generated with the seedless taxonomy-driven GLAN pipeline and Phi-3.5-MoE-instruct as the producer model. It documents the corpus schema, length/difficulty/discipline statistics, duplicate status, a two-layer contamination audit against KMMLU, KoBEST, and HAE-RAE-Bench using character-trigram Jaccard and multilingual-E5 cosine similarity, and licensing considerations. The central claims are that the corpus is essentially duplicate-free (one exact duplicate; zero near-duplicate clusters in a 5k probe) and substantially contamination-free against the audited benchmarks, with zero test items at Jaccard ≥0.8 and a single item at cosine ≥0.90.

Significance. If the claims hold, the corpus is a useful open resource for Korean SFT: it is the largest single-pipeline synthetic Korean instruction corpus released under OpenRAIL, and the contamination audit addresses a real practical concern for downstream evaluation. The paper is appropriately modest about not introducing a new method and about not providing SFT-gain evaluations. Its strengths are the concrete, reproducible statistics, the explicit licensing discussion, and the honest limitations section. The main risk is that the headline contamination and near-dedup claims are based on samples (20,000 of 303,581 GLAN rows; 5,000-row dedup probe; 500 of 35,030 KMMLU questions across 8 of 45 subjects), while the interpretation is worded as a corpus-level guarantee.

major comments (3)
  1. [§5 Protocol and Interpretation] The contamination audit compares each test question against only 20,000 uniformly sampled GLAN rows (6.6% of 303,581). This cannot certify 'zero test items at Jaccard ≥0.8' for the full corpus. If 10 contaminated rows exist in the 303k corpus, a 20k uniform sample misses all of them with probability ≈52%; if only 1 exists, the miss probability is ≈94%. The §5 Interpretation sentence that scores 'can be read without a contamination-adjustment' is therefore a sample-bound statement, not a corpus-level guarantee. Please either run a full MinHash-LSH scan over all rows, provide exact upper confidence bounds, or rescope the abstract and interpretation to the audited sample.
  2. [§4 Dedup cleanliness] The exact-duplicate count (1 in 303,581 rows) is a full-corpus statement, but the near-duplicate claim rests on a 5,000-question probe. A 5k sample cannot rule out Jaccard≥0.9 clusters in the remaining 298k rows. The sentence 'The taxonomy-driven seedless generation procedure therefore produces essentially non-overlapping prompts at this scale' overstates the evidence. Please either add a full near-duplicate scan or explicitly state that the near-duplicate guarantee is sample-based only.
  3. [§5, Table 3, and Abstract] The KMMLU audit uses a 500-question slice across 8 of 45 subjects, yet the abstract and §5 present the result as an audit 'against KMMLU.' A reader could reasonably infer that all 35,030 KMMLU questions were checked. Unaudited subjects—especially those overlapping the 1,084-discipline taxonomy—could contain a Jaccard≥0.8 pair. Please either audit all KMMLU subjects or consistently qualify the claim as 'eight KMMLU subjects' in the abstract, tables, and interpretation.
minor comments (4)
  1. [§5] Typo: 'We release a audit' should be 'We release an audit.'
  2. [§5 Protocol / Table 3] The protocol says counts at Jaccard ≥0.7 and ≥0.8 are reported, but Table 3 only has a J≥0.8 column. Since the maximum observed J is 0.163, the ≥0.7 count is implied to be zero, but the table should either include the column or the protocol text should be adjusted.
  3. [§3 / §7] The statement 'We release the released corpus without also releasing the proprietary prompt templates' is awkward; also, §7 says the audit is 'reproducible from the released dataset alone' but the audit scripts are not released. A script release or a more precise statement about the exact sampling seed and procedure would improve reproducibility.
  4. [Table 1] The table caption says sizes and licenses were verified on 2026-05-12, but the paper is dated May 2026; this is fine, but the verification date should be stated in the text as well, not only in the caption, for clarity.

Circularity Check

0 steps flagged

No circularity: the corpus statistics and contamination audit are direct measurements against external artifacts, with no fitted quantity or self-citation chain doing the work.

full rationale

The paper's central claims are (1) the release of a 303,581-row Korean instruction corpus, (2) dedup statistics, and (3) a contamination audit against KMMLU, KoBEST, and HAE-RAE-Bench. These are measurements of the released artifact and of external benchmark splits, not derivations from a fitted parameter. The audit compares GLAN-QnA-KR questions to benchmark test questions using character-trigram Jaccard and multilingual-E5 cosine, and reports observed maxima; no quantity is fit to the benchmarks and then renamed as a prediction. The 'seedless' property is inherited from the GLAN protocol [7], which is authored by Li et al. and is external to this paper; the paper explicitly disclaims a new synthesis method. The only self-citation [5] is the dataset card, which is not load-bearing for any claim. The sampling limitation (20,000 of 303,581 rows; KMMLU slice of 8 subjects) is a statistical coverage issue and is candidly acknowledged in the Limitations section, including the statement that a paraphrasing producer or answer-level matches could evade detection. Sampling risk is a correctness concern, not circularity. No equation reduces to its own input, no fitted parameter is presented as a prediction, and no load-bearing argument rests on a self-citation chain. The audit is externally falsifiable and reproducible from the released dataset, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper's claims rest mainly on sampling assumptions and hand-chosen thresholds, not on fitted physics/ML parameters. The free parameters are audit thresholds and sample sizes; the axioms are representativeness assumptions about samples and the adequacy of the similarity measures.

free parameters (3)
  • Jaccard near-duplicate thresholds (0.7 / 0.8) = 0.7 / 0.8
    Chosen by hand to match community practice; the core 'zero contaminated items' statement is defined at these thresholds, so a different threshold changes the conclusion. No null distribution or power analysis is supplied.
  • Cosine thresholds (0.90 / 0.95) = 0.90 / 0.95
    The paper calls 0.95 'the operationally meaningful threshold' after observing the median baseline 0.81–0.84; the single 0.901 hit lies between the two thresholds, making the boundary outcome threshold-dependent.
  • Audit/probe sample sizes (20,000 GLAN rows; 5,000-row dedup probe; 2,500 test items) = 20k / 5k / 2.5k
    Chosen by the author, not derived from a required confidence level. All dedup and contamination guarantees are conditional on these samples; the full corpus and full KMMLU are not scanned.
axioms (4)
  • domain assumption The 20,000-question uniform sample of GLAN-QnA-KR is representative of the full 303,581-row corpus for duplicate and contamination purposes.
    Invoked in §4 Dedup cleanliness and §5 Protocol; if the sample misses a contaminated or duplicated region, the reported maxima do not bound the corpus.
  • domain assumption The audited subsets (KMMLU 8 subjects, HAE-RAE 500, KoBEST 300 per task) represent the full benchmarks.
    Tables 3/4 use subset test items; KMMLU in full is 35,030 questions across 45 subjects, so the audit does not cover most KMMLU questions.
  • domain assumption Question-level character-trigram Jaccard and multilingual-E5 cosine are sufficient to detect meaningful contamination for downstream eval scores.
    Limitation 5 admits a paraphrasing producer or fact-only leakage (named entities, numeric answers) would evade detection, so the audit is a lower bound on contamination risk.
  • domain assumption The released rows are exactly the non-empty outputs of the described pipeline; no undisclosed filtering or selection was applied besides non-empty question/answer.
    §3 Post-filtering; the dedup cleanliness and length tails are interpreted relative to this premise, which cannot be independently verified without the exact generator.

pith-pipeline@v1.3.0-alltime-deepseek · 8038 in / 13636 out tokens · 133270 ms · 2026-08-02T14:11:07.232139+00:00 · methodology

0 comments
read the original abstract

We release GLAN-QnA-KR, a 303,581-row openly redistributable Korean instruction-QA corpus produced via the seedless taxonomy-driven GLAN synthesis pipeline with Microsoft's Phi-3.5-MoE-instruct as the producer model (generation: 2024-12; release: 2024-12; licence: OpenRAIL). The corpus spans a flat taxonomy of 1,084 English-labelled disciplines paired with Korean question/answer text, a 100-900 difficulty scale, and a median of 313 question characters and 1,098 answer characters per record. Two properties are atypical for synthetic instruction data at this scale: (i) exact duplicate questions number only 1 in 303,581 rows and character-trigram near-duplicate clusters at Jaccard >= 0.9 number zero in a 5,000-sample probe, and (ii) a two-layer contamination audit against KMMLU, KoBEST (five sub-tasks), and HAE-RAE-Bench shows a maximum test-vs-corpus question-level character-trigram Jaccard of 0.163 with zero test items at Jaccard >= 0.7, and a maximum multilingual-E5 cosine of 0.901 with a single test item at cosine >= 0.90 and zero at >= 0.95, across 20,000 sampled GLAN questions and seven evaluation sets. At the time of release, this is, to our knowledge, the largest single-pipeline synthetic Korean instruction corpus verifiable on the Hugging Face Hub and the only Korean >=100k-row corpus built under a seedless taxonomy-driven protocol. This note documents the generation protocol, corpus statistics, the contamination audit, and the licensing boundary in a form suitable for downstream citation.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 11 linked inside Pith

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.NeurIPS, 2020

    2020

  3. [3]

    Smith, and Jesse Dodge

    Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. What’s in my big data?arXiv preprint arXiv:2310.20707, 2023. 8

    Pith/arXiv arXiv 2023

  4. [4]

    Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C´ esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    GLAN-qna-kr-300k dataset card

    Daekeun Kim. GLAN-qna-kr-300k dataset card. https://huggingface.co/datasets/ daekeun-ml/GLAN-qna-kr-300k, 2024

    2024

  6. [6]

    KoBEST: Korean balanced evaluation of significant tasks

    Dohyeong Kim, Myeongjun Jang, Deuk Sin Kwon, and Eric Davis. KoBEST: Korean balanced evaluation of significant tasks. InCOLING, 2022

    2022

  7. [7]

    Synthetic data (almost) from scratch: Generalized instruction tuning for language models.arXiv preprint arXiv:2402.13064, 2024

    Haoran Li, Qingxiu Dong, Zhengyang Chen, Huanqi Wang, Haoyang Meng, Jingjing Liu, Jiaxing Lv, Yaoyao Liu, Hang Xu, Zhongkai Sun, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models.arXiv preprint arXiv:2402.13064, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Phi-3.5-MoE-instruct model card

    Microsoft. Phi-3.5-MoE-instruct model card. https://huggingface.co/microsoft/ Phi-3.5-MoE-instruct, 2024

    2024

  9. [9]

    Azure/synthetic-qa-generation: Generate synthetic qnas from real-world data.https://github.com/Azure/synthetic-qa-generation, 2024

    Microsoft Azure. Azure/synthetic-qa-generation: Generate synthetic qnas from real-world data.https://github.com/Azure/synthetic-qa-generation, 2024

    2024

  10. [10]

    Orca 2: Teaching small language models how to reason.arXiv preprint arXiv:2311.11045, 2023

    Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, et al. Orca 2: Teaching small language models how to reason.arXiv preprint arXiv:2311.11045, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    KMMLU: Measuring massive multitask language understanding in Korean.arXiv preprint arXiv:2402.11548, 2024

    Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. KMMLU: Measuring massive multitask language understanding in Korean.arXiv preprint arXiv:2402.11548, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    HAE-RAE bench: Evaluation of Korean knowledge in language models.arXiv preprint arXiv:2309.02706, 2023

    Guijin Son, Hanwool Lee, Suwan Kim, Jaecheol Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. HAE-RAE bench: Evaluation of Korean knowledge in language models.arXiv preprint arXiv:2309.02706, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Multilingual E5 text embeddings: A technical report.arXiv preprint arXiv:2402.05672, 2024

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual E5 text embeddings: A technical report.arXiv preprint arXiv:2402.05672, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.arXiv preprint arXiv:2212.10560, 2022

    Pith/arXiv arXiv 2022

  15. [15]

    Magicoder: Source code is all you need.arXiv preprint arXiv:2312.02120, 2023

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need.arXiv preprint arXiv:2312.02120, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    WizardLM: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardLM: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023. 9

    Pith/arXiv arXiv 2023