{"id":"d8a4de2f-4068-43dd-a626-90094270e010","arxiv_id":"2509.02332","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EMCO oversamples minority text by estimating word-transition probabilities from both minority and majority documents, expanding the synthetic minority vocabulary.","lead":"This paper introduces EMCO, a Markov chain oversampling method for imbalanced text classification that borrows word-transition statistics from the majority class to let the synthetic minority vocabulary expand. It reports competitive balanced accuracy on small, severely imbalanced text sets, which is useful when deep learning is impractical.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's 'every row' balanced-accuracy advantage for EMCO γ=1 is not backed by error bars or significance tests; the very-low-frequency differences may be sampling noise.","rationale":"The reader's verdict was CONDITIONAL, citing the absence of error bars and significance tests as a reason. My stress-test identifies that same missing statistical support as the single most load-bearing concern: the central claim is an empirical row-by-row superiority claim, and the evidence for it consists of point estimates with no quantification of variability. This is more directly load-bearing than the paper's stated topic/sequence-independence assumption, because even if that conceptual assumption were questionable, the empirical BA claim could still hold; conversely, without statistical support, the empirical claim is not established regardless of the mechanism. I chose 'partial' agreement because the reader's 'weakest_assumption' field names the conceptual assumption, although the rationale does mention missing significance tests. I recommend keeping the verdict CONDITIONAL rather than rejecting: the issue is addressable by rerunning with paired resampling and reporting confidence intervals, and the code release makes such a check feasible.","tokens_in":22604,"tokens_out":6735,"duration_ms":81050,"concrete_test":"Recompute the very-low-frequency rows of Table 3 using paired resampling over the existing repetitions/categories: for each data set and sampling ratio, draw 10,000 bootstrap samples of the category-level BA difference between EMCO γ=1 and the best non-EMCO method, and run a Wilcoxon signed-rank test across categories. If any row's 95% bootstrap CI for the difference includes 0, or the paired test is not significant for the majority of rows, the 'outperforms every row' claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that EMCO with γ=1 achieves the best balanced accuracy in very-low-frequency categories on every row of Table 3. For this claim to hold, the reported macro-averaged BAs must be stable enough to support row-by-row comparisons. This condition is not secured. Reuters and 20 Newsgroups results are averages of only five repetitions per category; the HuffPost 'Titles' rows without an asterisk come from a single random train/test split. No confidence intervals, standard deviations, or paired significance tests are reported anywhere. Many decisive margins are small: e.g., HuffPost Titles* at 10% sampling is .638 (EMCO γ=1) vs .630 (EMCO γ=0.1) and .615 (EDA); Reuters Full at 10% is .801 vs .791; several low-frequency rows differ by .001–.002. In very-low-frequency categories, test sets contain very few positive examples, so macro-averaged BA is high-variance. The 'every row except one' claim is thus a deterministic reading of point estimates that may not survive resampling. This is compounded by the fact that the BA advantage is largely a recall advantage (Appendix B), while precision drops sharply for γ=1 (e.g., Reuters Titles precision 0.212 vs SMOTE 0.508 at 10% ratio), so the practical interpretation of the BA gain depends on metric choice; but the primary problem is that the headline comparison is statistically unquantified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EMCO, a Markov-chain oversampling method for imbalanced text classification. The transition matrix is estimated from minority-class documents and augmented with transitions observed in majority-class documents from minority-vocabulary words to majority-only words, with a weight parameter γ. Synthetic minority documents are sampled from the resulting chain, allowing the minority feature space to expand beyond the convex hull of the minority training sample. The method is compared with ROS, SMOTE, ADASYN, DRO, DECOM, EDA, MCO (EMCO with γ=0), and a no-oversampling SVM baseline on Reuters-21578, HuffPost, and 20 Newsgroups data, with an additional LSTM experiment on Reuters headlines. The headline empirical claim is that EMCO with γ=1 achieves the best balanced accuracy in very low frequency categories on essentially every row of Table 3, at the cost of lower precision than several competitors. Section 7 reports vocabulary-growth analyses, including a comparison of synthetic vocabulary growth with Heaps' law.","tokens_in":23025,"tokens_out":6587,"duration_ms":85728,"significance":"If the empirical claim holds, EMCO is a practically useful, language-independent oversampling method for severely imbalanced text classification in small-data regimes. The method is clearly specified, the code is released, and the construction is not circular: γ is fixed before evaluation and the Heaps' law material is an external reference, not an input to the method. The paper also honestly identifies the recall-precision tradeoff controlled by γ and the scalability limitations of a vocabulary-squared transition matrix. However, the central empirical claim currently rests on point estimates without error bars or significance tests, and the extrapolation assumption is only indirectly validated. The method is therefore promising, but the evidence as reported does not yet support the 'every row' claim at the strength asserted.","major_comments":[{"comment":"The headline claim that 'EMCO with γ=1 outperforms the other approaches in very low frequency categories on every row, except for one' is not supported by the evidence as reported. Reuters and 20 Newsgroups results are averages of five repetitions per category; HuffPost Titles rows without an asterisk come from a single random split; no standard deviations, confidence intervals, or paired significance tests are given anywhere. Several decisive margins are small (e.g., HuffPost Titles* at 10%: .638 vs .630; Reuters Full at 10%: .801 vs .791; several rows differ by .001–.002), and macro-averaged balanced accuracy on very low frequency categories is high-variance because positive test sets are tiny. The deterministic reading of point estimates is therefore not secure. Please add per-category variances or confidence intervals, use repeated or nested splits for all data sets, and report paire","section":"Section 6, Table 3"},{"comment":"The central extrapolation assumption—that majority-only words following minority vocabulary words in majority documents are reasonable additions to the synthetic minority vocabulary—is only indirectly validated. Section 7's binary classification of synthetic vocabulary is a useful start, but it reports only macro-averaged recall, TNR, and balanced accuracy, not precision per category, so the degree of off-topic injection is not directly quantified. Appendix B shows that at γ=1 precision drops sharply (e.g., Reuters Titles 10%: EMCO .212 vs SMOTE .508) while recall rises; the balanced-accuracy advantage is thus partly a re-labeling of the recall-precision tradeoff. A direct diagnostic of the generated vocabulary—for example, precision for predicting held-out minority test words per category, or a topic-coherence sample of generated words—would substantiate the assumption. As the paper its","section":"Section 4.1, with Section 7 and Appendix B"}],"minor_comments":[{"comment":"The Heaps' law comparison is qualitative: parameters k≈63 and θ≈0.378 are fitted on the same training data, with no fitting procedure, fit diagnostics, or uncertainty reported. Please state how the parameters were estimated and treat the match as illustrative rather than as a formal validation.","section":"Section 7, Figure 3"},{"comment":"For reproducibility, please report random seeds for the five repetitions and for the random train-test splits. The code release is a strength, but seed information is necessary for exact replication and for others to compute standard errors.","section":"Section 5"},{"comment":"There are several typographical errors and inconsistencies, e.g., 'exaclty' in Section 5, 'also form the majority class' in Section 8, and inconsistent use of the † and ∗ table symbols. These should be cleaned up.","section":"Throughout"},{"comment":"Please clarify in the text or pseudocode that the loop keeps drawing tokens until the document contains the drawn number of non-<stop> tokens, and that <stop> is only a control token not included in the final document. The current pseudocode can be misread as allowing documents shorter than the drawn length.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations and I see no circularity or fabrication. The main obstacle is the statistical support for the central empirical claim: the repeated runs have no uncertainty quantification, and some key comparisons are based on a single split. The method itself is clearly specified, the code is released, and the paper is likely acceptable after the authors add error bars or significance tests and moderate the language of the headline claim. The self-citation to Avela (2024) is peripheral and not a concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: EMCO is a genuine, well-specified oversampling method, and the balanced-accuracy gains in very low frequency classes are plausible and consistent across several datasets. But the central 'every row' claim is not backed by any significance testing, and a few rows rest on a single split, so the headline is less robust than the text suggests. Still, the method is worth a serious referee.\n\nWhat's new: EMCO extends a Markov-chain minority model with transitions from the majority class, but only from words that already appear in the minority vocabulary. That is a simple, language-agnostic way to let the synthetic minority vocabulary grow beyond the minority sample's convex hull. It is distinct from SMOTE-style interpolation, EDA's synonym replacement, and DECOM's flat prior. The method is described cleanly, Algorithm 1 is straightforward, and the code and experimental pipeline are released. That is reproducible value.\n\nWhat it does well: the experiments cover three corpora, two classifier families, and a sensible low/very-low frequency split. The gamma sweep in Figure 2 shows the recall/tnr/precision trade-off directly. The Heaps' law comparison is a nice idea, though only illustrative.\n\nSoft spots: the main problem is uncertainty. Tables 3-5 give point estimates only; there are no confidence intervals, standard deviations, or paired tests anywhere. The HuffPost Titles rows without an asterisk come from a single random split. In very low frequency categories the test set contains few positive examples, so macro-averaged balanced accuracy is high-variance. Several decisive margins are a couple of thousandths (e.g., .638 vs .630, or Reuters Full .801 vs .791), so the 'every row except one' claim in Section 6 is a deterministic reading of noisy numbers and may not survive resampling. Also, the BA gain at gamma=1 comes with a large precision drop (Reuters Titles precision 0.212 vs SMOTE 0.508 at 10%), so the practical benefit depends on how much you care about false positives. The paper acknowledges this trade-off, but the strong phrasing in the abstract and Section 6 should be softened. The Heaps' law validation fits parameters on the same training data as an external reference, which is acceptable but not strong. None of these are fatal; they are addressable. The method itself is not circular, and the self-citation is peripheral.\n\nWho this is for: practitioners working with small, imbalanced text sets, especially where neural augmentation or pretrained models are unavailable. The paper does not reshape the field, but it provides a usable, language-independent tool.\n\nRecommendation: send it to peer review. The referees should ask for significance tests or confidence intervals on the headline rows, and a more careful 'wins' claim. This is a desk-reject no, a revise-and-resubmit yes.","headline":"EMCO is a real, reproducible oversampling method, but the paper's 'every row' claim needs significance testing before it can be taken at face value.","tokens_in":23404,"tokens_out":4008,"would_cite":true,"duration_ms":42839,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Oversampling a rare text class should grow its vocabulary, and a Markov chain that borrows word transitions from the majority class tops balanced accuracy on severely imbalanced text data.","keywords":["imbalanced text classification","synthetic oversampling","Markov chain","feature space growth","Heaps' law","balanced accuracy","minority class","bag-of-words"],"falsifier":"Build a binary text corpus in which the minority and majority classes share only function words but use disjoint topical nouns, oversample the minority with EMCO, and count how many introduced majority-only words belong to the majority's topic lexicon. If removing those off-topic words does not change balanced accuracy, the topic-sequence independence premise is not doing the work; if the synthetic minority vocabulary contains words that no genuinely new minority document would contain, the extrapolation premise fails.","tokens_in":22550,"feed_emoji":"📈","tokens_out":7554,"duration_ms":82333,"temperature":0.7,"pith_summary":"The paper's central claim is that synthetic oversampling of a rare text category should not be confined to the words that happen to appear in the minority training documents. Because real text vocabularies grow with sample size (Heaps' law), a synthetic minority sample should also grow; the paper proposes EMCO, a first-order Markov chain over words whose transition probabilities are estimated from the minority class plus, with weight gamma, from majority documents at words that also occur in the minority class. This lets generated minority documents contain words never seen in the minority training set. In experiments with linear SVM on Reuters, HuffPost, and 20 Newsgroups, EMCO with gamma=1 reports the best balanced accuracy in very low frequency categories on every table row except one, where gamma=0.1 is best, and it also improves recall and F2 scores. A practical virtue is that EMCO requires no external resources such as pretrained embeddings or synonym dictionaries, and the code is released.","feed_headline":"Markov oversampling tops balanced accuracy on rare classes","feed_subtitle":"By letting synthetic minority texts borrow word transitions from the majority class, EMCO beats standard oversampling where imbalance is sev","key_machinery":"The central object is the gamma-weighted word-transition matrix with three blocks: minority-to-all transitions (minority counts plus gamma-weighted majority counts), majority-only-to-minority returns (minority marginal distribution), and stop-token start/end rows from minority documents. The mechanism that carries the argument is treating topic and sequence as partly independent: any word common to both classes can act as a bridge, so majority-only words that follow that bridge word in majority documents are treated as plausible next words in synthetic minority documents. The scalar gamma controls how much extrapolation the sampler is allowed and, in the experiments, trades recall against pr","core_discovery":"EMCO models each minority document as a path in a finite Markov chain whose states are the training vocabulary plus a stop token. The transition matrix has a block structure: from minority-vocabulary states, transition weights are the minority transition counts plus gamma times majority counts for the same word pair; from majority-only states, the chain returns to minority words according to the minority marginal word distribution; and starts and ends are taken from minority documents. Self-transitions are zeroed. Sampling a synthetic document means drawing a length, then drawing words row by row from this matrix. Because majority transitions are included only for pairs whose first word is i","pith_inferences":["A testable extension is to replace the first-order chain with an n-gram or learned language model that still restricts extrapolation to majority transitions from shared words; the same bridge idea should transfer.","The gamma choice could be set by comparing synthetic minority vocabulary growth against Heaps' law fitted on the training corpus rather than by grid search; in one reported example gamma=0.1 matched Heaps' law closely.","If topic and sequence are not sufficiently independent—for example, if the majority-only words following a shared word are strongly tied to majority topics—EMCO will inject off-topic vocabulary and its balanced-accuracy advantage should shrink; a held-out study comparing human ratings of synthetic documents across domains would expose the boundary.","The comparison relies on tf-idf bag-of-words and linear SVM; whether the balanced-accuracy ranking transfers to transformer-based classifiers trained from scratch is untested by the paper."],"forward_implications":["On severely imbalanced multiclass text tasks in the tested document-embedding setting, EMCO can be the best available oversampling choice when the evaluation target is balanced accuracy or recall-heavy F2.","The gamma hyperparameter gives a direct, interpretable control over the recall-precision trade-off; users who need precision can lower gamma, while users who need recall or balanced accuracy can raise it to about one.","The method works without pretrained word vectors, synonym dictionaries, or LLM generation, making it usable for low-resource languages and private or small datasets.","Feature-space growth under EMCO tracks Heaps' law better than EDA or DECOM in the reported Reuters example, suggesting the method reproduces a natural vocabulary-growth dynamic rather than arbitrary expansion.","Because the sampling happens before vectorization, EMCO can be combined with any downstream classifier; the paper also reports improved balanced accuracy with a bidirectional LSTM over no oversampling for very low frequency Reuters headline categories."],"supporting_citations":[{"why":"Supplies the Heaps'-law vocabulary-growth relation that motivates allowing the minority feature space to expand during oversampling.","marker":"Heaps, 1978"},{"why":"Introduces SMOTE, the canonical synthetic oversampling baseline and the convex-hull-bound approach EMCO is designed to improve upon.","marker":"Chawla et al., 2002"},{"why":"Introduces ADASYN, a density-based SMOTE variant used as a general-purpose oversampling baseline.","marker":"He et al., 2008"},{"why":"Introduces DECOM topic-model oversampling, the semantic text-specific baseline whose vocabulary expansion comes from a flat prior.","marker":"Chen et al., 2011"},{"why":"Supplies DRO, a distributional latent-space oversampling baseline for text.","marker":"Moreo et al., 2016"},{"why":"Supplies EDA, an augmentation baseline that expands minority vocabulary by synonym replacement.","marker":"Wei and Zou, 2019"},{"why":"Supplies the reference implementations of ROS, SMOTE, and ADASYN used in the experiments.","marker":"Lemaître et al., 2017"}],"fun_headline_variants":["Borrow majority transitions to fix text imbalance","EMCO: synthetic samples grow from both classes","Markov oversampling expands rare-class vocabulary","New text oversampling leverages majority word patterns"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that word-order patterns are similar enough across topics that a majority-only word which follows a shared word in majority documents is a plausible continuation of a minority document; if this fails, EMCO injects off-topic words and degrades precision at high gamma.","fun_headline_variants_meta":{"raw":{"variants":["Borrow majority transitions to fix text imbalance","EMCO: synthetic samples grow from both classes","Markov oversampling expands rare-class vocabulary","New text oversampling leverages majority word patterns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1051,"prompt_tokens":715,"completion_tokens":336,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":280}},"tokens_in":459,"tokens_out":336,"duration_ms":4416,"temperature":1.0,"reasoning_tokens":280,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:35:10.022504+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a binary text corpus in which the minority and majority classes share only function words but use disjoint topical nouns, oversample the minority with EMCO, and count how many introduced majority-only words belong to the majority's topic lexicon. If removing those off-topic words does not change balanced accuracy, the topic-sequence independence premise is not doing the work; if the synthetic minority vocabulary contains words that no genuinely new minority document would contain, the extrapolation premise fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Heaps'-law vocabulary-growth relation that motivates allowing the minority feature space to expand during oversampling."},{"cited_title":"V ., Bowyer, K","cited_arxiv_id":null,"evidence_quote":"Introduces SMOTE, the canonical synthetic oversampling baseline and the convex-hull-bound approach EMCO is designed to improve upon."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces DECOM topic-model oversampling, the semantic text-specific baseline whose vocabulary expansion comes from a flat prior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies DRO, a distributional latent-space oversampling baseline for text."},{"cited_title":"and Zou, K","cited_arxiv_id":null,"evidence_quote":"Supplies EDA, an augmentation baseline that expands minority vocabulary by synonym replacement."}],"review_version":1}