{"id":"bbe55198-64ee-4877-8f73-5c9d94787d0d","arxiv_id":"2411.14896","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Category-aware paraphrasing was the most consistently effective LLM data augmentation prompt in three of four tested settings for Russian green-practice classification.","lead":"The authors tested four ways of using a Russian AI model to create extra training texts for a multi-label classifier that detects mentions of eco-friendly practices in social media. All four approaches improved accuracy, and the most consistently effective was paraphrasing a post while naming the categories it belongs to.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified label inheritance for generated sentences is load-bearing: the Ptext+topics gain could be an artifact of label noise, not prompt quality.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing issue: generated sentences are assumed to inherit their source labels without verification. The reader's overall CONDITIONAL verdict is appropriate because the empirical design is otherwise sound, the data and code are public, and the main uncertainty is the label-quality confound plus the absence of significance testing. My stress-test pass did not find an additional independent flaw that would change the verdict. The label-inheritance concern is concrete: it can be tested by manual annotation of the public generated data, and it directly affects the headline comparison among prompts. I therefore agree with the reader and recommend keeping the verdict CONDITIONAL, with the condition being that the authors verify label correctness of the generated texts or at least report a sensitivity analysis that bounds the effect of label noise.","tokens_in":9784,"tokens_out":1343,"duration_ms":11586,"concrete_test":"Take a sample (e.g., 100–200 generated texts per prompt, stratified by source label) and have annotators judge whether each generated text actually expresses each of its assigned green-practice categories, using the same annotation guidelines as the original GreenRu dataset. Then recompute the Table IV F1-scores after removing or relabeling the generated samples that fail verification, and compare the prompt ordering. If Ptext+topics remains best after relabeling, the central claim survives; if the ordering shifts or the gains vanish, the unverified label-inheritance assumption is the cause.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central comparison assumes that every generated sentence is correctly labeled with the source sentence's (or topic list's) multi-label annotation. Section III-C and Table IV evaluate classifiers trained on augmented data, but no manual or automatic verification of label correctness is reported. Table V reports only lexical/semantic similarity (ROUGE-1, ROUGE-L, BERTScore), which does not measure whether the generated text actually mentions the assigned green-practice categories. For Gtopics, the prompt is 'Write a short post ... related to the following topics', so the generation is conditioned on the topic list; for Ptext+topics, the prompt conditions the paraphrase on the source categories. If the model only partially follows the topic instruction, the effective training labels contain noise whose level and structure differ across prompts. The observed F1 ordering (Ptext+topics best in 3/4 settings, Gtopics best for ruELECTRA at x1.5) could then partly reflect different rates of label noise rather than the intrinsic value of each prompting strategy. This is especially consequential for rare classes: Table II shows 'repairing' has only 10 training mentions and 'signing petitions' 22, so a handful of mislabeled generated examples could dominate the learning signal for those classes. The abstract's stronger claim that 'all strategies improved classification performance' also rests on this assumption, because random duplication and back translation preserve source labels exactly, whereas LLM-generated texts may not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares four LLM prompt strategies for augmenting a Russian multi-label social media dataset (GreenRu) with mentions of green practices. The prompts are: paraphrasing the source text (Ptext), generating a new post from a topic list (Gtopics), paraphrasing the source while explicitly conditioning on its categories (Ptext+topics), and generating from a topic list plus a source example (Gtopics+text). Training sets are augmented to 1.5x and 2x their original size using T-lite-instruct-0.1, and ruELECTRA-large and ruBERT are fine-tuned as multi-label classifiers. Evaluation is macro-averaged multi-label F1 on a held-out test set, with three repeated runs per condition. The paper reports that every augmentation strategy improves over training on the original data, that Ptext+topics is the best prompt in three of the four model/scale combinations, and that category-aware paraphrasing is therefore the most effective strategy.","tokens_in":10123,"tokens_out":5164,"duration_ms":50741,"significance":"If the result holds, it offers a practical, low-cost recommendation for prompt-based data augmentation in multi-label, class-imbalanced Russian text classification: explicitly include the target categories when paraphrasing. The study is one of the first to examine this question for Russian-language data, uses an open Russian-specific instruct model, compares against standard baselines (random duplication and back translation), reports three repeated runs, and makes the generated texts publicly available. The main claims, however, currently rest on point estimates without any significance testing and on an unverified assumption that generated sentences inherit their assigned labels. Both issues bear directly on the central recommendation and need to be addressed before the conclusions are fully supported.","major_comments":[{"comment":"The central comparison assumes that every generated sentence is added to the training set with the labels of its source sentence (Ptext, Ptext+topics, Gtopics+text) or with the topic list (Gtopics). Section III-C describes adding generated sentences but reports no human or automatic verification that the generated text actually expresses the assigned green-practice categories. Table V reports only ROUGE-1, ROUGE-L, and BERTScore, which measure lexical and semantic similarity to the source, not whether the target labels are present. This matters because the prompts differ in how strongly they condition on categories: Gtopics and Ptext+topics explicitly instruct the model to write about given topics, while Ptext does not. If the LLM partially ignores the topic instruction, the four strategies introduce different levels and structures of label noise, and the observed F1 ordering could partly be an artifact of that noise rather than of prompt quality. The risk is especially acute for rare classes: Table II shows only 10 training mentions of 'repairing' and 22 of 'signing petitions', so a small number of mislabeled generated examples could dominate the learning signal for those classes. I ask the authors to assess label correctness on a sample of generated texts (e.g., human annotation or rule-based/classifier-based checks), report per-prompt label fidelity, and either filter generated examples before training or re-run the comparison with this verification in place.","section":"III-C and Table IV"},{"comment":"The reported differences are not supported by significance testing. For example, at the 1.5x scale for ruELECTRA, Ptext achieves 74.09±0.86, Ptext+topics 73.47±2.07, and Gtopics 74.17±0.47; the gap between the best and second-best conditions is 0.70 points with heavily overlapping standard deviations based on only three runs. The abstract's statement that 'all strategies improved classification performance compared to the models fine-tuned only on the original dataset' is likewise a point estimate statement; with the given variances, several of the improvements over the original data may not be statistically distinguishable from zero. Because the paper's main conclusion is a rank ordering of prompts, please report pairwise significance tests (e.g., paired bootstrap or a nonparametric test across seeds) or honestly state that the observed ordering is not statistically significant at the available sample size. If no test is reported, the wording in the abstract and Section IV should be softened accordingly.","section":"Table IV and Section IV"},{"comment":"The standard deviations in Table IV appear to reflect only variability in classifier fine-tuning, not variability in the LLM-generated augmentation data. Generation was performed once with temperature 0.5 and maximum 400 tokens, so the reported intervals are conditional on a single generated corpus per prompt. Since the paper compares prompts by the data they generate, the effective uncertainty is understated. Please either generate multiple augmentation sets per prompt and report variance across those sets, or state explicitly that the results and intervals are conditional on one generation pass.","section":"III-B and Table IV"}],"minor_comments":[{"comment":"The sentence 'the scores for Ptext are significantly higher than for Ptext+topics' uses 'significantly' without a statistical test; please replace it with 'substantially' or provide confidence intervals for the similarity metrics.","section":"Section IV, after Table V"},{"comment":"The captions for Figures 4 and 5 are missing, and the figures are referenced only indirectly in the text; please add captions identifying which bars correspond to which augmentation condition and model.","section":"Figures 4 and 5"},{"comment":"Please state whether the reported F1 scores come from the final fine-tuning epoch or from the best epoch on a validation split, and specify whether any early stopping was used; this affects the comparability of the three runs.","section":"Section III-E"},{"comment":"The paper defines the multi-label F1-score as the average of per-class scores but does not explicitly call it macro-averaged; please state this explicitly in the metric definition.","section":"Section III-F"}],"recommendation":"major_revision","confidential_remarks":"This is a straightforward empirical comparison with modest novelty but a useful applied setting (Russian, multi-label, class-imbalanced, open-source instruct model). The label-inheritance concern raised in my report is not a circularity issue; it is an omitted verification step that the authors can address within the scope of the manuscript. I would not reject the paper, but the central recommendation needs either stronger statistical support or more cautious wording, and the label-fidelity analysis should be added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a careful, reproducible empirical study of four prompt types for LLM data augmentation on a Russian multi-label ecological text dataset (GreenRu). It fine-tunes ruELECTRA and ruBERT on augmented data, compares against random duplication and back translation, and reports three-run averages with standard deviations. The generated data is public. If you work on data augmentation for imbalanced multi-label text, it's worth a look.\n\nWhat's actually new: it applies prompt-based augmentation to a Russian multi-label task with a Russian-specific open-source model (T-lite). The four prompts are not novel in themselves — combinations of paraphrase/generate with or without category conditioning, all seen in earlier work — but the comparison on this dataset and language is new, and the conclusion is concrete: paraphrasing while explicitly stating the source categories (Ptext+topics) wins three of four settings, with Gtopics winning the fourth.\n\nThe methodology is solid in design: held-out test set, generation only from training sentences, baselines included, hyperparameters reported. The paper is transparent about its own limitations, e.g., random sentence selection. That's good practice.\n\nThe soft spots are real but not disqualifying. First, no significance testing. The headline difference for ruELECTRA x1.5 is Gtopics 74.17±0.47 vs Ptext+topics 73.47±2.07 — overlapping. Several pairwise differences are within one standard deviation. The abstract's \"best results\" wording overstates what the table shows. Second, the paper assumes generated sentences inherit their source labels or topic lists without checking. Table V only reports ROUGE/BERTScore similarity, not label correctness. If the LLM only partially follows the topic instruction, the effective label noise differs across prompts. That could partly explain the ranking. For rare classes like 'repairing' (10 training mentions), a few mislabeled generated examples would matter. This is worth naming, but it cuts both ways: if Ptext+topics produces more on-topic texts, its success is real for practitioners even if the mechanism is label accuracy rather than something subtler.\n\nBottom line: the paper is a useful empirical datapoint, not a definitive proof. It deserves a serious referee — the design is clean enough to publish after addressing significance testing and label verification. I'd send it to review, with a request for significance checks or at least a clear statement about effect sizes, and some manual or automatic label inspection on generated samples.\n\nRecommendation: engage with it. If you're doing LLM augmentation for low-resource or imbalanced classification, the public data and clear setup make it a citable baseline.","headline":"Careful, reproducible comparison of LLM augmentation prompts on Russian multi-label data; the ranking is plausible but statistical support is thin.","tokens_in":10567,"tokens_out":3456,"would_cite":true,"duration_ms":28625,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that prompting an instruction-tuned Russian-language LLM to paraphrase a text while explicitly naming its topic categories is the most consistently effective data-augmentation strategy for multi-label classification of…","keywords":["data augmentation","large language models","prompt-based data augmentation","multi-label classification","ecological texts","Russian social media","green practices","class imbalance"],"falsifier":"Take the augmented training sets used in the reported F1 experiments, sample about 100 generated sentences from each of the four prompts, and have annotators judge which of the nine green-practice labels are actually present. If Ptext+topics sentences do not contain their assigned labels more often than sentences from the other prompts, or if removing label-incorrect generations eliminates the F1 advantage, then the comparison is measuring label noise rather than prompt quality.","tokens_in":9522,"feed_emoji":"🌱","tokens_out":6848,"duration_ms":63667,"temperature":0.7,"pith_summary":"Prompting an instruction-tuned language model to rewrite a sentence while explicitly stating which categories it belongs to is the most consistently effective data-augmentation recipe tested here. The paper compares four prompt strategies—plain paraphrase, category-only generation, category-aware paraphrase, and category-plus-example generation—on an imbalanced Russian dataset of social-media mentions of green practices, using two fine-tuned transformer classifiers and two training-set growth factors. All augmentation strategies raised multi-label F1 relative to fine-tuning on the original data alone, and most beat random duplication and back translation. In three of the four model-by-growth configurations, the category-aware paraphrase prompt produced the highest F1, which is what the paper is trying to establish.","feed_headline":"Category-aware paraphrases win 3 of 4 LLM augmentation tests","feed_subtitle":"Adding LLM-rewritten green-practice sentences to training data lifted multi-label F1 for both classifiers.","key_machinery":"The load-bearing mechanism is the category-conditioned paraphrase prompt, Ptext+topics, which instructs the model to paraphrase the original text while taking into account its relevance to a given list of categories. It combines the source sentence with explicit label information, so the LLM rewrites the text while keeping topical content and preserving more lexical and semantic similarity than generation-only prompts, but less than a plain paraphrase. The paper's similarity metrics show this intermediate position: Ptext+topics has ROUGE-1 of 18.94 percent and BERTScore of 70.40 percent, compared with 32.58 percent and 76.88 percent for plain paraphrase and 4.03 percent and 56.40 percent for category-only generation. This suggests the category conditioning changes the generated distribution in a way that helps the downstream multi-label classifier.","core_discovery":"The paper's central claim is that category-conditioned paraphrasing is the most useful of the four tested augmentation prompts for this task. In three out of four configurations—both classifiers and both augmentation sizes (1.5x and 2x)—the Ptext+topics prompt achieved the highest multi-label F1; the remaining best score went to the generation-only prompt. The best absolute F1 for ruELECTRA was 74.17 percent with the generation-only prompt at 1.5x growth, and for ruBERT 71.78 percent with Ptext+topics at 2x growth. All augmentation approaches improved over the original-data F1 of 69.96 percent (ruELECTRA) and 58.16 percent (ruBERT), and LLM augmentation usually outperformed random duplication and back translation.","pith_inferences":["A natural extension the paper leaves implicit is to combine Ptext+topics with rare-class oversampling: the study uses random sentence selection, so rare green-practice categories remain rare, and the conclusion itself flags sampling strategy as future work.","The reported gains could partly reflect label inheritance rather than prompt quality, since generated sentences are added with their source labels without verification; an annotation study of label fidelity in generated texts would separate these explanations.","The same category-conditioned paraphrase design may transfer to other languages and domains whenever a strong instruction-tuned model is available, but that transfer is not established by these Russian-only experiments."],"forward_implications":["Prompt-based augmentation with this Russian instruction-tuned LLM is beneficial for imbalanced multi-label text: all four prompts improved F1 over the original-data baseline, and most beat random duplication and back translation.","A category-aware paraphrase prompt is a strong default for such tasks, since it delivered the best result in three of four model-by-growth configurations and never reduced performance.","Generation-only prompts can be competitive even though their outputs are least similar to the original texts, so lexical diversity alone is not a drawback for augmentation.","Paraphrase-based prompts retain higher similarity to the source, but the best F1 came from the prompt that also names categories, indicating that category conditioning, not similarity per se, is what helps.","The approach offers a practical template for other class-imbalanced multi-label classification settings where training data are scarce."],"supporting_citations":[{"why":"Supplies the Russian social-media dataset of green-practice mentions with the nine multi-label categories used in all experiments.","marker":"[22]"},{"why":"Supplies ruELECTRA, one of the two transformer classifiers whose multi-label F1 carries the comparison.","marker":"[24]"},{"why":"Supplies ruBERT, the second classifier used to check that prompt results hold across architectures.","marker":"[26]"},{"why":"Introduces the ChatGPT-based paraphrasing augmentation approach that the Ptext strategy adapts.","marker":"[13]"},{"why":"Provides the earlier comparison of paraphrasing versus zero-shot generation with ChatGPT that frames this paper's four-prompt design.","marker":"[14]"},{"why":"Proposes combining paraphrasing and generation by rewriting the generated sample, which motivates the Gtopics+text strategy.","marker":"[15]"},{"why":"Defines the back-translation augmentation baseline that the LLM prompts are required to beat.","marker":"[10]"}],"fun_headline_variants":["Category-aware paraphrases win 3 of 4 augmentation tests","LLM prompts that know categories top data augmentation","Category-aware paraphrasing beats four other augmentation prompts","LLM-rewritten eco-texts boost multi-label F1 with right prompt","Paraphrase with category cues wins LLM augmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that every generated sentence carries the labels of its source sentence, or for the generation-only prompt the labels in the topic list, without checking whether the generated text actually expresses those categories.","fun_headline_variants_meta":{"raw":{"variants":["Category-aware paraphrases win 3 of 4 augmentation tests","LLM prompts that know categories top data augmentation","Category-aware paraphrasing beats four other augmentation prompts","LLM-rewritten eco-texts boost multi-label F1 with right prompt","Paraphrase with category cues wins LLM augmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2822,"prompt_tokens":881,"completion_tokens":1941,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":1858}},"tokens_in":497,"tokens_out":1941,"duration_ms":15421,"temperature":1.0,"reasoning_tokens":1858,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:44:45.639914+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the augmented training sets used in the reported F1 experiments, sample about 100 generated sentences from each of the four prompts, and have annotators judge which of the nine green-practice labels are actually present. If Ptext+topics sentences do not contain their assigned labels more often than sentences from the other prompts, or if removing label-incorrect generations eliminates the F1 advantage, then the comparison is measuring label noise rather than prompt quality.","supporting_citations":[{"cited_title":"GreenRu: A Russian Dataset for Detecting Mentions of Green Practices in Social Media Posts,","cited_arxiv_id":null,"evidence_quote":"Supplies the Russian social-media dataset of green-practice mentions with the nine multi-label categories used in all experiments."},{"cited_title":"A family of pretrained transformer language models for Russian,","cited_arxiv_id":null,"evidence_quote":"Supplies ruELECTRA, one of the two transformer classifiers whose multi-label F1 carries the comparison."},{"cited_title":"Adaptation of deep bidirectional multilin- gual transformers for Russian language,","cited_arxiv_id":null,"evidence_quote":"Supplies ruBERT, the second classifier used to check that prompt results hold across architectures."},{"cited_title":"Is ChatGPT the ultimate Data Augment- ation Algorithm?","cited_arxiv_id":null,"evidence_quote":"Provides the earlier comparison of paraphrasing versus zero-shot generation with ChatGPT that frames this paper's four-prompt design."},{"cited_title":"Improving Text Classification with Large Language Model-Based Data Augmentation,","cited_arxiv_id":null,"evidence_quote":"Proposes combining paraphrasing and generation by rewriting the generated sample, which motivates the Gtopics+text strategy."}],"review_version":1}