{"id":"c401669d-a235-442c-a97b-08af36f39c88","arxiv_id":"2501.00334","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-level curriculum, batch ordering by loss and instance/token reweighting by Monte Carlo dropout confidence, yields about 0.5 to 1.2 F0.5 gains for BART, mT5, and SynGEC on NLPCC and MuCGEC.","lead":"Chinese grammar correction models are trained by sorting sentences from easy to hard and reweighting their losses, using the model's own predictions as a difficulty signal. The paper reports small F0.5 gains on two Chinese GEC benchmarks, but the key loss-weighting equation, error bars, and code are missing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Instance-Level CL mechanism is never defined; the +0.64 F0.5 ablation and all main gains rest on an unspecified loss-weighting formula, making the central claim unverifiable.","rationale":"The reader's weakest assumption—that training loss monotonically measures intrinsic correction difficulty—is a genuine concern, but the more immediate and load-bearing problem is that the paper's Instance-Level CL component is not actually specified. The central contribution is a multi-granularity framework, and Table V explicitly credits +0.64 F0.5 to Instance-Level CL. Yet Section II-C stops at describing Monte Carlo dropout and confidence/variance estimation; it never writes down the loss-weighting equation, nor does it state how the Q sampled probabilities are aggregated into sentence-level and token-level weights. This is not a stylistic omission: the reader cannot tell whether the reported gains come from reweighting, from some other training detail, or from noise. Without that formula, the main result is not falsifiable and the paper cannot be accepted as a reproducible scientific claim. The reader's rationale already notes that the central loss-weighting equation is missing, but their weakest_assumption field focuses on the difficulty criterion; I agree partially because both issues undermine the central claim, yet the missing equation is the more fundamental blocker. A concretely testable path is to use the authors' implementation and compare against a shuffled-weight control; that would settle whether the stated mechanism is what produces the improvement. Until then, the paper's reported F0.5 gains remain unverifiable, so the current REJECT verdict is appropriate.","tokens_in":8214,"tokens_out":2782,"duration_ms":30722,"concrete_test":"Obtain the authors' code or the exact loss-weight formula from the authors. Re-run the BART 'Only Instance-Level CL' ablation with that formula, and also run a control in which the per-instance weights are randomly permuted within each batch while keeping the same training schedule and seed. Compare F0.5 across the five seeds used in the paper; if the proposed Instance-Level CL gain over the shuffled-weight control is not statistically significant, the claimed +0.64 improvement is not attributable to the stated mechanism.","verdict_should_be":"REJECT","load_bearing_attack":"Section II-C, which is supposed to describe Instance-Level Curriculum Learning, never actually defines the loss-weighting mechanism. The text says Monte Carlo dropout sampling yields Q conditional probabilities per instance and that their variance 'reflects the confidence' of the model, but it gives no equation, normalization, temperature, or threshold that maps those probabilities (or variances) to modified training losses. The phrase 'we utilize the probabilities to assess the complexity' is not a method. Since the headline improvements in Tables III, IV, and the +0.64 F0.5 ablation in Table V all depend on this component, the central claim cannot be independently checked or reproduced from the paper as written. In addition, the difficulty-scoring model in Section II-B1 is not specified ('a trained CGEC model'—which checkpoint, which architecture, trained on what), so it is unclear whether the batch-level ordering is even stable across the three base models or whether each model receives a different curriculum. The reported gains are small (0.5–1.2 F0.5) and the parameter k is tuned on the evaluation sets, which amplifies the risk that the missing formula hides a selection or implementation artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-granularity curriculum learning framework for Chinese grammatical error correction (CGEC). Batch-level curriculum learning scores training instances by the cross-entropy loss of a trained CGEC model and presents them in an easy-to-hard order controlled by a competence function. Instance-level curriculum learning is described as using Monte Carlo dropout sampling to obtain conditional probabilities and then adjusting loss weights based on the variance of those probabilities. The method is applied to BART, mT5, and SynGEC, and results are reported on NLPCC-test and MuCGEC-dev, with ablations, a parameter study, and a difficulty-interval analysis. The central claim is that this training-only, architecture-agnostic procedure yields consistent F0.5 improvements of roughly 0.5 to 1.2 points over baselines.","tokens_in":8461,"tokens_out":4024,"duration_ms":41686,"significance":"If the improvements were established, the proposed curriculum would be a useful, model-agnostic training strategy for CGEC and would connect curriculum learning more tightly to error-correction difficulty. The paper's intended contribution is plausible and the experimental design covers multiple base architectures, which is a strength. However, as written, the instance-level component is not specified as a reproducible mechanism, the evaluation setup includes selection of the hyperparameter k on the test sets, and no statistical significance evidence accompanies the reported seed averages. These issues currently prevent the claimed results from being verified, so the significance of the contribution is not yet demonstrated.","major_comments":[{"comment":"The Instance-Level Curriculum Learning mechanism is not defined. The text states that Monte Carlo dropout sampling yields Q conditional probabilities per instance and that the variance of these probabilities reflects model confidence, but it never gives the equation that maps these probabilities or variances to modified training losses. The abstract and conclusion say the loss function is regulated, but no formula, normalization, temperature, or threshold is provided. Because the main results in Tables III and IV and the +0.64 F0.5 ablation in Table V all depend on this component, the central claim cannot be independently checked or reproduced from the paper as written.","section":"II-C"},{"comment":"The hyperparameter k is selected by scanning F0.5 directly on the NLPCC-test and MuCGEC-dev datasets. The reported CL(BART) scores are therefore the best k values on the evaluation sets, which introduces selection bias into the claimed improvements. The authors should either use a held-out validation split for selecting k and report test results only for the selected value, or report results across all k values on a separate test set. Without this, the magnitudes of the reported gains are not reliable evidence for the method's effectiveness.","section":"III-F, Table VI"},{"comment":"The difficulty-scoring model is underspecified. The paper says 'Given a trained CGEC model and a dataset... we can acquire the difficulty of each instance by calculating the cross-entropy loss function,' but it does not state which architecture, which checkpoint, or which training data produced this scorer. This ambiguity matters because if each of the three base models is used to score its own training data, the batch-level curriculum differs per model and the comparison is confounded; the authors should fix a scoring model and report its provenance, or alternatively demonstrate that the loss-based difficulty ordering is stable across initializations and architectures.","section":"II-B1"},{"comment":"The claim that the method 'achieves consistent improvements with a significant margin over all baselines' is not supported by statistical evidence. The paper states that results are averages of five distinct experiments with different random seeds, but no standard deviations, confidence intervals, or significance tests are reported. Given that the observed differences are on the order of 0.5 to 1.2 F0.5, run-to-run variance in fine-tuning could substantially affect the conclusions, so the significance claim should be substantiated or softened.","section":"III-D, Table III and Table IV"},{"comment":"The most closely related baselines are absent from the experimental comparison. The introduction cites [10] (mixed-grained weighted training) and [11] (data weighted training strategies) as relevant prior work on addressing instance difficulty in GEC, but neither is included in Tables III or IV. Since these methods directly weight training data based on difficulty, they are the natural competitors for the proposed approach, and their omission undermines the 'all baselines' claim in Section III-D.","section":"I, III-D"}],"minor_comments":[{"comment":"The abbreviation 'PLMS' appears in the abstract; it should be 'PLMs' (pre-trained language models). This typo recurs in the introduction and should be corrected.","section":"Abstract and throughout"},{"comment":"Monte Carlo dropout sampling is introduced without a citation. Please provide a reference (e.g., Gal and Ghahramani, 2016) and specify how dropout is applied during inference and which layers are affected.","section":"II-C"},{"comment":"The notation for the k-th root is unclear; 'k√' should be typeset as \\(\\sqrt[k]{...}\\) or written in an equivalent unambiguous form.","section":"II-B2, Eq. (3)"},{"comment":"The table caption and text contain the phrase 'valuation of k'; this should be 'value of k'.","section":"III-F, Table VI"},{"comment":"The figure is not referenced in the text at the point where the difficulty-interval analysis is introduced. Additionally, the y-axis label 'Improvements' should be quantified as ΔF0.5 with units specified.","section":"III-G, Figure 2"},{"comment":"The learning rate is printed as '5e−5' with an ambiguous minus sign; please use standard scientific notation, e.g., \\(5\\times 10^{-5}\\).","section":"III-C"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from a clear statement of which hyperparameters were tuned on which splits; the current Table VI shows direct test-set selection, which is a serious methodological concern. The related-work section cites several papers from the same research group, and the novelty relative to existing data-weighting GEC methods (e.g., Lichtarge et al., 2020 and Li et al., 2023) should be clarified. If the authors can specify the instance-level loss-weighting formula and re-run the experiments with proper validation and significance testing, the approach may be salvageable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [name],\n\nQuick take on arXiv:2501.00334. The headline is that the paper's main claim—that a curriculum learning wrapper improves CGEC models by ~0.5–1.2 F0.5—is plausible but unverifiable as written, because the instance-level loss-weighting mechanism is never actually defined. That's the real problem, not the size of the gains.\n\nWhat's new: the specific combination of CDF-normalized loss difficulty, Platanios-style competence scheduling, and MC-dropout confidence weighting for Chinese GEC is not in the cited literature. The experiments cover three different base models (BART, mT5, SynGEC) on two datasets, and the improvements are consistent across all of them. That consistency is a point in its favor. The paper also clearly cites the closest existing work on weighted training for GEC (refs [10] and [11]).\n\nWhat's soft: Section II-C is supposed to describe how MC-dropout probabilities become loss weights, but it gives no equation, normalization, temperature, or threshold. \"We utilize the probabilities to assess the complexity\" is not a method. Since the ablation attributes +0.64 F0.5 to this component, the central result cannot be reproduced. The difficulty-scoring model in II-B is also unspecified—which checkpoint, which architecture, trained on what—so it's unclear whether the batch-level ordering is stable across the three base models. The hyperparameter k is tuned directly on the evaluation sets (Table VI), and the paper reports five-seed averages without standard deviations or significance tests. The phrase \"significant margin\" in Section III-D is therefore unsupported. The absence of comparisons with refs [10] and [11] further weakens the claim of novelty, since one of those (mixed-grained weighted training) is very close in spirit.\n\nNone of this means the idea is dead on arrival. The core intuition—easy-to-hard ordering plus loss reweighting—is reasonable, and the batch-level part is clearly described. But the manuscript as it stands does not allow an independent check of its main result.\n\nWho this is for: people working on GEC or curriculum learning for sequence generation. It's worth engaging with, but only after the authors fill in the missing formula, specify the difficulty-scoring model, report variance, and run honest parameter selection (e.g., tuning k on a held-out development set). I'd send it to peer review with a request for major revision rather than desk-reject it, because the idea is new enough and the gains are consistent enough to deserve a proper vetting.","headline":"Plausible curriculum learning idea for CGEC, but the instance-level mechanism is undefined, making the reported gains unverifiable as written.","tokens_in":8982,"tokens_out":2461,"would_cite":false,"duration_ms":22848,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that training Chinese grammatical error correction models on example pairs sorted from easy to hard by model loss, and then reweighting individual examples by a dropout-based confidence estimate, consistently improves…","keywords":["Chinese grammatical error correction","curriculum learning","loss-aware difficulty","batch-level curriculum","instance-level reweighting","Monte Carlo dropout","pre-trained language models","grammatical error correction"],"falsifier":"Sort a held-out set by a trained model's per-sentence loss and check whether correction accuracy improves monotonically as loss decreases; then retrain the whole model with the curriculum reversed, hard to easy. If the loss–difficulty correlation is weak or the reversed curriculum produces the same F0.5 gains, the easy-to-hard ordering is not the mechanism driving the reported improvement.","tokens_in":8019,"feed_emoji":"📝","tokens_out":8340,"duration_ms":75247,"temperature":0.7,"pith_summary":"Chinese grammatical error correction systems usually train on every sentence equally, even though some sentences are much harder to fix than others. This paper proposes a two-level curriculum: first, sort training pairs by a model-loss difficulty score and feed them in batches from easy to hard; then, inside each batch, reweight instances according to a dropout-based confidence estimate. The authors test the recipe on BART, mT5, and SynGEC and report consistent F0.5 gains on NLPCC-test and MuCGEC-dev. A sympathetic reading is that difficulty-aware training alone, with no change to model architecture or data, can push CGEC performance upward.","feed_headline":"Easy-to-hard training lifts Chinese grammar correction by 1.17 points","feed_subtitle":"The same training-only recipe lifts BART, mT5, and SynGEC scores on NLPCC-test and MuCGEC-dev.","key_machinery":"The paper's central object is a two-level curriculum driven by a loss-based difficulty score. At batch level, each sentence pair $(x^i,y^i)$ receives a difficulty score $d(x^i,y^i)$ equal to its cross-entropy loss under a trained CGEC model, normalized to $(0,1]$ by a cumulative density function; a competence schedule $C(t)=\\min(1,\\sqrt[k]{t(1-c_0^k)/T + c_0^k})$ determines which fraction of the difficulty-sorted data is exposed at training step $t$. At instance level, $Q=5$ Monte Carlo dropout forward passes yield $Q$ conditional probabilities per sentence and token; the variance of these probabilities acts as a confidence measure that reweights the loss so hard instances contribute more. The negative log-likelihood loss in Equation (1) is the quantity both levels operate on.","core_discovery":"On the paper's own terms, the discovery is that a loss-aware, two-granularity curriculum—batch-level ordering plus instance-level reweighting—improves CGEC models independent of the underlying architecture. Reported F0.5 gains on NLPCC-test are 1.17 points for BART, 1.11 for mT5, and 0.73 for SynGEC; on MuCGEC-dev they are 0.97, 0.55, and 0.87 respectively. The authors attribute the gains to gradually shifting model attention toward hard examples, and the ablation study shows both curriculum levels contribute.","pith_inferences":["The paper does not test whether the loss-based difficulty ranking is stable across random seeds; if rankings fluctuate, a cheaper variant would be to compute difficulty once from a small proxy model or to smooth scores over an ensemble of checkpoints.","The same two-level recipe should transfer to English grammatical error correction or spell-checking, since neither the CDF-normalized difficulty score nor the dropout-confidence reweighting is Chinese-specific.","A direct extension would be to replace the single-score CDF ordering with token-level or error-type-aware curricula, since Figure 2 shows hard sentences benefit most; error-type annotation could make the easy-to-hard path linguistically interpretable."],"forward_implications":["On NLPCC-test, adding the curriculum to BART raises F0.5 from 44.64 to 45.81, to mT5 from 42.81 to 43.92, and to SynGEC from 45.32 to 46.05.","On MuCGEC-dev, the same recipe raises mT5 from 36.87 to 37.42, BART from 37.62 to 38.59, and SynGEC from 38.31 to 39.18.","Ablations on BART show the combined gain (+1.17) is larger than batch-level only (+0.38) or instance-level only (+0.64), so the two granularities are complementary.","The competence-growth parameter $k$ has a sweet spot: F0.5 peaks at $k=2$ on both NLPCC-test and MuCGEC-dev, and larger $k$ degrades the gain, consistent with overfitting hard examples.","Figure 2 indicates gains appear across all difficulty intervals, not only the hardest sentences."],"supporting_citations":[{"why":"Introduces curriculum learning, the paradigm the paper adapts to CGEC.","marker":"[12]"},{"why":"Supplies the competence-based scheduling function used to feed batches from easy to hard.","marker":"[21]"},{"why":"Defines the NLPCC-test benchmark used for the main evaluation.","marker":"[17]"},{"why":"Provides the MuCGEC-dev benchmark and the SynGEC baseline model.","marker":"[18]"},{"why":"Supplies the SynGEC architecture and the training hyperparameters adopted in all experiments.","marker":"[23]"},{"why":"mT5, one of the three pre-trained models the curriculum wrapper is applied to.","marker":"[32]"},{"why":"CPT/BART, another of the three pre-trained models the curriculum wrapper is applied to.","marker":"[33]"},{"why":"Prior work on mixed-grained weighted training for GEC that motivates reweighting by difficulty.","marker":"[10]"},{"why":"Prior data-weighted training for GEC, the line of work this method extends.","marker":"[11]"}],"fun_headline_variants":["Loss-aware curriculum lifts CGEC scores across multiple models","Easy-to-hard training improves Chinese grammar correction","Multi-granularity curriculum boosts CGEC by 1.17 points","Training order matters: Easy-to-hard curriculum for CGEC","Loss-aware curriculum gives 1.17-point boost in Chinese GEC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The curriculum ordering stands or falls on the assumption that the model's per-sentence training loss is a reliable, stable measure of how hard a sentence is to correct, so that low-loss sentences are genuinely easy and high-loss sentences genuinely hard.","fun_headline_variants_meta":{"raw":{"variants":["Loss-aware curriculum lifts CGEC scores across multiple models","Easy-to-hard training improves Chinese grammar correction","Multi-granularity curriculum boosts CGEC by 1.17 points","Training order matters: Easy-to-hard curriculum for CGEC","Loss-aware curriculum gives 1.17-point boost in Chinese GEC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000938,"raw_usage":{"total_tokens":3933,"prompt_tokens":793,"completion_tokens":3140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":409,"completion_tokens_details":{"reasoning_tokens":3066}},"tokens_in":409,"tokens_out":3140,"duration_ms":20646,"temperature":1.0,"reasoning_tokens":3066,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:53:22.291404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sort a held-out set by a trained model's per-sentence loss and check whether correction accuracy improves monotonically as loss decreases; then retrain the whole model with the curriculum reversed, hard to easy. If the loss–difficulty correlation is weak or the reversed curriculum produces the same F0.5 gains, the easy-to-hard ordering is not the mechanism driving the reported improvement.","supporting_citations":[{"cited_title":"Competence-based curriculum learning for neural machine translation,","cited_arxiv_id":null,"evidence_quote":"Supplies the competence-based scheduling function used to feed batches from easy to hard."},{"cited_title":"Overview of the nlpcc 2018 shared task: Grammatical error correction,","cited_arxiv_id":null,"evidence_quote":"Defines the NLPCC-test benchmark used for the main evaluation."},{"cited_title":"Mucgec: a multi-reference multi-source evaluation dataset for chinese grammatical error correction,","cited_arxiv_id":null,"evidence_quote":"Provides the MuCGEC-dev benchmark and the SynGEC baseline model."},{"cited_title":"Syngec: Syntax-enhanced grammatical error correction with a tailored gec- oriented parser,","cited_arxiv_id":null,"evidence_quote":"Supplies the SynGEC architecture and the training hyperparameters adopted in all experiments."},{"cited_title":"mT5: A massively multilingual pre-trained text-to-text transformer,","cited_arxiv_id":null,"evidence_quote":"mT5, one of the three pre-trained models the curriculum wrapper is applied to."},{"cited_title":"Grammatical error correction via mixed-grained weighted training,","cited_arxiv_id":null,"evidence_quote":"Prior work on mixed-grained weighted training for GEC that motivates reweighting by difficulty."},{"cited_title":"Data weighted training strategies for grammatical error correction,","cited_arxiv_id":null,"evidence_quote":"Prior data-weighted training for GEC, the line of work this method extends."}],"review_version":1}