{"id":"16c66cb3-7b67-4607-bebd-2b47ce1fefbd","arxiv_id":"2502.09648","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding 294 essay-level lexical features to a KoBERT+BiGRU model raises Korean essay scoring accuracy from 0.649 to 0.657 and quadratic weighted kappa from 0.509 to 0.538 on the AI-HUB Essay Evaluation Dataset.","lead":"UKTA is a Korean writing evaluation system that combines morpheme analysis, 294 lexical features, and an attention-based deep learning model to score essays on 10 rubric criteria. The paper reports that adding these features improves accuracy and agreement over a raw-text baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Performance gain over baseline is small (0.008 accuracy, 0.029 QWK) and reported without variance or significance tests; it may be within run-to-run noise.","rationale":"The reader's identified weakest assumption concerns attention-weight faithfulness for explainability, which is a legitimate secondary issue. However, the paper's central empirical claim is the performance improvement in Table 1, and that claim rests entirely on small average differences without any variance or significance reporting. If those differences are within noise, the feature contributions and explainability discussion lose their foundation. The reader's rationale does mention missing error bars and significance tests, so there is partial overlap, but the reader did not elevate this to the primary load-bearing concern. The appropriate verdict remains CONDITIONAL: the paper should be accepted only if the authors supply the missing statistical evidence (or the results survive a proper significance test). No stronger verdict is warranted because the gap is addressable and the observed differences might be real; no weaker verdict is appropriate because the central claim is currently unverified.","tokens_in":12146,"tokens_out":3117,"duration_ms":27340,"concrete_test":"Obtain per-run predictions from the five runs for baseline and UKTA. Compute per-essay accuracy for each run and perform a paired bootstrap or McNemar's test on the 6,000 test essays between baseline and UKTA; for QWK, use a bootstrap over essays to obtain a 95% confidence interval for the difference. Report mean ± standard deviation over the five runs for each rubric. If the 95% CI for accuracy or QWK includes zero, the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 reports averages over five runs, but no standard deviations, per-run values, or significance tests are given. The headline result is a mean accuracy increase from 0.649 to 0.657 (Δ=0.008) and QWK from 0.509 to 0.538 (Δ=0.029) on the AI-HUB test set (≈6,000 essays per Section 4.1). For an accuracy near 0.65 with n≈6,000, the standard error is roughly sqrt(0.65*0.35/6000) ≈ 0.0062, so Δ=0.008 is only about 1.3 standard errors. With only five independent runs, the standard error of the mean difference could be larger. No confidence interval, paired test, or effect size is reported, so the observed advantage is not distinguishable from chance. The paper's own Section 4.2 calls the improvement 'significant' but provides no statistical inference. If the true difference is zero or negative, the central claim—that using all suggested features improves writing evaluation—is unsupported. This is an internal evidence gap, not a disagreement with consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UKTA, a Korean text analysis and automated writing evaluation system with three layers: morpheme analysis using the Bareun analyzer, extraction of 294 lexical diversity and cohesion features, and a neural scoring model that combines KoBERT+BiGRU sentence representations with an attention-weighted essay-level feature vector. On the AI-HUB Essay Evaluation Dataset, the authors report that the feature-augmented model improves average accuracy from 0.649 to 0.657 and quadratic weighted kappa from 0.509 to 0.538 over five runs, with 9 of 10 rubric scores improving. The paper also presents a qualitative analysis of attention weights for one low-scoring and one high-scoring essay.","tokens_in":12409,"tokens_out":4420,"duration_ms":40616,"significance":"If the reported improvements are reliable, UKTA is a useful multi-view tool for Korean writing evaluation, an under-resourced area relative to English. The feature pipeline is computed from essay text independently of rubric labels, so the comparison against the raw-text baseline is not circular. The integration of low-level morpheme analysis, mid-level lexical/cohesion features, and high-level rubric scoring is a sensible architecture, and the authors are explicit about using standard metrics. The main value is practical: a comprehensive Korean system with explainability-oriented outputs. However, the evidence for the central claim is currently thin because the quantitative comparison lacks uncertainty quantification, and the explainability claim depends on an unvalidated interpretation of attention weights.","major_comments":[{"comment":"The central quantitative claim is not supported by the reported statistics. The text states a \"significant improvement,\" but Table 1 reports only five-run averages with no standard deviations, per-run values, confidence intervals, or significance tests. For accuracy near 0.65 on roughly 6,000 test essays, the standard error of a proportion is about 0.006, so the average gain of 0.008 (0.649 to 0.657) is only about 1.3 standard errors; with only five runs, the sampling error of the mean difference is larger. The QWK gain of 0.029 (0.509 to 0.538) is likewise reported without uncertainty. The authors should provide per-rubric and overall variance, paired significance tests (for example, bootstrap over essays or McNemar for accuracy), and effect sizes, and they should temper the wording if the improvement does not survive such analysis.","section":"Section 4.2, Table 1"},{"comment":"The feature set, which is the treatment in the comparison, is incompletely specified. Section 3.2 states that 294 features are extracted and names three broad groups, but only Equations (1)-(6) and a few cohesion concepts are described; the full list of 294 features and their exact definitions are not given. Since the baseline comparison is \"with all suggested features\" versus raw text, the experiment is not reproducible and the reader cannot tell which components drive the result. A full feature inventory, the precise computation of each feature, and the hyperparameter settings used for the feature pipeline are needed.","section":"Section 3.2 and Section 3.3"},{"comment":"The explainability contribution rests on an unvalidated assumption. Attention weights from the essay-level attention layer are interpreted as feature importance and used to explain the model's predictions for low- and high-scoring essays. No evidence is provided that these attention weights faithfully reflect influence on predictions; attention can be an unreliable importance measure. The authors should validate the interpretation with an alternative such as permutation importance, leave-one-feature-out, or gradient-based attribution, or explicitly reframe the qualitative analysis as a case study of the model's attention patterns rather than as an explanation of scoring behavior.","section":"Section 3.3 and Section 4.2, qualitative results"},{"comment":"The paper says each experiment was performed five times and averages were used, but no random seed information or variance across the five runs is reported. Since the differences are small, run-to-run variability is a plausible alternative explanation for the observed gains. Reporting per-run results for all ten rubrics would also clarify whether the improvement is consistent or driven by a few rubrics; for example, the grammar rubric is unchanged at 0.601 accuracy and 0.280 QWK, and the originality QWK jumps from 0.069 to 0.172, which suggests heterogeneity that the current aggregate presentation obscures.","section":"Section 4.1, implementation details"}],"minor_comments":[{"comment":"The caption says \"both table (A-1) and list (A-1)\" and contains the phrase \"intuitive intuitive interface\"; the duplicate label and the typo should be fixed.","section":"Figure 3 caption"},{"comment":"The definition of MTLD is ambiguous: it states that K is the largest number of segments where the TTR of each segment is below a threshold, which is not the standard MTLD formulation. Please clarify how segments are constructed and how the threshold is applied.","section":"Equation (4), Section 3.2"},{"comment":"The table notes contain typos: \"Content nown\" should be \"Content noun,\" \"Content coupla\" should be \"copula,\" and \"Top-10 feature\" should be \"Top-10 features.\"","section":"Table 2 notes"},{"comment":"The prose and equations use \"token\" and \"morpheme\" interchangeably for the counting unit; since the system is morpheme-based, the counting unit should be defined once and used consistently.","section":"Equations (1)-(3), Section 3.2"},{"comment":"The description of topic-consistency computation is unclear: it says the topic sentence is identified by comparing the extracted keyword with each sentence, but the algorithm is not specified. Please state how keywords are matched to sentences and how the similarity scores are aggregated.","section":"Section 3.2, Cohesion"}],"recommendation":"major_revision","confidential_remarks":"This is a system paper with a promising architecture and a practically useful tool. The main gap is statistical: the central accuracy/QWK improvements are small and are presented without uncertainty quantification, so the claimed advantage over baseline is not currently established. The feature inventory and attention-weight validation issues are also fixable in revision. I have no concerns about attribution or scope; the paper fits an applied NLP/AWE venue, though a journal submission will need stronger evidence than the current SAC-style short paper provides."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a decent systems paper for Korean automated writing evaluation, and the tool itself looks useful. But the main empirical claim—that adding 294 lexical features improves scoring over raw text—is not statistically established as reported.\n\nWhat's genuinely new: the integration. The pieces (KoBERT, BiGRU, handcrafted features with attention, lexical diversity and cohesion measures) are all known, but putting them together for Korean with a morpheme analyzer and making it available as a multi-view tool is a real contribution. The qualitative analysis in Table 2 is a nice attempt to show what the attention weights pick out. The writing is clear, and they use the largest Korean essay dataset available.\n\nThe soft spots are real. Table 1 gives averages over five runs but no standard deviations, no per-run values, and no significance tests. The accuracy gain is 0.008 (0.649 to 0.657) on a test set of about 6,000 essays; that's roughly one standard error. The QWK gain is larger (0.509 to 0.538), but without variance you can't tell if either is beyond run-to-run noise. The paper calls the improvement 'significant' in Section 4.2, but there is no statistical test behind that word. That's an overclaim. Second, they compare only against their own KoBERT+BiGRU baseline, not against existing Korean AES systems like PASTA-I, so the practical value relative to prior tools is unclear. Third, the explainability argument assumes attention weights are faithful feature-importance measures; that assumption is asserted, not validated. Finally, the full 294-feature set is not listed, which makes reproduction harder.\n\nNone of this is fatal. The architecture is sensible and the system fills a gap for Korean text analysis. The missing statistics and comparisons are the kinds of things a referee can reasonably ask for. The paper is aimed at the Korean language education and NLP community; those readers will get value from the tool design and the qualitative findings, even if the quantitative claim needs more support.\n\nI'd send it to peer review rather than desk-reject; with added error bars, a significance test, and a comparison to at least one existing Korean AES model, the central claim could be put on solid ground.","headline":"A useful Korean AES system paper whose central empirical claim is under-supported by missing error bars and significance tests.","tokens_in":12936,"tokens_out":2771,"would_cite":false,"duration_ms":24740,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feeding Korean essays through a morpheme analyzer and 294 lexical features improves automated scoring over raw-text-only baselines.","keywords":["Automated writing evaluation","Korean text analysis","Morpheme analysis","Lexical diversity","Cohesion","Attention-based scoring","Rubric scoring","AI-HUB Essay Evaluation Dataset"],"falsifier":"Take the trained UKTA model, replace the learned attention weights with uniform or randomly permuted weights, and re-evaluate on the same test split. If accuracy and QWK stay near 0.657 and 0.538, the performance claim survives but the attention-based explanation is not load-bearing; if they drop, the weights matter but still need comparison against an independent attribution method (e.g., leave-one-out or SHAP) to confirm they rank features correctly.","tokens_in":11978,"feed_emoji":"📝","tokens_out":4333,"duration_ms":35940,"temperature":0.7,"pith_summary":"The paper tries to establish that a Korean automated writing evaluation system can be made both more accurate and more explainable by combining a high-accuracy morpheme analyzer with 294 mid-level lexical features and an attention-weighted scoring model. It reports that feeding these features into the model improves average accuracy from 0.649 to 0.657 and quadratic weighted kappa from 0.509 to 0.538 on the AI-HUB Essay Evaluation Dataset, with 9 of 10 rubric scores improving. If true, this matters because Korean writing evaluation has lagged English-language tools, and educators would gain not just scores but visible reasons for them.","feed_headline":"Adding 294 morpheme features lifts Korean essay scoring","feed_subtitle":"Accuracy rises to 0.657 and quadratic weighted kappa to 0.538 on AI-HUB essays.","key_machinery":"The central mechanism is a three-level pipeline: Bareun morpheme analysis produces morphemes; those morphemes feed 294 lexical features (basic counts and densities, lexical diversity indices, and cohesion via KeyBERT and SBERT); and an attention layer over the normalized features is combined with a KoBERT+BiGRU sentence encoder to predict rubric scores. The attention weights double as the paper's explainability device, ranking which features mattered for each essay.","core_discovery":"UKTA claims that using all suggested features—basic lexical statistics, lexical diversity measures such as TTR, MSTTR, MATTR, MTLD, HD-D, and vocd-D, plus cohesion scores from KeyBERT topic keywords and SBERT sentence similarity—improves automated Korean essay scoring over a baseline that reads raw text only. The system pipelines Bareun morpheme analysis into these 294 features, normalizes them, weights them with an attention layer, and concatenates the result with a KoBERT-BiGRU sentence representation before predicting 10 rubric scores. The attention weights are then presented as the features that contributed to each score.","pith_inferences":["Editorial inference: The attention-weights-as-importance claim is untested; a permutation or uniform-attention ablation would show whether the explainability story holds. This is my inference, not the paper's.","Editorial inference: If the feature enrichment generalizes, it could transfer to other morphologically rich languages (e.g., Turkish, Finnish, Japanese) with their own morpheme analyzers.","Editorial inference: The feature set of 294 is large relative to the dataset; pruning or grouping features could reveal which families (diversity vs cohesion) drive the gain.","Editorial inference: QWK gains are concentrated in a few rubrics; a rubric-by-rubric significance test would tell whether the average improvement is robust. This is my inference."],"forward_implications":["Korean essay scoring can be improved without changing the deep learning backbone, simply by adding morpheme-derived features.","The same feature set gives a built-in explanation channel: every score comes with a list of the top lexical and cohesion features that influenced it.","Accurate morpheme analysis reduces error propagation into later features, which matters for agglutinative Korean morphology.","The system provides a reusable analysis tool for educators, with morpheme, feature, and rubric views of the same text.","The reported gains set a new baseline for Korean automated writing evaluation on the AI-HUB dataset."],"supporting_citations":[{"why":"Provides the morpheme-based Korean cohesion analysis that UKTA builds on for mid-level features.","marker":"[18]"},{"why":"KeyBERT extracts the keywords used for topic-consistency cohesion measurement.","marker":"[11]"},{"why":"SBERT supplies the sentence-similarity scores used in cohesion features.","marker":"[32]"},{"why":"A prior Korean automated essay scoring model using argument structure; one of the systems UKTA extends and compares against.","marker":"[21]"},{"why":"PASTA-I, a KoELECTRA-based Korean essay scorer; the other prior system UKTA situates itself against.","marker":"[37]"},{"why":"Defines MTLD, one of the lexical diversity features included in the 294-feature set.","marker":"[23]"},{"why":"Validates MTLD, vocd-D, and HD-D, the diversity measures UKTA adopts.","marker":"[25]"},{"why":"Defines MSTTR, another diversity feature used in the feature vector.","marker":"[17]"}],"fun_headline_variants":["UKTA: unified Korean analyzer lifts essay scoring","294 morpheme features sharpen Korean essay grading","Korean AWE gets transparent morpho-lexical boost","Multi-view Korean text analyzer beats baseline scoring","Explainable Korean writing eval with richer features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The explainability claim assumes that the attention weights over the 294 features truthfully show which features drove each score; the paper does not validate this, and the score improvement could hold even if the attention weights were meaningless.","fun_headline_variants_meta":{"raw":{"variants":["UKTA: unified Korean analyzer lifts essay scoring","294 morpheme features sharpen Korean essay grading","Korean AWE gets transparent morpho-lexical boost","Multi-view Korean text analyzer beats baseline scoring","Explainable Korean writing eval with richer features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":1081,"prompt_tokens":775,"completion_tokens":306,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":391,"completion_tokens_details":{"reasoning_tokens":235}},"tokens_in":391,"tokens_out":306,"duration_ms":3372,"temperature":1.0,"reasoning_tokens":235,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:21:13.478528+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained UKTA model, replace the learned attention weights with uniform or randomly permuted weights, and re-evaluate on the same test split. If accuracy and QWK stay near 0.657 and 0.538, the performance claim survives but the attention-based explanation is not load-bearing; if they drop, the weights matter but still need comparison against an independent attribution method (e.g., leave-one-out or SHAP) to confirm they rank features correctly.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KeyBERT extracts the keywords used for topic-consistency cohesion measurement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A prior Korean automated essay scoring model using argument structure; one of the systems UKTA extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PASTA-I, a KoELECTRA-based Korean essay scorer; the other prior system UKTA situates itself against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines MTLD, one of the lexical diversity features included in the 294-feature set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Validates MTLD, vocd-D, and HD-D, the diversity measures UKTA adopts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines MSTTR, another diversity feature used in the feature vector."}],"review_version":1}