{"id":"7e8dbd30-2613-4f93-9ed9-f55fccbd49a3","arxiv_id":"2509.01640","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TransGAT fuses a fine-tuned Transformer's essay-level prediction with a graph attention network run over syntactic dependency edges, reporting an average QWK of 0.854 on ELLIPSE.","lead":"This paper proposes TransGAT, a hybrid model that combines Transformer language models (BERT, RoBERTa, DeBERTaV3) with graph attention networks to score essays on six analytic traits. On the ELLIPSE dataset, the best variant reaches an average quadratic weighted kappa of 0.854, but the comparison to prior baselines is confounded by model size and absent ablations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparison is not controlled: TransGAT's best row is RoBERTa-large vs. base baselines, with no reported split or ablation, so the 0.854 QWK margin cannot be attributed to GAT.","rationale":"The paper's intended contribution is a hybrid Transformer-GAT model for analytic AES, with the headline that it outperforms prior baselines on ELLIPSE. For that claim to hold, the comparison must isolate the effect of adding GAT to a Transformer backbone. It does not. The baselines are borrowed from prior papers and use smaller base models; no same-protocol reimplementation, no ablation, and no cross-prompt experiment are provided. The reader's weakest assumption identifies exactly this: the 0.854 vs. 0.825 margin may be an artifact of protocol or model capacity. Our stress test agrees and adds that the missing RoBERTa-large-without-GAT ablation is the decisive missing control. The paper does contain a reasonable architecture and some useful engineering details (syntactic dependency graph, two-stream fusion, Stanza), but the evidence is insufficient to accept the superiority claim as stated. This is not an internal inconsistency or a fraud concern; it is an under-specified, confounded evaluation. Conditional acceptance with mandatory matched baselines and ablations is the appropriate verdict, matching the reader's original judgment.","tokens_in":14596,"tokens_out":3219,"duration_ms":37044,"concrete_test":"Fix one train/val/test split (e.g., prompt-disjoint split for cross-prompt; within-prompt split for the reported QWK) and run four systems under identical training settings: RoBERTa-base without GAT, RoBERTa-base with GAT, RoBERTa-large without GAT, and RoBERTa-large with GAT, each over at least three seeds. If RoBERTa-large without GAT reaches or exceeds 0.854 average QWK, the gain attributed to GAT collapses. If RoBERTa-base-GAT does not beat RoBERTa-base by a stable margin, the GAT component is not supported. Also report whether any experiment uses held-out prompts; if not, remove the cross-prompt claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'TransGAT outperforms baseline models' (abstract) and 'first to integrate Transformer-based embeddings with GATs for AES while jointly addressing cross-prompt and multi-dimensional scoring' (Section 1)—rests on Table 3. That table is confounded in two ways. First, no train/validation/test split, prompt partitioning, or random seeds are reported. The baseline QWK values are taken directly from Sun et al. (roberta-base) and Chen et al. (debertaV3-base), which may use different evaluation protocols; without reimplementation under a common split, the 0.029 QWK margin over Sun et al. is not a controlled comparison. Second, the best TransGAT row uses RoBERTa-large-GAT, while the strongest baseline uses RoBERTa-base. The paper never runs RoBERTa-large without GAT or RoBERTa-base with GAT, so the improvement could be entirely due to model scale. Additionally, the claimed cross-prompt contribution is not evaluated anywhere: Section 4 reports only per-dimension QWK on ELLIPSE, with no held-out-prompt protocol, so the 'jointly addressing cross-prompt' novelty claim is unsupported. The GAT graph is also intra-essay syntactic dependency (Section 3.2.2), not cross-essay relations, which undercuts the Section 1 motivation about relationships between essays. The architecture may be sound, but the headline evidence for superiority is not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TransGAT, an architecture for multi-dimensional automated essay scoring (AES) on the ELLIPSE corpus. In each variant, a fine-tuned Transformer (BERT-large, RoBERTa-large, or DeBERTaV3-large) is used in two streams: a [CLS]-based prediction stream and a token-level GAT stream whose edges come from Stanza syntactic dependency parses. The two streams are combined by sum fusion (Eq. 8) and trained with MSE loss over six analytic traits. The authors report QWK scores for the three TransGAT variants and compare them with two previously published baselines, claiming that RoBERTa-large-GAT achieves the best average QWK of 0.854 and outperforms the baseline models. The paper also claims to be the first to integrate Transformer embeddings with GATs for AES while jointly addressing cross-prompt and multi-dimensional scoring.","tokens_in":14968,"tokens_out":3500,"duration_ms":40063,"significance":"If the claimed performance advantage were established under a controlled protocol, the combination of contextual Transformer embeddings with a GAT over syntactic dependency graphs would be a reasonable and potentially useful contribution to analytic AES. The GAT formulation in Section 3.2.3 is standard and correctly presented, the choice of ELLIPSE is appropriate for multi-dimensional scoring, and the architecture is clearly described. However, as presented, the headline empirical claim rests on an uncontrolled comparison with published baseline numbers, the contribution of the GAT stream is not isolated by any ablation, and the cross-prompt novelty claim is not evaluated anywhere in the experiments. The central scientific significance is therefore not yet established.","major_comments":[{"comment":"The central claim that 'TransGAT outperforms baseline models' compares RoBERTa-large-GAT (avg QWK 0.854) against roberta-base (0.825) and debertaV3-base (0.685) taken directly from Sun et al. and Chen et al. The manuscript never specifies its own train/validation/test split, prompt partitioning, or random seeds, and the baselines are not re-run under a common protocol. The compared models differ simultaneously in backbone size and evaluation setup, so the 0.029-point margin over Sun et al. cannot be attributed to the GAT component. To support the abstract's claim, the authors must either re-run the baselines under the same split and training procedure or provide a precise statement of the protocol and demonstrate that it matches the baselines.","section":"§4.5, Table 3"},{"comment":"No ablation isolates the contribution of the GAT stream. The only novel component over a fine-tuned Transformer is the second stream and its fusion (Eq. 8), yet the paper never reports the performance of the Transformer-only stream with the same backbone (e.g., RoBERTa-large without GAT) or the GAT stream alone. Without such a control, the statement in Section 4.5 that 'the results highlight the advantage of incorporating GAT with Transformer architectures' is unsupported. This is a load-bearing point for the paper's main architectural claim.","section":"§3.2.4 and §4.5"},{"comment":"The paper claims to be 'the first to integrate Transformer-based embeddings with GATs for AES while jointly addressing cross-prompt and multi-dimensional scoring.' No cross-prompt experiment is reported: Section 4 evaluates only within the ELLIPSE dataset with no held-out-prompt protocol, and the GAT graph is built from intra-essay syntactic dependencies (Eq. 2), not from cross-essay relations. The motivation in Section 1 repeatedly refers to 'relationships between essays,' but the model does not model such relationships. The cross-prompt novelty claim should be either removed or substantiated with a true cross-prompt evaluation.","section":"§1 and §3.2.2"},{"comment":"The reported results are selected from 'extensive experiments' in which the authors varied dependency parsers, dropout, learning rates, optimizers, schedulers, number of GAT layers and heads, activation functions, and batch sizes, and then present 'the best results from these experiments.' No number of runs, random seeds, confidence intervals, or significance tests are reported. The headline QWK of 0.854 is therefore a best-of-many configuration and may be optimistically biased. The authors should report the evaluation protocol, the number of configurations tried, and either standard errors or the results of an appropriate significance test.","section":"§4.4"}],"minor_comments":[{"comment":"The sentence 'Building on this, the present study integrates Transformer-based contextual embeddings with GCNs...' should refer to GATs, since the proposed method uses Graph Attention Networks, not GCNs.","section":"§2.2"},{"comment":"The row label 'debertaV-large-GAT' is inconsistent with the text's 'DeBERTaV3-large-GAT'. This should be corrected for clarity.","section":"Table 3"},{"comment":"The explanation for lower Vocabulary performance says the GAT integration 'emphasizes relational and contextual information across essays,' but the graph edges are intra-essay syntactic dependencies. The phrase should be 'across words' or 'across tokens' to be consistent with Section 3.2.2.","section":"§4.5"},{"comment":"Figure 1 is dense and not fully legible in the text. Labeling the fusion operation and the pooling step explicitly, or enlarging the relevant panels, would help readers connect the figure to Equations (1)–(8).","section":"Figure 1"},{"comment":"The statement that 'no dropout was applied during training, as this was found to improve regression performance' reports a modeling choice without showing the supporting experiment. Please provide the comparison or cite a prior result.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper builds a reasonable hybrid—a fine-tuned Transformer encoder feeding a GAT on syntactic dependency trees—for multi-trait essay scoring, using the ELLIPSE corpus. That is a legitimate extension of previous Transformer+GCN work. The architecture is spelled out clearly and the GAT machinery is standard. The choice of ELLIPSE over ASAP is sensible for analytic scoring, and the per-trait QWK table is a useful breakdown.\n\nThe problem is that the paper's main claim—'outperforms baseline models'—rests on Table 3, which is not a controlled comparison. The best TransGAT row is RoBERTa-large-GAT; the baselines are RoBERTa-base from Sun et al. and DeBERTaV3-base from Chen et al. There is no reimplementation of those baselines under the same train/validation/test split, and the split itself is not disclosed. So the 0.029 QWK margin over Sun et al. could be due to model scale, data handling, or random seeds. The paper also never ablates: no RoBERTa-large without GAT, and no RoBERTa-base with GAT, so we can't attribute the gain to the graph component. Section 4.4 says they selected the best results from many configurations, which further inflates the reported numbers.\n\nThe cross-prompt part of the claimed novelty is unsupported. The graph edges are intra-essay syntactic dependencies, not cross-essay relations, and no cross-prompt experiment is run. The Introduction motivates the method with 'relationships across multiple texts,' but the actual model models relationships within one essay. That mismatch should be fixed, either by removing the cross-prompt claim or by adding a held-out-prompt evaluation.\n\nNone of this means the architecture is bad. The components are well-known and the combination is plausible for capturing syntactic structure that a Transformer might underweight. The paper is honest about what it does, and the related work is adequate, though it does not engage closely with the closest Transformer+GCN baselines. The issue is evidence, not idea.\n\nWho gets value: someone entering analytic AES who wants a template for Transformer+GAT hybrids. I would not cite the empirical result as established, but the architecture description is usable.\n\nRecommendation: send it to peer review with a request for major revision. A referee should demand matched baselines, a disclosed split, an ablation of the GAT stream, and either a real cross-prompt evaluation or a revised scope statement. The paper deserves a serious referee because the architectural idea is sound and the field needs more multi-dimensional AES work on modern datasets.","headline":"Plausible architecture, but the headline result is confounded by model-scale and protocol differences, and the cross-prompt claim is not evaluated.","tokens_in":15417,"tokens_out":3150,"would_cite":false,"duration_ms":32784,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hybrid Transformer-GAT model scores essays at 0.854 average QWK","keywords":["Automated Essay Scoring","Graph Attention Networks","Transformer fine-tuning","Analytic scoring","ELLIPSE corpus","Syntactic dependency graph","Cross-prompt scoring","Quadratic Weighted Kappa"],"falsifier":"Run the same fine-tuned Transformer backbone (e.g., RoBERTa-large) with and without the GAT stream, keeping the identical train/validation/test split, hyperparameters, and random seeds on ELLIPSE; if adding the GAT does not raise QWK, the claimed contribution collapses. Reporting the split and seeds publicly would settle whether the reported margin over Sun et al. (2024) and Chen et al. (2024) is due to protocol differences.","tokens_in":14525,"feed_emoji":"📝","tokens_out":5654,"duration_ms":53498,"temperature":0.7,"pith_summary":"This paper tries to show that automated essay scoring improves when contextual Transformer embeddings are combined with Graph Attention Networks that use syntactic dependencies as edges. The proposed model, TransGAT, pairs each fine-tuned Transformer (BERT, RoBERTa, or DeBERTaV3) with a separate GAT: one stream predicts from the whole essay, the other from token-level graphs, and the two predictions are summed. On the ELLIPSE corpus of essays written by English language learners, RoBERTa-large-GAT reaches an average Quadratic Weighted Kappa of 0.854 across six analytic traits, which the authors report as surpassing two baseline studies. If true, this would support analytic, multi-dimensional scoring that can give learners feedback on grammar, vocabulary, and cohesion rather than a single holistic grade.","feed_headline":"Graph attention lifts essay scoring to 0.854 QWK","feed_subtitle":"Transformer embeddings plus syntactic dependency graphs beat baselines on six analytic writing traits.","key_machinery":"The load-bearing mechanism is the two-stream prediction pair: a fine-tuned Transformer provides both the essay-level [CLS] embedding and contextualized token embeddings, while a two-layer GAT with four attention heads propagates information over a syntactic dependency graph built by Stanza; global mean pooling converts node updates into a graph embedding, and the essay-level and graph-level outputs are added. The GAT is the component that differentiates this from a plain Transformer baseline: it learns a weight for each neighboring token rather than treating all neighbors equally, which the paper argues suits essays where relationships between words vary in importance.","core_discovery":"TransGAT claims that attaching a Graph Attention Network to a fine-tuned Transformer improves analytic essay scoring beyond what a Transformer alone can do. Each model pair is trained with two streams: the [CLS] embedding drives an essay-level regression, while the same model's token embeddings are treated as nodes in a graph with edges drawn from Stanza's syntactic dependency parse; two GAT layers update the nodes, mean pooling produces a graph-level vector, and the two outputs are summed to score six traits. On the ELLIPSE corpus, the paper reports that RoBERTa-large-GAT delivers an average QWK of 0.854, beating RoBERTa-base and DeBERTaV3-base baselines and posting top scores on four of si","pith_inferences":["The 0.029 average-QWK margin between RoBERTa-large-GAT and Sun et al.'s RoBERTa-base is consistent with the added capacity of a large model; the paper does not isolate the GAT's contribution, so a fair test would compare RoBERTa-large with and without the GAT stream.","If the GAT's syntactic edges are what help most, then other relation types—semantic similarity between sentences, coreference links, or cross-essay edges—may generalize the design to heterogeneous graphs, which the paper lists as future work.","The same architecture is directly testable on other analytic-scoring corpora such as ASAP++ or ICLE++, whose different prompt and genre mixes would show whether the ELLIPSE gains transfer.","Applying the trained graph stream to morphologically rich languages would require an Arabic-aware dependency parser and tokenizer; the paper notes this direction, but the extent of the gain there is untested."],"forward_implications":["If TransGAT's reported results hold, RoBERTa-large-GAT alone delivers near-perfect agreement (0.854 QWK) across six analytic traits on ELLIPSE, suggesting deployable multi-trait scoring for learner essays.","Analytic scoring becomes feasible for English language learners: the model produces separate scores for cohesion, syntax, vocabulary, phraseology, grammar, and conventions, enabling targeted feedback.","The syntactic-dependency graph construction can be swapped for other parsers or edge definitions without changing the two-stream architecture, since the paper reports that varying parsers, optimizers, and schedulers affected performance.","Because the method does not rely on prompt-specific features, cross-prompt and multi-dimensional scoring can be addressed in one system, which the paper identifies as its research gap."],"supporting_citations":[{"why":"Supplies the ELLIPSE corpus, the dataset on which all reported QWK scores are computed.","marker":"Crossley et al., 2023"},{"why":"Provides the Transformer architecture that the fine-tuned backbones (BERT, RoBERTa, DeBERTaV3) build on.","marker":"Vaswani, 2017"},{"why":"Introduces the Graph Attention Network whose attention-weighted neighbor aggregation is the paper's central relational mechanism.","marker":"Veličković et al., 2017"},{"why":"Defines BERT, the baseline transformer family whose large variant is fine-tuned in one TransGAT configuration.","marker":"Devlin et al., 2019"},{"why":"Introduces RoBERTa, which in its large form gives TransGAT its best average QWK.","marker":"Liu et al., 2019"},{"why":"Defines DeBERTaV3, the model that achieves the top scores on cohesion and syntax in the TransGAT experiments.","marker":"He et al., 2021"},{"why":"One of the two ELLIPSE baselines the paper compares against, using RoBERTa-base.","marker":"Sun and Wang, 2024"},{"why":"The other ELLIPSE baseline, using DeBERTaV3-base, which TransGAT is claimed to outperform.","marker":"Chen et al., 2024"}],"fun_headline_variants":["Transformer plus graph attention scores essays better","Syntactic graphs boost essay scoring to 0.854 QWK","Graph attention lifts analytic essay scoring to 0.854 QWK","TransGAT: Transformers meet graphs for essay scoring","Combining BERT and GAT improves essay scoring"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported advantage over baselines assumes the comparison was run on the same data split and training protocol; the paper never specifies its own split or seeds, so the 0.029 QWK lead may reflect model size or data handling rather than the graph attention mechanism itself.","fun_headline_variants_meta":{"raw":{"variants":["Transformer plus graph attention scores essays better","Syntactic graphs boost essay scoring to 0.854 QWK","Graph attention lifts analytic essay scoring to 0.854 QWK","TransGAT: Transformers meet graphs for essay scoring","Combining BERT and GAT improves essay scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000804,"raw_usage":{"total_tokens":3387,"prompt_tokens":781,"completion_tokens":2606,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2525}},"tokens_in":525,"tokens_out":2606,"duration_ms":19615,"temperature":1.0,"reasoning_tokens":2525,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:19:25.057718+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuned Transformer backbone (e.g., RoBERTa-large) with and without the GAT stream, keeping the identical train/validation/test split, hyperparameters, and random seeds on ELLIPSE; if adding the GAT does not raise QWK, the claimed contribution collapses. Reporting the split and seeds publicly would settle whether the reported margin over Sun et al. (2024) and Chen et al. (2024) is due to protocol differences.","supporting_citations":[{"cited_title":", author Tian, Y","cited_arxiv_id":null,"evidence_quote":"Supplies the ELLIPSE corpus, the dataset on which all reported QWK scores are computed."},{"cited_title":", year 2017","cited_arxiv_id":null,"evidence_quote":"Provides the Transformer architecture that the fine-tuned backbones (BERT, RoBERTa, DeBERTaV3) build on."},{"cited_title":", author Chang, M.W","cited_arxiv_id":null,"evidence_quote":"Defines BERT, the baseline transformer family whose large variant is fine-tuned in one TransGAT configuration."},{"cited_title":"Automatic Essay Multi-dimensional Scoring with Fine-tuning and Multiple Regression","cited_arxiv_id":"2406.01198","evidence_quote":"One of the two ELLIPSE baselines the paper compares against, using RoBERTa-base."},{"cited_title":", author Lan, Y","cited_arxiv_id":null,"evidence_quote":"The other ELLIPSE baseline, using DeBERTaV3-base, which TransGAT is claimed to outperform."}],"review_version":1}