{"id":"12a71c4d-c0fa-4e69-9e9d-40a3a6489a3a","arxiv_id":"1908.02505","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An embedding-based classifier, trained on cleaned learner response data, outperformed grammar-file-based baselines on the CALL v3 text scoring task without using the reference grammar at inference.","lead":"This system paper reports that an embedding-based classifier using BERT and NNLM text embeddings achieved the top score among text-task entries in the CALL v3 shared task, without using the provided reference grammar at inference time. It is worth reading for evidence that pre-trained language-model embeddings can replace hand-built grammar files for automated scoring of learner responses, and for the data-cleaning choices behind that result.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The grammar-free margin over the baseline may rest on a test set that has only ~300 unique non-overlapping utterances; without a decontaminated re-scoring, the central claim is not settled.","rationale":"The reader's weakest assumption is that the 1,000-utterance test set is a stable, unbiased measure after deduplication and overlap removal, and the paper's own Section 3.2 explicitly flags the leakage danger. My stress test converges on the same point but sharpens it: the claimed superiority is precisely over a grammar baseline, and if the evaluation set contains many entries whose exact normalized text appears in training, an embedding-based classifier with a memorizing-capable hidden layer can score those entries from stored label associations. This does not make the claim false, but it makes the reported margins uninterpretable without a unique-entry, decontaminated rescoring. The conclusion/table mismatch further weakens trust in the reported numbers. Independent support from Table 2 is limited because the v2 test set is also reported to intersect the training set. Given no code, no error bars, and no unique-entry analysis, the central claim should be marked unverified rather than conditionally accepted on the current evidence.","tokens_in":6882,"tokens_out":5120,"duration_ms":60022,"concrete_test":"Use the official CALL v3 scoring script on the submitted outputs (FFF, DDD, EEE) and the grammar baseline. Build a decontaminated test set by applying the paper's Section 4.1 preprocessing to the v3 test transcriptions, removing any prompt+response text string that appears in the final 4,481-entry training set, then collapsing duplicate normalized texts to one representative utterance per unique group, yielding approximately 300 items. Recompute Dfull for each system and the baseline on this subset, and bootstrap over the unique groups to obtain a confidence interval for each margin over baseline. If DDD or EEE no longer exceeds the baseline by a clearly positive margin (e.g., lower bound above zero), the grammar-free advantage is not supported.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim (Section 1) is that embedding-only submissions beat the grammar baseline on the CALL v3 text task. The load-bearing evidence is Table 1, scored on the 1,000-utterance v3 test set. The authors themselves state in Section 3.2 that after duplicate removal and excluding overlap with training, 'only about 300 unique entries out of 1000' remain, and they warn of 'a danger to create a system with seemingly acceptable performance that would merely overfit the training dataset.' Since the submitted systems were scored on the full contaminated set, the reported margins (FFF 4.413, DDD 4.403, EEE 4.227 vs. baseline 4.09 on Dfull) may be inflated by repeated test templates whose labels are already present in the training set after the paper's own concatenation of RecResult/Transcription and deduplication. The classifier is a 128-neuron MLP over fixed embeddings; identical or near-identical normalized test strings map to the same input vector as training strings, so outputs for those entries can copy training labels rather than reflect generalization. Table 2's v2 validation is subject to the same overlap concern. No confidence intervals or unique-entry scores are reported, and the conclusion's Dfull values (4.192/4.178/4.025) do not match Table 1 (4.413/4.403/4.227), so the numerical basis is not internally consistent. The claim may still be true, but as reported it is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a text-subtask scoring system for the CALL v3 shared task. The system uses pre-trained text embeddings (BERT and NNLM) concatenated into fixed vectors, feeds them into a shallow MLP classifier, and predicts one of three classes (correct, wrong language, wrong meaning). The authors claim that their submission was the only text-task entry to beat the grammar-file-based baseline, and that variants which do not use the grammar file at inference also surpass the baseline, demonstrating that grammar-free embedding-based scoring is competitive. The paper details data preprocessing, model training, validation on the v2 test set, and final results on the v3 test set, and argues that relying on the reference grammar makes CALL scoring systems non-scalable.","tokens_in":7195,"tokens_out":4822,"duration_ms":47922,"significance":"If the central claim holds, the result is significant for CALL: it suggests that a text-scoring system built entirely on generic embeddings plus a small classifier can match or beat a task-specific grammar reference, with obvious benefits for portability to new prompts and languages. The paper's strengths include the external shared-task evaluation, the use of standard and reproducible embedding models, a concrete description of data preparation, and an unusually candid discussion of dataset quality issues in Section 3.2. However, the reported evidence has load-bearing gaps: the effective test set is small and contaminated by duplicates and train/test overlap, no uncertainty quantification is given, and the numerical results in Section 6 do not match Table 1. These issues must be resolved before the central claim can be considered established.","major_comments":[{"comment":"The authors state that after removing duplicates and entries overlapping with the training set, the 1000-utterance v3 test set contains 'only about 300 unique entries out of 1000' and warn about the 'danger to create a system with seemingly acceptable performance that would merely overfit the training dataset.' Nevertheless, the Dfull scores in Table 1 (FFF 4.413, DDD 4.403, EEE 4.227 vs. baseline 4.09) are computed on the full set, which includes repeated and training-overlapping transcriptions. Since the classifier takes fixed embedding vectors as input, test strings that also appear in training can effectively copy training labels rather than test generalization. Please report scores restricted to the unique, non-overlapping subset, or otherwise quantify how much of the margin over the baseline is contributed by duplicate and overlapping entries. Without such a decontaminated evaluation, the paper's central claim is not established.","section":"Section 3.2 / Table 1"},{"comment":"The conclusions report Dfull scores of 4.192, 4.178, and 4.025 for the best, BERT+nnlm, and nnlm submissions, while Table 1 lists the corresponding highlighted text submissions as 4.413 (FFF), 4.403 (DDD), and 4.227 (EEE). Because the Dfull metric is presumably the same, these numbers cannot both be correct without an explanation. Please reconcile the two sets of numbers, state which ones are the official v3 leaderboard scores, and clarify the relationship between the anonymized submission IDs in Table 1 and the model names in Section 6.","section":"Section 6 vs. Table 1"},{"comment":"The abstract and Section 1 emphasize that the presented system 'does not rely on the reference grammar file for scoring,' but Section 4.4 and Section 6 identify the best-scoring submission as using 'nnlm and the updated grammar' (the nnlm+ model). Only the other two submissions (DDD and EEE) are actually grammar-free. Please state explicitly which of the three highlighted submissions are grammar-free and ensure that the abstract, introduction, and conclusions are not stronger than the evidence for a grammar-free system.","section":"Abstract / Section 1 vs. Section 4.4 / Section 6"},{"comment":"No confidence intervals, error bars, or significance tests are reported for any of the Dfull differences. With an effective test size of roughly 300 unique utterances, a difference of 0.137 (EEE vs. baseline) or even 0.323 (FFF vs. baseline) may well be within sampling noise. Please provide at least bootstrap confidence intervals or a significance test on the unique subset, so that the reader can assess whether the claimed superiority over the grammar baseline is statistically reliable.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"There is a typographical error: 'organazires' should be 'organizers'.","section":"Section 3.2"},{"comment":"'differnet' should be 'different' in the description of ensemble initialization.","section":"Section 4.3"},{"comment":"The table caption says 'scores of our systems are highlighted,' but the highlighting is not visible in the text version of the paper; please add a visible formatting cue (bold or an asterisk).","section":"Table 1"},{"comment":"The phrase 'the relatively high performance of this model is surprising' for nnlm would benefit from a brief discussion of why averaging word embeddings might still work well in this task, since the authors later rely on this result.","section":"Section 4.2"},{"comment":"The sentence 'On the CALL v3 test set (Table 1), the model nnlm+ (FFF) achieves the best performance' is clear, but the following sentence should explicitly note that lower Dfull values are better or that the authors are using the convention from the shared task, to avoid confusion for readers not familiar with the metric.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"This is a short system-description paper with an externally evaluated result. The main obstacle to acceptance is not the system design but the integrity of the evaluation: the authors themselves flag that only ~300 unique test entries remain after de-duplication and overlap removal, and the reported leaderboard numbers are not internally consistent with the conclusion. If the authors can supply decontaminated scores and reconcile the numerical discrepancy, the paper could be acceptable; as it stands, the central claim is not yet established. I would also ask the editor to verify whether the abstract's grammar-free claim is consistent with the authors' best official submission, since the paper states that the best-scoring variant uses the updated grammar."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that its main claim—embedding-only classifiers can beat a grammar-file baseline for CALL text scoring—is genuinely new for this task but not actually settled by the evidence presented. Prior shared-task systems all leaned on the reference grammar; this one shows BERT and NNLM embeddings through a 128-neuron MLP scoring competitively without grammar features at inference. That is a real idea and the authors deserve credit for trying it and for being unusually candid about the data mess.\n\nThe paper does several things well. It documents the duplicate and overlap problems in the CALL v3 text data, including the fact that only about 300 unique non-overlapping utterances remain out of 1000. It warns that a system could appear acceptable while merely overfitting. It also describes preprocessing choices in enough detail to reproduce the pipeline. Those are assets.\n\nThe soft spots are real and load-bearing. First, the leaderboard scores in Table 1 were computed on the full contaminated set, so the margins over the grammar baseline (4.413, 4.403, 4.227 vs. 4.09) may largely reflect memorized training labels for repeated ASR transcripts rather than generalization. The paper has no decontaminated re-scoring, no confidence intervals, and no unique-entry analysis. Second, the conclusion reports Dfull scores of 4.192, 4.178, and 4.025 that do not match Table 1; that is a factual inconsistency a reader can't wave away. Third, the best submission (FFF) uses the updated grammar at inference, so the no-grammar claim rests on the second and third systems, DDD and EEE, which still beat the baseline. The grammar-free point is supported only if those scores are trustworthy, and the contamination problem directly threatens them. Fourth, early stopping on Dfull on the v2 test set is a form of tuning on a held-out set; it doesn't invalidate the result but it weakens the validation story.\n\nWho is this for? Researchers working on CALL shared tasks and automated scoring of learner language. It's a useful data point and a good critique of the task design. I would not cite it as evidence that grammar-free scoring works, because the evaluation is too shaky. But I would bring it to a reading group to discuss evaluation contamination and shared-task design.\n\nFor peer review: yes, it deserves a serious referee, but only with major revision. The authors should re-score on a deduplicated, decontaminated set, fix the numerical inconsistency, and clearly separate the grammar-using and grammar-free systems. Without that, the headline claim stays conditional.","headline":"The grammar-free embedding result is plausible but not established—the test set is heavily contaminated and the paper's own numbers don't line up.","tokens_in":7729,"tokens_out":1693,"would_cite":false,"duration_ms":20253,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A text-embedding classifier can score learner responses in the CALL v3 text task above the grammar-based baseline, with no reference grammar at inference.","keywords":["computer-assisted language learning","CALL shared task","text embeddings","BERT","NNLM","learner response scoring","grammar-free scoring","Dfull metric"],"falsifier":"Re-score the CALL v3 text task using only the roughly 300 unique test utterances that do not overlap the training set; if the embedding-based system's Dfull score no longer exceeds the grammar baseline, the reported superiority is an artifact of duplicate leakage and template overfitting.","tokens_in":6663,"feed_emoji":"🧠","tokens_out":10855,"duration_ms":96294,"temperature":0.7,"pith_summary":"This paper tries to establish that a scoring system built from pre-trained text embeddings can grade language-learner responses in the CALL v3 text task without consulting the reference grammar during inference, and that it performs as well as or better than systems that do use the grammar. On the shared task's test set, the authors report their submission is the only one among text-task competitors that beats the grammar-based baseline, and a grammar-free variant also surpasses it. If true, this would make CALL scoring systems easier to port to new prompts, domains, or language pairs, because no hand-written grammar must be extended. The paper also argues that careful data preparation — removing duplicates, noisy entries, and ASR artifacts — is what makes the embedding approach viable.","feed_headline":"Grammar-free embeddings beat reference grammar in CALL text","feed_subtitle":"A shallow classifier on BERT and NNLM vectors matched or topped grammar-based scoring in a shared-task challenge.","key_machinery":"The mechanism is the text-embedding estimator. BERT provides a 768-dimensional contextual vector for the German prompt machine-translated to English, concatenated with the learner's response via the '|||' separator; NNLM provides three 50-dimensional averaged vectors from the original German prompt, the translated prompt, and the response. These are concatenated into a 918-dimensional input to a neural network with one hidden layer of 128 ReLU units, regularized by dropout and early stopping tuned on the Dfull metric, with predictions averaged over an ensemble of models. The embeddings themselves are not fine-tuned, so the reference grammar file never enters inference.","core_discovery":"The central claim, on the paper's own terms, is that concatenating BERT's 768-dimensional contextual vector with three 50-dimensional NNLM vectors (for the German prompt, its machine translation, and the learner's English response) and feeding the resulting 918-dimensional vector into a single-hidden-layer neural network yields accept/reject decisions for learner utterances that match or beat hand-built grammar matching. The authors report Dfull scores above the grammar baseline on the CALL v3 text test set, and they emphasize that grammar-free variants (nnlm-only and BERT+nnlm) also outperform the baseline, showing the reference grammar is not load-bearing for good scoring. The contribution is an existence proof rather than a new learning architecture: off-the-shelf embeddings plus a shallow classifier can replace an explicit grammar in a constrained CALL setting.","pith_inferences":["Not claimed in the paper: the leaderboard edge may not survive a test set built from only unique, non-overlapping utterances; the authors' own warning about overfitting makes this the main threat to the result.","Because both contextual BERT and order-insensitive NNLM averaging work, the model may be exploiting lexical overlap with grammar-file entries rather than deep understanding; a test with novel phrasings would separate these possibilities.","An extension the authors mention but do not try is fine-tuning BERT with data augmentation, which could push the grammar-free variants further; this is a plausible next step given their setup."],"forward_implications":["The same embedding-classifier recipe can be applied to new prompts or language pairs without writing or extending a reference grammar.","Because inference does not depend on the grammar, combining this scorer with improved ASR could raise overall CALL system performance.","The results suggest the grammar file is a 'low-hanging fruit' that sets a performance ceiling; text-task scoring may reward representation choice and data cleaning more than sophisticated modeling.","The paper's preprocessing steps — deduplication, artifact removal, and merging transcription fields — are part of the pipeline, and the reported gains may be partly due to these steps."],"supporting_citations":[{"why":"Supplies the NNLM word-embedding model that, when averaged, yields one of the two best-performing input representations.","marker":"[1]"},{"why":"Supplies the pre-trained BERT model that produces the 768-dimensional contextual vectors used as the other input component.","marker":"[2]"},{"why":"Defines the CALL shared task's Dfull evaluation metric and the grammar-based baseline that the system must beat.","marker":"[8]"},{"why":"Provides the preprocessing steps (whitespace, ASR artifact and filler removal, contraction expansion) that the paper adapts for its training data.","marker":"[13]"},{"why":"Describes the previous year's doc2vec-and-word-mover-distance approach, the similarity-based method the authors tried and found inferior to direct embeddings.","marker":"[14]"},{"why":"Another prior system that relied on the reference grammar and doc2vec embeddings, representing the grammar-dependent approach the paper argues against.","marker":"[15]"}],"fun_headline_variants":["Embeddings replace grammar in top CALL text scorer","No grammar needed: embeddings top CALL shared task","BERT and NNLM vectors beat hand-built grammar in CALL","Shallow net on embeddings outperforms grammar file in CALL","Grammar-free scoring wins on CALL text subset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking on which the conclusion rests comes from a 1000-utterance test set that contains only about 300 unique entries after removing duplicates and overlaps with the training set, so if duplicates drive the score, the claimed advantage over the grammar baseline may not hold for genuinely new learner responses.","fun_headline_variants_meta":{"raw":{"variants":["Embeddings replace grammar in top CALL text scorer","No grammar needed: embeddings top CALL shared task","BERT and NNLM vectors beat hand-built grammar in CALL","Shallow net on embeddings outperforms grammar file in CALL","Grammar-free scoring wins on CALL text subset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1267,"prompt_tokens":798,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":414,"tokens_out":469,"duration_ms":5335,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:41:24.289185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the CALL v3 text task using only the roughly 300 unique test utterances that do not overlap the training set; if the embedding-based system's Dfull score no longer exceeds the grammar baseline, the reported superiority is an artifact of duplicate leakage and template overfitting.","supporting_citations":[{"cited_title":"the re- search for and study of applications of the computer in language teaching and learning","cited_arxiv_id":null,"evidence_quote":"Supplies the NNLM word-embedding model that, when averaged, yields one of the two best-performing input representations."},{"cited_title":"For example , one of the last year’s submissions [15] processed the ASR out - put and up to 10 entries from the reference grammar file using the doc2vec model","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained BERT model that produces the 768-dimensional contextual vectors used as the other input component."},{"cited_title":"Bengio, R","cited_arxiv_id":null,"evidence_quote":"Defines the CALL shared task's Dfull evaluation metric and the grammar-based baseline that the system must beat."},{"cited_title":"CALL-SLT: A Spoken CALL System Based on Grammarand Speech Recognition,","cited_arxiv_id":null,"evidence_quote":"Provides the preprocessing steps (whitespace, ASR artifact and filler removal, contraction expansion) that the paper adapts for its training data."},{"cited_title":"Overview of the 2017 Spoken CALL Shared- Task,","cited_arxiv_id":null,"evidence_quote":"Describes the previous year's doc2vec-and-word-mover-distance approach, the similarity-based method the authors tried and found inferior to direct embeddings."},{"cited_title":"Overview of the 2018 Spoken CAL L Shared Task,","cited_arxiv_id":null,"evidence_quote":"Another prior system that relied on the reference grammar and doc2vec embeddings, representing the grammar-dependent approach the paper argues against."}],"review_version":1}