Embedding-based system for the Text part of CALL v3 shared task

Dmytro Tkanov; Ievgen Karaulov; Olga Zvyeryeva; Volodymyr Sokhatskyi

REVIEW 4 major objections 5 minor 29 references

Embedding-based system for the Text part of CALL v3 shared task

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A text-embedding classifier can score learner responses in the CALL v3 text task above the grammar-based baseline, with no reference grammar at inference.

desk verdict The grammar-free embedding result is plausible but not established—the test set is heavily contaminated and the paper's own numbers don't line up. read the letter →

arxiv 1908.02505 v1 pith:G5I7VO2I submitted 2019-08-07 cs.CL

Volodymyr Sokhatskyi , Olga Zvyeryeva , Ievgen Karaulov , Dmytro Tkanov This is my paper · ORCID

classification cs.CL

keywords computer-assistedlanguagelearningCALLsharedtasktextembeddingsBERTNNLMlearnerresponsescoringgrammar-freeDfullmetric

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a scoring system built from pre-trained text embeddings can grade language-learner responses in the CALL v3 text task without consulting the reference grammar during inference, and that it performs as well as or better than systems that do use the grammar. On the shared task's test set, the authors report their submission is the only one among text-task competitors that beats the grammar-based baseline, and a grammar-free variant also surpasses it. If true, this would make CALL scoring systems easier to port to new prompts, domains, or language pairs, because no hand-written grammar must be extended. The paper also argues that careful data preparation — removing duplicates, noisy entries, and ASR artifacts — is what makes the embedding approach viable.

What carries the argument

The mechanism is the text-embedding estimator. BERT provides a 768-dimensional contextual vector for the German prompt machine-translated to English, concatenated with the learner's response via the '|||' separator; NNLM provides three 50-dimensional averaged vectors from the original German prompt, the translated prompt, and the response. These are concatenated into a 918-dimensional input to a neural network with one hidden layer of 128 ReLU units, regularized by dropout and early stopping tuned on the Dfull metric, with predictions averaged over an ensemble of models. The embeddings themselves are not fine-tuned, so the reference grammar file never enters inference.

What would settle it

Re-score the CALL v3 text task using only the roughly 300 unique test utterances that do not overlap the training set; if the embedding-based system's Dfull score no longer exceeds the grammar baseline, the reported superiority is an artifact of duplicate leakage and template overfitting.

Watch

Extended reading notes

Core claim

The central claim, on the paper's own terms, is that concatenating BERT's 768-dimensional contextual vector with three 50-dimensional NNLM vectors (for the German prompt, its machine translation, and the learner's English response) and feeding the resulting 918-dimensional vector into a single-hidden-layer neural network yields accept/reject decisions for learner utterances that match or beat hand-built grammar matching. The authors report Dfull scores above the grammar baseline on the CALL v3 text test set, and they emphasize that grammar-free variants (nnlm-only and BERT+nnlm) also outperform the baseline, showing the reference grammar is not load-bearing for good scoring. The contribution is an existence proof rather than a new learning architecture: off-the-shelf embeddings plus a shallow classifier can replace an explicit grammar in a constrained CALL setting.

Load-bearing premise

The ranking on which the conclusion rests comes from a 1000-utterance test set that contains only about 300 unique entries after removing duplicates and overlaps with the training set, so if duplicates drive the score, the claimed advantage over the grammar baseline may not hold for genuinely new learner responses.

Editorial extensions

If this is right

The same embedding-classifier recipe can be applied to new prompts or language pairs without writing or extending a reference grammar.
Because inference does not depend on the grammar, combining this scorer with improved ASR could raise overall CALL system performance.
The results suggest the grammar file is a 'low-hanging fruit' that sets a performance ceiling; text-task scoring may reward representation choice and data cleaning more than sophisticated modeling.
The paper's preprocessing steps — deduplication, artifact removal, and merging transcription fields — are part of the pipeline, and the reported gains may be partly due to these steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Not claimed in the paper: the leaderboard edge may not survive a test set built from only unique, non-overlapping utterances; the authors' own warning about overfitting makes this the main threat to the result.
Because both contextual BERT and order-insensitive NNLM averaging work, the model may be exploiting lexical overlap with grammar-file entries rather than deep understanding; a test with novel phrasings would separate these possibilities.
An extension the authors mention but do not try is fine-tuning BERT with data augmentation, which could push the grammar-free variants further; this is a plausible next step given their setup.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know about this paper is that its main claim—embedding-only classifiers can beat a grammar-file baseline for CALL text scoring—is genuinely new for this task but not actually settled by the evidence presented. Prior shared-task systems all leaned on the reference grammar; this one shows BERT and NNLM embeddings through a 128-neuron MLP scoring competitively without grammar features at inference. That is a real idea and the authors deserve credit for trying it and for being unusually candid about the data mess.

The paper does several things well. It documents the duplicate and overlap problems in the CALL v3 text data, including the fact that only about 300 unique non-overlapping utterances remain out of 1000. It warns that a system could appear acceptable while merely overfitting. It also describes preprocessing choices in enough detail to reproduce the pipeline. Those are assets.

The soft spots are real and load-bearing. First, the leaderboard scores in Table 1 were computed on the full contaminated set, so the margins over the grammar baseline (4.413, 4.403, 4.227 vs. 4.09) may largely reflect memorized training labels for repeated ASR transcripts rather than generalization. The paper has no decontaminated re-scoring, no confidence intervals, and no unique-entry analysis. Second, the conclusion reports Dfull scores of 4.192, 4.178, and 4.025 that do not match Table 1; that is a factual inconsistency a reader can't wave away. Third, the best submission (FFF) uses the updated grammar at inference, so the no-grammar claim rests on the second and third systems, DDD and EEE, which still beat the baseline. The grammar-free point is supported only if those scores are trustworthy, and the contamination problem directly threatens them. Fourth, early stopping on Dfull on the v2 test set is a form of tuning on a held-out set; it doesn't invalidate the result but it weakens the validation story.

Who is this for? Researchers working on CALL shared tasks and automated scoring of learner language. It's a useful data point and a good critique of the task design. I would not cite it as evidence that grammar-free scoring works, because the evaluation is too shaky. But I would bring it to a reading group to discuss evaluation contamination and shared-task design.

For peer review: yes, it deserves a serious referee, but only with major revision. The authors should re-score on a deduplicated, decontaminated set, fix the numerical inconsistency, and clearly separate the grammar-using and grammar-free systems. Without that, the headline claim stays conditional.

Referee Report

4 major / 5 minor

Summary. The paper describes a text-subtask scoring system for the CALL v3 shared task. The system uses pre-trained text embeddings (BERT and NNLM) concatenated into fixed vectors, feeds them into a shallow MLP classifier, and predicts one of three classes (correct, wrong language, wrong meaning). The authors claim that their submission was the only text-task entry to beat the grammar-file-based baseline, and that variants which do not use the grammar file at inference also surpass the baseline, demonstrating that grammar-free embedding-based scoring is competitive. The paper details data preprocessing, model training, validation on the v2 test set, and final results on the v3 test set, and argues that relying on the reference grammar makes CALL scoring systems non-scalable.

Significance. If the central claim holds, the result is significant for CALL: it suggests that a text-scoring system built entirely on generic embeddings plus a small classifier can match or beat a task-specific grammar reference, with obvious benefits for portability to new prompts and languages. The paper's strengths include the external shared-task evaluation, the use of standard and reproducible embedding models, a concrete description of data preparation, and an unusually candid discussion of dataset quality issues in Section 3.2. However, the reported evidence has load-bearing gaps: the effective test set is small and contaminated by duplicates and train/test overlap, no uncertainty quantification is given, and the numerical results in Section 6 do not match Table 1. These issues must be resolved before the central claim can be considered established.

major comments (4)

[Section 3.2 / Table 1] The authors state that after removing duplicates and entries overlapping with the training set, the 1000-utterance v3 test set contains 'only about 300 unique entries out of 1000' and warn about the 'danger to create a system with seemingly acceptable performance that would merely overfit the training dataset.' Nevertheless, the Dfull scores in Table 1 (FFF 4.413, DDD 4.403, EEE 4.227 vs. baseline 4.09) are computed on the full set, which includes repeated and training-overlapping transcriptions. Since the classifier takes fixed embedding vectors as input, test strings that also appear in training can effectively copy training labels rather than test generalization. Please report scores restricted to the unique, non-overlapping subset, or otherwise quantify how much of the margin over the baseline is contributed by duplicate and overlapping entries. Without such a decontaminated evaluation, the paper's central claim is not established.
[Section 6 vs. Table 1] The conclusions report Dfull scores of 4.192, 4.178, and 4.025 for the best, BERT+nnlm, and nnlm submissions, while Table 1 lists the corresponding highlighted text submissions as 4.413 (FFF), 4.403 (DDD), and 4.227 (EEE). Because the Dfull metric is presumably the same, these numbers cannot both be correct without an explanation. Please reconcile the two sets of numbers, state which ones are the official v3 leaderboard scores, and clarify the relationship between the anonymized submission IDs in Table 1 and the model names in Section 6.
[Abstract / Section 1 vs. Section 4.4 / Section 6] The abstract and Section 1 emphasize that the presented system 'does not rely on the reference grammar file for scoring,' but Section 4.4 and Section 6 identify the best-scoring submission as using 'nnlm and the updated grammar' (the nnlm+ model). Only the other two submissions (DDD and EEE) are actually grammar-free. Please state explicitly which of the three highlighted submissions are grammar-free and ensure that the abstract, introduction, and conclusions are not stronger than the evidence for a grammar-free system.
[Tables 1 and 2] No confidence intervals, error bars, or significance tests are reported for any of the Dfull differences. With an effective test size of roughly 300 unique utterances, a difference of 0.137 (EEE vs. baseline) or even 0.323 (FFF vs. baseline) may well be within sampling noise. Please provide at least bootstrap confidence intervals or a significance test on the unique subset, so that the reader can assess whether the claimed superiority over the grammar baseline is statistically reliable.

minor comments (5)

[Section 3.2] There is a typographical error: 'organazires' should be 'organizers'.
[Section 4.3] 'differnet' should be 'different' in the description of ensemble initialization.
[Table 1] The table caption says 'scores of our systems are highlighted,' but the highlighting is not visible in the text version of the paper; please add a visible formatting cue (bold or an asterisk).
[Section 4.2] The phrase 'the relatively high performance of this model is surprising' for nnlm would benefit from a brief discussion of why averaging word embeddings might still work well in this task, since the authors later rely on this result.
[Section 4.4] The sentence 'On the CALL v3 test set (Table 1), the model nnlm+ (FFF) achieves the best performance' is clear, but the following sentence should explicitly note that lower Dfull values are better or that the authors are using the convention from the shared task, to avoid confusion for readers not familiar with the metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the system is scored against the externally released CALL v3 test set and uses fixed public embeddings, with the no-grammar variants excluding the reference grammar at inference.

full rationale

The paper's central claim is an empirical leaderboard result on the shared task's externally scored test set (Table 1), not a derivation from fitted inputs. The no-grammar submissions (DDD, EEE) are trained on ASR transcripts and embeddings and do not consult the reference grammar at inference, so their reported superiority over the grammar baseline is not built into the model by construction. The best-scoring submission FFF does use the updated grammar, but the paper explicitly separates this variant from the grammar-free ones and the abstract and conclusion are clear about which systems use the grammar file. Training-data preparation (deduplication, removal of Group C, merging RecResult and Transcription, filtering via the baseline grammar) is a data-cleaning step, not a target-value injection. The v2 validation and the v3 test are external held-out outcomes, and the reported Dfull metric is the shared task's official measure. The paper's own warning in Section 3.2 that only about 300 unique non-overlapping entries remain in the test set is a generalization and validity concern about test-set contamination, not a circularity of the derivation: the model does not define its target in terms of its features, nor does any prediction reduce to a fitted parameter. No load-bearing self-citation or imported uniqueness claim is present; citations to previous shared-task systems are background. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central empirical claim rests on the assumed validity of the shared task labels and Dfull metric, on hand-made preprocessing decisions (excluding Group C, removing duplicates and ASR artifacts, merging transcript columns), and on the reliability of the updated grammar as a training filter. No new entities are introduced.

free parameters (4)

Hidden layer size of classifier = 128
Chosen by hand for the final neural network; no ablation reported.
Dropout rate and early stopping threshold
Used for regularization but exact values are not reported; selected by validation performance.
Ensemble composition
Ensemble averages models trained on different train subsets and random seeds; the number of models and the specific subsets are not given.
Preprocessing word-removal rules
The list of removed words ('yes', 'thanks', 'no', 'and', ASR fillers) and the rule to remove Group C are hand-set choices that affect training data and therefore scores.

assumptions (4)

domain assumption The Dfull metric and labels provided by the CALL shared task organizers are a valid measure of scoring quality.
Used throughout Section 1 and 4.4 to compare systems; if the metric is flawed, the ranking claim fails.
ad hoc to paper The Updated grammar file's judgments are reliable enough to use as training filters and in the nnlm+ scoring system.
Section 4.1 says the authors improved the grammar by removing entries with mistakes and then used it to include 'part of CALL 1st edition training dataset that was correctly judged by the baseline grammar'.
ad hoc to paper Excluding Group C and duplicated transcripts improves generalization rather than removing signal.
Section 3.1 states inclusion of Group C usually decreased accuracy; this motivates a data selection that is not independently validated.
domain assumption Merging RecResult and Transcription columns and treating ASR artifacts as removable preserves the label semantics.
Section 4.1: each entry is duplicated with the same judgment; this assumes the two ASR transcript forms are equivalent for text scoring.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Embedding-based system for the Text part of CALL v3 shared task." pith.science (2026). https://pith.science/paper/G5I7VO2I

@misc{pith2026190802505,
  author       = {Pith},
  title        = {Pith review of: Embedding-based system for the Text part of CALL v3 shared task},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G5I7VO2I}},
  note         = {Machine review of arXiv:1908.02505}
}

read the original abstract

This paper presents a scoring system that has shown the top result on the text subset of CALL v3 shared task. The presented system is based on text embeddings, namely NNLM~\cite{nnlm} and BERT~\cite{Bert}. The distinguishing feature of the given approach is that it does not rely on the reference grammar file for scoring. The model is compared against approaches that use the grammar file and proves the possibility to achieve similar and even higher results without a predefined set of correct answers. The paper describes the model itself and the data preparation process that played a crucial role in the model training.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 25 canonical work pages

[1]

the re- search for and study of applications of the computer in language teaching and learning

Introduction Computer Assisted Language Learning, or CALL, is “the re- search for and study of applications of the computer in language teaching and learning” [3]. However, rapid developments in technologies and machine learning methods in recent years have transformed CALL from a simple request-response system based on certain predeﬁne d rules to a compl...

work page 2016
[2]

For example , one of the last year’s submissions [15] processed the ASR out - put and up to 10 entries from the reference grammar ﬁle using the doc2vec model

Previous work All proposed solutions for CALL v1 and v2 relied heavily on the reference grammar ﬁle [9, 13, 14, 15, 16, 17, 18]. For example , one of the last year’s submissions [15] processed the ASR out - put and up to 10 entries from the reference grammar ﬁle using the doc2vec model. Afterwards, they used the word mover dis- tance to get 10 distances r...

work page
[3]

correct” or “incorrect

Dataset 3.1. Overview The data provided for the third edition of CALL shared task was collected from an online CALL tool used to help young Swiss German students improve their English ﬂuency. The training data was the same as the data provided for the second edition of the task. Each participant was asked to respond verbally i n English to a given German ...

work page 2019
[4]

can I pay with credit card

Text scoring system 4.1. Dataset resampling One of our key efforts was to form a high-quality training set . First, we improved the reference grammar ﬁle by removing a number of entries with mistakes. For example, there are 15 “can I pay with credit card” entries in test set for the second CALL shared task, as well as many similar ones like “I would like ...

work page
[5]

low hanging fruit

Discussion In our opinion, the allowance of grammar ﬁle renders text sub - task unattractive in comparison to audio subtask. The reaso n is that any increase in ASR performance would result in much more noticeable score improvements. The grammar ﬁle pro- vides a “low hanging fruit” that gives results that are hard t o improve upon. As a result, the work o...

work page
[6]

We also discussed the dataset and pro- posed changes to data formation routines for future competi - tions

Conclusions In this paper we presented a text-based scoring system for CALL v3 shared task. We also discussed the dataset and pro- posed changes to data formation routines for future competi - tions. Our best submission to the challenge obtained Df ull score of 4.192. The system achieved such result using nnlm and the updated grammar. Two other submission...

work page
[7]

Acknowledgments We would like to thank Andrey Osetrov for his valuable com- ments and suggestions

work page
[8]

Bengio, R

Y . Bengio, R. Ducharme, P . Vincent, C. Jauvin, ” A Neural P rob- abilistic Language Model,” in textitJournal of Machine Lea rning Research, 3:1137-1155, 2003

work page 2003

Show all 29 references

[9]

Devlin, M

J. Devlin, M. W. Chang, K. Lee K. Toutanova, ”BERT: Pre-tr aining of Deep Bidirectional Transformers for Language Understan ding,” in textitGoogle AI Language, 2018

2018
[10]

Computer-assisted language learning: Contex t and con- ceptualization.,

M. Levy, “Computer-assisted language learning: Contex t and con- ceptualization.,” in Oxford University Press., 1997

1997
[11]

Spoken grammar practice in an ASR-based CALL sys- tem,

B. Penning de Vries, S. Bodnar, C. Cucchiarini, H. Strik, and R. v.Hout, “Spoken grammar practice in an ASR-based CALL sys- tem,” in Speech and Language Technology in Education (SLaTE), Grenoble, France, pp. 60–65, 2013

2013
[12]

ASR-based CALL Systems and Learner Speech Data: New Resources and Opportunities for Research and Developmentin Sec- ond Language Learning,

C. Cucchiarini, S. Bodnar, B. Penning de Vries, R. V . Hout , and H. Strik, “ASR-based CALL Systems and Learner Speech Data: New Resources and Opportunities for Research and Developmentin Sec- ond Language Learning,” in EuropeanLanguage Resources Associ- ation (ELRA), Reykjavi...

2014
[13]

CALL-SLT: A Spoken CALL System Based on Grammarand Speech Recognition,

E. Rayner, N. Tsourakis, C. Baur, P . Bouillon, and J. Gerl ach, “CALL-SLT: A Spoken CALL System Based on Grammarand Speech Recognition,” Linguistic Issues in LanguageTechnology, vol. 10, no. 2, 2014

2014
[14]

Overview of the 2017 Spoken CALL Shared- Task,

C. Baur, C. Chua, J. Gerlach, M. Rayner, M. Russell, H. Strik, X. Wei, “Overview of the 2017 Spoken CALL Shared- Task,” in Proc. 7th ISCA W orkshop on Speech and Lan- guageTechnology in Education, pp. 71–78 , 2017. [Online]. Available:http://dx.doi.org/10.21437/SLaTE.2017-13

2017 doi
[15]

Overview of the 2018 Spoken CAL L Shared Task,

C. Baur, A. Caines, C. Chua, J. Gerlach, M. Qian, M. Rayner , M. Russell, H. Strik and X. Wei, “Overview of the 2018 Spoken CAL L Shared Task,” in Interspeech 2018, India Sep. 2018

2018
[16]

The University o f Birm- ingham 2017 SLaTE CALL Shared Task Systems,

M. Qian, X. Wei, P . Jancovic M. Russell, “The University o f Birm- ingham 2017 SLaTE CALL Shared Task Systems,” in Proceedings of the Seventh SLaTE W orkshop, Stockholm, Sweden 2017

2017
[17]

Magooda and D

A. Magooda and D. Litman, ”Syntactic and semantic featu res forhuman like judgement in spoken call,” in Proceedings of the Seventh SLaTE W orkshop,Stockholm, Sweden, 2017

2017
[18]

Y . R. Oh, H.-B. Jeon, H. J. Song, B. O. Kang, Y .-K. Lee, J.- G. Park, Y .-K. Lee, ”Deep-Learning based automatic spon-tane ous speech assessment in a data-driven approach for the 2017SLa TE CALL Shared Challenge,” in textitProceedings of the Sevent h SLaTE Workshop, Stockhol...

2017
[19]

Evanini, M

K. Evanini, M. Mulholland, E. Tsuprun, and Y . Qian, ”Usi ng anautomated content scoring system for spoken CALL respons es: The ETS submission for the Spoken CALL Challenge,” in textit - Proceedings of the Seventh SLaTE Workshop, Stockholm, Sweden, 2017

2017
[20]

D. Julg, M. Kunstek, C. Freimoser, K. Berkling, M. Qian, ”The CSU-K Rule-Based System for the 2nd Edition Spoken CALL SharedTask,” in textitInterspeech 2018, Hyderabad, India , 2018

2018
[21]

Nguyen, L

H. Nguyen, L. Chen, R. Prieto, C. Wang, and Y . Liu, ”Liuli shuo’s System for the Spoken CALL Shared Task 2018,” in textitInter - speech 2018, Hyderabad, India, 2018

2018
[22]

Freimoser, M

C. Freimoser, M. Kunstek, D. J¨ ulg, K. Berkling, M. Qian , ”The CSU-K DNN-Based System for the 2nd Edition Spoken CALL Shared Task,” 2018

2018
[23]

Ateeq, A

M. Ateeq, A. Hanani, A. Qaroush, ”An Optimization Based Ap- proach for Solving Spoken CALL Shared Task,” in textitInter - speech 2018, Hyderabad, India, 2018

2018
[24]

M. Qian, X. Wei, P . Jancovic, M. Russell, ”The Universit y of Birmingham 2018 Spoken CALL Shared Task Systems,” in tex- titInterspeech 2018, Hyderabad, India, 2018

2018
[25]

Evanini, M

K. Evanini, M. Mulholland, R. Ubale, Y . Qian, R. Pugh, V . Ra- manarayanan, A. Cahill, ”Improvements to an Automated Cont ent Scoring System for Spoken CALLResponses: The ETS Submis- sion to the Second Spoken CALL Shared Task,” in textitInterspeech 2018, Hyderabad, India, 2018

2018
[26]

M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, ”Deep contextualized word representa tions,” arXiv:1802.05365, 2018

2018 arXiv
[27]

Q. Le, T. Mikolov ”Distributed Representations of Sent ences and Documents,” arXiv:1405.4053v2 22 May 2014

2014 arXiv
[28]

Mikolov, K

T. Mikolov, K. Chen, G. Corrado, J. Dean, ”Efﬁcient Esti mation of Word Representations in V ector Space,” arXiv:1301.3781 v3 7 Sep 2013

2013 arXiv
[29]

D. Cer, Y . Y ang, S. -yi Kong, N. Hua, N. Limtiaco, R. St. John, N. Constant, M. Guajardo-C´ espedes, S. Y uan, C. Tar, Y .- H. Sung, B. Strope, R. Kurzweil, ”Universal Sentence Encode r,” arXiv:1803.11175, 2018

2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

the re- search for and study of applications of the computer in language teaching and learning

Introduction Computer Assisted Language Learning, or CALL, is “the re- search for and study of applications of the computer in language teaching and learning” [3]. However, rapid developments in technologies and machine learning methods in recent years have transformed CALL from a simple request-response system based on certain predeﬁne d rules to a compl...

work page 2016

[2] [2]

For example , one of the last year’s submissions [15] processed the ASR out - put and up to 10 entries from the reference grammar ﬁle using the doc2vec model

Previous work All proposed solutions for CALL v1 and v2 relied heavily on the reference grammar ﬁle [9, 13, 14, 15, 16, 17, 18]. For example , one of the last year’s submissions [15] processed the ASR out - put and up to 10 entries from the reference grammar ﬁle using the doc2vec model. Afterwards, they used the word mover dis- tance to get 10 distances r...

work page

[3] [3]

correct” or “incorrect

Dataset 3.1. Overview The data provided for the third edition of CALL shared task was collected from an online CALL tool used to help young Swiss German students improve their English ﬂuency. The training data was the same as the data provided for the second edition of the task. Each participant was asked to respond verbally i n English to a given German ...

work page 2019

[4] [4]

can I pay with credit card

Text scoring system 4.1. Dataset resampling One of our key efforts was to form a high-quality training set . First, we improved the reference grammar ﬁle by removing a number of entries with mistakes. For example, there are 15 “can I pay with credit card” entries in test set for the second CALL shared task, as well as many similar ones like “I would like ...

work page

[5] [5]

low hanging fruit

Discussion In our opinion, the allowance of grammar ﬁle renders text sub - task unattractive in comparison to audio subtask. The reaso n is that any increase in ASR performance would result in much more noticeable score improvements. The grammar ﬁle pro- vides a “low hanging fruit” that gives results that are hard t o improve upon. As a result, the work o...

work page

[6] [6]

We also discussed the dataset and pro- posed changes to data formation routines for future competi - tions

Conclusions In this paper we presented a text-based scoring system for CALL v3 shared task. We also discussed the dataset and pro- posed changes to data formation routines for future competi - tions. Our best submission to the challenge obtained Df ull score of 4.192. The system achieved such result using nnlm and the updated grammar. Two other submission...

work page

[7] [7]

Acknowledgments We would like to thank Andrey Osetrov for his valuable com- ments and suggestions

work page

[8] [8]

Bengio, R

Y . Bengio, R. Ducharme, P . Vincent, C. Jauvin, ” A Neural P rob- abilistic Language Model,” in textitJournal of Machine Lea rning Research, 3:1137-1155, 2003

work page 2003