RECORD
Learning from Partially Annotated Data: Example-aware Creation of Gap-filling Exercises for Language Learning
Computation and Language · Submitted 2 Jun 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We show that our model for this French grammar gap-filling exercise generation outperforms a competitive baseline classifier by 8% in F1 percentage points, achieving an average F1 score of 82%.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Since performing exercises (including, e.g., practice tests) forms a crucial component of learning, and creating such exercises requires non-trivial effort from the teacher, there is a great value in automatic exercise generation in digital tools in education. In this paper, we particularly focus on automatic creation of gapfilling exercises for language learning, specifically grammar exercises. Since providing any annotation in this domain requires human expert effort, we aim to avoid it entirely and explore the task of converting existing texts into new gap-filling exercises, purely based on an example exercise, without explicit instruction or detailed annotation of the intended grammar topics. We contribute (i) a novel neural network architecture specifically designed for aforementioned gap-filling exercise generation task, and (ii) a real-world benchmark dataset for French grammar. We show that our model for this French grammar gap-filling exercise generation outperforms a competitive baseline classifier by 8% in F1 percentage points, achieving an average F1 score of 82%. Our model implementation and the dataset are made publicly available to foster future research, thus offering a standardized evaluation and baseline solution of the proposed partially annotated data prediction task in grammar exercise creation.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Since performing exercises (including, e.g., practice tests) forms a crucial component of learning, and creating such exercises requires non-trivial effort from the teacher, there is a great value in automatic exercise generation in digital tools in education. | background 1.00 | 0.04 | 0.03 | asserted 1.00 |
| s1 | In this paper, we particularly focus on automatic creation of gapfilling exercises for language learning, specifically grammar exercises. | background 0.64 | 0.29 | 0.03 | asserted 0.90 |
| s2 | Since providing any annotation in this domain requires human expert effort, we aim to avoid it entirely and explore the task of converting existing texts into new gap-filling exercises, purely based on an example exercise, without explicit instruction or detailed annotation of the intended grammar topics. | method 0.69 | 0.27 | 0.05 | asserted 0.86 |
| s3 | We contribute (i) a novel neural network architecture specifically designed for aforementioned gap-filling exercise generation task, and (ii) a real-world benchmark dataset for French grammar. | contribution 1.00 | 0.95 | 0.05 | asserted 0.99 |
| s4 | We show that our model for this French grammar gap-filling exercise generation outperforms a competitive baseline classifier by 8% in F1 percentage points, achieving an average F1 score of 82%. | result 1.00 | 0.97 | 0.61 | asserted 1.00 |
| s5 | Our model implementation and the dataset are made publicly available to foster future research, thus offering a standardized evaluation and baseline solution of the proposed partially annotated data prediction task in grammar exercise creation. | future work 0.56 | 0.86 | 0.07 | asserted 0.97 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Since performing exercises (including, e.g., practice tests) forms a crucial component of learning, and creating such exercises requires non-trivial effort from the teacher, there is a great value in automatic exercise generation in digital tools in education. In this paper, we particularly focus on automatic creation of gapfilling exercises for language learning, specifically grammar exercises. Since providing any annotation in this domain requires human expert effort, we aim to avoid it entirely and explore the task of converting existing texts into new gap-filling exercises, purely based on an example exercise, without explicit instruction or detailed annotation of the intended grammar topics. We contribute (i) a novel neural network architecture specifically designed for aforementioned gap-filling exercise generation task, and (ii) a real-world benchmark dataset for French grammar. We show that our model for this French grammar gap-filling exercise generation outperforms a competitive baseline classifier by 8% in F1 percentage points, achieving an average F1 score of 82%. Our model implementation and the dataset are made publicly available to foster future research, thus offering a standardized evaluation and baseline solution of the proposed partially annotated data prediction task in grammar exercise creation.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 1 sentence without a confident label
Uses the method or tool of 1 paper
-
An input text $𝐭$, consisting of $N$ tokens $𝐭=[t_0, t_1, …,t_N-1]$ is encoded by a transformer based masked language model (MLM), in our experiments the multilingual [ref] .
Unsupervised Cross-lingual Representation Learning at Scale · Example-aware span detection model · 1.00
Cites as background 2 papers
-
More recently, [ref] , have developed sequence labeling model to automate the process of generating gap-filling exercises.
Learning to Automatically Generate Fill-In-The-Blank Quizzes · Introduction · 0.87
-
In terms of technology, recent developments in natural language processing, BERT [ref] , GPT-3 [ref] , InstructGPT [ref] , open up new opportunities for further upscaling and improving automatic generation of such tests/exercises.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · Introduction · 0.88
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.