Focusing on Students, not Machines: Grounded Question Generation and Automated Answer Grading

G\'er\^ome Meyer; Philip Breuer

REVIEW 6 major objections 6 minor 59 references

Focusing on Students, not Machines: Grounded Question Generation and Automated Answer Grading

T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A bachelor's thesis argues that large language models, led by GPT-4o, can grade short student answers as well as or better than specialized automated grading systems, and that layout-aware PDF chunking improves the grounding of generated…

desk verdict A useful, honest bachelor's thesis with a reusable ASAG benchmark and layout-aware chunking; the GPT-4o result holds up broadly, but the evaluation needs error bars and a sensitivity check on ordinal labels. read the letter →

arxiv 2506.12066 v1 pith:CXDTHMLA submitted 2025-06-02 cs.CL

Gérôme Meyer , Philip Breuer This is my paper

classification cs.CL

keywords automatedshortanswergradingquestiongenerationretrievalaugmenteddocumentlayoutanalysisPDFchunkinglargelanguagemodelsGPT-4oBloom'staxonomy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The thesis builds the foundation for a study platform that turns course PDFs into grounded questions and automatically grades short student answers. It claims three things: that chunking PDFs by visual layout using document layout analysis cuts context loss compared with page-based splitting and improves topic classification; that questions and reference answers generated from study materials plus learning objectives are high quality; and that on a new combined benchmark of seven answer-grading datasets, large language models, especially GPT-4o, grade short answers better than specialized grading systems, with performance growing as model size grows. The authors conclude that no tested system is yet mature enough for fully autonomous examination grading.

What carries the argument

Two mechanisms carry the argument. First, DLA chunking: a YOLOv8 model fine-tuned on DocLayNet labels PDF blocks by layout type (heading, list, table, text, and so on), blocks are associated with predicted instances via Intersection over Union, heading heights are clustered to find the document's hierarchy, and chunks break at the largest headings, falling back to fixed-size windows when needed. This preserves semantic cohesion across pages. Second, the combined ASAG2024 benchmark: seven public grading datasets are normalized to a 0-1 scale and scored with a range-weighted MAE and RMSD that gives each grade range equal weight, counteracting the skewed grade distributions that let trivial mean-prediction baselines look competitive.

What would settle it

Take a stratified sample of answers from Beetle II and SciEntsBank, have human raters assign continuous scores on the same 0-3 scale instead of picking categories, and recompute the weighted MAE and RMSD for GPT-4o and the baseline; if the rank order flips or the gap closes, the equal-interval conversion was load-bearing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that automated short-answer grading is a generalizable capability of large pretrained language models rather than a specialty that requires task-specific fine-tuning. The thesis's benchmark evaluation shows GPT-4o reaching the lowest weighted mean absolute error of 0.21 and weighted RMSD of 0.27 across the seven datasets, outperforming both the mean-grade baseline and specialized systems such as a BART model fine-tuned on the SAF dataset, which collapses to predicting the most common grade. Scale is the driver: Llama-3-8B barely clears the baseline, GPT-3.5-turbo improves substantially, and GPT-4o leads. The thesis also claims that its layout-aware chunking achieves zero lost contexts on the annotated 45-page document, matching optimal chunking, while single-page chunking lost 27 contexts.

Load-bearing premise

The comparison assumes that the categorical grades in Beetle II and SciEntsBank can be converted to equally spaced numeric values (0, 1, 2, 3) that behave like a continuous scale; if the true distance between 'partially correct' and 'correct' differs from the distance between 'incorrect' and 'partially correct', the error metrics misrepresent every system's accuracy on those datasets.

Editorial extensions

If this is right

In self-study and practice settings, an LLM-based grader can provide immediate feedback on open-ended answers without retraining, at an error level comparable to or better than the best specialized systems on this benchmark.
Model scale, not prompt or task specialization, is the main driver of grading quality, so practitioners should expect larger general-purpose models to keep improving automated short-answer grading performance as they are released.
Specialized fine-tuned graders that are not validated on target data can collapse to in-distribution predictions; the benchmark's weighted metrics expose this in a way raw MAE does not.
Layout-aware chunking reduces context loss in RAG pipelines for PDF documents, which should improve the grounding and relevance of generated questions.
A human-in-the-loop remains necessary for examinations because even the best system's RMSD on the SAF dataset of 0.27 is far from the human annotator agreement of 0.09.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The equal-interval conversion of categorical labels (incorrect, partially correct, correct) in Beetle II and SciEntsBank is a substantive modeling choice; a future benchmark could test ordinal metrics or ask human raters to assign continuous scores to a sample to see whether the ranking of systems survives.
The weighting scheme's 0.1 bins are a parameter; other bin widths could change the relative standing of systems whose errors cluster near grade boundaries.
The same weighting idea could transfer to other imbalanced automated scoring tasks, such as essay scoring or clinical note grading, where predicting the majority class inflates apparent accuracy.
DLA chunking's benefit is demonstrated on one 45-page document; the claim that it generalizes across PDF types would be strengthened by testing on slides, multi-column papers, and scanned documents with OCR.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this is a bachelor's thesis, and it reads like one—a bit wide, some rough edges—but it has two genuinely useful pieces. The combined ASAG2024 benchmark, with seven normalized datasets and a grade-balancing weighted metric, is a real contribution to ASAG evaluation. The DLA chunking method, using YOLOv8/DocLayNet layout instances and IoU-based block association, is a concrete implementation of an idea that until now was mostly a blog post. They also credit the blog post that inspired the chunking and acknowledge GradeAid's overlapping effort, so the novelty claims are not overstated. I'd use both.

The grading evaluation is honestly done. The authors show the mean baseline and unweighted metrics, then explain why those are misleading and introduce the weighted metric. That's the right instinct. They also report the BART model's failure mode (predicting grade 1 everywhere) rather than hiding it. The conclusion that LLM grading improves with scale and that current systems still need human oversight is supported by their data.

Soft spots, in proportion. The ordinal-to-interval conversion for Beetle and SciEntsBank is the one the stress-test flagged, and it is a real assumption: treating 'contradictory' as equidistant to 'irrelevant' and 'partially correct' is not self-evident. That said, I don't think it overturns the headline. In the weighted MAE tables, GPT-4o and GPT-3.5 are tied or near-tied on exactly those two datasets; the GPT-4o advantage comes from DigiKlausur, Mohler, SAF, and Stita. So the concern is real but not load-bearing for the central comparison. Still, a sensitivity analysis with alternative monotone mappings should be added before the benchmark is adopted.

The other two soft spots are missing error bars (no confidence intervals or significance tests anywhere) and the question-generation evaluation: 75 samples per model, one rater, no inter-annotator agreement, and a post-hoc exclusion of bad chunks after the fact. The authors flag the exclusion themselves, which is good, but it means the question-quality numbers should be read as indicative, not conclusive. The weighting scheme was also introduced after seeing the baseline parity; that's an understandable response but it should be labeled as post-hoc.

Who is this for? People building RAG chunkers or benchmarking ASAG systems. It's not a breakthrough paper, but it's a solid, honest engineering contribution with a reusable artifact. I'd send it to review, with a request for the sensitivity analysis and error bars. A serious referee would get value from it.

Referee Report

6 major / 6 minor

Summary. The manuscript presents a two-part system for educational support: (1) grounded question generation from PDF course materials using document layout analysis (DLA) chunking, topic classification via embeddings, and LLM-based generation of questions and reference answers; and (2) automated short-answer grading, evaluated on a newly assembled benchmark, ASAG2024, that combines seven existing datasets into a normalized grade scale. The grading evaluation compares a mean baseline, a semantic-similarity baseline, two specialized grading systems (BART-SAF and PrometheusII-7B), and three LLMs (Llama-3-8B, GPT-3.5-turbo, GPT-4o), reporting per-dataset MAE, RMSD, and weighted versions. The authors conclude that GPT-4o achieves the best weighted error metrics, that larger LLMs generalize better to grading, and that current systems still require human oversight.

Significance. If the results hold, the paper makes a useful contribution by releasing a combined benchmark for automated short-answer grading and by providing a controlled comparison across several system families. The per-dataset reporting and the explicit attempt to address grade imbalance through a weighting scheme are positive features, as is the use of external, publicly available datasets rather than fitted internal quantities. The question-generation work is less central but offers a reasonable engineering contribution. However, the benchmark's validity depends on several assumptions that are not tested, and the evaluation lacks statistical rigor, so the headline numerical claims should be treated as provisional.

major comments (6)

[§5.1.4–§5.1.5, Tables 12 and 14] The conversion of the Beetle II and SciEntsBank categorical labels ('irrelevant', 'contradictory', 'partially correct', 'correct') to the equal-interval numeric scale (0, 1, 2, 3) is load-bearing for the headline comparison, because these two datasets contribute two of the seven per-dataset scores that are averaged in the reported means. The paper provides no evidence that the distances between adjacent categories are equal, and the specific ordering is debatable (e.g., assigning 'contradictory' a higher score than 'irrelevant'). Because MAE and RMSD are linear functions of this encoding, any monotone remapping, such as (0, 1, 3, 4), changes every system's per-dataset error and could alter the aggregate ranking. A sensitivity analysis over plausible monotone encodings, or the use of ordinal-appropriate metrics, is needed before the numerical superiority claim for GPT-4o can be accepted.
[§5.1, Tables 11–14] The DigiKlausur dataset appears throughout the results (Tables 11–14, Figures 9–10) but is never described in Section 5.1. Its original grade scale, provenance, preprocessing, and any normalization steps are missing. Without this information, the benchmark is not reproducible and the reported aggregate numbers are not fully interpretable. This is a straightforward completeness issue that must be fixed.
[§5.2.4, Figure 11] The LLM prompt instructs models to 'grade it with a score of 0, 1, 2, 3, 4 or 5', but all benchmark grades are normalized to the [0, 1] interval. The paper does not state how the model's 0–5 output was mapped to the 0–1 scale used in the MAE/RMSD calculations. Similarly, Section 5.1.6 states that CU-NLP does not contain the original questions, yet the prompt in Figure 11 unconditionally injects a 'Question' field; the handling of this missing input is not described. Both details are essential for the evaluation to be well-defined.
[§5.3.2, Figure 12] The weighting scheme that produces the headline wMAE and wRMSD values is introduced without a principled justification for why equalizing the total weight across ten 0.1-wide grade bins yields an unbiased comparison. The scheme also has arbitrary elements, such as the bin width and the redistribution of weight from empty bins. Since the unweighted results in Table 13 show the baseline, Nomic-embed-text-v1, and GPT-4o tied at 0.31, and the weighted metric is introduced only after observing this parity, the independence of the main comparison is weakened. At minimum, the authors should justify the weighting choice on independent grounds and report sensitivity to the bin width.
[§4.2.3, Tables 7–8] The question-generation evaluation is based on 75 randomly sampled questions per model, assessed by a single annotator with no reported inter-annotator agreement. Table 8 then recomputes scores after removing 10 questions for Llama-3-8B and 20 questions for GPT-4o because they originated from 'a select few chunks', but the removal criterion is applied asymmetrically and post hoc. This makes the adjusted comparison between models unfair and weakens the claim that DLA chunking produces high-quality questions. The paper should report the original annotation protocol, include at least a second annotator or a clear adjudication procedure, and justify any exclusion rule before presenting adjusted scores.
[§5.3, Tables 11–14] No confidence intervals, standard errors, or significance tests are reported for any of the grading comparisons. Given that the main differences among GPT-3.5-turbo and GPT-4o are small (e.g., weighted MAE 0.23 vs. 0.21 and weighted RMSD 0.30 vs. 0.27), the claim that 'the scale of the LLM is the main driver of grading quality' is not supported without some measure of uncertainty, especially since the per-dataset scores show considerable overlap across systems.

minor comments (6)

[§5.3.1] The MAE formula is missing the absolute value; it should read |y_i − ŷ_i|, and the weighted version in §5.3.2 has the same omission.
[Figure 7] The 'adjusted F1-score' is not defined in the text or caption. Please specify how it is computed and how it differs from the standard F1-score.
[§5.1, Tables 11–14] The 'Mean' row in each results table is an unweighted average of the per-dataset scores, not a dataset-size-weighted mean. This should be stated explicitly in the table captions or in the text to avoid confusion.
[References] References [12] and [17] are identical; please merge or renumber them.
[§4.3] The sentence 'DLA chunking performs up to 28% above the naive approach baseline' refers to Figure 7, but it is not clear whether this is the standard or adjusted F1-score for which model and which comparison; please clarify.
[§4.2.1, Table 3] The caption says 'classification report of the predicted block types', but the rows for 'caption' and 'picture' have very low support; this is not a problem, but the low support should be acknowledged when interpreting the F1-scores.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluations are empirical comparisons against external datasets and manual annotations, with no fitted quantity renamed as a prediction.

full rationale

The paper's central results are empirical evaluations rather than derivations from self-defined quantities. The ASAG2024 benchmark is a collection of seven existing datasets, and the headline comparisons report MAE/RMSD of various grading systems on those external datasets. The conversion of categorical labels in Beetle II and SciEntsBank to a numeric 0-3 scale (Section 5.1.4) is an explicit methodological assumption, not a circular reduction: no system's output is defined in terms of that encoding, and the limitation paragraphs (5.4.1) note validity concerns without claiming to derive the encoding from the results. The weighted metric introduced in Section 5.3.2 is a fixed rule (equal weight per 0.1-interval decile, empty intervals redistributed equally), not a parameter fitted to make GPT-4o win; it is applied uniformly to all systems and datasets. The DLA chunking evaluation compares against manual annotations of one document, and the question-generation assessment is a human evaluation of generated outputs; neither step fits a parameter and then re-predicts that same parameter. No load-bearing step relies on a self-citation: the cited prior work on SAF, Beetle II, and embedding models is external, and the thesis authors' own benchmark is the contribution being introduced rather than a presupposed authoritative result. The equal-interval encoding and the post-hoc adoption of weighting after seeing unweighted baseline parity (Table 13) are legitimate methodological choices subject to correctness scrutiny, but they do not make any prediction equivalent to its input by construction. Therefore no circularity is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central evaluation rests on the reliability of pre-existing datasets and the chosen normalization and weighting. The chunking depends on layout model quality and learning objective availability. No new theoretical entities are introduced; the main free parameters are thresholds left unspecified.

free parameters (4)

z-score threshold for topic filtering = not specified
Used to filter irrelevant chunks in §4.1.2; the value is not reported, so the classifier behavior is not fully reproducible.
minimum chunk length threshold = not specified
In §4.1.1, too-small chunks are appended or deleted based on a threshold that is not quantified.
Bloom's taxonomy level selection = Remember, Understand, Apply
Choosing only the first three levels restricts question generation and grading to short answers, a design decision that shapes the entire system.
grade weighting decile count = 10
The new weighting scheme in §5.3.2 assigns equal total weight to each 0.1-wide grade band; the number of bands (10) is a methodological choice that affects the ranking of systems.

assumptions (5)

domain assumption A pre-trained YOLOv8 model fine-tuned on DocLayNet accurately identifies layout instances in the target PDF.
Invoked in §4.1.1; if layout detection fails, the chunking degenerates to fixed-size windows, as the authors admit in §4.3.1.
domain assumption Learning objectives are available and accurately define relevant topics.
Stated in §3.1; both question relevance filtering and topic classification depend on these objectives.
domain assumption Human grades across the seven datasets are reliable and can be normalized to a common 0-1 scale.
The combined benchmark in §5.1 assumes that different grading rubrics and categorical mappings (e.g., Beetle II's four classes) are commensurable after linear scaling.
ad hoc to paper The 10-bin weighting scheme gives an unbiased comparison of grading systems.
Introduced in §5.3.2; it redistributes weight to rare grade ranges and is the main methodological novelty, but it is not derived from any theory of grading fairness.
domain assumption The prompt shown in Figure 11 is neutral across LLMs.
§5.2.4 states the same prompt was used for all models, but no evidence is provided that smaller models are not disadvantaged by the instruction style.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Focusing on Students, not Machines: Grounded Question Generation and Automated Answer Grading." pith.science (2026). https://pith.science/paper/CXDTHMLA

@misc{pith2026250612066,
  author       = {Pith},
  title        = {Pith review of: Focusing on Students, not Machines: Grounded Question Generation and Automated Answer Grading},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CXDTHMLA}},
  note         = {Machine review of arXiv:2506.12066}
}

read the original abstract

Digital technologies are increasingly used in education to reduce the workload of teachers and students. However, creating open-ended study or examination questions and grading their answers is still a tedious task. This thesis presents the foundation for a system that generates questions grounded in class materials and automatically grades student answers. It introduces a sophisticated method for chunking documents with a visual layout, specifically targeting PDF documents. This method enhances the accuracy of downstream tasks, including Retrieval Augmented Generation (RAG). Our thesis demonstrates that high-quality questions and reference answers can be generated from study material. Further, it introduces a new benchmark for automated grading of short answers to facilitate comparison of automated grading systems. An evaluation of various grading systems is conducted and indicates that Large Language Models (LLMs) can generalise to the task of automated grading of short answers from their pre-training tasks. As with other tasks, increasing the parameter size of the LLMs leads to greater performance. Currently, available systems still need human oversight, especially in examination scenarios.

Figures

Figures reproduced from arXiv: 2506.12066 by the authors.

**Figure 2.** A mockup of the question generation user interface This section provides an overview of the learning platform that influenced the design decisions for this thesis. Existing learning platforms feature a straightforward, albeit time-consuming, approach. Students or Teachers can manually create flashcards with terms and definitions. After creating a list of cards, they can be answered and self-assessed. Depending on th… view at source ↗

**Figure 3.** Illustration of context loss in fixed window and page break chunking A naive but straightforward approach is to split a document based on a fixed-size window. This could be multiple blocks or entire pages. As seen on the left in [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗

Figures from the paper (12 more)

**Figure 4.** Figure 4: PDF blocks in grey overlaid with a few examples of layout structures, including their types and bounding boxes. This requires a model that performs Document Layout Analysis (DLA) to extract the bounding boxes of individual layout structures like headings or paragraphs.…

**Figure 5.** Figure 5: LLM prompt for question generation The prompt in [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]

**Figure 6.** Figure 6: LLM prompt for reference answer generation The prompt generating the reference answer is shown in [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

**Figure 7.** Figure 7: Comparison of different chunking algorithms and embedding models for topic classification 4.2.3. Question and Reference Answer The question and their reference answer quality of GPT-4o and Llama-3-8B are evaluated manually based on the criteria outlined in [PITH_FULL_…

**Figure 8.** Figure 8: Overview diagram of the answer grading step [PITH_FULL_IMAGE:figures/full_fig_p031_8.png]

**Figure 9.** Figure 9: The datasets that make up the combined ASAG2024 benchmark together with the number of answers, reference answer, and grade triplets. The grades in the different datasets are usually not distributed evenly (see [PITH_FULL_IMAGE:figures/full_fig_p032_9.png]

**Figure 10.** Figure 10: Box plot of the grade distribution with the grade scaled between 0 and 1. The DigiKlausur and SAF datasets have a median grade of 1. These two datasets also contribute a large portion of the total number of entries (as seen in [PITH_FULL_IMAGE:figures/full_fig_p033_10.png]

**Figure 11.** Figure 11: Prompt used for the evaluation of Llama-3-8b, GPT-3.5-turbo and GPT-4o. The three variables “question”, “reference_answer” and “provided_answer” marked in blue are injected for each prediction with the corresponding values. 5.3. Results Two metrics are reported. RMSD …

**Figure 12.** Figure 12: , so that the total weight sums up to 100% [PITH_FULL_IMAGE:figures/full_fig_p040_12.png]

**Figure 13.** Figure 13: Grades predicted on the different datasets by the BART model trained on the SAF data. On a variety of datasets, it predicts only grades of one, which is the most common grade in its original training data. While there is a trend for generalist systems to become better…

**Figure 1.** Figure 1: Illustration of text being translated into vectors via an embedding model. ..................................................................................................................... 4 [PITH_FULL_IMAGE:figures/full_fig_p051_1.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 52 canonical work pages

[1]

D. A. Petrusevich, ‘Modern trends in the digitalization of education’, Journal of Physics: Conference Series, vol. 1691, 2020, [Online]. Available: https://api.sem anticscholar.org/CorpusID:230650493

work page 2020
[2]

P. C. Brown, H. L. Roediger III, and M. A. McDaniel, Make it stick: The science of successful learning. Harvard University Press, 2014

work page 2014
[3]

Semantic Similarity

System Design Figure 2: A mockup of the question generation user interface This section provides an overview of the learning platform that inﬂuenced the design decisions for this thesis. Existing learning platforms feature a straightforward, albeit time-consuming, approach. Students or Teachers can manually create ﬂashcards with terms and deﬁnitions. Afte...

work page 2023
[4]

context loss

Question Generation This section describes the steps for generating questions and their reference answers that are eﬀective, grounded, accurate and relevant. To ground questions in the study material, documents need to be fed to LLMs. However, since LLMs are constrained by speciﬁc context window sizes, documents cannot be passed in their entirety. Models ...

work page
[5]

grading rubrics

Answer Grading Figure 8: Overview diagram of the answer grading step Figure 8 shows the diﬀerent input and output parts of the grading task: The reference answer and provided student answer (from here on provided answer) are taken as input to the grading system which returns a grade. Optionally, the question can be included in the input, which has been sh...

work page 2024
[6]

Using Document Layout Structure for Eﬃcient RAG

Related Work This section gives insight into similar and concurrent research in the ﬁelds of question generation and automated grading. 6.1. RAG Document Chunking RAG is a new and rapidly evolving area of research, with numerous novel approaches emerging. The unique approach to using the document’s layout for chunking explored in this thesis was inspired ...

work page
[7]

question

Conclusion Our thesis presents a more sophisticated approach for chunking documents with a visual layout, such as PDF documents, for downstream tasks, including RAG. We show that questions and reference answers generated with a combination of study material and learning objectives are of high quality. Furthermore, a AQG system with LLMs that receive instr...

work page
[8]

del Gobbo, A

E. del Gobbo, A. Guarino, B. Cafarelli, L. Grilli, and P. Limone, ‘Automatic evaluation of open-ended questions for online learning. A systematic mapping’, Studies in Educational Evaluation, vol. 77, p. 101258–101259, 2023, doi: https:// doi.org/10.1016/j.stueduc.2023.101258

arXiv 2023

Show all 59 references

[9]

Mohler, R

M. Mohler, R. Bunescu, and R. Mihalcea, ‘Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments’, in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, D. Li...

2011
[10]

Filighera, S

A. Filighera, S. Parihar, T. Steuer, T. Meuser, and S. Ochs, ‘Your Answer is Incorrect… Would you like to know why? Introducing a Bilingual Short Answer Feedback Dataset’, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

2022
[11]

Dzikovska, N

M. Dzikovska, N. Steinhauser, E. Farrow, J. Moore, and G. Campbell, ‘BEETLE II: Deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics’, International Journal of Artiﬁcial Intelligence in Education, v...

2014
[12]

R. D. Nielsen, W. H. Ward, J. H. Martin, and M. Palmer, ‘Annotating Students’ Understanding of Science Concepts’, in International Conference on Language Resources and Evaluation, 2008. [Online]. Available: https://api.semanticscho lar.org/CorpusID:12938607

2008
[13]

C. N. Tulu, O. Ozkaya, and U. Orhan, ‘Automatic Short Answer Grading With SemSpace Sense Vectors and MaLSTM’, IEEE Access, vol. 9, no. , pp. 19270– 19280, 2021, doi: 10.1109/ACCESS.2021.3054346. 46

2021
[14]

del Gobbo, A

E. del Gobbo, A. Guarino, B. Cafarelli, and L. Grilli, ‘GradeAid: a framework for automatic short answers grading in educational contexts—design, implementation and evaluation’, Knowledge and Information Systems, vol. 65, no. 10, pp. 4295–4334, Oct. 2023, doi: 10.1007/s10115-0...

2023 doi
[15]

Hu et al., ‘Do Large Language Models Know about Facts?’

X. Hu et al., ‘Do Large Language Models Know about Facts?’. 2023

2023
[16]

Wei et al., ‘Emergent Abilities of Large Language Models’, ArXiv, 2022, [Online]

J. Wei et al., ‘Emergent Abilities of Large Language Models’, ArXiv, 2022, [Online]. Available: https://api.semanticscholar.org/CorpusID:249674500

2022
[18]

Radford, K

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, and others, ‘Improving language understanding by generative pre-training’, 2018

2018
[19]

Nussbaum, J

Z. Nussbaum, J. X. Morris, B. Duderstadt, and A. Mulyar, ‘Nomic Embed: Training a Reproducible Long Context Text Embedder’. 2024

2024
[20]

Steck, C

H. Steck, C. Ekanadham, and N. Kallus, ‘Is cosine-similarity of embeddings really about similarity?’, in Companion Proceedings of the ACM on Web Conference 2024, 2024, pp. 887–890

2024
[21]

Sahoo, A

P. Sahoo, A. K. Singh, S. Saha, V. Jain, S. Mondal, and A. Chadha, ‘A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications’, arXiv preprint arXiv:2402.07927, 2024

2024 arXiv
[22]

Radford et al., ‘Language models are unsupervised multitask learners’, OpenAI blog, vol

A. Radford et al., ‘Language models are unsupervised multitask learners’, OpenAI blog, vol. 1, no. 8, p. 9–10, 2019

2019
[23]

Wei et al., ‘Chain-of-thought prompting elicits reasoning in large language models’, Advances in neural information processing systems, vol

J. Wei et al., ‘Chain-of-thought prompting elicits reasoning in large language models’, Advances in neural information processing systems, vol. 35, pp. 24824– 24837, 2022

2022
[24]

Lewis et al., ‘Retrieval-augmented generation for knowledge-intensive nlp tasks’, Advances in Neural Information Processing Systems, vol

P. Lewis et al., ‘Retrieval-augmented generation for knowledge-intensive nlp tasks’, Advances in Neural Information Processing Systems, vol. 33, pp. 9459– 9474, 2020

2020
[25]

Bienz, R

T. Bienz, R. Cohn, and C. Adobe Systems (Mountain View, Portable document format reference manual. Addison-Wesley Boston^ eMA MA, 1993

1993
[26]

L. W. Anderson, A taxonomy for learning, teaching, and assessing : a revision of Bloom's taxonomy of educational objectives, Complete ed. New York: Longman, 2001

2001
[27]

Elkins, E

S. Elkins, E. Kochmar, J. C. K. Cheung, and I. Serban, ‘How Useful are Educational Questions Generated by Large Language Models?’. 2023. 47

2023
[28]

N. F. Liu et al., ‘Lost in the Middle: How Language Models Use Long Contexts’. 2023

2023
[29]

D. Reis, J. Kupec, J. Hong, and A. Daoudi, ‘Real-Time Flying Object Detection with YOLOv8’. 2023

2023
[30]

Redmon, S

J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, ‘You Only Look Once: Uniﬁed, Real-Time Object Detection’. 2016

2016
[31]

Pﬁtzmann, C

B. Pﬁtzmann, C. Auer, M. Dolﬁ, A. S. Nassar, and P. Staar, ‘DocLayNet: A Large Human-Annotated Dataset for Document-Layout Segmentation’, in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, in KDD ’22. ACM, Aug. 2022. doi: 10.1145/3534678.3539043

2022
[32]

[Online]

Maik Thiele, ‘ documentlayoutsegmentation_YOLOv8_ondoclaynet (Revision 25486d5) ’. [Online]. Available: https://huggingface.co/DILHTWD/documen tlayoutsegmentation_YOLOv8_ondoclaynet
[33]

L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei, ‘Multilingual E5 Text Embeddings: A Technical Report’, arXiv preprint arXiv:2402.05672, 2024

2024 arXiv
[34]

W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, ‘MiniLM: Deep Self- Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers’. 2020

2020
[35]

Lightman et al., ‘Let's Verify Step by Step’, ArXiv, 2023, [Online]

H. Lightman et al., ‘Let's Verify Step by Step’, ArXiv, 2023, [Online]. Available: https://api.semanticscholar.org/CorpusID:258987659

2023
[36]

Doddapaneni, G

S. Doddapaneni, G. Ramesh, M. M. Khapra, A. Kunchukuttan, and P. Kumar, ‘A Primer on Pretrained Multilingual Language Models’. 2021

2021
[37]

Raﬀel et al., ‘Exploring the Limits of Transfer Learning with a Uniﬁed Text- to-Text Transformer’, Journal of Machine Learning Research, vol

C. Raﬀel et al., ‘Exploring the Limits of Transfer Learning with a Uniﬁed Text- to-Text Transformer’, Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020, [Online]. Available: http://jmlr.org/papers/v21/20-074.html

2020
[38]

L. Xue et al., ‘mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer’, in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, K. Toutanova, A. Rumshisky, L. Zettlemoyer, D...

2021 doi
[39]

Caines et al., ‘The Teacher-Student Chatroom Corpus’, in Proceedings of the 9th Workshop on NLP for Computer Assisted Language Learning, D

A. Caines et al., ‘The Teacher-Student Chatroom Corpus’, in Proceedings of the 9th Workshop on NLP for Computer Assisted Language Learning, D. Alfter, E. Volodina, I. Pilan, H. Lange, and L. Borin, Eds., Gothenburg, Sweden: LiU 48 Electronic Press, Nov. 2020, pp. 10–20. [Onlin...

2020
[40]

Pilan, J

I. Pilan, J. Lee, C. Y. Yeung, and J. Webster, ‘A Dataset for Investigating the Impact of Feedback on Student Revision Outcome’, in Proceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S...

2020
[41]

M. A. Sultan, C. Salazar, and T. Sumner, ‘Fast and easy short answer grading with high accuracy’, in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1070–1075

2016
[42]

Ruan et al., ‘Quizbot: A dialogue-based adaptive learning system for factual knowledge’, in Proceedings of the 2019 CHI conference on human factors in computing systems, 2019, pp

S. Ruan et al., ‘Quizbot: A dialogue-based adaptive learning system for factual knowledge’, in Proceedings of the 2019 CHI conference on human factors in computing systems, 2019, pp. 1–13

2019
[43]

C. Sung, T. I. Dhamecha, and N. Mukhi, ‘Improving short answer grading using transformer-based pre-training’, in Artiﬁcial Intelligence in Education: 20th International Conference, AIED 2019, Chicago, IL, USA, June 25-29, 2019, Proceedings, Part I 20, 2019, pp. 469–481

2019
[44]

S. Basu, C. Jacobs, and L. Vanderwende, ‘Powergrading: a clustering approach to amplify human eﬀort for short answer grading’, Transactions of the Association for Computational Linguistics, vol. 1, pp. 391–402, 2013

2013
[45]

M. Dzikovska et al., ‘SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge’, in Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic E...

2013
[46]

Muennighoﬀ, N

N. Muennighoﬀ, N. Tazi, L. Magne, and N. Reimers, ‘MTEB: Massive Text Embedding Benchmark’. 2023

2023
[47]

Lewis et al., ‘BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension’

M. Lewis et al., ‘BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension’. 2019

2019
[48]

Kim et al., ‘Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models’

S. Kim et al., ‘Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models’. 2024. 49

2024
[49]

Wijanto and H

M. Wijanto and H. Yong, ‘Combining Balancing Dataset and SentenceTransformers to Improve Short Answer Grading Performance’, Applied Sciences, 2024, [Online]. Available: https://api.semanticscholar.org/ CorpusID:270072875

2024
[50]

Frost, ‘Root Mean Square Error (RMSE)’

J. Frost, ‘Root Mean Square Error (RMSE)’. Accessed: May 31, 2024. [Online]. Available: https://statisticsbyjim.com/regression/root-mean-square- error-rmse/

2024
[51]

Chang and F

L. Chang and F. Ginter, ‘Automatic Short Answer Grading for Finnish with ChatGPT’, Proceedings of the AAAI Conference on Artiﬁcial Intelligence, vol. 38, no. 21, pp. 23173–23181, Mar. 2024, doi: 10.1609/aaai.v38i21.30363

2024 doi
[52]

Borji, ‘A Categorical Archive of ChatGPT Failures’, ArXiv, 2023, [Online]

A. Borji, ‘A Categorical Archive of ChatGPT Failures’, ArXiv, 2023, [Online]. Available: https://api.semanticscholar.org/CorpusID:256627571

2023
[53]

A. Abid, M. Farooqi, and J. Y. Zou, ‘Persistent Anti-Muslim Bias in Large Language Models’, Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021, [Online]. Available: https://api.semanticscholar.org/ CorpusID:231603388

2021
[54]

Nadeem, A

M. Nadeem, A. Bethke, and S. Reddy, ‘StereoSet: Measuring stereotypical bias in pretrained language models’, in Annual Meeting of the Association for Computational Linguistics, 2020. [Online]. Available: https://api.semanticsch olar.org/CorpusID:215828184

2020
[55]

P. P. Liang, C. Wu, L.-P. Morency, and R. Salakhutdinov, ‘Towards Understanding and Mitigating Social Biases in Language Models’, in International Conference on Machine Learning, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:235623756

2021
[56]

Welbl et al., ‘Challenges in Detoxifying Language Models’, ArXiv, 2021, [Online]

J. Welbl et al., ‘Challenges in Detoxifying Language Models’, ArXiv, 2021, [Online]. Available: https://api.semanticscholar.org/CorpusID:237513578

2021
[57]

Filighera, ‘Automatic Short Answer Grading Using Neural Models

A. Filighera, ‘Automatic Short Answer Grading Using Neural Models. Examining Adversarial Robustness and Elaborated Feedback Generation’, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:267201224

2023
[58]

Filighera, T

A. Filighera, T. Steuer, and C. Rensing, ‘Fooling Automatic Short Answer Grading Systems’, in Artiﬁcial Intelligence in Education, I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, and E. Millán, Eds., Cham: Springer International Publishing, 2020, pp. 177–190

2020
[59]

Sukla, ‘Using Document Layout Structure for Eﬃcient RAG’

A. Sukla, ‘Using Document Layout Structure for Eﬃcient RAG’. Accessed: May 31, 2024. [Online]. Available: https://ambikasukla.substack.com/p/eﬃcient- rag-with-document-layout 50

2024
[60]

Steuer, ‘Automatic Question Generation to Support Reading Comprehension of Learners - Content Selection, Neural Question Generation, and Educational Evaluation’, 2023

T. Steuer, ‘Automatic Question Generation to Support Reading Comprehension of Learners - Content Selection, Neural Question Generation, and Educational Evaluation’, 2023. doi: https://doi.org/10.26083/tuprints-00023032. 51 B. Appendix B.1. Oﬃcial Assignment Focusing on Student...

2023 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

D. A. Petrusevich, ‘Modern trends in the digitalization of education’, Journal of Physics: Conference Series, vol. 1691, 2020, [Online]. Available: https://api.sem anticscholar.org/CorpusID:230650493

work page 2020

[2] [2]

P. C. Brown, H. L. Roediger III, and M. A. McDaniel, Make it stick: The science of successful learning. Harvard University Press, 2014

work page 2014

[3] [3]

Semantic Similarity

System Design Figure 2: A mockup of the question generation user interface This section provides an overview of the learning platform that inﬂuenced the design decisions for this thesis. Existing learning platforms feature a straightforward, albeit time-consuming, approach. Students or Teachers can manually create ﬂashcards with terms and deﬁnitions. Afte...

work page 2023

[4] [4]

context loss

Question Generation This section describes the steps for generating questions and their reference answers that are eﬀective, grounded, accurate and relevant. To ground questions in the study material, documents need to be fed to LLMs. However, since LLMs are constrained by speciﬁc context window sizes, documents cannot be passed in their entirety. Models ...

work page

[5] [5]

grading rubrics

Answer Grading Figure 8: Overview diagram of the answer grading step Figure 8 shows the diﬀerent input and output parts of the grading task: The reference answer and provided student answer (from here on provided answer) are taken as input to the grading system which returns a grade. Optionally, the question can be included in the input, which has been sh...

work page 2024

[6] [6]

Using Document Layout Structure for Eﬃcient RAG

Related Work This section gives insight into similar and concurrent research in the ﬁelds of question generation and automated grading. 6.1. RAG Document Chunking RAG is a new and rapidly evolving area of research, with numerous novel approaches emerging. The unique approach to using the document’s layout for chunking explored in this thesis was inspired ...

work page

[7] [7]

question

Conclusion Our thesis presents a more sophisticated approach for chunking documents with a visual layout, such as PDF documents, for downstream tasks, including RAG. We show that questions and reference answers generated with a combination of study material and learning objectives are of high quality. Furthermore, a AQG system with LLMs that receive instr...

work page

[8] [8]

del Gobbo, A

E. del Gobbo, A. Guarino, B. Cafarelli, L. Grilli, and P. Limone, ‘Automatic evaluation of open-ended questions for online learning. A systematic mapping’, Studies in Educational Evaluation, vol. 77, p. 101258–101259, 2023, doi: https:// doi.org/10.1016/j.stueduc.2023.101258

arXiv 2023