Pre-Training Curriculum for Multi-Token Prediction in Language Models

Alan Akbik; Ansar Aynetdinov

REVIEW 5 major objections 4 minor 13 references

Pre-Training Curriculum for Multi-Token Prediction in Language Models

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small language models trained with a forward curriculum — starting at next-token prediction and adding heads over time — keep multi-token prediction's speedups while improving on static MTP baselines.

desk verdict Interesting and honest empirical study, but the abstract overclaims the forward curriculum's NTP gains; the reverse curriculum is the one that actually improves NTP, and the paper needs claim-narrowing and variance reporting before it is publishable. read the letter →

arxiv 2505.22757 v1 pith:4E7KDX7B submitted 2025-05-28 cs.CL cs.AI

Ansar Aynetdinov , Alan Akbik This is my paper

classification cs.CLcs.AI

keywords multi-tokenpredictioncurriculumlearningpre-trainingobjectiveself-speculativedecodingsmalllanguagemodelsbyte-leveltokenizationnext-tokeninferenceacceleration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-token prediction (MTP) — training a model to predict the next $k$ tokens at once through several heads on a shared backbone — promises faster inference through self-speculative decoding, but small language models have been reported to learn the objective poorly. The paper asks whether ordering that objective during pre-training fixes the problem, and proposes two curricula that change the number of active prediction heads over time: a forward curriculum that starts with ordinary next-token prediction ($k = 1$) and adds one head at regular intervals up to $k = k_{\max}$, and a reverse curriculum that does the opposite. The paper's claim is that the forward curriculum gives 1.3B and 3B models the best trade-off — better downstream next-token performance and output quality than a static MTP baseline, with self-speculative decoding speedups nearly as large — while the reverse curriculum yields stronger next-token models but renders its extra heads useless for decoding. If the claim is right, a small model can get both quality and fast generation from a single fixed training schedule, with no architectural changes and no auxiliary draft model.

What carries the argument

The object that carries the argument is a schedule over the active prediction-head count $k$, defined per training epoch $e$ as $k_{\text{current}}(e) = \min(k_{\max}, \lfloor e/(E/k_{\max}) \rfloor + 1)$ for the forward curriculum and $k_{\text{current}}(e) = \max(1, k_{\max} - \lfloor e/(E/k_{\max}) \rfloor)$ for the reverse, where $E$ is the total number of training epochs. The schedule decides which of the $k$ independent language-modeling heads contribute to the multi-token loss $L_{\text{MTP}} = -\sum_{t=1}^{T}\sum_{i=1}^{k} \log P(x_{t+i} \mid x_1, \dots, x_t; \theta)$ at each stage of pre-training. The payoff mechanism is self-speculative decoding: the trained heads draft $k$ tokens in one forward pass and a verification pass accepts or rejects them, so a model's per-head acceptance rate measures how usable the auxiliary heads actually are. That acceptance rate is what separates the two curricula — forward-trained heads are accepted almost as often as static MTP's, reverse-trained heads almost never.

What would settle it

Train identical 1.3B and 3B models on MiniPile under four schedules matched for total head-time: forward, reverse, static MTP, and a randomly shuffled order of the same $k$ values, plus an adaptive schedule that advances $k$ only when the current heads' draft-acceptance rate plateaus. If the shuffled schedule reproduces the forward curriculum's main-head gains and near-static-MTP acceptances, ordering is not the causal ingredient and the curriculum claim fails; if the adaptive schedule clearly beats the fixed one, the fixed schedule — the paper's acknowledged weak point — is the binding constraint.

Watch

Extended reading notes

Core claim

Training Llama-style 1.3B and 3B models for a single epoch on MiniPile, with subword and byte vocabularies, two head designs (linear layers and transformer layers), and $k = 2, 4,$ or $8$ predicted tokens, the paper reports a consistent split between the two directions of curriculum. The forward curriculum ($k$ grows from 1 to $k_{\max}$) improves over the static MTP baseline on next-token benchmarks in most configurations — for four-head models LAMBADA bits-per-byte improves by roughly 4–5 percent relative and BLiMP accuracy rises in most cases — and its drafted tokens are accepted during self-speculative decoding almost as often as static MTP's, keeping inference speed as a close second. The reverse curriculum ($k$ shrinks from $k_{\max}$ to 1) produces the stronger next-token model, and for byte-level tokens it matches or beats plain NTP on BLiMP, ARC-E, and OBQA, but its auxiliary heads are almost never accepted at decoding time, so its speed effectively falls back to the NTP rate. The paper's conclusion is that the forward direction is the useful one: it is the only curriculum that keeps the MTP speed benefit while improving over the static MTP quality baseline, and the pattern persists when the training budget is raised to 10B tokens of FineWeb-Edu.

Load-bearing premise

The load-bearing premise is that changing the number of predicted tokens on a fixed schedule — one head added or removed every $E/k_{\max}$ epochs, with no signal from the model's own loss or acceptance rate — is a faithful realization of curriculum learning; the paper itself concedes this spacing is 'somewhat arbitrary' and can under- or over-train individual heads.

Editorial extensions

If this is right

Small models trained with the forward curriculum can act as their own speculative decoder: draft acceptance nearly matches static MTP while main-head next-token performance improves on it, removing the usual quality-for-speed trade-off of MTP on sub-7B models.
The reverse curriculum effectively yields a next-token model with dead auxiliary heads: its main head is stronger, but its drafted tokens win almost no acceptances, so nothing is gained at inference time — the choice between the two curricula is a choice between speed-plus-quality and quality-only.
Byte-level tokenization is where dynamic MTP earns its keep: reverse-curriculum byte-level models match or beat plain NTP on BLiMP, ARC-E, and OBQA, even though a static MTP objective with the same tokens does not.
The curriculum advantage over static MTP is clearest with four heads and survives a tenfold larger data budget, since the 10B-token FineWeb-Edu run reproduces the same ordering of results.
Linear-layer and transformer-layer heads behave similarly under the curricula, so the observed effect is attributable to the schedule rather than to the extra parameters that linear heads add.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A matched-exposure control — forward versus static MTP with identical total head-epochs — would separate the benefit of ordering from the benefit of spending early training on easier one-token targets; the paper does not run this control, so part of the forward gain could be extra effective training, not curriculum ordering.
The per-head draft-acceptance rate the paper measures at the end of training could serve as an online signal: an adaptive curriculum that adds a head only once the previous head's acceptance plateaus is a direct, testable upgrade to the fixed schedule the authors themselves flag as arbitrary.
Because the clearest wins appear with byte-level tokens, the forward curriculum is most promising for small-vocabulary regimes — multilingual scripts, morphologically rich languages, and code — where subword units are scarce or irregular.
The reverse curriculum's pattern of a good main head with dead auxiliary heads suggests MTP's extra heads can fail silently even when the backbone benefits, so future MTP pre-training evaluations should report draft acceptance alongside perplexity and downstream scores.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, quick take on arXiv:2505.22757. The new thing here is simple and testable: instead of training with a fixed number of MTP heads, you change k on a fixed epoch-based schedule. Forward curriculum (NTP→MTP) and reverse (MTP→NTP) are both evaluated, and the forward/reverse comparison is genuinely new relative to Gloeckle et al. That deserves credit. The experimental sweep is broad — 1.3B and 3B, subword and byte tokenization, linear and transformer heads, several NTP and generation metrics — and the paper is honestly written, with a Limitations section that openly calls the schedule 'somewhat arbitrary.' The FineWeb-Edu check is also a good-faith scaling probe.

Now the soft spots. The abstract's central claim — that the forward curriculum improves downstream NTP performance — does not survive contact with Table 1. For byte-level models, forward degrades MiniPile BPB in all four configurations and degrades or ties LAMBADA in four of four. For subword models, it helps only at 4 heads and hurts at 2 heads. The paper's own Section 3.2.2 admits the curricula don't beat the NTP baseline for subword models. The reverse curriculum is the one that consistently improves NTP over static MTP, but it forfeits the self-speculative decoding speedups. So the honest headline would be: reverse improves NTP but no speedup; forward preserves speedup but doesn't reliably help NTP. That is a defensible trade-off story, but it is not what the abstract says.

The other weaknesses are more standard. No repeated seeds, no error bars, no significance tests, and only a promised code release. The output-quality claim is mixed: forward beats static on at least two metrics in 8/12 configurations, but there are negative signs and no variance. The schedule is admittedly arbitrary, so the forward/reverse difference could be an artifact of that specific schedule rather than evidence about curriculum direction. That said, the paper acknowledges this and frames the result as a proof-of-principle, which is fair.

On the plus side, the SemScore self-reference is not a problem; it's a fixed sentence-embedding metric, not fit to these results. The citation pattern looks appropriate, including Gloeckle et al., Salhan et al., and the curriculum-learning literature.

Verdict: this deserves a serious referee. The idea is new enough and the experiments are extensive enough to warrant engagement. But a referee should push for a rewritten abstract and conclusion that match the actual tables, repeated runs or at least explicit variance reporting, and ideally a release of the training code. I'd be happy to see it in the reading group as a case study in claim-evidence mismatch.

Referee Report

5 major / 4 minor

Summary. This paper proposes two curriculum schedules for multi-token prediction (MTP) pre-training of small language models: a forward curriculum that starts with next-token prediction (k=1) and adds one prediction head every E/kmax epochs, and a reverse curriculum that starts with k=kmax and removes heads over time. The authors train 1.3B and 3B Llama-style models on MiniPile with subword and byte tokenizers, using linear-layer (LL) and transformer-layer (TL) prediction heads, and compare against static MTP and NTP baselines on NTP benchmarks, self-speculative decoding speed, and output quality metrics. The abstract claims that the forward curriculum improves downstream NTP performance and output quality while retaining self-speculative decoding benefits, and that the reverse curriculum gives stronger NTP and output quality but no speedups. The evidence in the paper is configuration-dependent, and the broad claims are not consistently supported by the reported tables.

Significance. If the reported effects were robust, the paper would make a useful practical contribution: a simple fixed schedule for changing the MTP prediction depth could let small models obtain inference speedups without sacrificing next-token quality. The paper includes a wide configuration matrix, a scale check on FineWeb-Edu, and a Limitations section, which are strengths. The SemScore metric comes from the authors' prior work but is a fixed external embedding-based similarity measure, so it is not fitted to the current results and is not a circularity concern. Nevertheless, the headline claim of forward-curriculum improvements is not supported by the paper's own tables, and the absence of repeated runs prevents a strong conclusion.

major comments (5)

[Abstract; Table 1; Section 3.2.2] The central claim that the forward curriculum improves downstream NTP performance is contradicted by the reported numbers. For byte-level models, forward training degrades MiniPile BPB relative to the static MTP baseline in all four configurations (1.3B 4-LL +3.16%, 1.3B 8-LL +0.81%, 3B 4-LL +2.62%, 3B 8-LL +0.79%) and degrades or ties LAMBADA in all four. For subword-level models, forward improves both MiniPile and LAMBADA only in the 4-head configurations (4-LL and 4-TL), while the 2-head configurations show degradations (e.g., 1.3B 2-LL +0.15%/+0.37%, 3B 2-TL +0.88%/+0.26%). Section 3.2.2 itself states that the curricula 'do not lead to performance improvements over the NTP baseline when applied to subword-level models.' The abstract's unqualified claim should be revised to reflect the configuration-specific evidence; the FineWeb-Edu check in Table 5 also shows the NTP model remains the best performer.
[Table 2; Section 3.4] The claim that the forward curriculum improves generative output quality is not supported by any uncertainty quantification, and the signs are mixed. While forward wins on at least two metrics in 8/12 configurations, there are substantial degradations in several others: 1.3B subword 2-LL BLEU -2.70% and ROUGE -0.06%; 3B subword 2-TL BLEU -7.44% and ROUGE -3.97%; 3B byte 4-LL BLEU -6.98% and SemScore -5.88%; 1.3B byte 8-LL SemScore -5.20%. Without repeated runs or significance tests, the 'at least two metrics' criterion is not a reliable basis for the qualitative claim that output quality improves.
[Section 3.1.4; Tables 1 and 2] All results come from a single training run per configuration. Many of the reported differences are under 3%, and several are under 1%, which is within the range that can arise from optimizer stochasticity and data-ordering effects. The paper should provide at least three seeds for the key comparisons (e.g., forward versus static MTP for the 4-head subword and byte configurations) or otherwise justify why single runs are sufficient. Without this, the central improvement claims cannot be evaluated.
[Section 2.2; Limitations] The curriculum is defined by a single uniform schedule that changes k every E/kmax epochs. The Limitations section acknowledges that this choice is 'somewhat arbitrary' and may under- or overfit the prediction heads. Because no alternative schedule (different step sizes, loss-triggered scheduling, or a random-schedule control) is tested, the observed differences between forward and reverse curricula could be artifacts of this specific schedule rather than evidence about easy-to-hard versus hard-to-easy learning. A sensitivity analysis with at least one alternative schedule is needed before the paper can claim that curriculum-based MTP training fundamentally benefits SLMs.
[Section 3.1.2] The Transformer Layer (TL) head setup is underspecified for dynamic k. The text says 'we use l - k transformer layers as the model backbone and dedicate k out of l layers to handling token prediction,' but when k changes during a curriculum the partition between backbone and prediction layers must change during training. The paper does not explain which layers are added or removed, how the shared output linear layer is applied, or whether parameters are re-initialized when k changes. This is necessary for reproducibility and for interpreting the TL results in Tables 1 and 2.

minor comments (4)

[Section 3.2.2] The first bullet states that 'Byte-level dynamic MTP models outperform subword-level ones,' but Table 1 does not directly support this as a general statement about all dynamic models; the reverse curriculum drives most of the byte-level gains, and the forward curriculum often degrades byte-level BPB.
[Abstract] The abstract says the forward curriculum improves 'downstream NTP performance,' but the comparisons in Table 1 are made against the static MTP baseline, not against the NTP baseline. The comparison target should be stated explicitly.
[Section 3.4.1] The paper uses gpt-3.5-turbo as the G-Eval judge but does not report the prompt temperature or the number of evaluation samples per generation; adding these details would improve reproducibility.
[Section 2.2] The notation for the epoch-based schedule uses E/kmax, but the Limitations section later refers to 'm/n steps' without connecting this to the earlier definition; the notation should be unified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical comparison with externally defined objectives, metrics, and baselines; the only self-citation (SemScore) is a fixed external similarity metric and is not load-bearing.

full rationale

This paper is an empirical study comparing static MTP, forward-curriculum MTP, and reverse-curriculum MTP training schedules. The proposed curricula are defined by explicit epoch-based formulas (Section 2.2) that do not incorporate the experimental outcomes being predicted. The NTP benchmarks, inference speed measurements, and output-quality metrics are all external to the training procedure: MiniPile and LAMBADA are standard corpora, self-speculative decoding follows Stern et al. (2018), and the quality metrics (BLEU, ROUGE-L, SemScore, TTR, G-Eval) are fixed evaluation functions applied after training. The only self-citation is SemScore (Aynetdinov and Akbik, 2024), which is a pre-existing sentence-embedding similarity metric with no fitted parameters tied to this paper's results; using it as one of several quality metrics does not make the comparison circular. No equation in the paper defines the curricula in terms of their measured effects, and no fitted parameter is renamed as a prediction. The acknowledged arbitrariness of the uniform schedule (Limitations) is a validity concern about schedule design, not circularity. Any disagreement between the abstract's broad claims and Table 1's mixed results is a claim-evidence mismatch, which belongs under correctness risk rather than circularity analysis. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper rests on background assumptions from prior MTP work and its own evaluation metric; no free parameters are fitted to data, but the curriculum schedule is an arbitrary hand-picked hyperparameter and the evaluation relies on the authors' SemScore.

free parameters (1)

Curriculum step interval = E/kmax epochs (e.g., E/4 when kmax=4)
The schedule adds or removes a head at uniform intervals; the interval is a hand-picked hyperparameter that defines the curriculum and is acknowledged as arbitrary in the Limitations section.

assumptions (5)

domain assumption Multi-token prediction with multiple independent heads improves large language model performance.
The paper extends this prior result to small models; see Section 1, citing Gloeckle et al. (2024).
domain assumption Self-speculative decoding with MTP heads yields inference speedups proportional to saved forward passes.
Section 2.1 and Experiment 2 measure speedups solely in forward passes, assuming wall-clock speedup follows.
domain assumption MiniPile is a sufficient corpus to evaluate SLM pre-training curricula.
All main experiments train for one epoch on MiniPile; Section 3.1.3 acknowledges models are not competitive LLMs.
domain assumption SemScore is a valid measure of generated output quality.
The authors' own metric from Aynetdinov and Akbik (2024) is used in Experiment 3; it is a fixed embedding similarity metric, but self-cited.
ad hoc to paper The fixed uniform schedule for changing k is a valid curriculum.
Section 2.2 defines kcurrent(e) with uniform intervals; the paper acknowledges the choice is arbitrary (Limitations).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Pre-Training Curriculum for Multi-Token Prediction in Language Models." pith.science (2026). https://pith.science/paper/4E7KDX7B

@misc{pith2026250522757,
  author       = {Pith},
  title        = {Pith review of: Pre-Training Curriculum for Multi-Token Prediction in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4E7KDX7B}},
  note         = {Machine review of arXiv:2505.22757}
}

abstract

Multi-token prediction (MTP) is a recently proposed pre-training objective for language models. Rather than predicting only the next token (NTP), MTP predicts the next $k$ tokens at each prediction step, using multiple prediction heads. MTP has shown promise in improving downstream performance, inference speed, and training efficiency, particularly for large models. However, prior work has shown that smaller language models (SLMs) struggle with the MTP objective. To address this, we propose a curriculum learning strategy for MTP training, exploring two variants: a forward curriculum, which gradually increases the complexity of the pre-training objective from NTP to MTP, and a reverse curriculum, which does the opposite. Our experiments show that the forward curriculum enables SLMs to better leverage the MTP objective during pre-training, improving downstream NTP performance and generative output quality, while retaining the benefits of self-speculative decoding. The reverse curriculum achieves stronger NTP performance and output quality, but fails to provide any self-speculative decoding benefits.

Figures

Figures reproduced from arXiv: 2505.22757 by the authors.

**Figure 2.** Inference speed-ups achieved by MTP models in relation to the NTP baseline. We measure the speed-ups [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Breakdown of the per-head acceptance rate [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

[1]

An input sequence (text or code) that serves as the starting point

work page
[2]

An output sequence written as a continuation of the input

work page
[3]

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve

A framework for few-shot language model evaluation. Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024. Bet- ter & faster large language models via multi-token prediction. Preprint, arXiv:2404.19737. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha ...

arXiv 2024
[4]

In International Conference on Learning Representations

Measuring massive multitask language under- standing. In International Conference on Learning Representations. 10 Michael Y . Hu, Aaron Mueller, Candace Ross, Ad- ina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the sec- ond babylm challenge: Sample-efficient pretrain- ing...

arXiv 2024
[9]

Your task is to rate the written output sequence on one metric

A target sequence that represents the expected continuation of the input sequence. Your task is to rate the written output sequence on one metric. Please make sure you read and understand these instructions carefully. Keep this document open while reviewing, and refer to it as needed. Evaluation Criteria: Overall Quality (1-5) - how well does the output s...

work page
[10]

Carefully read the input, output, and target sequences

work page
[11]

Compare the output sequence to both the input sequence (continuity) and the target sequence (alignment)

work page
[12]

Rate the output on a scale of 1-5 for Quality, according to the criteria above. ### Input Sequence: {Input Sequence} ### Output Sequence: {Output Sequence} ### Target Sequence: {Target Sequence} Evaluation Form (scores ONLY): - Quality: B Additional Results B.1 Result tables with absolute values Tables 3 and 4 report the absolute scores that corre- spond ...

work page 2024

Show all 13 references

[13]

Please refer to Warstadt et al

We show the performance on each linguistic phenomenon considered in the benchmark. Please refer to Warstadt et al. (2020) for the precise defini- tion of each phenomenon and additional details on how to interpret them. Below we discuss the results of one of the particularly in...

2020
[2016]

In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany

Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin- guistics. Petru Soviany, Radu...

2022 arXiv
[2018]

(?:[^"\\]|\\.)*

Blockwise parallel decoding for deep autore- gressive models. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman ...

2023 arXiv
[2021]

Algorithmica, 83(9):2833–2858

Best fit bin packing with random order revis- ited. Algorithmica, 83(9):2833–2858. Ansar Aynetdinov and Alan Akbik. 2024. Sem- score: Automated evaluation of instruction-tuned llms based on semantic textual similarity. Preprint, arXiv:2401.17072. Yoshua Bengio, Jérôme Louradou...

2024 arXiv
[2024]

Preprint, arXiv:2401.10774

Medusa: Simple llm inference acceleration framework with multiple decoding heads. Preprint, arXiv:2401.10774. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc,...

2018 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

An input sequence (text or code) that serves as the starting point

work page

[2] [2]

An output sequence written as a continuation of the input

work page

[3] [3]

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve

A framework for few-shot language model evaluation. Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024. Bet- ter & faster large language models via multi-token prediction. Preprint, arXiv:2404.19737. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha ...

arXiv 2024

[4] [4]

In International Conference on Learning Representations

Measuring massive multitask language under- standing. In International Conference on Learning Representations. 10 Michael Y . Hu, Aaron Mueller, Candace Ross, Ad- ina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the sec- ond babylm challenge: Sample-efficient pretrain- ing...

arXiv 2024

[5] [9]

Your task is to rate the written output sequence on one metric

A target sequence that represents the expected continuation of the input sequence. Your task is to rate the written output sequence on one metric. Please make sure you read and understand these instructions carefully. Keep this document open while reviewing, and refer to it as needed. Evaluation Criteria: Overall Quality (1-5) - how well does the output s...

work page

[6] [10]

Carefully read the input, output, and target sequences

work page

[7] [11]

Compare the output sequence to both the input sequence (continuity) and the target sequence (alignment)

work page

[8] [12]

Rate the output on a scale of 1-5 for Quality, according to the criteria above. ### Input Sequence: {Input Sequence} ### Output Sequence: {Output Sequence} ### Target Sequence: {Target Sequence} Evaluation Form (scores ONLY): - Quality: B Additional Results B.1 Result tables with absolute values Tables 3 and 4 report the absolute scores that corre- spond ...

work page 2024