Pith. sign in

REVIEW 3 major objections 5 minor 16 references

MockConf: A Student Interpretation Dataset: Analysis, Word- and Span-level Alignment and Baselines

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces MockConf, a seven-hour corpus of simultaneous interpreting in five European languages, manually aligned at span and word level with a divergence taxonomy, plus a web annotation tool and automatic alignment baselines.

desk verdict MockConf is a genuinely new, honestly reported interpreting corpus with manual span/word alignments, but gold-standard reliability rests on single-annotator labels with low agreement—still worth serious review. read the letter →

arxiv 2506.04848 v1 pith:BTNNND4A submitted 2025-06-05 cs.CL

Dávid Javorský , Ondřej Bojar , François Yvon This is my paper · ORCID
classification cs.CL
keywords simultaneousinterpretingspanalignmentwordannotationtooldivergencetaxonomyMockConfCzech-Englishspeechtranslationcorpus
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that simultaneous interpreting can be fruitfully studied and automatically benchmarked with a parallel corpus that is manually aligned at two levels: spans of text labeled by how the interpreter transformed the source, and word links marked sure or possible. It introduces MockConf, roughly seven hours of student mock-conference recordings in Czech, English, French, German, and Spanish, transcribed and annotated with a seven-label taxonomy of translation, paraphrase, summarization, generalization, addition, and replacement. The authors argue that existing parallel corpora and alignment tools fail for interpreting because the speech has no clean sentence boundaries and contains long-range divergences such as shortening and simplification. A sympathetic reader cares because a verified resource of this kind would let researchers measure interpreting behavior, monitor trainee interpreters, and test whether automatic alignment can reproduce human judgments on a hard, realistic task.

What carries the argument

The central objects are the span-label taxonomy and the alignment pipeline. The taxonomy, adapted from Barik's classification of translation departures, defines seven labels — translation, paraphrase, summarization, generalization, factual addition, uninformative addition, and replacement — applied to maximal-length corresponding spans between the source and interpreted transcripts. The machinery of the alignment method is a three-stage pipeline: BERTAlign produces n-m coarse span alignments without relying on sentence boundaries; punctuation-matched splits refine spans into sub-segments while SimAlign with XLM-R embeddings generates word links inside each span pair; and a small neural classifier using LaBSE similarity and span lengths assigns labels. InterAlign, the released annotation tool, is itself part of the machinery, since it is what makes word- and span-level annotation on long, unsegmented inputs feasible.

What would settle it

Re-transcribing a sample of the non-Czech recordings with a different speech recognizer plus native-speaker revision, then comparing word error rates and re-running the alignment, would test transcript fidelity directly; if alternate transcripts shift span boundaries or word links significantly, the gold labels are not trustworthy. Independently, double-annotating additional recordings and checking whether inter-annotator agreement rises above the reported target-side label Kappa of 0.25 would test the reliability of the single-annotator gold standard.

Watch

Extended reading notes

Core claim

The central claim is that MockConf provides a reusable reference for the alignment and analysis of simultaneous interpreting, with a span-label taxonomy adapted from translation departures in interpreting studies and word links that mark sure versus context-dependent correspondences. The paper reports quantitative properties of the corpus: translation spans cover roughly half of tokens, paraphrase about a fifth, and summarization spans are shorter on the target side (ratio ~0.6), while relay interpreting shows a higher proportion of translations and fewer additions than direct interpreting. On the evaluation side, the paper claims that a three-stage baseline system (coarse BERTAlign, punctuation-driven sub-segmentation with XLM-R word alignment, and LaBSE feature labeling) beats random baselines and in some inter-annotator comparisons rivals a single human annotator for segmentation, but that exact match with manual links remains low, and standard statistical word alignment (SimAlign) performs poorly on this data.

Load-bearing premise

The entire dataset rests on the transcripts being faithful: non-Czech speech was transcribed automatically by WhisperX and then only revised by native Czech speakers with self-reported proficiency in each foreign language, so any transcription errors these revisers missed are inherited by every span and word alignment in the corpus.

Editorial extensions

If this is right

  • Researchers can use MockConf to quantify interpreting strategies, for example the finding that summarization spans shrink to roughly 0.6 times the source length while translation spans stay near 1.0.
  • The InterAlign tool makes it practical to align long parallel speech transcripts without any pre-existing sentence segmentation.
  • The proposed metrics (segmentation F1, relaxed and exact span match, word AER, token-level label accuracy) give the community a shared way to compare future alignment systems on interpreting data.
  • The reported baselines show that standard word alignment (SimAlign) performs much worse on interpreting than on written MT, indicating that dedicated models are needed.
  • The low inter-annotator agreement (target-label Kappa 0.25; exact link match 14.85–30.46 percent) defines an upper bound on how well any automatic system can be expected to match an individual annotator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is to train a supervised divergence classifier on the aligned spans directly from speech or transcript features, instead of relying on punctuation-based sub-segmentation, which the paper itself flags as unreliable.
  • The sure/possible word-link distinction could support studies of ear-voice span and cognitive load by revealing which target words depend on context beyond a one-to-one translation.
  • The corpus's design could transfer to consecutive interpreting or medical settings to test whether the same divergence categories recur outside conference-style mock scenarios.
  • As the authors collected additional recordings without full consent, a plausible roadmap is to use MockConf's test split as a held-out benchmark while extending training data from the broader pool.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents MockConf, a new publicly released corpus of simultaneous interpreting collected from student mock conferences. The corpus comprises roughly 7 hours of recordings in Czech, English, French, German, and Spanish, with transcripts revised from WhisperX output and manual annotations at both span and word levels. A seven-label taxonomy (Translation, Paraphrase, Summarization, Generalization, Factual/Uninformative Addition, Replacement) is applied to spans, and word links are marked as sure or possible. The authors also release InterAlign, a web-based annotation tool designed for long, non-sentence-aligned inputs. They report descriptive analyses of span lengths, label distributions, relay versus direct interpreting, and multi-track interpreting, and they propose evaluation metrics and baselines for automatic span and word alignment, comparing a BERTAlign-based pipeline (with optional sub-segmentation and label classification) against random baselines and against inter-annotator agreement.

Significance. If the annotations are reliable, MockConf fills a genuine gap: it is, to my knowledge, the first publicly released simultaneous-interpreting corpus with manual span-level labels and word-level alignment links, and the InterAlign tool addresses a real practical need. The paper is transparent in reporting inter-annotator agreement, including low agreement figures, and it honestly includes random baselines. The resource and tool will likely be useful for corpus-based interpreting studies, for benchmarking alignment methods, and for educational monitoring. However, the significance is currently bounded by concerns about annotation stability and transcript fidelity, which the paper itself documents but does not resolve; the quantitative corpus analyses and baseline comparisons inherit those uncertainties.

major comments (3)
  1. [Section 3.1, Table 4] The inter-annotator reliability evidence is thin and partly discouraging: only one cs-en recording was double-annotated, target-side label agreement is Cohen's Kappa 0.25, and exact alignment-link agreement ranges from 14.85% to 30.46% depending on the reference annotator. Because every other recording in the corpus is annotated by a single annotator, the corpus-level analyses in Sections 3.2-3.4 and the baseline scores in Table 7 aggregate annotations that may be strongly influenced by individual annotator style (as Figure 2 itself shows for span lengths and link counts). The paper should either provide a quantitative control for annotator identity (for example, mixed-effects models or per-annotator breakdowns of the key statistics), or double-annotate a stratified sample of recordings and show that the reported trends and baseline rankings are stable across annotators. Without this, the central claim that MockConf is a usable reference corpus for evaluating automatic alignment is not fully supported.
  2. [Section 4.1 and Section 4.4, Table 7] The label classifier is trained on 80% of the development set ('we use devset for training where we take 80% of devset for actual training and 20% as held-out data for the evaluation during the training'), yet Table 7 reports results for the full development set. If the development-set rows for BA+sub+lab are computed on data that include the training portion, the label-match scores are in-sample estimates and overstate performance. This should be clarified: either the devset rows must be restricted to the 20% held-out portion, or the evaluation should use cross-validation on the development set. The same issue may affect the development-set label F1 and accuracy numbers reported in Table 7.
  3. [Section 2.1] The transcriptions are produced by WhisperX and then manually revised by native Czech speakers with self-reported proficiency in English, French, German, and Spanish. Since every alignment and downstream analysis is performed on these transcripts, undetected ASR errors in the non-Czech sides will propagate into the span labels, word links, and all corpus statistics. The paper should report the revisers' actual language proficiency or, more usefully, validate transcript fidelity on a sample with a native-speaker check per language and direction, and state the resulting error rates. This is particularly important for the French, German, and Spanish recordings, where the revision pool may be small and self-reported proficiency is not externally verified.
minor comments (5)
  1. [Table 2] The label 'Summariaztion' is a typo and should read 'Summarization'.
  2. [Table 1 caption] The caption says 'ISO-632-2'; the correct standard is ISO 639-2.
  3. [Ethics Statement] The phrase 'e.i.' should be 'i.e.' in 'fully anonymized, e.i. they do not contain'.
  4. [Section 4.3 and Limitations] The random baseline is said not to use Reformulation or Replacement labels, which means its label-match scores are not a fully comparable lower bound; this should be stated where the baseline is introduced, not only in the Limitations section.
  5. [Section 3.2, Figure 3] The claim that the span-length distribution 'seems to be uniform' would be more convincing with error bars or a statistical test, especially given the small corpus size and the annotator variability documented in Figure 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the corpus analyses and baselines are evaluated against held-out human annotations, and the only self-citation supplies background terminology rather than a load-bearing premise.

full rationale

MockConf’s derivation chain consists of corpus construction, descriptive analysis, and automatic alignment baselines; none of these steps defines a predicted quantity as its own input. The label classifier is trained on 80% of the development set and its final numbers in Table 7 are reported on separate development/test evaluation splits, so no fitted parameter is dressed up as a prediction. The random baseline deliberately mirrors reference label counts but is presented as a non-informative comparison, not as a learned predictor. The word-alignment definition of sure/possible links cites Bojar and Prokopová (2006), but this is standard external annotation terminology and is not used to justify any result that depends on the present paper’s own claims. The Limitations section acknowledges low inter-annotator agreement (target label Kappa 0.25, exact link agreement 14.85–30.46% in Table 4), which is a gold-standard reliability limitation rather than a circular step. No equation in the paper reduces by construction to an input, and no load-bearing uniqueness or ansatz is imported through self-citation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim (a reliable, aligned interpreting corpus) rests on transcript fidelity, on the validity of the span-label taxonomy, and on the trustworthiness of single-annotator gold labels. The listed hyperparameters are hand-chosen baseline configurations only; they do not affect the dataset claim. No invented entities are introduced; InterAlign is released software, not a postulated entity.

free parameters (4)
  • Label classifier hidden layer size = 100
    Section 4.1: the span-label classifier uses two hidden layers of size 100. A hand-chosen hyperparameter; it affects the baseline label results (Table 7), not the dataset itself.
  • SimAlign link distance cutoff = 50 tokens
    Section 4.3: word links with |p_ws - p_wt| > 50 are discarded for the word-alignment baseline.
  • BERTAlign configuration = max_align=10, top_k=10, window=10, skip=0.0
    Section 4.1: coarse alignment parameters taken from the tool's conventions; they shape the BA baseline.
  • SimAlign embedding window and stride = window=128, stride=64
    Section 4.3: sliding-window settings for contextual embeddings in the word-alignment baseline.
assumptions (5)
  • domain assumption The manually revised WhisperX transcripts faithfully reproduce what was said, including disfluencies
    Section 2.1 and Appendix C: the transcription step is intended to capture exact wording; all alignments and analyses are defined on these transcripts.
  • domain assumption Native Czech revisers with sufficient self-reported proficiency can reliably revise English, French, German, and Spanish transcripts
    Section 2.1: transcript revision was done by native Czech speakers; no independent verification of foreign-language transcript accuracy is provided.
  • domain assumption The Barik (1994) omission/addition/replacement taxonomy, extended with paraphrase, summarization, and generalization labels, is a valid annotation scheme for simultaneous interpreting
    Section 2.3 and Appendix D define the span labels from this taxonomy; its reliability is measured on one double-annotated recording only.
  • domain assumption The consent-authorized subset of recordings is adequate for the reported analyses
    Section 2.1: only recordings with consent from both participants are included; the excluded recordings could differ systematically from the included ones.
  • domain assumption Moses tokenization is an acceptable unit for length statistics
    Table 1 and Section 3 compute token counts and length ratios with the Moses tokenizer; results could shift with a different tokenization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MockConf: A Student Interpretation Dataset: Analysis, Word- and Span-level Alignment and Baselines." pith.science (2026). https://pith.science/paper/BTNNND4A

@misc{pith2026250604848,
  author       = {Pith},
  title        = {Pith review of: MockConf: A Student Interpretation Dataset: Analysis, Word- and Span-level Alignment and Baselines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BTNNND4A}},
  note         = {Machine review of arXiv:2506.04848}
}
read the original abstract

In simultaneous interpreting, an interpreter renders a source speech into another language with a very short lag, much sooner than sentences are finished. In order to understand and later reproduce this dynamic and complex task automatically, we need dedicated datasets and tools for analysis, monitoring, and evaluation, such as parallel speech corpora, and tools for their automatic annotation. Existing parallel corpora of translated texts and associated alignment algorithms hardly fill this gap, as they fail to model long-range interactions between speech segments or specific types of divergences (e.g., shortening, simplification, functional generalization) between the original and interpreted speeches. In this work, we introduce MockConf, a student interpreting dataset that was collected from Mock Conferences run as part of the students' curriculum. This dataset contains 7 hours of recordings in 5 European languages, transcribed and aligned at the level of spans and words. We further implement and release InterAlign, a modern web-based annotation tool for parallel word and span annotations on long inputs, suitable for aligning simultaneous interpreting. We propose metrics for the evaluation and a baseline for automatic alignment. Dataset and tools are released to the community.

Figures

Figures reproduced from arXiv: 2506.04848 by the authors.

Figure 1
Figure 1. Examples of span-level annotation from our [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Span length (in tokens) distribution per label [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Left: Span length (in tokens) distribution per [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Relative proportion of each span label within [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Two alignment annotations (by two different [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: A screenshot of InterAlign for aligning transcripts of speech and their interpreting [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: List of chunks and word alignment links dis [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 15 canonical work pages

  1. [1]

    doc id 2. doc id character token 18 8 0.96 0.96 39 12 0.86 0.90 40 41 0.94 0.95 43 44 0.97 0.93 13 45 1.04 1.01 10 46 1.16 1.11 15 4 0.95 0.96 Table 10: Character and token ratios for multi-track interpreting. The first two columns denote ids of docu- ments that are interpretations of the same speech. More details about the documents are in Table 8 and Ta...

    work page 2003

  2. [2]

    • Transcripts already contain sentences

    You can change the segmentation to sentences. • Transcripts already contain sentences. It is possible that a different sentence divi- sion is suitable, but you are welcome to create your own sentence division (but this is not required). • Please edit the sentences so that each one is on a separate line

    work page

  3. [3]

    summarization

    Label proper names. • Recordings can contain the names of cities, organizations - it is important to mark these proper names with the [NAME] tag, for example, the sen- tence on the left will be the sentence on the right after the arrow: Vá- clav was then in the Czech Repub- lic. → [NAME](Václav) was then in [NAME](Czech Republic). D Annotation Guidelines ...

    work page

  4. [5]

    Annotation Style Guide for the Blinker Project

    Lost in Interpreting: Speech Translation from Source or Interpreter? In Proc. Interspeech 2021 , pages 2376–2380. Mana Makinae, Katsuhito Sudoh, Masaru Yamada, and Satoshi Nakamura. 2025. An automatic quality met- ric for evaluating simultaneous interpretation. Joel Martin, Howard Johnson, Benoit Farley, and Anna Maclachlan. 2003. Aligning and using an En...

    work page Pith review arXiv 2021

  5. [7]

    In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaL- iDa), pages 64–73, Reykjavik, Iceland (Online)

    CombAlign: a tool for obtaining high-quality word alignments. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaL- iDa), pages 64–73, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden. Craig Stewart, Nikolai V ogler, Junjie Hu, Jordan Boyd- Graber, and Graham Neubig. 2018. Automatic esti- mation of sim...

    work page 2018

  6. [11]

    inter- interna- tional

    Please correct the transcripts to match what is said in the recordings. • Do not correct grammar – if the speaker makes grammatical or any other lan- guage mistakes (stutters, repeats himself, uses the wrong form of a word or a whole word), the transcript should capture the exact notation of what is said. • For example, you can edit the stutter in the wor...

    work page

  7. [14]

    Translation

    “Translation” (Alignment): If a word in the source span directly corresponds to a transla- tion in the target span out of any additional context, mark it as a translation alignment. En- sure accuracy and precision in aligning words with their translations

    work page

  8. [15]

    Reformulation

    “Reformulation”: Identify phrases in the source span that convey the same meaning as phrases in the target span but are not direct translations. Use the reformulation label for such alignments with a specific category

    work page

Show all 16 references
  1. [16]

    Addition

    “Addition”: Highlight cases where phrases are present in one span that do not have a direct counterpart in the other segment. Mark these as addition alignments with a specific category. D.3 Word-level alignments Within each pair of aligned segments (so you can- not create word...

  2. [1999]

    Ma- chine Learning, 34(1-3):177–210

    Statistical models for text segmentation. Ma- chine Learning, 34(1-3):177–210. Silvia Bernardini, Adriano Ferraresi, and Maja Miliˇce- vi´c. 2016. From EPIC to EPTIC—Exploring sim- plification in interpreting and translation from an intermodal perspective. Target, 28(1):61–86....

    2016

  3. [2000]

    Meta, 45(3):548–557

    On the use of compensatory strategies in si- multaneous interpretation. Meta, 45(3):548–557. Fatemeh Azadi, Heshaam Faili, and Mohammad Javad Dousti. 2023. PMI-align: Word alignment with point- wise mutual information without requiring parallel training data. In Findings of th...

    2023

  4. [2004]

    In Proceedings of the Workshop on Multilingual Lin- guistic Resources, pages 57–64, Geneva, Switzerland

    Multilingual aligned parallel treebank corpus reflecting contextual information and its applications. In Proceedings of the Workshop on Multilingual Lin- guistic Resources, pages 57–64, Geneva, Switzerland. COLING. Henk van den Heuvel, Khalid Choukri, Christian Gol- lan, Asunc...

    2006

  5. [2012]

    In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC‘12), pages 125–129, Istanbul, Turkey

    TED-LIUM: an automatic speech recogni- tion dedicated corpus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC‘12), pages 125–129, Istanbul, Turkey. European Language Resources Association (ELRA). Annalisa Sandrelli and Claudio B...

    2006

  6. [2014]

    In Pro- ceedings of the Ninth International Conference on Language Resources and Evaluation (LREC‘14) , pages 2913–2919, Reykjavik, Iceland

    SWIFT aligner, a multifunctional tool for par- allel corpora: Visualization, word alignment, and (morpho)-syntactic cross-language transfer. In Pro- ceedings of the Ninth International Conference on Language Resources and Evaluation (LREC‘14) , pages 2913–2919, Reykjavik, Icel...

    2008

  7. [2021]

    In Proceedings of the 18th International Conference on Spoken Language Trans- lation (IWSLT 2021), pages 226–235, Bangkok, Thai- land (online)

    Large-scale English-Japanese simultaneous in- terpretation corpus: Construction and analyses with sentence-aligned data. In Proceedings of the 18th International Conference on Spoken Language Trans- lation (IWSLT 2021), pages 226–235, Bangkok, Thai- land (online). Association ...

    2021

  8. [2023]

    In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (V olume 3: System Demonstrations), pages 169–178, Toronto, Canada

    Japanese-to-English simultaneous dubbing prototype. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (V olume 3: System Demonstrations), pages 169–178, Toronto, Canada. Association for Computational Lin- guistics. ADDF ADDU GEN PARA ...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.