{"id":"584f4c56-7fce-4ae4-a4f1-4111aa8eccb0","arxiv_id":"2501.05963","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A color-marker method for translating span annotations with DeepL yields a high-coverage Finnish SQuAD2.0 whose trained QA models beat prior translated datasets.","lead":"Researchers used a commercial translation service's document-formatting feature to carry question-answer span annotations from English SQuAD into a new Finnish version, preserving 97.2% of examples. The method could let low-resource languages get question-answering datasets without manual annotation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is not independently anchored for Finnish: models are trained and evaluated on the same DeepL-translated dev split, and the headline F1 advantage over the prior Finnish SQuAD is not cross-validated on a shared or human-annotated Finnish test set, so systematic translation…","rationale":"The reader's weakest_assumption focuses on DeepL's formatting fidelity, but the deepest load-bearing uncertainty is the evaluation design: the strongest quantitative evidence for the Finnish dataset is generated by training and testing on the same translation pipeline, and the comparison to the prior Finnish dataset is not cross-validated. This is not a fatal flaw because the paper provides several honest complementary evaluations: a 97.2% preservation rate, a 321-example manual span-transfer check, a backtranslation experiment on untainted English test data, and a Spanish cross-dataset comparison. The Spanish experiment is the most direct support for the general claim, but the headline Finnish claim lacks the equivalent control. The concern is concrete and testable: Kylliäinen's dataset is cited in the paper, so a shared Finnish evaluation is feasible. A conditional accept is appropriate: the resource and method are valuable, but the Finnish superiority claim should be confirmed by a cross-dataset or human-annotated evaluation before it is taken at face value. The reader's rationale already mentions the missing human-annotated Finnish test set and the self-referential nature of the headline scores, so this stress-test sharpens that point into a specific load-bearing check rather than introducing an entirely new objection.","tokens_in":9777,"tokens_out":5158,"duration_ms":54969,"concrete_test":"Run a Finnish cross-dataset evaluation: train two FinBERT-base models with identical hyperparameters and training setup on (a) the released TurkuNLP/squad_v2_fi data and (b) Kylliäinen's Finnish SQuAD2.0, then evaluate both checkpoints on (i) each dataset's dev split and (ii), if available, a small human-translated Finnish SQuAD dev subset; if the DeepL-trained model does not beat the Kylliäinen-trained model on the Kylliäinen dev split or the human-translated dev subset, the reported F1 advantage is an artifact of same-pipeline train/dev consistency rather than evidence of superior data quality.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The key comparison in Section 4.3 (Table 1) reports F1 73.7 on the new dataset versus 61.9 on Kylliäinen's Finnish SQuAD, but these numbers come from models trained and evaluated on different Finnish dev sets produced by different translation pipelines. A model trained on the DeepL-translated data is scored on the DeepL-translated dev set, meaning that any systematic translation-error pattern present in both train and dev can be learned by the model and inflate the score without improving answers on naturally occurring Finnish text. The manual error analysis in Section 4.4 validates span-transfer fidelity relative to the English source, not the quality or naturalness of the Finnish answer text, and it samples only answerable questions. The backtranslation experiment in Section 4.2 is a helpful sanity check, but it evaluates an English model on original English SQuAD and estimates the one-round penalty only under the unverified assumption that errors are evenly distributed between the en->fi and fi->en translation rounds. The Spanish cross-dataset experiments in Table 3 directly support the method's general superiority, but no equivalent Finnish cross-dataset evaluation is reported for the language that is the paper's primary contribution. Without either a cross-evaluation against Kylliäinen's data or a manually translated Finnish test set, the statement that the method produces 'consistently better translated data' remains strongest for Spanish and only indirectly supported for Finnish.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a method for transferring span-level question answering annotations during machine translation by exploiting DeepL's formatted-document translation: answer spans are encoded as colored text in the input document, the translated document preserves the colors, and the spans are reconstructed after translation, with only trailing punctuation stripped. The authors release a Finnish translation of SQuAD2.0 that preserves 90,233 of 92,749 answerable question-answer pairs (97.2%), train extractive QA readers on it (FinBERT, FinALBERT, BERT-large), and evaluate the resource through downstream QA performance on the translated validation set, an English-Finnish-English backtranslation experiment, a Spanish head-to-head comparison with the TAR method, and manual inspection of 321 answer spans. They report that their Finnish dataset gives F1 73.7 with BERT-base versus F1 61.9 for the prior Finnish SQuAD by Kylliäinen, and argue that the method is simple, cheap, and broadly applicable to other span-annotated datasets and languages.","tokens_in":10037,"tokens_out":4780,"duration_ms":47302,"significance":"If the claims hold, the contribution is practically valuable: a low-cost, near-lossless annotation-transfer method for span-labeled datasets, together with an openly released Finnish QA resource and trained model. The paper ships code and data under open licenses, and it reports several evaluation angles rather than a single metric. The manual evaluation (87.2% correct in a 321-example sample), the two-round backtranslation penalty (8.4 EM / 5.1 F1), and the Spanish cross-dataset comparison (Table 3) are meaningful positive evidence. The main caveat is that the headline Finnish quality claim rests on training and evaluating within the same DeepL-translated pipeline, and lacks a shared or manually annotated Finnish test set; consequently the general superiority claim is strongest for Spanish and only indirect for Finnish.","major_comments":[{"comment":"The claim that the new Finnish data are 'consistently better' than Kylliäinen's Finnish SQuAD is not established by the reported F1 comparison, because the two models are trained and evaluated on different translation pipelines. A model trained on DeepL output can learn systematic DeepL translation patterns present in both train and dev, inflating scores without improving performance on naturally occurring Finnish. Please report cross-evaluation (training on one dataset and evaluating on the other's dev set) or a manually translated Finnish test set; the Spanish experiment in Table 3 demonstrates that this design is feasible and should be replicated for Finnish.","section":"Sec. 4.3, Table 1"},{"comment":"The manual error analysis only samples answerable questions and judges span fidelity relative to the English original; it does not assess the grammaticality or naturalness of the Finnish answer text, nor whether unanswerable questions remain unanswerable after translation. It therefore supports the mechanism of span transfer but not the target-language quality, which is the load-bearing part of the resource claim for Finnish.","section":"Sec. 4.4, Table 4"},{"comment":"The backtranslation experiment estimates the single-round penalty by halving the two-round drop in EM and F1, which relies on the unverified assumption that translation errors are approximately evenly distributed between the en->fi and fi->en rounds. Since Finnish QA is the primary contribution, this assumption should be validated, for example on a small manually translated Finnish test set, or the derived 4.2 EM / 2.6 F1 figures should be presented explicitly as a rough heuristic rather than as the expected one-round loss.","section":"Sec. 4.2, Table 2"},{"comment":"In the Spanish head-to-head, training on the new method's data yields lower EM on the TAR test set (64.5 vs 66.3) while improving F1 (74.0 vs 73.7). The paper notes the 'opposite tendency' of EM but does not investigate it, even though the abstract claims the method produces 'consistently better translated data'. A short analysis of span-boundary statistics, answer lengths, or entity types would help clarify whether the F1 gain reflects overall translation quality rather than a systematic boundary bias.","section":"Sec. 4.3, Table 3"}],"minor_comments":[{"comment":"Please specify the grid-search space and the criterion used to select the FinALBERT pretraining learning rate (5.28e-3), including whether a held-out split was used, so that the negative FinALBERT result can be reproduced and interpreted.","section":"Sec. 3.3"},{"comment":"The paper reports preserving 90,233 of 92,749 answerable QA pairs but does not state how many of the 49,434 unanswerable SQuAD2.0 questions are preserved in the released dataset; please clarify the full dataset statistics.","section":"Sec. 3.2"},{"comment":"The cross-dataset comparison in Table 1 mixes model sizes, pretrained backbones, and evaluation splits; adding a column or footnote stating the model type, training split, and eval split for each row would make the comparison more transparent.","section":"Sec. 4.1, Table 1"},{"comment":"The statement 'the training data used was identical to that used to train the FinBERT model' is too coarse for reproducibility; please report the pretraining steps, batch size, and checkpoint selection procedure for FinALBERT.","section":"Sec. 3.3"},{"comment":"The cost figure 'approximately 20C' should be written as '€20' or '20 EUR', with a note on the conversion rate or pricing tier used.","section":"Sec. 3.3"},{"comment":"The 'web source' entries in Table 1 lack full publication details; consider moving the URLs into the bibliography or a dedicated data-availability section so that the cited online resources remain traceable.","section":"Table 1, references"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a resource paper at this venue. The main risk is that the Finnish evaluation does not rule out train/test consistency effects from the DeepL translation pipeline; I would require a cross-evaluation against Kylliäinen's Finnish SQuAD or an explicitly labeled human-translated Finnish test set before acceptance. The related-work coverage is adequate and I see no novelty or attribution concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper turns a small feature of DeepL—translating formatted documents—into a cheap way to build span-annotated QA datasets in new languages. The trick is not new; the authors already used it for relation extraction in Bassignana et al. (2023), and they cite it. What is new is the QA adaptation with overlapping spans, the resulting Finnish SQuAD2.0, and a genuinely multi-pronged evaluation. That is a solid, reproducible contribution within the subfield.\n\nCredit where due: 87.2% of 321 manually inspected answer spans are fully correct; the backtranslation drops EM by 8.4 points over two rounds, suggesting roughly half that per direction if errors are symmetric; and the Spanish head-to-head against Translate-Align-Retrieve is the strongest evidence, with F1 improving even when training on the DeepL data and testing on TAR data. The dataset also retains 97.2% of the original question-answer pairs, compared with 71.9% for the earlier Finnish SQuAD. Code, data, and a trained model are released openly.\n\nThe soft spots are real but not fatal. The headline Finnish comparison (73.7 vs 61.9 F1) is not apples-to-apples: train and dev come from different translation pipelines, and there is no human-annotated Finnish test set or Finnish cross-dataset evaluation. Because the train and dev splits share one DeepL pass, systematic translation errors can be learned and inflate the score relative to performance on naturally occurring Finnish. The manual error analysis checks fidelity to the English source, not naturalness of the Finnish, and samples only answerable questions. The backtranslation estimate leans on an unverified assumption about symmetric error rates. And the method depends on a closed commercial service with no formatting-fidelity guarantee. The authors acknowledge all of these in the Limitations section; they do not hide the dependence, which is more than many dataset papers do.\n\nNet: the resource is worth having, the method is likely to transfer to other languages, and the evaluation is honest. The 'consistently better' claim is directly supported for Spanish and only indirectly for Finnish—a reason to soften the wording, not to reject. I would send this to review, and I would cite the dataset if I were doing Finnish QA.","headline":"A practical, low-cost trick for span-preserving MT of QA datasets, honestly evaluated; the Finnish resource is useful even if the headline advantage is partly self-referential.","tokens_in":10588,"tokens_out":3502,"would_cite":true,"duration_ms":34906,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a span-annotated question-answering dataset can be machine-translated by coloring the answer spans and letting a commercial translation service preserve the formatting, producing a Finnish SQuAD2.0 that keeps 97.2%…","keywords":["Finnish SQuAD","machine translation","span annotations","DeepL","question answering","dataset translation","annotation transfer"],"falsifier":"Manually audit a larger random sample, including passages with overlapping answer spans, long answers, and translations that reorder clauses; if the color-to-span mapping is lost in more than a few percent of overlapping cases, or if the 87.2% correct rate drops sharply on the larger sample, the central claim weakens. A quantitative test would be to have native Finnish speakers re-annotate the same translated passages and compare a QA model trained on the machine-translated spans against one trained on the human re-annotations, holding everything else fixed.","tokens_in":9554,"feed_emoji":"🎨","tokens_out":7697,"duration_ms":70246,"temperature":0.7,"pith_summary":"The paper sets out to show that the hardest part of translating a question-answering dataset, keeping the answer spans aligned with the translated passages, can be avoided entirely by encoding the spans as colored text and asking a commercial machine-translation service to translate a formatted document. Applied to SQuAD2.0, the method preserves 97.2% of the original question-answer pairs and produces a Finnish dataset on which a standard Finnish BERT reader reaches F1 73.7, compared with 61.9 on the previous translate-and-align Finnish SQuAD. If the claim holds, any span-annotated dataset can be ported to any language supported by the service for roughly the cost of the API calls, without hand-written alignment heuristics.","feed_headline":"Color-coded answer spans beat translate-and-align for Finnish SQuAD","feed_subtitle":"DeepL keeps 97.2% of the pairs; Finnish QA models gain about 12 F1 points.","key_machinery":"The mechanism is DeepL's formatting-preserving document translation, treated as a span-transfer channel. Input documents encode each answer as a distinct color, with overlapping answers split so the shared region gets its own color; after translation the colors are mapped back to answer spans, trailing punctuation is stripped, and discontinuous spans produced by translation are stored both as-is and as a continuous first-to-last span so standard QA architectures can consume them. This single mechanism replaces the usual translate-and-align machinery, and it is why the method is simple, cheap, and language-agnostic within the service's language coverage.","core_discovery":"The central discovery is that formatted-document transfer through DeepL carries answer-span annotations across languages with high fidelity: when each answer span is rendered in its own color in the source document, the translated document arrives with the coloring intact, and the color-to-span mapping can be reversed mechanically. On SQuAD2.0 to Finnish, 90,233 of 92,749 question-answer pairs survive, a 97.2% preservation rate, and a manual audit of 321 spans finds 87.2% flawless transfers with only 2.2% missing. The resulting Finnish QA models clearly outperform models trained on the earlier Finnish translate-and-align dataset, and a purpose-built Spanish comparison shows the same method beating the alignment-based TAR pipeline.","pith_inferences":["Editorial inference: the same pipeline should be directly applicable to other span-annotated tasks, including named entity recognition and event extraction, where the overlap-handling by extra colors removes the limitation noted in the earlier relation-extraction use of the method.","Editorial inference: because the backtranslation experiment estimates per-round noise, one could pre-screen candidate target languages for translation quality by running the method in reverse and measuring downstream F1, before spending any annotation effort.","Editorial inference: the 2.6% discontinuous answer spans are mostly translation artifacts; rephrasing questions or passages to avoid clause-boundary splits might reduce that rate, though the paper keeps discontinuous spans for compatibility with standard architectures."],"forward_implications":["The Finnish dataset preserves 90,233 of the original 92,749 question-answer pairs, a 97.2% retention rate that is far above the previous Finnish translate-and-align dataset.","A FinBERT-base reader trained on this dataset reaches EM 68.2 and F1 73.7, clearly above the 55.5 EM and 61.9 F1 reported for the previous Finnish SQuAD with a comparable model.","An English-Finnish-English backtranslation experiment loses 8.4 EM points and 5.1 F1 points over two translation rounds, implying a penalty of roughly 4.2 EM and 2.6 F1 points per round.","The same formatting-transfer method applied to Spanish SQuAD beats the alignment-based TAR dataset when trained and tested in-domain (F1 79.2 vs 73.7), and cross-training on the new dataset improves F1 on both test sets.","A manual error analysis of 321 sampled answers finds 87.2% transferred flawlessly, 9.0% over-extended, and none wrong, indicating the dominant error is minor span over-extension."],"supporting_citations":[{"why":"Source dataset; provides the SQuAD2.0 question-answer pairs and unanswerable questions that the translation method preserves.","marker":"Rajpurkar et al. (2018)"},{"why":"Previous Finnish SQuAD2.0 translate-and-align dataset; serves as the main baseline the new dataset and models are compared against.","marker":"Kylliäinen and Yangarber (2023)"},{"why":"Spanish SQuAD built with Translate-Align-Retrieve; baseline for the cross-method Spanish comparison.","marker":"Carrino et al. (2019)"},{"why":"Metadata-tagging annotation-preserving translation approach; comparison point for annotation preservation and motivation for formatting-based transfer.","marker":"Seinen et al. (2024)"},{"why":"Defines the BERT span-detection training setup used to fine-tune the Finnish QA readers.","marker":"Devlin et al. (2019)"},{"why":"Provides the Finnish BERT model that the main QA reader is fine-tuned from.","marker":"Virtanen et al. (2019)"},{"why":"Earlier instance of DeepL formatted-document transfer for relation extraction; supports the method's reuse on span-annotated data.","marker":"Bassignana et al. (2023)"},{"why":"ALBERT architecture; the paper attempts to replicate its reported English SQuAD gains for Finnish without finding them.","marker":"Lan et al. (2020)"}],"fun_headline_variants":["Color-coded answer spans survive DeepL transfer to Finnish","DeepL formatted-documents carry 97.2% of Finnish SQuAD spans","Simple DeepL trick beats align-and-translate for Finnish SQuAD","Finnish SQuAD via color-coded spans outperforms earlier dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that DeepL's formatted-document translation preserves the color-coded answer spans reliably and that the colored regions in the output are correct translations of the original answers, a property the paper checks on only 321 manually inspected examples and cannot verify from the vendor.","fun_headline_variants_meta":{"raw":{"variants":["Color-coded answer spans survive DeepL transfer to Finnish","DeepL formatted-documents carry 97.2% of Finnish SQuAD spans","Simple DeepL trick beats align-and-translate for Finnish SQuAD","Finnish SQuAD via color-coded spans outperforms earlier dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1504,"prompt_tokens":892,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":508,"tokens_out":612,"duration_ms":5305,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:18.442239+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually audit a larger random sample, including passages with overlapping answer spans, long answers, and translations that reorder clauses; if the color-to-span mapping is lost in more than a few percent of overlapping cases, or if the 87.2% correct rate drops sharply on the larger sample, the central claim weakens. A quantitative test would be to have native Finnish speakers re-annotate the same translated passages and compare a QA model trained on the machine-translated spans against one trained on the human re-annotations, holding everything else fixed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Previous Finnish SQuAD2.0 translate-and-align dataset; serves as the main baseline the new dataset and models are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Metadata-tagging annotation-preserving translation approach; comparison point for annotation preservation and motivation for formatting-based transfer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier instance of DeepL formatted-document transfer for relation extraction; supports the method's reuse on span-annotated data."}],"review_version":1}