{"id":"fed44aed-9ab9-4995-a575-0d5fb7a8b6ef","arxiv_id":"2502.07642","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FoQA provides the first extractive question-answering benchmark for Faroese, with 2,000 human-validated samples and baseline scores for nine models.","lead":"FoQA is a new Faroese extractive question-answering dataset with 2,000 validated question-answer pairs built from Faroese Wikipedia articles using GPT-4-turbo and native-speaker review. It is the first QA benchmark for Faroese and provides baseline scores for nine language models, from small multilingual BERTs to GPT-4-class systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset counts cannot be reconciled: 4,130 annotated, 2,000 validated, and 2,395 rejected do not partition cleanly, so the composition of the 2,000-sample benchmark is not yet established.","rationale":"This is a resource paper; the load-bearing statement is the abstract's claim of a dataset with 2,000 samples released in three versions. The generation pipeline is reproducible and the data and code are released, which is real support for the resource. The most insecure point is the sample accounting in Sections 5.2 and 5.3, because the text's own numbers cannot be reconciled on any straightforward reading. This is an internal inconsistency rather than a disagreement with consensus, and it directly concerns the size and composition of the benchmark. The annotation-quality concern (no inter-annotator agreement) is acknowledged and important, but it is a known limitation rather than a contradiction; the sample accounting is an objective check that should be settled first. The proposed Hugging Face audit would settle it. The reader already flagged the statistics ambiguity as one of the main issues, so my concern partly overlaps with the reader's reading; I do not change the CONDITIONAL verdict, because the requested clarifications would resolve or sharpen this issue either way.","tokens_in":12249,"tokens_out":9191,"duration_ms":86593,"concrete_test":"Load the three released splits from the Hugging Face repository (alexandrainst/foqa) and verify lengths: default should be 2,000, incorrect-samples 2,395, all-samples 10,001. Then audit id overlap: take the 241 entries flagged corrected in the released files and check whether they are disjoint from the 1,759 entries labelled initially correct; recompute the partition of the 4,130 annotated samples. Also count how many unique rejected ids appear in incorrect-samples and compare with 2,130. If default has exactly 2,000 unique entries and the overlap/partition resolves as described, the reporting error is harmless; if not, the 2,000-sample claim and the three-version description need revision before the benchmark is used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2 reports 4,130 annotated samples split as 1,759 Correct, 1,908 Incorrect, and 222 Incorrect Answer, and then adds that 241 samples have the label Corrected, including 41 of the 1,759 initially Correct samples. Since 1,759 + 1,908 + 222 + 241 = 4,130, the four labels are treated as disjoint; but if 41 of the 1,759 are also in Corrected, the Correct count and the Corrected count overlap, so the validated default set of 2,000 = 1,759 + 241 cannot be a count of unique samples. Section 5.3 further says the released incorrect-samples version contains 2,395 rejected samples, whereas the manual review described can account for only 1,908 + 222 = 2,130 annotated rejects. These discrepancies leave the relationship between the three released versions, and the exact composition of the 2,000-sample validated benchmark, ambiguous. Because the central claim is precisely that FoQA contains 2,000 validated samples that form a usable benchmark, this is load-bearing: either the validation categories are misreported or the released splits are not the ones described.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FoQA, a Faroese extractive question-answering dataset constructed semi-automatically from Faroese Wikipedia using GPT-4-turbo for question and answer generation, question rephrasing, and native-speaker validation. The authors report a validated set of 2,000 QA pairs, release three dataset versions, evaluate nine models (LLMs and encoder-only models) on the benchmark, analyze question type distribution, and discuss qualitative error patterns. The central claim is that FoQA is the first Faroese QA evaluation resource and serves as a useful benchmark for Faroese language understanding.","tokens_in":12400,"tokens_out":9310,"duration_ms":87648,"significance":"If the central claim holds, FoQA fills a clear gap: no dedicated Faroese extractive QA benchmark currently exists, and the paper provides a reproducible semi-automated pipeline, open-source code, released datasets, and baseline scores across nine models. These are meaningful strengths for a low-resource language resource. The impact is moderate: the dataset is small (2,000 validated pairs) and the evaluation is not accompanied by human performance or inter-annotator agreement, but the resource itself is potentially useful for the NLP community.","major_comments":[{"comment":"The reported sample counts do not reconcile. Section 5.2 reports 4,130 annotated samples with 1,759 Correct, 1,908 Incorrect, and 222 Incorrect Answer, and then says that 241 samples have the label Corrected, including 41 of the 1,759 Correct samples. If Corrected is a disjoint fourth category, the arithmetic 1,759 + 1,908 + 222 + 241 = 4,130 is consistent, but then the statement that the 41 are among the 1,759 Correct is contradictory. If the 41 overlap, then the default set described in Section 5.3 as 1,759 initially correct plus 241 corrected contains only 1,959 unique pairs, not 2,000. Section 5.3 also states that the incorrect-samples version has 2,395 rejected examples, but the manual review can account for only 1,908 + 222 = 2,130 annotated rejects (or 2,171 if the 41 double-counted samples are removed from the validated set). The composition of the released default and incorrect sets must be clarified before the central claim of a 2,000-sample validated benchmark can be accepted.","section":"Sections 5.2 and 5.3"},{"comment":"The validation protocol does not provide a quantitative measure of annotation reliability. Initial validation is performed by a single annotator, and the second pass is applied only to samples initially marked Correct; moreover, that second pass is split between the original annotator and a second annotator, so half of the Correct samples are re-reviewed by the same person who made the original judgment. The Limitations section acknowledges that traditional inter-annotator agreement measurements were not performed, but the abstract and conclusion nevertheless describe the dataset as 'validated' and as a 'valuable benchmark.' I would ask the authors to either add a small inter-annotator agreement study (for example, on a random subset of 100-200 samples) or to soften the validation claim in the abstract and conclusion so that it accurately reflects the single-annotator-plus-partial-review protocol.","section":"Sections 3.2, 3.3, and 5.2; Limitations"}],"minor_comments":[{"comment":"In the second contribution bullet, the phrase 'F aroeseusing' appears to be a spacing error and should read 'Faroese using.'","section":"Section 1"},{"comment":"The bibliography entries for TyDi QA (Clark et al., 2020) and DeBERTa (He et al., 2021b) are truncated; the full titles should be restored.","section":"References"},{"comment":"The text says 'with a consistent random seed (4242) to maintain reproducibility,' but it does not specify how the seed is passed to the OpenAI API; please clarify the exact API parameter used or state that reproducibility refers to the codebase rather than the model outputs.","section":"Section 4"},{"comment":"The description of the GPT-4-turbo comparison as 'before and after the correction' is confusing; please clarify that the model is evaluated on the original and corrected versions of the same questions, and note that the reported p-values would not survive a strict multiple-comparison correction, although this does not affect the main qualitative conclusion.","section":"Section 6"},{"comment":"The dataset format includes an answer_start character index, but the paper does not state whether the shortened contexts used in the default split were created before or after computing answer_start indices; please specify this for reproducibility.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The core contribution is potentially valuable, but the count inconsistencies in Sections 5.2 and 5.3 are load-bearing: they directly affect the claimed size and composition of the released benchmark. The annotation reliability concern is also important for a resource whose main value is being 'validated.' If the authors can reconcile the counts and either provide a small inter-annotator agreement study or temper the validation claims, I would view the paper as acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FoQA is a useful, honest resource paper: it gives the first Faroese extractive QA dataset, ships code, data, and an annotation tool, and reports baselines from nine models. That is a real contribution for a language with almost no evaluation resources. The pipeline—GPT-4-turbo generation, rephrasing, then native-speaker validation—isn't novel in pieces, but the authors implement it carefully and release the artifacts, so the resource itself is usable.\n\nThe soft spots are real. The dataset accounting in Section 5 does not reconcile. 4,130 annotated samples are split into 1,759 Correct, 1,908 Incorrect, 222 Incorrect Answer, and 241 Corrected, where the 241 includes 41 of the 1,759. That means the four labels are not disjoint, so the default set of 2,000 = 1,759 + 241 double counts 41 samples. And 2,395 rejected examples in the released incorrect-samples version cannot be derived from 2,130 annotated rejects plus the 41 corrected errors. This is not a cosmetic typo; the central claim is that the benchmark contains 2,000 validated pairs, and the released versions don't yet line up with the described annotation process. I assume it's fixable, but it needs a clear table.\n\nSecond, there is no human baseline or trivial baseline. The model scores (GPT-4 at F1 77.6, encoder models around 30) are only meaningful if we know what a native speaker or a simple string-matching system would score. Without that, 'effectiveness' is uncalibrated. The authors address the main circularity concern—GPT-4-turbo generated the data—by testing on corrected questions and finding higher scores there, which is the right check. Other models tested did not generate the data.\n\nMinor points: no evaluation of original versus rephrased questions, despite rephrasing being a stated method; no inter-annotator agreement, which they acknowledge; and the related work skips other LLM-generated QA datasets for low-resource languages, so the methodological novelty is a bit overstated.\n\nWho is this for: anyone working on Faroese or low-resource Nordic NLP, and people building QA datasets with LLM generation. It deserves a serious referee. I would send it to review and ask for the count table, a human or trivial baseline, and a sentence or two acknowledging the rephrasing-evaluation gap.","headline":"First real Faroese QA benchmark with released code and data, but Section 5's numbers do not add up and need to be fixed before the 2,000-sample claim holds.","tokens_in":13041,"tokens_out":3525,"would_cite":true,"duration_ms":33015,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces FoQA, a 2,000-pair extractive question-answering dataset for Faroese built with GPT-4 and native-speaker validation, plus baseline scores for nine models.","keywords":["Faroese","extractive question answering","low-resource NLP","QA benchmark","GPT-4-turbo","native-speaker validation","Wikipedia","dataset release"],"falsifier":"Sample, say, 200 question-answer pairs from FoQA's validated set, have two independent native Faroese speakers re-apply the paper's three-tier annotation rules, and count how often they mark a published pair as not Correct; if disagreement is substantial (around 10 percent or more), the benchmark's gold labels are too noisy for reliable model ranking. A separate check would be to run a simple word-overlap baseline that selects the source sentence most similar to the question: if it approaches GPT-4-turbo's F1, the rephrasing did not remove the surface-matching weakness the paper set out to fix.","tokens_in":11972,"feed_emoji":"📚","tokens_out":11725,"duration_ms":108718,"temperature":0.7,"pith_summary":"FoQA is an extractive question-answering dataset for Faroese, a low-resource language with few evaluation resources: it contains 2,000 question-answer pairs drawn from Faroese Wikipedia, with every answer appearing verbatim in the source article. The paper argues that this is the first QA benchmark for Faroese and that it can measure how well language models understand the language, while also demonstrating a semi-automated recipe that needs only one native-speaker annotator rather than a large annotation team. If the dataset is sound, it gives the Faroese language-technology community a shared evaluation set, gives model developers concrete baseline scores to beat, and offers other low-resource languages a low-effort path to building their own QA benchmarks. The paper reports baselines for nine models, with GPT-4-turbo at F1 77.6 and BERT-style encoders around 30, and releases both the validated pairs and the rejected samples for error analysis.","feed_headline":"First Faroese QA benchmark ships 2,000 validated question-answer pairs","feed_subtitle":"Built from Wikipedia with GPT-4 and native-speaker checks, FoQA gives Faroese its first QA scoreboard.","key_machinery":"The mechanism that carries the argument is the generation-rephrasing-validation pipeline. A generative model is prompted to return, for each Wikipedia article, 2-10 questions and verbatim answers in a strict JSON structure; outputs that break the format are discarded automatically. A second prompt rewrites each question using synonyms or different word order while keeping the answer fixed, to reduce surface overlap between question and source text. Then a single native Faroese annotator applies three successive checks (grammar, meaning and context, final label) to each candidate, and every Correct item is reviewed a second time by the original annotator or a second native speaker. The 241 corrected samples are included in the released benchmark so model performance on machine-generated versus corrected questions can be compared.","core_discovery":"The paper's central contribution is FoQA, a Faroese extractive question-answering dataset of 2,000 validated samples in which every answer appears verbatim in a Faroese Wikipedia article. The samples were generated by gpt-4-turbo-2024-04-09, then a rewriting step paraphrased the questions to reduce simple text matching, and native-speaker validation filtered and corrected the results. The paper reports that on this benchmark GPT-4-turbo reaches F1 77.6 and exact match 55.6, while BERT-style encoder models score near 30, and it releases three versions of the dataset: 2,000 validated samples, all 10,001 generated samples, and 2,395 rejected samples. It also releases the open-source generation and annotation code, with the claim that the same single-annotator pipeline can create QA datasets for other low-resource languages.","pith_inferences":["Editorial extension: because inter-annotator agreement was not measured, the 2,000 gold labels rest on one initial reviewer's judgment; re-annotating a random subset with two independent native speakers would quantify how much label noise the baselines contain.","Editorial extension: the paper never compared rephrased questions to the original ones, so we do not yet know whether the rephrasing step actually increased difficulty; a head-to-head evaluation on both versions would test the method's core assumption.","Editorial extension: if the pipeline is adopted for other languages, the corrected-question result warns that an LLM's own questions may be a hidden confound in the benchmark, so released datasets should always include rejected and corrected subsets alongside the validated one."],"forward_implications":["Faroese QA evaluation now has a shared, SQuAD-compatible test set, so results from different models and future systems can be compared directly.","Published baselines give concrete targets: the strongest decoder model scores F1 77.6, while encoder models sit near 30, and any claimed improvement needs to be measured against these numbers.","The 2,395 rejected samples enable error analysis of LLM-generated Faroese, revealing patterns such as grammatical gender errors and Icelandic-influenced wording.","The 241 corrected samples offer a built-in control for measuring bias toward machine-generated question phrasing, and the paper finds the generator scored higher on corrected questions than on its own.","Releasing all 10,001 generated pairs lets other researchers train or fine-tune on the raw data instead of only the cleaned subset."],"supporting_citations":[{"why":"Defines the SQuAD extractive-QA format and evaluation approach that FoQA adopts, making the dataset compatible with existing QA pipelines.","marker":"Rajpurkar et al. 2016"},{"why":"Provides Natural Questions in Icelandic, the closest low-resource relative that FoQA extends and compares against for question-writing strategy.","marker":"Snæbjarnarson and Einarsson 2022b"},{"why":"TyDi QA's natural-question-first methodology motivates FoQA's rephrasing step, which aims to avoid questions that merely mirror source text.","marker":"Clark et al. 2020"},{"why":"Supplies the guided JSON generation approach FoQA uses to enforce the strict output format that filters malformed model output.","marker":"Willard and Louf 2023"},{"why":"Identifies gpt-4-turbo-2024-04-09, the generative model that produces the initial QA pairs and rewritten questions.","marker":"OpenAI 2023"},{"why":"Provides the ScandEval benchmark used to select GPT-4-turbo as the generator based on its Faroese performance.","marker":"Nielsen 2023"}],"fun_headline_variants":["Faroese gets its first Q&A dataset: 2,000 validated pairs","GPT-4-built Faroese Q&A benchmark: 2,000 human-verified items","FoQA: 2,000 human-checked Q&A pairs for Faroese","Testing Faroese AI: 2,000 human-verified Q&A pairs released"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's usefulness depends on the native-speaker reviewers having labelled the 2,000 released pairs correctly, even though the initial pass was made by a single annotator and no inter-annotator agreement was measured.","fun_headline_variants_meta":{"raw":{"variants":["Faroese gets its first Q&A dataset: 2,000 validated pairs","GPT-4-built Faroese Q&A benchmark: 2,000 human-verified items","FoQA: 2,000 human-checked Q&A pairs for Faroese","Testing Faroese AI: 2,000 human-verified Q&A pairs released"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00179,"raw_usage":{"total_tokens":7016,"prompt_tokens":868,"completion_tokens":6148,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":6056}},"tokens_in":484,"tokens_out":6148,"duration_ms":41786,"temperature":1.0,"reasoning_tokens":6056,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:02:35.868041+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample, say, 200 question-answer pairs from FoQA's validated set, have two independent native Faroese speakers re-apply the paper's three-tier annotation rules, and count how often they mark a published pair as not Correct; if disagreement is substantial (around 10 percent or more), the benchmark's gold labels are too noisy for reliable model ranking. A separate check would be to run a simple word-overlap baseline that selects the source sentence most similar to the question: if it approaches GPT-4-turbo's F1, the rephrasing did not remove the surface-matching weakness the paper set out to fix.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SQuAD extractive-QA format and evaluation approach that FoQA adopts, making the dataset compatible with existing QA pipelines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies gpt-4-turbo-2024-04-09, the generative model that produces the initial QA pairs and rewritten questions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ScandEval benchmark used to select GPT-4-turbo as the generator based on its Faroese performance."}],"review_version":1}