{"id":"92c36bd8-8d82-4bce-b095-b7c221f9fd8c","arxiv_id":"2509.06065","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"KatotohananQA is a Filipino translation of TruthfulQA; seven LLMs scored 94.72% in English versus 83.87% in Filipino, with GPT-5 and GPT-5 mini showing the smallest gap.","lead":"This paper introduces KatotohananQA, a Filipino translation of the TruthfulQA truthfulness benchmark, and evaluates seven proprietary LLMs on it. Across models, accuracy averaged 94.72% in English but 83.87% in Filipino, a gap that matters for fair, safe multilingual AI deployment.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"English-Filipino gap may be a translation artifact; no equivalence or annotator agreement checks are reported.","rationale":"The central claim is the cross-lingual truthfulness gap. Since the Filipino benchmark is a translation, the gap is interpretable only if the translation preserves item content and difficulty. The paper provides no quantitative evidence for this: no back-translation, no equivalence ratings, no inter-annotator agreement, and no check that code-mixing in retained English terms does not alter difficulty. The examples suggest the translation intentionally keeps English for cultural reasons, which could introduce a systematic difference. This is not an external consensus issue but an internal validity issue. The reader's weakest assumption already identified translation parallelism; my analysis agrees. A targeted equivalence check would settle whether the gap is real. Therefore the verdict stays conditional pending that check.","tokens_in":9018,"tokens_out":4492,"duration_ms":51473,"concrete_test":"Independently back-translate all 790 KatotohananQA items to English (e.g., with a professional translator or COMET-based MT evaluation) and compare semantic equivalence to the original TruthfulQA. Then recompute the English-vs-Filipino accuracy gap on only the items rated as near-equivalent; also correlate low-equivalence items with per-item model errors. If the gap disappears or shrinks materially on the equivalent-only subset, the headline gap is a translation artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim that English truthfulness overstates Filipino truthfulness depends on KatotohananQA items being parallel to the English binary-choice TruthfulQA in meaning and difficulty. The methodology (§3.1) uses Google Translate plus native-speaker revision, but reports no back-translation, no difficulty-equivalence measurement, and no inter-annotator agreement. The examples in Tables 1–3 show deliberate retention of English words/phrases ('Penny', 'Council Housed and Violent', 'eating an apple a day'), resulting in code-mixed Filipino prompts. If these prompts are less natural, more ambiguous, or systematically harder to parse than their English counterparts, the observed overall gap (−10.85 points, Table 4) and the category/topic gaps (Tables 5–7) could be artifacts of the adaptation rather than evidence of lower truthfulness in Filipino. The Discussion (§5.3) attributes the gap to training-data disparity, but never rules out translation quality as a confound. Without an equivalence check, the central comparison is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces KatotohananQA, a Filipino adaptation of the 790-item binary-choice TruthfulQA benchmark, built by machine-translating with Google Translate and then revising by native speakers. The authors evaluate seven free-tier proprietary LLMs in English and Filipino, reporting an overall accuracy of 94.72% in English versus 83.87% in Filipino, a mean gap of +10.85 percentage points. They further break down performance by model, question type, category, and topic, and use McNemar's test and Cohen's g to assess significance and effect size. The paper concludes that LLMs are less truthful in Filipino than in English, that GPT-5 and GPT-5 mini are relatively robust multilingually, and that certain question categories and topics are especially sensitive to language transfer.","tokens_in":9260,"tokens_out":4288,"duration_ms":50845,"significance":"KatotohananQA addresses a real gap: truthfulness evaluation in low-resource languages is scarce, and the binary-choice TruthfulQA format is a useful testbed. The dataset is publicly released, which is a concrete contribution to multilingual evaluation. If the reported gap is real, the finding that English accuracy systematically overstates Filipino truthfulness for several proprietary models is important for deployment and fairness. The paper also usefully connects to prior multilingual TruthfulQA work. However, the central quantitative claim depends on the translated items being parallel to the English items in meaning and difficulty, and that parallelism is not demonstrated. As it stands, the paper establishes an association between language and accuracy, but not yet that the gap is a language-specific truthfulness deficit rather than a translation artifact.","major_comments":[{"comment":"The headline claim that English truthfulness overstates Filipino truthfulness assumes that KatotohananQA items are parallel to the English binary-choice TruthfulQA items in meaning and difficulty. The paper reports a two-step translation process but provides no back-translation check, no difficulty-equivalence analysis, and no inter-annotator agreement metrics. The examples in Tables 1–3 show that many items deliberately retain English words or whole phrases (e.g., 'Council Housed and Violent', 'eating an apple a day'), producing code-mixed prompts. If these prompts are less natural, more ambiguous, or systematically harder to parse for the models than the English originals, the observed overall gap (−10.85 points, Table 4) and the category- and topic-level gaps could be artifacts of the adaptation. Section 5.3 attributes the gap to training-data disparity, but this confound is not ruled","section":"§3.1, Tables 1–3"},{"comment":"Each model was run once per item, yet statistical significance is claimed and effect sizes are reported. Moreover, GPT-5 and GPT-5 mini were evaluated at temperature 1.0 while all other models used temperature 0.0. With a single stochastic sample at temperature 1.0, the reported 0.00-point gap for GPT-5 and the 1.01-point gap for GPT-5 mini could be partly sampling noise. The paper should either match decoding settings across models, report repeated sampling with confidence intervals, or at least quantify the variance of accuracy under the actual sampling scheme. This is directly relevant to the 'multilingual robustness of OpenAI models' conclusion in §5.2.","section":"§3.3, §3.5"},{"comment":"The text states that across 5,530 instances, English was correct and Filipino incorrect 650 times, with 70 of the reverse pattern. Table 8 reports 670 in the English-correct/Filipino-incorrect cell. The table is internally consistent with the reported overall accuracies (5238/5530 ≈ 94.72%; 4638/5530 ≈ 83.87%), so the number 650 appears to be a typographical error, but the inconsistency must be corrected. If 670 is correct, the point is actually strengthened, but the text and table should agree.","section":"§5.1 vs Table 8"},{"comment":"The paper does not report the actual McNemar p-values or Cohen's g values for the individual models, categories, or topics; it only states how many groups were significant. Without these values, the reader cannot assess the strength of evidence, and no correction for multiple comparisons is mentioned. Given that 37 categories and 19 topics are tested, some significant results would be expected by chance. Please provide full statistics in a supplementary table or appendix.","section":"§3.5, §4"}],"minor_comments":[{"comment":"The 'Average' rows in these tables are unweighted means over the category/topic rows, not pooled accuracies across all items. This should be stated in the table caption or text to avoid confusion with the overall accuracy in Table 4.","section":"Tables 6 and 7"},{"comment":"The sentence 'These findings suggest the performance gap between English and Filipino isn’t about a lack of knowledge, but a disparity in training data' overstates what the data can show. The benchmark does not directly measure training data proportions or knowledge separation; this is a plausible hypothesis but should be framed as such.","section":"§5.3"},{"comment":"The GPT-5 and GPT-5 mini results (97.72% and 93.67% in Filipino) are near the English ceiling. The interpretation that this reflects 'multilingual robustness' should acknowledge possible ceiling effects and the fact that temperature settings differ.","section":"§5.2"},{"comment":"Minor typographical/style issues: 'GPT-5 Mini' and 'GPT-5 mini' are used inconsistently; Table 7 lists a topic called 'Language' with different values from the category 'Language' in Table 6, which may confuse readers; and the reference to the OpenAI blog [17] could be supplemented by a more detailed technical source.","section":"General"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you're working on multilingual evaluation or truthfulness benchmarks. The dataset is the contribution: a 790-item Filipino translation of binary-choice TruthfulQA, human-verified by native speakers, with clear translation guidelines. That's useful. The evaluation of seven proprietary models is straightforward and the results are reported per model, type, category, and topic, with McNemar and Cohen's g. The headline gap—English 94.72, Filipino 83.87—is large and the discordant pairs (650/70, though Table 8 says 670) back it up.\n\nThe soft spots are not fatal but they matter. The translation-equivalence worry is real: no back-translation, no inter-annotator agreement, and the examples show intended code-mixing where English phrases are kept for cultural fidelity. That could make Filipino items harder or easier in ways that aren't measured. The paper doesn't claim to have solved that, and it lists limitations, but the central interpretation—that the gap reflects training-data disparity—would be stronger with a difficulty check. Also, the OpenAI robustness claim rests on two models, single runs, temperature 1.0, and no contamination check; that's a thin reed for a \"strong multilingual robustness\" conclusion. And the 650/670 mismatch in the contingency table should be fixed.\n\nCredit where due: the dataset is publicly available, the method is clearly described, and the statistical reporting is better than most benchmark papers. The related work is appropriately cited, including Uhura's warning that translations may not be perfectly parallel—so the authors were aware of the risk.\n\nWho is it for? People building multilingual benchmarks and anyone studying cross-lingual truthfulness. It deserves a serious referee, with requests for the equivalence analysis and corrected numbers. My recommendation: send it to review, conditionally. It's a solid resource with an honest limitations section, not a paper that needs to be reshaped.","headline":"A useful Filipino TruthfulQA resource with a plausible but not fully pinned-down English-Filipino gap; the translation-equivalence worry is real but the paper is honest about its limits.","tokens_in":9666,"tokens_out":2154,"would_cite":true,"duration_ms":25313,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Filipino translation of TruthfulQA shows LLMs are systematically less truthful in Filipino, averaging an 11-point accuracy gap.","keywords":["KatotohananQA","TruthfulQA","Filipino","multilingual evaluation","truthfulness","large language models","low-resource languages","benchmark translation"],"falsifier":"Back-translate the 790 Filipino items to English and have independent native Filipino-English bilingual raters flag every item whose meaning, option difficulty, or cultural reference shifted in translation. If the accuracy gap on flagged items is comparable to the gap on un-flagged items, the translation-parallelism explanation is supported; if flagged items account for most of the errors, the comparison is not parallel and the central claim would need to be re-assessed.","tokens_in":8972,"feed_emoji":"🇵🇭","tokens_out":10750,"duration_ms":108009,"temperature":0.7,"pith_summary":"This paper introduces KatotohananQA, a Filipino translation of the 790-question binary-choice TruthfulQA benchmark, and uses it to ask whether large language models are as truthful in Filipino as they are in English. Seven free-tier proprietary models were evaluated in both languages with matched prompts. The central finding is a consistent gap: average accuracy is 94.72% in English but 83.87% in Filipino, and every model except GPT-5 scores lower in Filipino, with the largest drops on questions that require logical reasoning, cultural knowledge, or identifying people. The paper argues the gap is not uniform: some categories and topics lose far more accuracy than others, and the newest GPT-5 models are the most multilingual-robust. If the dataset is a faithful parallel translation, English truthfulness scores materially overstate how truthful these models are for Filipino speakers.","feed_headline":"Filipino test: AI truthfulness drops 11 points vs English","feed_subtitle":"A 790-question Filipino version of TruthfulQA shows the gap is worst on logic and cultural-knowledge questions.","key_machinery":"The central object is KatotohananQA itself, a parallel Filipino version of TruthfulQA's binary-choice setting containing 790 questions in 37 categories and 19 latent topics. Binary-choice means each question is presented with exactly one correct and one incorrect answer of similar length, so a model cannot win by length-based heuristics. The dataset carries the argument because every result is a paired English-versus-Filipino accuracy comparison on the same question instances; the two-step translation protocol — machine translation plus native-speaker revision, with explicit rules for proper nouns, acronyms, and proverbs — is what licenses treating the two versions as measuring the same know","core_discovery":"KatotohananQA is a parallel Filipino adaptation of the binary-choice TruthfulQA benchmark: 790 questions, each with one correct and one incorrect option of similar length, machine-translated and then revised by native Filipino speakers under guidelines meant to preserve meaning and cultural context. Across seven free-tier proprietary models evaluated zero-shot, English accuracy averaged 94.72% and Filipino accuracy 83.87%, a mean difference of +10.85 percentage points. The gap was positive for six of the seven models, from +1.01 points for GPT-5 mini to +27.46 for DeepSeek V3, and zero for GPT-5. Paired significance tests and effect sizes show the differences are non-trivial for most models,","pith_inferences":["The translation-parallelism assumption is untested: a back-translation or difficulty-equivalence study of the 790 items could change how much of the 11-point gap is attributed to language-specific truthfulness versus translation artifacts.","The concentration of the gap in logic puzzles, riddles, and clue-based identification suggests the bottleneck may be Filipino-mediated reasoning rather than factual coverage; fine-tuning on Filipino reasoning data would be a targeted probe of that hypothesis.","Because the new benchmark is public, future model releases can be checked against it directly, turning the reported gap into a trackable metric that does not require re-translating the questions.","A natural extension is to measure whether the binary-choice gap persists in free-form generation, where a model's tendency to hedge, switch languages, or produce false explanations may be even more visible."],"forward_implications":["If the central claim is right, English-only TruthfulQA scores overstate Filipino truthfulness for most current free-tier models by roughly 11 points on average.","The gap is largest on reasoning-heavy and culturally embedded items, so a model that looks reliable on English logic or trivia should not be assumed reliable for reasoning in Filipino.","Newer GPT-5 and GPT-5 mini show near-zero gaps, suggesting multilingual truthfulness may be improving in the newest training runs, but the improvement is not yet consistent across vendors.","Safety and deployment evaluations in Filipino-speaking contexts should include native-language truthfulness items rather than relying only on translated general benchmarks.","Category- and topic-level results can flag where a model is least trustworthy in Filipino, guiding targeted data collection or training interventions."],"supporting_citations":[{"why":"Supplies the original TruthfulQA benchmark and the imitative-falsehood construct that KatotohananQA adapts.","marker":"[4]"},{"why":"Defines the binary-choice TruthfulQA setting (790 questions, 37 categories) used as the translation source.","marker":"[5]"},{"why":"Documents Filipino's low share of Common Crawl, grounding the low-resource-language premise.","marker":"[7]"},{"why":"Filipino-language NLP benchmark cited as evidence that Filipino is under-resourced and underrepresented in training data.","marker":"[9]"},{"why":"Prior professional translation of TruthfulQA into several Western European languages; gives the multilingual-gap comparison and translation method.","marker":"[10]"},{"why":"Low-resource African truthfulness benchmark supporting the expectation that accuracy drops in low-resource languages.","marker":"[12]"},{"why":"Topic-modeling technique used to induce the 19 latent topics for the topic-level analysis.","marker":"[13]"},{"why":"Provides the paired-nominal test and effect-size procedures used to establish statistical significance and magnitudes.","marker":"[14]"},{"why":"Large multilingual benchmark supporting the observation that English performance dominates low-resource languages.","marker":"[16]"}],"fun_headline_variants":["AI truthfulness drops 11 points in Filipino benchmark","KatotohananQA shows Filipino gap in AI honesty","Filipino test: GPT-5 closes gap, others lag","New Filipino benchmark exposes truthfulness divide","AI less truthful in Filipino by 11 points"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The translated Filipino items are parallel to the English binary-choice items in meaning and difficulty, so the observed accuracy gap measures language-specific truthfulness rather than translation artifacts.","fun_headline_variants_meta":{"raw":{"variants":["AI truthfulness drops 11 points in Filipino benchmark","KatotohananQA shows Filipino gap in AI honesty","Filipino test: GPT-5 closes gap, others lag","New Filipino benchmark exposes truthfulness divide","AI less truthful in Filipino by 11 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":2966,"prompt_tokens":683,"completion_tokens":2283,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":427,"completion_tokens_details":{"reasoning_tokens":2207}},"tokens_in":427,"tokens_out":2283,"duration_ms":18449,"temperature":1.0,"reasoning_tokens":2207,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:29:47.858349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Back-translate the 790 Filipino items to English and have independent native Filipino-English bilingual raters flag every item whose meaning, option difficulty, or cultural reference shifted in translation. If the accuracy gap on flagged items is comparable to the gap on un-flagged items, the translation-parallelism explanation is supported; if flagged items account for most of the errors, the comparison is not parallel and the central claim would need to be re-assessed.","supporting_citations":[{"cited_title":"LessWrong(2025)","cited_arxiv_id":null,"evidence_quote":"Defines the binary-choice TruthfulQA setting (790 questions, 37 categories) used as the translation source."},{"cited_title":"https://commoncrawl.github.io/ cc-crawl-statistics/plots/languages","cited_arxiv_id":null,"evidence_quote":"Documents Filipino's low share of Common Crawl, grounding the low-resource-language premise."},{"cited_title":"Batayan: A Filipino NLP benchmark for evaluating Large Language Models","cited_arxiv_id":"2502.14911","evidence_quote":"Filipino-language NLP benchmark cited as evidence that Filipino is under-resourced and underrepresented in training data."},{"cited_title":"arXiv preprint arXiv:2502.09387(2025)","cited_arxiv_id":null,"evidence_quote":"Prior professional translation of TruthfulQA into several Western European languages; gives the multilingual-gap comparison and translation method."},{"cited_title":"Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languages","cited_arxiv_id":"2412.00948","evidence_quote":"Low-resource African truthfulness benchmark supporting the expectation that accuracy drops in low-resource languages."},{"cited_title":"S.: Chapter 5: Tests for Paired Nominal Data.R Companion Hand- book(2025)","cited_arxiv_id":null,"evidence_quote":"Provides the paired-nominal test and effect-size procedures used to establish statistical significance and magnitudes."}],"review_version":1}