{"id":"55df7c21-7b87-46a2-9be4-0c2f36c22656","arxiv_id":"2501.05981","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across five LLMs and three languages, answers to the same 13 fact-checkable North Korea prompts differ substantially by model and by language, with refusal rates and accuracy varying.","lead":"This paper tests five large language models on 13 North Korea topics in English, Korean, and Chinese, and finds that answers vary sharply by model and language. It uses North Korea as a case where reliable sources are scarce and false rumours spread easily, arguing that standard hallucination fixes are not enough for such settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Korean false-rumour prompt asserts 'clear fact' while English says 'Apparently'; reported language differences in Table 1 may partly be prompt-strength effects.","rationale":"The reader's weakest_assumption already identifies the Section 4.1.2 Korean prompt as a potential confound, and I agree this is the single most load-bearing issue. The paper's central qualitative finding — that outputs differ across model and language — is plausible and the dataset is transparently small, but the specific inference about language differences requires that only the language changes between conditions. Here the false-rumour prompts differ in assertiveness ('Apparently' vs 'clear fact'), and because the paper itself invokes sycophancy as the reason for using these markers, the directional risk is clear: stronger assertions of a false premise are more likely to elicit agreement, lowering measured accuracy in Korean and possibly Chinese independently of any language-specific knowledge. The concern does not require assuming the authors intended this; it is an internal design issue that can be resolved empirically. The proposed ablation is straightforward and directly tests whether the language effect survives prompt-strength matching. If it does not, the conclusion should be softened to 'model and prompt framing jointly shape answers' rather than 'language leads to vastly different understandings.' Since the reader already conditioned the verdict on this issue, I do not propose moving the verdict; the existing CONDITIONAL assessment is the right level of caution.","tokens_in":22602,"tokens_out":2414,"duration_ms":26650,"concrete_test":"Re-run the 7 false-rumour topics on the three trilingual models with matched epistemic framing across languages: use a back-translated English 'Apparently...', Chinese '据说...', and Korean '듣기로는...' for all conditions, keeping the same 5 trials per topic and the same annotation rubric. If the Korean accuracy gap versus English observed in Table 1 shrinks or disappears, the reported language effect for false rumours is substantially a prompt-framing artifact. Also report bootstrap confidence intervals over the 13 topics for the language contrasts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of RQ2 is that the choice of language leads to materially different LLM outputs about North Korea. The most load-bearing assumption behind this claim is that the prompts used across languages are equivalent in epistemic framing. Section 4.1.2 violates this assumption for the false-rumour category. The English prompt begins 'Apparently {rumour}', signaling hearsay or uncertainty; the Korean prompt instead states '{rumour}는 것은 명확한 사실이야' ('it is a clear fact that {rumour}'), an unhedged assertion of truth. The Chinese prompt uses '显然是' ('obviously is'), which is also stronger than 'Apparently'. Since the paper explicitly motivates the use of discourse markers through the phenomenon of sycophancy — models tend to agree with users even when users are mistaken — the stronger Korean (and possibly Chinese) framing could push models toward confirming a false premise more than the English framing does. This makes the language variable entangled with prompt assertiveness. For false rumours in particular, the observed accuracy gaps across languages (e.g., ChatGPT-3.5 Korean FR accuracy 44.62 vs English 56.92) could reflect prompt strength rather than language-specific knowledge or worldviews. Because the abstract and conclusion generalize to 'choice of model and language' causing 'vastly different understandings,' an uncontrolled prompt confound directly weakens the evidence for the language part of that claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a multilingual, multi-model evaluation of how large language models handle North Korea-related information. The authors construct 13 topics with verifiable ground truth, split into false rumours and lesser-known information, and prompt ChatGPT-3.5, Gemini, Claude 3 Sonnet, Qwen-72B-Chat, and Solar Mini in English, Mandarin Chinese, and Korean with five repetitions per condition. They report accuracy, refusal-to-answer rate, and consistency in Table 1, supplemented by qualitative examples in the appendix. The central claim, advanced in the abstract and conclusion, is that the choice of model and language can lead to materially different factual understandings of North Korea, with implications for misinformation and geopolitical information access.","tokens_in":22966,"tokens_out":6844,"duration_ms":61862,"significance":"The paper addresses a genuinely important and understudied problem: LLM hallucination in an information-scarce, geopolitically sensitive domain where common mitigation strategies (grounding, abstention) may be counterproductive. The design has notable strengths: the ground truth for the 13 topics is externally verifiable, the topic-generation step using GPT-4 is not circular because the evaluated models are different systems, the inclusion of language-specific open models (Qwen for Chinese, Solar Mini for Korean) provides a useful contrast to the multilingual closed models, and the paper is transparent about its annotation workflow. If the observed differences are reliable, the study would be a valuable early demonstration that users of different LLMs in different languages may receive different factual pictures of North Korea. The contribution is largely empirical and descriptive rather than methodological, but the domain relevance is high.","major_comments":[{"comment":"The cross-linguistic prompts for the false-rumour category are not semantically equivalent. The English prompt hedges with \"Apparently\", while the Korean prompt asserts \"{description}는 것은 명확한 사실이야\" (\"it is a clear fact that ...\") and the Chinese prompt uses \"显然是\" (\"obviously is\"), both of which express a stronger epistemic commitment. Because the paper's RQ2 conclusion attributes accuracy differences to language (e.g., ChatGPT-3.5 Korean FR accuracy 44.62 vs. English FR accuracy 56.92 in Table 1), prompt assertiveness is a direct confound for the language effect. The authors should either re-run the false-rumour prompts with matched epistemic framing across languages, or explicitly re-analyse the data excluding false rumours; otherwise the language comparison for this category is not clean.","section":"§4.1.2 and Appendix A.1"},{"comment":"The abstract and conclusion assert \"significant differences\" and \"vastly different understandings,\" but the paper reports no significance tests, confidence intervals, or effect sizes. With only five regenerations per model-by-language-by-topic cell and 13 topics, the observed differences may be within sampling variability; for example, Gemini's Korean FR RtA ratio (38.46) is nearly equal to its English FR RtA ratio (40.00). The authors should add a permutation or bootstrap analysis over trials and topics, or explicitly weaken the wording to \"observed differences in this exploratory set.\" The Limitations section's assertion that dataset size \"becomes less critical\" once differences are apparent needs statistical backing rather than an argument from belief.","section":"§5.2 and Table 1"},{"comment":"The accuracy metric requires subjective judgment about the balance of evidence in model outputs, yet the paper reports no inter-annotator agreement. The text says there was \"minimal ambiguity\" but provides no kappa or other agreement measure, and the annotation process relied on at most three annotators with different language expertise. Because accuracy is the primary quantitative comparison in Table 1, an agreement statistic is needed to support the reliability of the labels. Without it, the reader cannot distinguish measurement error from genuine model differences.","section":"§5 (annotation paragraph) and §4.3.1"},{"comment":"The paper does not release the raw model outputs, annotations, or the exact model versions used (e.g., which GPT-3.5 snapshot, which Gemini version, which Claude 3 Sonnet release), and the access window is only described as March 2024. Given that the central claim is empirical and that the authors themselves note the topics are a small, expert-filtered set without a sampling frame, these omissions limit reproducibility and make it difficult to assess whether the findings generalize beyond the 13 topics. At minimum, exact model identifiers and the full prompt/output/annotation files should be provided in a supplement or repository.","section":"§4.2 and Limitations"}],"minor_comments":[{"comment":"Each metric cell appears to contain three numbers (e.g., 56.92, 34.29, 83.33 for ChatGPT-3.5 English accuracy), but the caption only defines (FR) and (LKI). Please clarify what the third number represents.","section":"Table 1"},{"comment":"The Gemini Korean consistency cell contains \"0.50 (I)\"; this should be \"0.50 (LKI)\" to match the other entries.","section":"Table 1"},{"comment":"The phrase \"artibrary generation\" is a typo for \"arbitrary generation.\"","section":"§4.1.1"},{"comment":"The Chinese false-rumour prompt contains duplicated text \"你对你对自己的答案\" and the English prompt has a doubled period after \"your answer..\"; these should be corrected.","section":"§4.1.2"},{"comment":"The caption claims \"The semantics for the input text is the same for all languages,\" which conflicts with the differing epistemic strength documented in major comment 1; the claim should be revised or the prompts aligned.","section":"Appendix A.1 caption"},{"comment":"The statement that \"both Qwen-72B and Solar Mini achieved 100% accuracy for verified topics\" is not visible in Table 1; either include subcategory results in the table or refer explicitly to the supplementary material.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The prompt confound in §4.1.2 is the most serious issue and is fixable: the authors can add a matched-prompt control in which all languages use the same epistemic hedge, or they can restrict the language claim to the lesser-known-information category. If such a control is not feasible, the conclusions should be reframed as strictly exploratory. I would not recommend rejection because the case study is motivated and the domain is important; however, the statistical and transparency gaps must be addressed before the claims in the abstract can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core observation here is real and worth taking seriously: LLM outputs about North Korea vary by model and by language on fact-verifiable topics. The paper builds a small but genuinely new benchmark—13 topics, split into false rumours and lesser-known facts—and tests five models across English, Korean, and Chinese, including language-specific models. That setting is under-explored, and the qualitative examples in the appendix make the variation concrete. The finding that Claude 3 Sonnet leads on accuracy while Gemini refuses at high rates is useful, and the authors are honest about the dataset's limits.\n\nThe soft spot is the prompt-translation confound, and it is real. In the false-rumour category, the English prompt says \"Apparently\", inviting skepticism, while the Korean prompt asserts the rumour is \"a clear fact\" and the Chinese prompt says \"obviously is\". Because the paper itself motivates these discourse markers through sycophancy, part of the reported \"language effect\" could be a prompt-assertiveness effect rather than language-specific knowledge or worldview. That directly weakens RQ2 as stated. The absence of significance tests, confidence intervals, inter-annotator agreement, and data release compounds the problem. With 13 topics and five trials per cell, a single topic can swing accuracy by twenty points, so the headline numbers are noisier than they look.\n\nThat said, the central qualitative claim—that some models and languages produce materially different factual pictures of North Korea—probably survives the confound. The gaps in Table 1 are large and appear across categories, not just false rumours. The paper is an honest exploratory study, not a definitive one.\n\nWho should read it: people working on hallucination evaluation, multilingual LLM behavior, and misinformation in high-stakes domains. With revisions—matching the prompts across languages, adding basic statistical grounding, and releasing the annotation data—it would be a solid workshop or short conference paper. As it stands, it deserves a serious referee rather than a desk rejection. I'd send it to review, with the prompt confound as the main revision request.","headline":"Worth a serious referee: the paper documents real model- and language-linked variation in LLM answers about North Korea, but the language comparison is partly confounded by unequal prompt assertiveness.","tokens_in":23351,"tokens_out":2309,"would_cite":true,"duration_ms":24947,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that which LLM you ask, and in which language, materially changes the factual picture users get of North Korea.","keywords":["LLM hallucination","North Korea","multilingual evaluation","false rumours","geopolitical misinformation","refusal to answer","sycophancy","AI sovereignty"],"falsifier":"If the prompts were back-translated to equalize assertiveness across languages (e.g. using the same 'Apparently' hedge in Korean) and the Korean/English accuracy differences disappeared, the central claim that language itself shapes understanding would be falsified. Alternatively, a larger sample of real user queries about North Korea that showed no model-by-language interaction would undercut the generalization.","tokens_in":22375,"feed_emoji":"🌏","tokens_out":6247,"duration_ms":55353,"temperature":0.7,"pith_summary":"The paper takes North Korea as an extreme test case for LLM hallucination: a country with scarce verifiable information and many uncorrected false rumours. It asks whether five widely used chatbots — ChatGPT-3.5, Gemini, Claude 3 Sonnet, Qwen-72B, and Solar Mini — produce accurate, consistent, and non-refusing answers about North Korea in English, Korean, and Mandarin Chinese. Across 13 topics with verifiable ground truth, the paper measures large differences in accuracy, refusal-to-answer, and consistency depending on the model and the language. The central claim is that the choice of model and language can yield materially different factual pictures of North Korea, which matters because users increasingly rely on LLMs for information about a geopolitically sensitive country. If true, hallucination-mitigation strategies that simply add credible sources or encourage abstention are not enough for closed-information environments.","feed_headline":"Which chatbot you ask changes the North Korea story","feed_subtitle":"Five chatbots, three languages, 13 checked topics: accuracy and refusal rates diverge enough to shape public understanding.","key_machinery":"The measurement apparatus is a small curated dataset of 13 North Korea topics with verifiable ground truth, built by prompting GPT-4 for topic candidates, keeping only topics that appeared more than once, and filtering by a domain expert. The topics are split into false rumours (widely circulated but debunked claims, e.g. Kim Jong Un's death or a unicorn lair) and lesser-known information (true but under-documented facts, e.g. the five-year election cycle or the Arduous March). To provoke hallucination, the prompts deliberately use discourse markers that pressure the model: 'Apparently' for false rumours and 'I heard' for lesser-known information, exploiting sycophancy. Each model is prompted five times per topic, and three metrics are computed: accuracy against ground truth, refusal-to-answer rate, and answer-type consistency. This design lets the paper attribute answer differences to model and language while holding topic content fixed.","core_discovery":"On the paper's own terms, the discovery is that 'the choice of model and language can lead to vastly different understandings of North Korea.' For 13 topics with verifiable truths, accuracy ranged widely: Claude 3 Sonnet was generally most accurate, ChatGPT-3.5 next, Gemini lowest, while language-specialised models (Qwen-72B for Chinese, Solar Mini for Korean) were highly accurate for verified topics but not for false rumours. Gemini refused to answer frequently across all languages even on non-sensitive topics, whereas Qwen almost never refused. The paper also reports that models were less accurate on false rumours than on lesser-known true information, and that prominent death rumours about Kim Jong Un, Hyon Song Wol, and Jang Song Thaek persisted in model outputs despite available corrections. The authors attribute part of the language gap to English-centric training data, and they interpret the high accuracy of language-specific models as consistent with the case for 'AI sovereignty.'","pith_inferences":["Not in the paper: the 'language effect' is partly confounded with prompt force. The Korean false-rumour prompt asserts the rumour is a clear fact, while English uses 'Apparently'; a controlled back-translation study could isolate whether language or prompt assertiveness drives the accuracy gap.","Not in the paper: because all 13 topics were generated by GPT-4 and filtered by a single expert, the dataset is a convenience sample, not a representation of real user queries; a larger query log from search or chat data would test whether the patterns generalise.","Not in the paper: the same method could be applied to other closed or partially closed information environments (e.g. Myanmar, Eritrea, Turkmenistan) where rumour persistence and source scarcity may produce similar model-by-language divergence.","Not in the paper: consistency as measured (1/number of answer types) rewards repeated refusal, so a model that always declines can score as highly consistent; future metrics should separate engagement from uniformity."],"forward_implications":["Users of different LLMs, or the same LLM in different languages, can receive contradictory factual information about North Korea on the same topic.","Refusal-to-answer behaviour does not reliably signal lack of knowledge: Gemini's high refusal rate inflated its consistency score while lowering its accuracy, so users may see a conservative model as a consistent one.","False rumours that were widely reported but weakly corrected remain a persistent hallucination source even for prominent cases with available corrections.","Language-specific models can outperform general multilingual models in their own language on verified facts, lending support to 'AI sovereignty' arguments.","Hallucination mitigation via retrieval augmentation or uncertainty expression may fail in closed-information domains, where expert-curated sources and careful source design are needed."],"supporting_citations":[{"why":"Documents how North Korea misinformation persists because corrections are scarce and unprofitable, grounding the choice of North Korea as the case study.","marker":"Seo (2023)"},{"why":"Shows sensationalist UK coverage of North Korea and the 'Kim Jong-un effect', explaining why false rumours enter model training data.","marker":"Seo et al. (2023)"},{"why":"Supplies the 'arbitrary generation' method used to prompt GPT-4 for candidate topics.","marker":"Chen and Shu (2024a)"},{"why":"Establishes sycophancy in LLMs, the behaviour the discourse-marker prompts are designed to provoke.","marker":"Sharma et al. (2024a)"},{"why":"Provides evidence that users trust LLM outputs without critical scrutiny, motivating a user-perspective evaluation.","marker":"Spatharioti et al. (2023)"},{"why":"Representative of retrieval-augmented generation, the main mitigation strategy the paper argues is insufficient for closed information environments.","marker":"Lewis et al. (2020)"},{"why":"Shows hallucination leaderboards with limited North Korea coverage, motivating the paper's topic-specific evaluation.","marker":"Hong et al. (2024)"},{"why":"Shows pre-existing attitudes amplify misinformation belief, the mechanism that makes North Korea rumours consequential.","marker":"Ecker et al. (2014)"}],"fun_headline_variants":["Chatbot choice reshapes North Korea answers","Ask five AIs about North Korea, get five realities","LLM accuracy on North Korea varies by model and tongue","North Korea facts differ sharply across chatbots","Pick your AI: North Korea answers split by model and language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The findings rest on the assumption that the 13 expert-selected topics and the translated prompts represent how users actually ask about North Korea, and that the translated prompts are equivalent in how strongly they assert the rumour.","fun_headline_variants_meta":{"raw":{"variants":["Chatbot choice reshapes North Korea answers","Ask five AIs about North Korea, get five realities","LLM accuracy on North Korea varies by model and tongue","North Korea facts differ sharply across chatbots","Pick your AI: North Korea answers split by model and language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1271,"prompt_tokens":938,"completion_tokens":333,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":258}},"tokens_in":554,"tokens_out":333,"duration_ms":3825,"temperature":1.0,"reasoning_tokens":258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:34.721995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If the prompts were back-translated to equalize assertiveness across languages (e.g. using the same 'Apparently' hedge in Korean) and the Korean/English accuracy differences disappeared, the central claim that language itself shapes understanding would be falsified. Alternatively, a larger sample of real user queries about North Korea that showed no model-by-language interaction would undercut the generalization.","supporting_citations":[],"review_version":1}