{"id":"92757aeb-b068-4300-9638-24ef6750c95c","arxiv_id":"2607.04029","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Hallucination signals from LLM internals transfer across English–Arabic and Arabic domains for most models, depending on class separability and feature-space language alignment.","lead":"Internal signals from LLM hidden states, attention maps, and token probabilities can detect hallucinations and transfer across English–Arabic and across Arabic QA datasets for most of six models. This matters for building detectors that work in lower-resource languages without full retraining.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged GPT-4o labeling premise; that premise remains the softest link but does not overturn the transfer observations.","rationale":"The reader correctly isolates the GPT-4o-as-judge step (§3.2, Appendix D) as the load-bearing assumption that underpins every reported metric. No stronger internal inconsistency appears: the feature extraction (hidden-state Wasserstein/cosine, attention, token-probability statistics), the nine transfer settings, the language-specific normalization, and the t-SNE diagnostics are all coherent with the claim that transfer success tracks class separability plus language alignment. The small data sizes, two-language scope, and missing multi-seed bars are real limitations but secondary; they justify the CONDITIONAL verdict already assigned and do not require a further downgrade. The concrete human-agreement check above would settle the residual doubt without altering the present reading of the manuscript.","tokens_in":27654,"tokens_out":528,"duration_ms":5155,"concrete_test":"On a stratified random sample of 100 Arabic + 100 English items (balanced across TruthfulQA-AR, HalluScore, and the English counterpart), obtain independent binary labels from two native-speaker annotators using the identical four rules. Compute Cohen’s κ against GPT-4o and recompute the En→Ar and Ar→HS AUC-ROC columns of Table 1 using only the human-consensus subset. If κ < 0.6 or any transferred AUC drops >10 points relative to the GPT-4o numbers, the transfer claim is compromised; otherwise the labeling premise holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (internal-state hallucination signals transfer across languages/domains for most of the six models) rests on binary labels produced solely by GPT-4o under the four rules in §3.2 and Appendix D. Because every F1/AUC number in Table 1 is computed against those labels, any systematic language- or culture-dependent bias in the judge would be absorbed into the feature-space geometry and could be misread as genuine signal transfer. The paper already notes this limitation and supplies no human agreement numbers, multi-judge consensus, or BLEURT/GPT-4o correlation on the Arabic subset. That is the single weakest premise; once the labels are taken as given, the experimental design (monolingual baselines, both transfer directions, t-SNE alignment checks, QuantileTransformer normalization) is internally consistent and the reported patterns follow.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"CrossHallu is an empirical study of whether internal-state features used for hallucination detection (hidden-state and attention Wasserstein/cosine shifts plus token-probability statistics, following HalluShift) transfer across languages and domains. The authors evaluate six Arabic-capable LLMs on generative QA under monolingual baselines, English↔Arabic transfer on aligned TruthfulQA, Arabic cross-domain transfer (TruthfulQA-AR ↔ HalluScore), and combined language+domain transfer, reporting F1, AUC-ROC, and PR-AUC for a fixed three-layer MLP. They conclude that internal hallucination signals transfer for most models, that cross-lingual success tracks class separability and English–Arabic feature-space alignment (supported by t-SNE and layer-wise cosine trajectories), and that Arabic cross-domain transfer is asymmetric (Ar→HS often better than HS→Ar).","tokens_in":27895,"tokens_out":1357,"duration_ms":17906,"significance":"The paper fills a clear gap: internal-state hallucination detectors have been studied almost exclusively in English, while multilingual deployment and higher hallucination rates in lower-resource languages make transferability practically important. The experimental matrix cleanly isolates language shift, domain shift, and their combination; the public code, multi-model coverage, and supporting t-SNE / layer-wise analyses are genuine strengths. If the transfer patterns hold under more carefully validated labels and stricter normalization, the work would provide useful guidance on which model families support language-agnostic hallucination probes and on when training on a cleaner source domain helps a harder target. The contribution is primarily empirical and methodological rather than theoretical.","major_comments":[{"comment":"§4.4 Feature normalization: for cross-lingual (and by implication CL-CD) runs, a QuantileTransformer is “fit independently on the training and test feature sets.” Using test-set quantiles adapts the target representation with target statistics and is not pure zero-shot transfer of a detector trained only on the source. This choice is load-bearing for the central transfer claim in Table 1 (CL and CL-CD blocks). Please either (i) report the main numbers with a scaler fit only on the source/train features and applied to the target, or (ii) reframe those settings as unsupervised feature alignment / domain adaptation and add the pure-transfer numbers as the primary result. The monolingual StandardScaler protocol is fine.","section":null},{"comment":"§3.2 and Appendix D: every F1/AUC number in Table 1 is defined against binary labels from a single GPT-4o judge (temperature 0, four rules, bilingual CoT prompts). There is no human agreement, multi-judge consensus, or even GPT-4o–BLEURT/human correlation on the Arabic subset, despite the paper’s own note that BLEURT is weak on Arabic and that TruthfulQA/HalluScore contain misconception and culturally nuanced items. This is the softest load-bearing premise for the transfer conclusions. At minimum, provide human validation (or multi-judge agreement) on a stratified subset of English and Arabic labels, report agreement statistics, and discuss residual language- or culture-dependent judge bias as a threat to interpreting “signal transfer.”","section":null},{"comment":"Table 1 and §5 Cross-domain: HalluScore is introduced as “currently the only available Arabic dataset specifically designed for hallucination detection” and is authored by two of the present authors (Alansari & Luqman, 2026). Using predefined splits is appropriate, but the Ar→HS gains (e.g., Phi4-mini 57.19→72.59, Aya 80.37→86.90) and the asymmetry argument rest on treating HalluScore as an independent harder domain. Please add an explicit discussion of possible construction/overlap effects between TruthfulQA-style misconception items and HalluScore, and, if feasible, a small external Arabic QA set or a leave-one-category-out check so that cross-domain claims are not solely internal to the authors’ own benchmark.","section":null}],"minor_comments":[{"comment":"Table 1 header and caption: “1st, 2nd, 3rd best AUC-ROC per row” is stated but the manuscript text does not specify how ties or near-ties are broken; a short note would help.","section":null},{"comment":"§3.1: the token-probability feature list mentions “max/min entropy, low-prob count, gradients, percentiles” in Figure 2 but the prose derives eight scalars from pmax/pmin; a single explicit feature-dimension table (per model if dimensions differ) would improve reproducibility.","section":null},{"comment":"Figure 4 / Figure 6: layer-pair indices differ across models (different depths); stating L and the exact even/odd index sets per model in the caption or appendix would reduce ambiguity.","section":null},{"comment":"§4.1: 772/817 TruthfulQA items retained after translation; a brief characterization of excluded items (categories, failure modes) would help assess selection bias in the bilingual subset.","section":null},{"comment":"PR-AUC is repeatedly high due to class imbalance (§5); consider reporting positive-class prevalence per model/dataset in Table 1 or Appendix E so readers can calibrate F1/PR-AUC.","section":null},{"comment":"Acknowledgments section still contains the ACL template boilerplate (“This document has been adapted by Steven Bethard…”); replace with actual acknowledgments or remove.","section":null},{"comment":"Code link in the abstract points to github.com/aishaalansari57/CrossHal while Appendix C cites an anonymous.4open URL; align the public repository reference for the camera-ready version.","section":null}],"recommendation":"major_revision","confidential_remarks":"The work is a solid first empirical study and is a reasonable fit for a CL venue if the normalization protocol and label-validation gaps are fixed. The independent QuantileTransformer fit on test is the most under-discussed methodological issue; without correction or reframing, the cross-lingual numbers overstate pure transfer. HalluScore authorship is disclosed via citation but should be handled carefully in review of the cross-domain claims. I do not see circularity in the detector design itself."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is the first systematic look at whether internal-state hallucination detectors actually transfer across languages and domains. They take the HalluShift-style feature stack (hidden-state Wasserstein/cosine, attention, token-prob stats), train a small MLP, and run the full matrix: monolingual baselines, EN↔AR on aligned TruthfulQA, AR↔HalluScore, and the combined EN↔HalluScore setting, on six Arabic-capable models. That design is clean and the numbers are useful.\n\nWhat is new is the transfer evidence itself, not a new detector. Cross-lingual success tracks both class separability and language alignment in the feature space (Phi-4-mini and Aya look better aligned; Allam/Silma less so). Cross-domain inside Arabic is asymmetric: training on cleaner TruthfulQA-AR helps on HalluScore more than the reverse. The t-SNE and layer-wise cosine plots back the story; Wasserstein is less discriminative, which they note. Code is public.\n\nSoft spots are real but already flagged. Every metric sits on GPT-4o binary labels with no human agreement or multi-judge check, and Arabic/cultural items are exactly where that can bite. Data are small (~600 train), only two languages, one detector architecture, no multi-seed error bars. HalluScore is from two of the same authors, but they use the predefined splits as an external set, so circularity is low. Free parameters (MLP size, τ, layer stride, QuantileTransformer) are fixed, not swept.\n\nThis is for people building multilingual reliability tools or diagnosing representation alignment. It does not reorganize theory, but it is the right empirical first step and the patterns look solid once you take the labels as given. I would send it to peer review; the labeling gap is fixable with agreement numbers or a second judge. Worth reading and citing if you work on internal probes or Arabic LLMs.","headline":"First clean transfer study of internal-state hallucination detectors across English/Arabic and two Arabic domains; results hold for most of six models once you accept the GPT-4o labels.","tokens_in":28477,"tokens_out":511,"would_cite":true,"duration_ms":5759,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Internal hallucination signals extracted from LLM activations transfer across languages and domains for most models.","keywords":["hallucination detection","internal representations","cross-lingual transfer","cross-domain transfer","large language models","Arabic NLP","TruthfulQA","generative QA"],"falsifier":"Replace the GPT-4o labels with a large human-verified set (or a second independent judge that agrees strongly with humans) for both languages and re-run every transfer experiment; if cross-lingual AUC-ROC collapses while monolingual scores stay high, the transfer claim is false.","tokens_in":28574,"feed_emoji":"🌐","tokens_out":886,"duration_ms":17090,"temperature":0.7,"pith_summary":"This paper asks whether the internal cues that let a small classifier detect hallucinations inside a large language model work only in English, or also across languages and datasets. The authors pull hidden-state, attention, and token-probability features from six Arabic-capable models, train a three-layer detector, and test it under monolingual, English–Arabic, Arabic-to-Arabic domain-shift, and combined transfer protocols on TruthfulQA and HalluScore. They find that the signals do transfer for most models when classes remain separable and the two languages sit close in feature space; cross-domain transfer inside Arabic is real but asymmetric and depends on which dataset supplies the training examples. If the result holds, detectors trained on high-resource or cleaner data can help lower-resource or harder settings without collecting new labels from scratch. A reader who cares about safer multilingual deployment should care, because hallucination rates are already higher outside English and annotation is expensive.","feed_headline":"Hallucination detectors transfer across languages for most LLMs","feed_subtitle":"English-trained internal-state classifiers can flag Arabic hallucinations when features align","key_machinery":"CrossHallu feature vectors: Wasserstein distances and cosine similarities between consecutive even/odd layers of hidden states and attention, plus eight token-probability statistics (min-of-max, max spread, normalized entropy, low-probability counts, mean gradient, and three percentiles), concatenated and fed to a three-layer MLP binary detector.","core_discovery":"Internal-state hallucination signals from hidden states, attention patterns, and token probabilities transfer across languages and domains for most of the six evaluated LLMs. Cross-lingual success is governed by class separability within each language and by how well English and Arabic embeddings overlap; cross-domain transfer within Arabic is feasible yet asymmetric, with training on TruthfulQA Arabic transferring better to HalluScore than the reverse.","pith_inferences":["The same transfer protocol could probe whether other reliability signals (toxicity, bias) also live in shared subspaces without new labeled data.","If late-layer cosine drops are the main discriminative cue, detectors could be restricted to upper layers only, cutting extraction cost.","Extending the pair set beyond Arabic–English would likely show sharper transfer collapse once morphological or script distance breaks feature-space alignment.","Asymmetric domain transfer implies that easier, cleaner hallucination corpora may be more valuable as source data than large noisy ones."],"forward_implications":["A detector trained only on English internal states can be applied to Arabic generations for models whose English and Arabic embeddings already interleave.","Training on a cleaner Arabic QA set can raise detection on a harder Arabic set, so data quality matters more than matching the exact target domain.","Multilingual models with shared language-agnostic subspaces are better zero-shot monitoring candidates than Arabic-centric models.","Combined language-plus-domain shift remains the hardest case, so production systems may still need light target calibration.","The same internal features can diagnose how language-agnostic a model’s deeper layers actually are."],"fun_headline_variants":["Internal hallucination signals transfer across languages in most LLMs","LLM hallucination detectors generalize cross-lingually for most models","English-trained internal detectors flag Arabic hallucinations when features align","Cross-lingual and cross-domain transfer of hallucination signals works for most LLMs","Hallucination internals transfer across languages and domains in six LLMs"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The GPT-4o binary labels for English and Arabic answers are accurate enough that measured transfer reflects real internal-signal generalization rather than shared judge artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Internal hallucination signals transfer across languages in most LLMs","LLM hallucination detectors generalize cross-lingually for most models","English-trained internal detectors flag Arabic hallucinations when features align","Cross-lingual and cross-domain transfer of hallucination signals works for most LLMs","Hallucination internals transfer across languages and domains in six LLMs"]},"model":"grok-4.5","effort":"low","cost_usd":0.005678,"raw_usage":{"total_tokens":1535,"prompt_tokens":787,"num_sources_used":0,"completion_tokens":89,"cost_in_usd_ticks":56780000,"prompt_tokens_details":{"text_tokens":787,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":659,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":787,"tokens_out":89,"duration_ms":4932,"temperature":1.0,"reasoning_tokens":659,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T22:12:10.400379+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Replace the GPT-4o labels with a large human-verified set (or a second independent judge that agrees strongly with humans) for both languages and re-run every transfer experiment; if cross-lingual AUC-ROC collapses while monolingual scores stay high, the transfer claim is false.","supporting_citations":[],"review_version":1}