{"id":"ae299b9d-6ae8-4dc1-9b17-e1b6c0baf16c","arxiv_id":"2608.00207","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Arabic medical failure in LLMs is a knowledge-routing breakdown, not a knowledge deficit, and TLoRA, a low-rank adapter on the mechanistically identified layer window, improves Arabic medical MCQA over full-network LoRA.","lead":"This paper diagnoses why an LLM answers Arabic medical questions worse than English ones, and finds the knowledge is in the model but fails to surface, tracing the problem to a specific layer range. It then uses that diagnosis to fine-tune only those layers, outperforming full-model fine-tuning on Arabic medical multiple-choice tests, and introduces a clinician-built Arabic dialogue benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TLoRA's headline advantage over full-network LoRA is not statistically established: in-domain CIs overlap (p=0.74) and OOD gains come from tiny benchmarks with no significance tests.","rationale":"The reader's weakest_assumption focused on the machine-translated English diagnostic reference. That is a legitimate concern about the mechanistic 'knowledge present' interpretation, but I see the more load-bearing threat to the paper's central claim in the empirical comparison: the paper explicitly reports p=0.74 on the in-domain benchmark and provides no significance evidence for the OOD benchmarks that carry the 'outperforms full-network LoRA' claim. The method may still be promising, and the mechanistic story may survive, but the headline practical contribution is not established by the reported statistics. This aligns with the reader's point (3), so I mark partial agreement. The remaining verdict should stay CONDITIONAL: the issues are addressable with targeted statistical analysis and possibly additional evaluation, but the current evidence does not support an unconditional accept.","tokens_in":35885,"tokens_out":9590,"duration_ms":98955,"concrete_test":"Recompute TLoRA vs LoRA v2 with paired per-example significance tests on each MCQA benchmark. For MedArabiQ (n=100), compute an exact McNemar or binomial test on the 5-question delta; for ArabicMMLU and AraSTEM, use paired bootstrap or McNemar tests and report 95% CIs on the accuracy difference and on the macro-average. If the OOD deltas and macro-average CI include zero (or p>0.05), the headline should be softened from 'outperforms' to 'competitive'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that TLoRA 'outperforms full-network LoRA' rests on a comparison that is not statistically supported. The only controlled full-network baseline with the same KL alignment objective is LoRA v2. On the in-domain benchmark (MedAraBench), TLoRA scores 62.1 vs LoRA v2's 61.9 with overlapping 95% confidence intervals and p=0.74 (§6.1). The large 20.6-point margin reported against 'Mistral + LoRA' in Table 2 is against plain LoRA without the KL alignment term, so it does not isolate layer selection. The OOD advantages are on small benchmarks: MedArabiQ has 100 questions (60.0 vs 55.0, a 5-question difference), ArabicMMLU-biology has 1,072 (61.6 vs 60.3), and AraSTEM-medicine has 721 (65.1 vs 60.7). No confidence intervals, p-values, or paired significance tests are reported for any OOD comparison. The macro-average margin (62.2 vs 59.5) is driven by these unreplicated, unadjusted differences. As reported, the data do not establish that the mechanistically-selected L1–34 window beats a full-network LoRA trained with the same objective. The mechanistic diagnosis may still be sound, but the paper's headline adaptation claim is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates why Mistral-Small-3.2-24B underperforms in Arabic medical MCQA. It reports that the model answers correctly in English on 29.6% of questions it fails in Arabic, and uses tuned lens probing, causal activation patching, and KL divergence profiling to localize the failure to a layer window bounded by Lpatch=24 and LKL=34, interpreting it as a knowledge-routing failure rather than a knowledge deficit. Based on this, it proposes TLoRA: LoRA restricted to layers L1–L34 with a cross-lingual KL alignment loss at layer 34, and reports improvements over full-network LoRA on Arabic medical MCQA. It also introduces AraClinicDialog, a clinician-constructed multi-dialectal Arabic medical dialogue benchmark, and evaluates short-answer generation and multi-turn dialogue transfer. The main empirical claims are the mechanistic diagnosis and the superiority of TLoRA over full LoRA.","tokens_in":36203,"tokens_out":7847,"duration_ms":71028,"significance":"If the claims are upheld, the contribution is valuable: a cheap diagnostic pipeline (3.63 GPU-hours, Table S22) that identifies a plausible representational locus for cross-lingual failure, an adaptation method with extensive ablations, and a new clinician-validated dialogue resource. The paper also includes useful robustness checks: exhaustive window ablation (Table 3), CE-only and fixed-learning-rate comparisons (Appendix F), c-threshold sensitivity (Appendix F.1), and a validated LLM-as-a-judge protocol (Pearson = 0.978 on short-answer generation). These strengths mean the manuscript is close to making its case; however, the statistical basis for the headline comparison and the translation-dependent diagnostic require additional support.","major_comments":[{"comment":"The claim that TLoRA 'outperforms full-network LoRA' is not established statistically. On the in-domain benchmark, TLoRA (62.1, CI [60.8, 63.4]) and LoRA v2 (61.9, CI [60.5, 63.2]) are statistically indistinguishable (p = 0.74). The OOD comparisons are presented without confidence intervals, p-values, or paired tests. MedArabiQ has only 100 items (60.0 vs 55.0, i.e. a 5-question difference); ArabicMMLU-biology differs by 1.3 pp (61.6 vs 60.3); AraSTEM-medicine by 4.4 pp (65.1 vs 60.7), which a naive normal approximation would put at p ≈ 0.08. The macro-average advantage of 2.7 pp is driven by these unreplicated differences. Please report per-benchmark CIs and paired significance tests (or a pre-specified combined analysis), and temper the abstract/§6.1 claims accordingly. The same issue affects the window-ablation claims in Table 3.","section":"§6.1, Table 2, Table 3"},{"comment":"The 'knowledge-routing breakdown' diagnosis depends entirely on the English reference being a Google Translate rendering of the Arabic question. The 29.6% English-correct/Arabic-incorrect figure and the subsequent patching and KL analyses all use this reference. If machine translation produces systematically easier, less ambiguous, or syntactically simplified English items, the result could reflect translation artifacts rather than a routing failure. Please add a human-translated or independently sourced English subset (e.g., 200–500 items) to re-estimate this rate; at minimum, check whether the English-correct/Arabic-fail set is stable under a second translation engine and whether it is predicted by MT quality scores. The mechanistic conclusion is load-bearing for the method's motivation, so this control is necessary.","section":"§3"},{"comment":"The final window L1–34 is selected by held-out MCQA performance after the candidate windows are generated from mechanistic boundaries; the manuscript says 'select the window with the best held-out performance' (§4.1) and 'optimal window is selected via held-out performance' (§5.1). Thus the coincidence between the selected window and the mechanistic hypothesis is partly self-fulfilling, and the claim that the design 'follows directly from the mechanistic evidence' (§1, §8) is stronger than the procedure warrants. In addition, sensitivity is reported only for c (Appendix F.1); Lpatch=24 depends on the arbitrary τpatch=0.5 in Eq. (1), and no sensitivity analysis for τpatch is provided. Please add a τpatch sweep and present the window selection more carefully as mechanism-constrained, validation-selected.","section":"§4.1, Eq. (1), Eq. (3), Appendix F.1"},{"comment":"The statement that CE-only and fixed-LR 'preserve the L1–34 ranking' is too strong. Under the fixed learning rate (Table S21), L1–34 is below full LoRA on ArabicMMLU-biology (59.6 vs 60.3), and the gap on MedAraBench is 0.1 pp; the CE-only comparison (Table S20) shows full LoRA collapsing at its tuned learning rate, but no CE-only comparison at a controlled learning rate is reported for full LoRA. These are exactly the controlled comparisons needed to isolate layer placement from optimization; please report them or soften the causal claim.","section":"§6.1, Appendices F.2–F.3"}],"minor_comments":[{"comment":"The 'TLoRA (optimal)' configuration appears to be selected on test performance; clarify the selection protocol or mark it as an oracle. As presented, the 38.1% vs 29.7% comparison is not a fair evaluation of a method choice.","section":"§6.2, Figure 2"},{"comment":"The naming is confusing: 'Mistral + LoRA' in Table 2 is CE-only full LoRA, while 'Mistral + LoRA v2 (with KL)' is the controlled full-LoRA baseline, and Table 3's 'Full LoRA (W5)' corresponds to the latter. Please make this explicit in the table caption and main text to avoid inadvertent comparison of TLoRA against the uncontrolled CE-only baseline.","section":"Table 2, §5.3, §6.1"},{"comment":"The LLM-as-a-judge protocol is validated on short-answer generation (Pearson = 0.978) but used directly for multi-turn dialogue without dialogue-specific human validation. The dialogue judge prompt also uses a 'Primary Reasoning Objective', which may inflate agreement. A small human agreement study on dialogue judgments would strengthen this part.","section":"§5.2.3, Appendix C"},{"comment":"On Llama-3.1-8B-Instruct, TLoRA is below full LoRA on MedAraBench (47.3 vs 49.6) and ties on MedArabiQ (39.0 vs 39.0). The sentence 'matches or exceeds full-network LoRA on MCQA' should be qualified per benchmark; the current wording overstates the generalizability evidence.","section":"Appendix I"}],"recommendation":"major_revision","confidential_remarks":"The experimental corpus is substantial and the resource contribution is real. The required revisions are feasible: add statistical tests and translation controls, and reframe the mechanistic and comparative claims. I would not reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. The mechanistic story is the real contribution: using tuned lens probing, causal activation patching, and KL divergence profiling, the authors make a plausible case that Arabic medical knowledge is present but fails to surface in Mistral-Small-3.2-24B, localizing the breakdown to a specific layer window (Lpatch=24, LKL=34). That is new, and the diagnostic pipeline is cheap (about 3.6 GPU-hours) and transfers to a second model family (Llama-3.1-8B). The paper is also honest about its main null: in-domain, TLoRA (62.1) and the matched full-network LoRA v2 (61.9) are statistically indistinguishable (p=0.74, CIs overlap). The window ablation is careful: five mechanistic windows, per-window LR sweeps, CE-only and fixed-LR controls, and a KL probe-layer sensitivity analysis. The new AraClinicDialog benchmark is a genuine asset, carefully constructed with clinician involvement and dialect variants.\n\nThe soft spots are real and cluster in the headline adaptation claim. The 'outperforms full-network LoRA' language in the abstract and Table 2 is not supported by the controlled comparison. The large 20.6-point margin against 'Mistral + LoRA' is against plain LoRA without the KL alignment term, so it does not isolate layer selection. The OOD advantages (60.0 vs 55.0 on MedArabiQ, 65.1 vs 60.7 on AraSTEM, 61.6 vs 60.3 on MMLU-Bio) come from small benchmarks with no confidence intervals or significance tests. The macro-average margin (62.2 vs 59.5) is driven by those unreplicated differences. So the stress-test note is correct: the data do not establish that the mechanistically-selected L1–34 window beats a full-network LoRA trained with the same objective.\n\nThe second soft spot is the translation-based diagnostic. The 'knowledge present' conclusion uses Google-Translate English as the reference; the reader's concern that translation artifacts could make English systematically easier is legitimate, though the causal patching results are somewhat independent of that and do support a routing story. The window is also ultimately chosen by held-out performance rather than purely by the causal analysis, so 'causal layer selection' overstates the design; the paper is transparent about this, but the framing invites the circularity critique the reader raises. No code or data are released, which limits reproducibility, and the judge is a closed model, though the human validation (Pearson 0.978) mitigates that.\n\nOverall: the mechanistic analysis is solid enough to deserve serious referee attention, and the benchmark is useful. The adaptation claim needs reworking — either run larger OOD benchmarks, add significance testing, or soften the conclusion to 'comparative' rather than 'outperforms.' This is a paper worth engaging with, not a desk reject.","headline":"Mechanistic diagnosis is the real contribution; the TLoRA advantage over full LoRA is not statistically established.","tokens_in":36726,"tokens_out":3271,"would_cite":true,"duration_ms":26952,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that Arabic medical failures in large language models are a knowledge-routing breakdown rather than a knowledge deficit, and that restricting low-rank adaptation to the layers where Arabic and English representations diver","keywords":["Arabic medical NLP","knowledge routing","low-rank adaptation","causal activation patching","tuned lens probing","cross-lingual divergence","medical question answering","clinical dialogue"],"falsifier":"Run the same English/Arabic diagnostic with human expert translations instead of Google Translate on a large sample of questions; if the English-correct/Arabic-incorrect rate drops to near zero, the routing-failure claim falls. Alternatively, patch Arabic hidden states into the English forward pass at layer 24 on questions the model gets wrong in English; if no such reverse gap exists, the knowledge access story is not language-symmetric and would need re-examination.","tokens_in":35778,"feed_emoji":"🩺","tokens_out":3724,"duration_ms":35351,"temperature":0.7,"pith_summary":"The paper sets out to prove that a large language model's poor Arabic medical exam performance is not simply missing knowledge. On paired English and Arabic versions of the same questions, the model answers correctly in English on about 30 percent of the cases it gets wrong in Arabic. Mechanistic probes show the correct answer is encoded mid-network in Arabic but lost in later layers, and that replacing Arabic internal states with English ones at a single layer recovers the answer. The authors then propose TLoRA, which applies low-rank adaptation only to layers below the divergence point, and show it beats full-network LoRA on Arabic medical multiple-choice QA while preserving generation and dialogue ability better than full fine-tuning. A reader should care because it suggests a cheap, diagnosis-driven route to improving medical access in under-resourced languages instead of treating the gap as an unavoidable data-scarcity problem.","feed_headline":"Arabic medical failures are a routing glitch, not missing knowledge","feed_subtitle":"Restricting adaptation to divergent layers beats full-model fine-tuning on Arabic medical exams and preserves generation.","key_machinery":"TLoRA (Targeted Low-Rank Adaptation) is low-rank adaptation restricted to a contiguous layer window, L1-L34, selected from the model's own internal signals: the causal boundary Lpatch=24, defined as the first layer where activation patching recovers at least 50 percent of the English-Arabic gap, and the divergence onset LKL=34, defined as the first layer where the cross-lingual KL divergence exceeds the mean plus one standard deviation. The training objective combines cross-entropy on Arabic medical MCQs with a KL alignment term between Arabic and English logit-lens distributions at LKL, with the alignment weight calibrated from the model's initialization losses. This machinery converts a me","core_discovery":"The paper's central discovery is that the Arabic medical failure of Mistral-Small-3.2-24B is a routing failure, not a knowledge deficit. Using tuned-lens probing, causal activation patching, and cross-lingual KL-divergence profiling, the authors find that the correct-answer probability in Arabic rises to English levels by mid-network and then collapses before the output; patching English hidden states at layer 24 recovers more than 80 percent of the English-Arabic gap, and Arabic-English divergence begins sharply at layer 34. These two boundaries define adaptation windows, and a window covering layers 1 through 34 yields higher Arabic medical MCQA accuracy than full-network LoRA or any other","pith_inferences":["If this diagnosis generalizes, multilingual medical evaluation should report not just accuracy gaps but also the layer-wise divergence profile, because two models with the same gap may need opposite interventions.","A testable extension the authors leave implicit: keep the adaptation window fixed but vary the alignment teacher (for example, dialectal Arabic instead of English) to see whether the routing fix is specifically tied to the English-centric latent pathway.","The paper's English reference is machine-translated; a careful reader would want a human-translated English set to confirm that the 30 percent 'knows in English but not Arabic' phenomenon is not inflated by translation artifacts."],"forward_implications":["If the claim is right, improving Arabic medical capability does not require more Arabic pretraining data; small adapters on a narrow layer window can unlock knowledge already in the model.","The same diagnostic pipeline can be run on other languages and models to locate their routing-failure layers before adaptation.","Full-network LoRA is not just wasteful but sometimes harmful: adapting beyond the divergence window can overfit away general capabilities, while targeted adaptation preserves downstream generation quality better.","Reducing the routing bottleneck narrows the English/Arabic access gap from 29.6 percent to 19.0 percent in this setting.","The mechanistic diagnosis costs under two hours on two GPUs per model, making it practical as a routine preprocessing step before adaptation."],"fun_headline_variants":["Arabic medical gap is a routing flaw, not missing knowledge","Targeted LoRA on divergent layers boosts Arabic medical QA","Causal patching reveals Arabic medical knowledge is present","Layer-specific adaptation outperforms full-model LoRA for Arabic","Fixing Arabic medical LLM output requires layer selection, not retraining"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The interpretation that Arabic knowledge is present but not routed to the output rests on the English translation of the Arabic questions being a fair measure of what the model knows; if those translated questions are systematically easier or phrased differently, the 'present but not surfaced' conclusion weakens.","fun_headline_variants_meta":{"raw":{"variants":["Arabic medical gap is a routing flaw, not missing knowledge","Targeted LoRA on divergent layers boosts Arabic medical QA","Causal patching reveals Arabic medical knowledge is present","Layer-specific adaptation outperforms full-model LoRA for Arabic","Fixing Arabic medical LLM output requires layer selection, not retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1233,"prompt_tokens":738,"completion_tokens":495,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":482,"tokens_out":495,"duration_ms":4898,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T00:37:09.332898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same English/Arabic diagnostic with human expert translations instead of Google Translate on a large sample of questions; if the English-correct/Arabic-incorrect rate drops to near zero, the routing-failure claim falls. Alternatively, patch Arabic hidden states into the English forward pass at layer 24 on questions the model gets wrong in English; if no such reverse gap exists, the knowledge access story is not language-symmetric and would need re-examination.","supporting_citations":[],"review_version":1}