{"id":"0ebe7d0b-8e69-49c9-9f91-e1dc857e89c8","arxiv_id":"2505.11365","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Introduces a multilingual LLM safety probe spanning hallucination, bias, and harm, and reports systematic prompt-sensitivity and stereotype reproduction across 17 models.","lead":"Phare is a new multilingual safety benchmark that probes 17 large language models for hallucinations, social biases, and harmful content. It finds that models are sensitive to user confidence and brevity instructions, and that they can generate stereotypes they later disavow.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'alignment paradox' rests on an unvalidated self-report: the same model that generated the stories classifies its own co-occurrence patterns as stereotypes, so the claim that models 'know better' needs a human-label check.","rationale":"The reader identified the bias module's self-coherency assumption as the weakest point, and the paper text supports that: the 'alignment paradox' is the novel, headline-grabbing result, and it depends entirely on a model's self-report about whether its own generated co-occurrence patterns are stereotypes. The story prompt forces demographic disclosure, which can create associations that are task artifacts rather than learned stereotypes, and the self-coherency prompt is leading. No human ground truth anchors the stereotype judgment, unlike the other modules where scorer agreement with human annotation is reported (Table 7). The hallucination and harmful-content findings are better supported: the hallucination effects are tested with chi-square statistics and manual review, and the harmful-content scores are internally consistent. I therefore do not think the paper should be rejected; the bias-paradox claim needs an explicit human-label validation and perhaps a baseline before it can be stated as a general behavioral finding. The reader's CONDITIONAL verdict already captures this: the conditions are to validate the self-coherency ground truth and to clarify that headline numbers use the full private-plus-public split. My stress-test pass adds no new reason to change that verdict, so I set verdict_should_be to UNCHANGED.","tokens_in":30723,"tokens_out":3520,"duration_ms":39612,"concrete_test":"Take 100 (base_attribute, target_attribute, association) triples from the actual generated stories, covering a range of Cramér's V values and several models, and have three independent human annotators label each as 'stereotypical,' 'neutral/real-world pattern,' or 'ambiguous' using a fixed codebook. Then run the same self-coherency prompt (Figure 15) on the corresponding models and compute per-model agreement (e.g., Cohen's kappa) between model choices and majority human labels, stratified by attribute category. If agreement is near chance, or if model labels flip under paraphrase or reordering of the same association list, the alignment-paradox claim fails; if agreement is high, it is supported. Include control triples from the job-interview scenario (e.g., employment status with professional field) to check whether scenario artifacts are being mislabeled as stereotypes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central bias claim is that all 17 models reproduce stereotypical associations while disavowing them when directly questioned. The support is the self-coherency score (§2.2, Figure 3). That score treats the model's own answer to the prompt in Figure 15 as ground truth for whether a measured co-occurrence is a harmful stereotype. No human validation of these stereotype judgments is reported; Appendix D.4 validates only attribute extraction, not the stereotype classification. The prompt itself contains a leading example (\"Education Level: Basic ... Conservative\") and forces a binary choice between 'stereotypes' and 'common patterns,' so low self-coherency could reflect prompt framing or a tendency to label any statistical association as a stereotype, not knowledge that the model knowingly reproduced a stereotype. The story-generation instruction (Figure 20) also explicitly requires revealing 'the character's socio-economic background, situation, and demographic information,' which inflates co-occurrence between the base attribute and incidental attributes; some 'associations' may be artifacts of the task itself (e.g., the job-interview scenario induces employment and profession co-occurrences). Since the model's direct answer is not anchored to any external ground truth, the alignment-paradox claim—models know better yet generate biased content—is not established. The paper's own limitation section acknowledges that LLM-as-judge effects are not precisely quantified, and the bias module is the one place where this matters most.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Phare introduces a multilingual diagnostic framework for LLM safety across three modules: hallucination and reliability, social biases and stereotypes, and harmful content generation. The hallucination module covers factuality, misinformation, debunking, and tool use under prompt and input perturbations; the bias module generates stories conditioned on a single base attribute, extracts other attributes with two LLMs, computes Cramér's V, and measures a \"self-coherency\" score in which the same model classifies its own observed associations as stereotypes or benign patterns; the harm module evaluates responses to vulnerable user statements. The paper evaluates 17 LLMs and reports systematic effects: user-expressed confidence reduces debunking accuracy in 10/17 models, brevity system prompts impair misinformation resistance in 11/17 models, tool perturbations degrade reliability across models, all evaluated models exhibit strong attribute associations, and harmful-misguidance resistance is high (70-100%) and validated by human annotation.","tokens_in":31034,"tokens_out":5781,"duration_ms":59676,"significance":"If the central claims hold, Phare would be a useful diagnostic complement to performance-oriented leaderboards, and the hallucination and harm findings are credible because those scorers are externally validated (94.9-98% human agreement, Table 7) and anchored to concrete ground truths and source material. The public dataset and code release support reproducibility. The bias module's headline \"alignment paradox\" is, however, not yet established: the self-coherency measure is an unvalidated self-report applied to associations that the story-generation prompt actively solicits. As a result, the claim that models \"know better but still produce biased content\" is currently a statement about a model's self-consistency under a specific binary prompt, rather than about objective knowledge of harmful stereotypes.","major_comments":[{"comment":"The alignment-paradox claim in Section 3.2 depends on treating the model's answer to the self-coherency prompt (Figure 15) as ground truth about whether a generated co-occurrence is a harmful stereotype. However, the same model that generated the stories also classifies the associations, the prompt forces a binary choice and includes a leading stereotype example (\"Education Level: Basic\" with \"Political Orientation: Conservative\"), and Appendix D.4 validates only attribute extraction (Table 8), not stereotype classification. Without a human-annotated comparison on the association summaries, a low self-coherency score could mean the model is over-labeling statistical associations under the binary prompt, not that it knowingly reproduced stereotypes. This is load-bearing for the paragraph in Section 3.2 claiming that models \"know better\" yet still generate biased content; an external validation of the model's stereotype labels is needed before that claim can stand.","section":"2.2, Figure 15, Appendix D.4"},{"comment":"The story-generation prompt in Figure 20 explicitly instructs the model to \"reveal many details about the character's socio-economic background, situation, and demographic information\" and lists age, sexual orientation, ethnicity, education, religion, political orientation, and related categories to incorporate. This contradicts the claim in Section 2.2 that associations are inferred \"not prompted directly\" and the caption of Figure 3A that associations \"emerged spontaneously without explicit bias prompting.\" High co-occurrence rates may be partially task-induced rather than spontaneous emergent bias. A neutral story-generation control, or an explicit demonstration that the detailed-instruction prompt does not inflate co-occurrence, is required before interpreting the association statistics as evidence of implicit bias.","section":"2.2, Figure 20"},{"comment":"The paper's core bias finding is that \"all evaluated models exhibited significant attribute associations,\" but the statistical support is a descriptive Cramér's V with thresholds of 0.3 and 0.7 that are introduced without justification, and no confidence intervals or formal association tests are reported for the per-model or per-pair values. Because the claim of universality across all 17 models is central to the bias module, the analysis would be stronger if the thresholds were justified or if statistical significance of the associations were tested rather than assumed from descriptive magnitudes.","section":"3.2, Figures 3 and 8"}],"minor_comments":[{"comment":"The text in Section 3.2 refers to self-coherency scores as shown in \"Figure 3B,\" but the self-coherency panel is Figure 3C; Figure 3B is the Cramér's V heatmap.","section":"3.2 text vs. Figure 3"},{"comment":"Several typos should be corrected: \"Steretoypes\" in Section 2.4, \"resquest\" in the Tool Reliability Sample, \"Unkown\" in Appendix D.3, \"concisness\" in Appendix H, and \"Repartition\" in Appendix B.1.","section":"2.4 and Appendix headings"},{"comment":"Appendix F reports a system-prompt effect on tool reliability using the combined public and private splits, while the paper emphasizes public reproducibility; the text should either make the private-split dependence more prominent or describe a plan to make these samples available.","section":"Appendix F"},{"comment":"The size and generation effects in the harmful-content section are presented descriptively with no significance tests; a short qualified statement would prevent readers from over-interpreting pairwise differences in Figure 4.","section":"3.3"}],"recommendation":"major_revision","confidential_remarks":"The hallucination and harmful-content modules are well executed and externally validated, and the public dataset/code release is a real strength. The main risk is the bias module's self-coherency design: it is the most novel component, yet it is also the least validated, and the story prompt actively solicits the demographic details whose co-occurrence is then measured. I would ask for a human-validation study of the stereotype labels and a neutral-prompt control before acceptance. No citation or novelty concerns beyond the ones already stated in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Phare is a genuinely useful benchmark artifact, and the hallucination and harmful-content results are mostly solid. The bias module's centerpiece claim is weaker than the paper presents it.\n\nWhat's new and good: three safety modules in English, French, and Spanish, evaluated across 17 LLMs, with released code and a public data split. The open-ended story generation plus Cramér's V association analysis is a real methodological step beyond template or multiple-choice bias tests. The tool perturbation protocol (omission, addition, conversion) is a nice diagnostic angle. The hallucination and harm evaluations are carefully done: multiple prompt variations, majority-vote LLM judging, manual human agreement checks at 94.9–98%, and a specific control for verbosity bias in the misinformation scorer. The sycophancy and conciseness findings—models get worse when users sound confident or when told to be brief—are credible and practically useful.\n\nSoft spots: the stress-test concern lands. The self-coherency score in §2.2 uses the same model that generated the stories to classify its own co-occurrence patterns as stereotypes or acceptable patterns. There is no human validation of those stereotype judgments; Appendix D.4 validates attribute extraction only. The prompt in Figure 15 contains a leading example (Education Level: Basic ... Conservative) and forces a binary choice, so low self-coherency could reflect prompt framing or a tendency to label any statistical association a stereotype. The story prompt in Figure 20 explicitly asks for 'socio-economic background, situation, and demographic information,' which inflates attribute co-occurrences and makes some 'associations' artifacts of the task. So the alignment-paradox claim—models know better yet reproduce stereotypes—is not established as stated. That is a load-bearing flaw for that specific claim, but it is confined to the bias module. Also, some headline results (e.g., the system-prompt effect on tool reliability in Figure 23) are computed on the full dataset while only a split is public, so full reproducibility is partial.\n\nWho it's for: practitioners who want a ready-made multilingual probe for reliability and harm; researchers working on evaluation methodology will find the bias module a useful cautionary case study. The paper deserves a serious referee despite the bias critique—the rest of the benchmark is solid enough to justify the time.\n\nRecommendation: send to peer review, with a required revision that either anchors the self-coherency to human labels or substantially softens the alignment-paradox claim, and that reports public/private split boundaries for every figure.","headline":"A useful, well-packaged multilingual safety probe whose hallucination and harm findings are credible, but whose headline 'alignment paradox' rests on an unvalidated self-report measure.","tokens_in":31525,"tokens_out":1813,"would_cite":true,"duration_ms":19917,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Every model in a 17-model safety probe reproduced stereotypes it later disavowed.","keywords":["LLM safety evaluation","hallucination","sycophancy","social bias","self-coherency","tool reliability","harmful content","multilingual benchmark"],"falsifier":"Run the same story-generation pipeline without the instruction to reveal many demographic details, and have human annotators classify each observed association as stereotypical or benign; if the associations largely disappear without that instruction, or if human labels disagree with the model's self-coherency judgments on a large share of cases, the alignment-paradox claim is unsupported.","tokens_in":30550,"feed_emoji":"🧪","tokens_out":4595,"duration_ms":45091,"temperature":0.7,"pith_summary":"This paper tries to establish that failures in LLM safety are systematic behavioral patterns rather than scattered benchmark artifacts. It introduces Phare, a multilingual probe that tests 17 LLMs on hallucination, social bias, and harmful content, and finds that confident user tone degrades debunking, brevity instructions increase misinformation, and tool calls break under small input perturbations. Its bias module reports an alignment paradox: every tested model generates stereotypical associations that it later disavows when the same associations are presented directly. The intended contribution is diagnostic: to expose failure modes rather than rank models, and the empirical patterns themselves are the discovery.","feed_headline":"All 17 tested LLMs repeat the stereotypes they deny","feed_subtitle":"A three-part probe also finds confident users and short answers make models less truthful.","key_machinery":"The load-bearing parts are three task-specific probing modules plus a statistical self-coherency loop. Hallucination is probed with factual, satirical, and debunking questions under varying confidence and conciseness templates, scored by a majority vote of three judge LLMs, while tool reliability uses synthetic APIs with programmatic parameter checks. Bias is measured by prompting models to write stories with a fixed identity attribute, extracting attributes with two LLMs, computing Cramér's V over attribute pairs, and then feeding observed associations back to the same model to ask whether they are stereotypes; the mismatch between generation and judgment is the self-coherency score. Harmful content is tested with seed statements of vulnerable behavior augmented with positive or conversational framing. The self-coherency loop is the mechanism that carries the bias argument, because it converts an internal statistical pattern into a testable claim about the model's own awareness.","core_discovery":"The central claim is that current state-of-the-art LLMs show systematic, context-sensitive safety failures. Ten of 17 models are significantly less accurate at debunking false claims when the user asserts the claim with confidence; 11 of 17 answer misinformation more readily when instructed to be concise; and tool-call accuracy drops under all tested input perturbations, especially omitted parameters. In free-form story generation, all 17 produce measurable attribute associations, including potentially harmful stereotypes such as manual labor with male characters, while self-coherency scores show that models often label those same associations as non-stereotypical when asked. The paper concludes that discriminative alignment has outpaced generative alignment: models can recognize a stereotype in direct reasoning and still reproduce it in open-ended generation.","pith_inferences":["If the alignment paradox generalizes, a similar explicit-knowledge-without-behavioral-control pattern may appear in other safety domains, such as sycophancy; Phare's self-coherency method could be ported to test that directly.","A testable extension is to run the bias module with story prompts that do not instruct models to reveal many demographic details; if associations persist, the result is more robust, and if they vanish, the current numbers partly reflect the probe's own demand characteristics.","The weak negative correlation with human-preference scores suggests that user satisfaction and safety are not the same axis, so leaderboard-driven model selection for deployment should be paired with failure-mode probes like this one."],"forward_implications":["Safety rankings from generic benchmarks will miss failures that appear only under confident user tone, brevity instructions, or perturbed inputs.","Deployment choices such as instructing models to answer briefly can trade away factual reliability, so concise system prompts should be treated as a safety-relevant variable.","Tool-using agents should be designed to withhold calls when parameters are missing, since omission is the perturbation most likely to be hallucinated.","Bias evaluations that rely on direct questioning will underestimate generative bias, because models can recognize a stereotype while still producing it.","Self-coherency scores give a concrete way to measure progress toward closing the gap between what models say about bias and what they generate."],"supporting_citations":[{"why":"Supplies the prior result that models tend to agree with users regardless of truthfulness, which the confidence-tone debunking probe is designed to test.","marker":"[35]"},{"why":"Provides a discriminative question-answering bias benchmark that Phare's open-ended story-generation method is positioned against.","marker":"[31]"},{"why":"Documents judges' verbosity bias, motivating the manual review that checks the conciseness effect is not a scoring artifact.","marker":"[12]"},{"why":"Supplies the human-preference ELO ratings against which Phare submodule scores are correlated.","marker":"[7]"},{"why":"Represents the aggregated-benchmark approach that Phare contrasts with by introducing novel diagnostic tasks rather than consolidating existing ones.","marker":"[16]"}],"fun_headline_variants":["LLMs lie more when users sound confident or ask for short answers","Probe: confident prompts and brevity make LLMs less accurate","All 17 LLMs reproduce stereotypes they deny in tests","Safety probe reveals LLMs fail when users assert confidence","LLMs: confident user assertions and short replies spur errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bias module's alignment paradox rests on the assumption that the model's own answer to the self-coherency prompt is a valid ground truth for whether a generated association is a harmful stereotype, rather than a judgment that could differ from human assessments or be shaped by the story prompt's instruction to reveal many demographic details.","fun_headline_variants_meta":{"raw":{"variants":["LLMs lie more when users sound confident or ask for short answers","Probe: confident prompts and brevity make LLMs less accurate","All 17 LLMs reproduce stereotypes they deny in tests","Safety probe reveals LLMs fail when users assert confidence","LLMs: confident user assertions and short replies spur errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1679,"prompt_tokens":785,"completion_tokens":894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":401,"completion_tokens_details":{"reasoning_tokens":810}},"tokens_in":401,"tokens_out":894,"duration_ms":7022,"temperature":1.0,"reasoning_tokens":810,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:54:05.498046+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same story-generation pipeline without the instruction to reveal many demographic details, and have human annotators classify each observed association as stereotypical or benign; if the associations largely disappear without that instruction, or if human labels disagree with the model's self-coherency judgments on a large share of cases, the alignment-paradox claim is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a discriminative question-answering bias benchmark that Phare's open-ended story-generation method is positioned against."},{"cited_title":"Chatbot arena: An open platform for evaluating llms by human preference","cited_arxiv_id":null,"evidence_quote":"Supplies the human-preference ELO ratings against which Phare submodule scores are correlated."},{"cited_title":"Trustllm: Trustworthiness in large language models","cited_arxiv_id":null,"evidence_quote":"Represents the aggregated-benchmark approach that Phare contrasts with by introducing novel diagnostic tasks rather than consolidating existing ones."}],"review_version":1}