{"id":"efb8e0f8-8c7d-4864-bc66-94d091c29b8a","arxiv_id":"2610.20690","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"ImpactBench is an open, expert-guided benchmark platform that evaluates AI models on 809 behavioral metrics across psychological, physical, and social well-being, finding that models are better at fostering positive outcomes than avoiding harmful ones.","lead":"This paper introduces ImpactBench, a large-scale benchmark that evaluates how generative AI models affect human well-being across psychological, physical, and social domains through simulated multi-turn conversations. It provides an open platform where domain experts can contribute metrics, and reveals that frontier LLMs promote desirable behaviors in 68.7% of tests but avoid harms in only 53.1%, with significant variation across domains and age groups.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The judge model (GPT-5.6 Luna) is also a target model, creating a potential conflict that could systematically bias evaluations in favor of its own family; the paper provides no analysis of this bias and does not test judge impartiality across model families.","rationale":"The reader's weakest_assumption focused on general judge reliability and generalizability of human validation, which are important but broader concerns. I identify a more specific, unaddressed methodological flaw: the judge model is also a target model, and the paper provides no sensitivity analysis for this family-conflict bias. This does not invalidate the entire platform—the infrastructure, metrics, and validation design remain valuable—but it undermines confidence in the exact numerical claims and model-ordering statements that are central to the paper's empirical contribution. A conditional acceptance requiring a judge-model sensitivity analysis (or an explicit defense of impartiality) is appropriate, because the core asymmetry finding (positive vs negative gap) may still be robust, but the detailed rankings and the claim that negative metrics separate models are at risk if judge bias exists. If the concrete test shows no systematic shift, the paper can revert to ACCEPT status.","tokens_in":42045,"tokens_out":2730,"duration_ms":27303,"concrete_test":"Re-evaluate a stratified sample of ~500 conversations (covering multiple metrics, polarities, and all target models) using an independent judge model that is not in the target set—e.g., Claude Opus 5 or a different family entirely. Compute pass rates and model rankings from the alternative judge, and compare them to the original GPT-5.6 Luna rankings. If the rank of GPT-5.6 Luna drops by 2 or more positions on negative metrics (or if its pass-rate advantage over similarly scored models shifts by >5 percentage points), the conflict concern is substantiated and the ranking claims in §4 become unreliable without adjustment.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest load-bearing concern is that the LLM judge (GPT-5.6 Luna, §3.5.2) appears in the target-model list (§3.4.6) alongside GPT-5.6 Terra and other models. Because the same model family is used both as the evaluation instrument and as a subject, the judge could systematically favor responses from its own checkpoint or family—for example, by interpreting GPT-5.6 Luna's outputs as more appropriate or overlooking subtle failures. This would inflate Luna's pass rates relative to models from other families, directly affecting the central empirical claims about model rankings, the magnitude of the positive–negative gap (e.g., §4.4 reports Luna's positive rate 74.9%, negative 66.3%), and the finding that 'negative metrics separate models far more than positive metrics.' The paper's human validation (18 metrics, 297 ratings) does not test for family-specific bias, and the bootstrap intervals (§4.10) condition on saved judgments and thus do not capture judge-induced systematic error. Although the positive–negative asymmetry may hold across models even with biased judgments, the specific percentages and the claim about between-model separation on negative metrics become uninterpretable without a judge-family bias check. The paper acknowledges evaluation awareness in target models (§6.1) but does not discuss judge-model conflict.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"ImpactBench is an open, expert-guided benchmark for evaluating AI impact on human flourishing across psychological, physical, and social domains. It translates 809 metrics from 26 benchmarks into multi-turn simulated conversations, scored by an LLM judge (GPT-5.6 Luna) and validated through human expert review. Across 48,540 conversations with 10 frontier LLMs, the paper finds that models promote desirable behaviors in 68.7% of evaluations but avoid harms in only 53.1%, with negative metrics better separating models than positive metrics. The benchmark infrastructure, open submission process, and public-facing platform are designed to address structural gaps in current AI evaluation practice.","tokens_in":42425,"tokens_out":3277,"duration_ms":31278,"significance":"If the results hold, ImpactBench provides a much-needed infrastructure for holistic, transparent evaluation of AI's human impact, addressing structural gaps in current benchmarking practice: who defines constructs, how rigorously they are measured, whether evaluation captures multi-turn behavior, and how results are communicated beyond the technical community. The scale (48,540 conversations, 809 metrics, 10 models), the inclusion of multi-turn simulation and matched teen/adult scenarios, the human validation effort, and the sensitivity analyses are notable strengths. The framework is designed to evolve through open submissions, which is important for a contested and rapidly developing area. The central empirical finding that models foster positive outcomes more reliably than they mitigate negative ones, and that negative metrics separate models far more than positive ones, is consequential if substantiated.","major_comments":[{"comment":"Judge-model conflict: GPT-5.6 Luna serves as both the judge (§3.5.2) and one of the ten target models (§3.4.6). The paper does not evaluate whether the judge systematically favors its own family or checkpoint. This could inflate Luna's pass rates and affect the central claims about model rankings and the magnitude of the positive–negative gap (e.g., §4.4 reports Luna's positive rate 74.9%, negative 66.3%). The human validation (§4.9) covers multiple target models but does not report agreement rates broken down by model family, and the bootstrap intervals (§4.10) condition on saved judgments and thus do not capture judge-induced systematic error. The authors should either (a) provide evidence that the judge's impartiality across model families is adequate (e.g., human-model agreement rates per family, or a re-evaluation with a different judge on a subset), or (b) explicitly discuss this limitation and its potential impact on the conclusions about model rankings and the positive–negative asymmetry.","section":"§3.5.2, §3.4.6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Human validation coverage: Expert review covered only 18 of 809 metrics (2.2%) and 297 of 48,540 conversations (0.6%), with only four metrics receiving overlapping reviews. The paper acknowledges this limitation (§5.6), but the central claim that the automated judge is reliable (83.3% majority agreement, Cohen's Kappa = 0.667) is based on this small, non-random sample. Given the low inter-reviewer agreement on some metrics (Krippendorff's alpha as low as 0.121 for scaffolded learning), the generalizability of the validation to the full metric suite is uncertain. The authors should clarify the sampling strategy for human review (was it random or stratified?) and provide a more precise statement about which kinds of metrics the validation supports. In particular, the validation should be analyzed for whether agreement varies by model family or metric polarity.","section":"§4.9","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The claim that 'negative metrics separate models far more than positive metrics' is supported by the range of pass rates (15.9 points for positive vs. 38.6 points for negative). However, this difference could be partly driven by the choice of metrics: negative metrics may be intrinsically more diverse or harder, and the paper does not control for metric difficulty or distribution. The authors should discuss whether the metric set is balanced in difficulty between polarities, or at least acknowledge that the separation may reflect metric selection rather than a fundamental property of model behavior. Additionally, the ranking correlation (Spearman ρ=0.94) suggests that ordering is similar across polarities, which somewhat tempers the claim about separation.","section":"§4.4, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The color scale (red to green) is not explicitly defined; adding a legend would improve interpretability for readers, especially those who may view the figure in grayscale.","section":"Figure 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The list of 26 benchmarks is cited only by name and references. While Table 2 provides a useful summary, consider adding a brief one-sentence description of each benchmark's focus in the main text or appendix to help readers unfamiliar with each source.","section":"§3.2.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper reports Krippendorff's alpha for four HumaneBench metrics based on 9 items per metric. Given the small sample size, the paper should explicitly caution that these coefficients have wide uncertainty and should not be over-interpreted. Also note that the table's caption states 'These evaluations used the earlier model set; they are not agreement estimates for the entire main production evaluation,' which is helpful but could be more prominent.","section":"Appendix C, Table 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The formatting of the table in the preprint shows slight misalignment in the column for Model names; ensure proper alignment in the camera-ready version for readability.","section":"Page 46, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses 'gpt-5.6-sol' for scenario generation and 'gpt-5.6-luna' as the judge. Clarify whether these are the same underlying model or different versions/checkpoints, and if possible report the exact version or checkpoint identifiers to support reproducibility.","section":"§3.5.2, §3.3.2","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision","confidential_remarks":"The paper is ambitious and represents a substantial engineering and empirical effort. The judge-model conflict is a legitimate concern that the skeptic correctly identifies; it is not addressed in the current manuscript. I recommend major revision rather than rejection because the issue is addressable (e.g., by re-running a subset with a different judge or by showing that human agreement rates are comparable across model families). The paper's contributions to the evaluation infrastructure are significant, and the limitations are generally well-acknowledged, but the load-bearing claim about model rankings on negative metrics is vulnerable without a bias check. I also note that the human validation, while commendable in design, covers too small a fraction of the metric suite to fully support the reliability claims made about the full pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"You should know this paper is a serious infrastructure effort: 809 metrics across 26 benchmarks, 10 models, 48k simulated conversations, with multi-turn design, expert validation, sensitivity analyses, and a public platform. That is real work. The central empirical claim—that models are better at promoting positive behaviors (68.7%) than avoiding negative ones (53.1%)—is likely robust as a qualitative pattern, because it holds across all ten models. The paper also gives credit where due: internal and external expert reviews, inter-judge reliability checks, ablation of anchors, and a candid limitations section that flags that these are behavioral proxies and that bootstrap intervals condition on saved judgments.\n\nNow the soft spot that matters most. The judge model (GPT-5.6 Luna) is also a target model. The same checkpoint family scores itself and its sibling (Terra). This is not a minor oversight—it is a potential systematic bias that could inflate Luna's pass rates relative to models from other families. The stress-test note is correct: the paper provides no analysis of whether the judge favors its own family. The human validation on 18 metrics (297 ratings) did not test for family-specific bias. Because the paper’s central findings include model rankings and the claim that \"negative metrics separate models far more than positive metrics,\" the specific percentages and between-model differences become uninterpretable without a judge-impartiality check. The positive–negative gap might survive a bias correction, but the paper does not show that.\n\nOther concerns are milder. Inter-reviewer agreement was low on some metrics (Krippendorff's alpha 0.121 for scaffolded learning), which the paper acknowledges. The bootstrap intervals do not capture variance from the simulator or the judge, also acknowledged. So the main unresolved issue is the judge-model conflict.\n\nThis paper deserves a serious referee: the infrastructure is valuable and the methodology is mostly sound. But a referee should push for an analysis of potential judge bias—for example, re-evaluating a subset with a judge from a different family, or at minimum a systematic discussion of why the conflict might not matter. Who benefits? Anyone building or using human-impact evaluations, including policymakers, developers, and researchers. I would cite the paper for its infrastructure design, not for the specific numerical results until the conflict is resolved.","headline":"A massive, well-executed benchmarking infrastructure for AI human impact, but the judge-model conflict is a load-bearing flaw that needs addressing before the numerical results can be fully trusted.","tokens_in":42984,"tokens_out":1632,"would_cite":true,"duration_ms":16813,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"ImpactBench evaluates 10 frontier LLMs across 809 metrics of human flourishing and finds a persistent gap: models promote good behaviors in 68.7% of cases but avoid harm in only 53.1%.","keywords":["AI Benchmarks","LLM evaluation","Psychology of Technology","Human Flourishing","AI safety","Multi-turn evaluation","Human-AI interaction","Expert-guided evaluation"],"falsifier":"Re-running the full evaluation with human expert judges instead of the LLM judge on a random sample of 1,000 conversations and finding systematic disagreement would falsify the claim that the automated judge provides reliable scores. Alternatively, a real-world longitudinal study showing no correlation between ImpactBench pass rates and actual user well-being outcomes would undermine the benchmark's predictive validity.","tokens_in":43575,"feed_emoji":"⚖️","tokens_out":4212,"duration_ms":39399,"temperature":0.7,"pith_summary":"ImpactBench is an open, expert-guided platform that evaluates how generative AI models affect human well-being across psychological, physical, and social dimensions. The paper establishes a framework for translating domain expertise and existing benchmarks into 809 observable behavioral metrics, tested through 48,540 simulated multi-turn conversations with ten frontier LLMs. Its central claim is that current models are substantially better at promoting desirable behaviors (68.7% pass rate) than at avoiding harmful ones (53.1% pass rate), with particularly large gaps in learning support, autonomy preservation, and emotional boundaries. The authors argue that human-impact evaluation requires separating positive and negative behavioral criteria, multi-turn interaction testing, and open infrastructure that allows diverse stakeholders to contribute and inspect metrics.","feed_headline":"AI models help well but avoid harm in only 53% of chats","feed_subtitle":"Open benchmark ImpactBench reveals frontier LLMs are far better at promoting good behaviors than at preventing harmful ones","key_machinery":"The ImpactBench evaluation pipeline converts expert-identified constructs and existing benchmark criteria into standardized behavioral metrics, each specifying a target behavior and a polarity (positive: desirable behavior present, or negative: harmful behavior absent). For each metric, a generator creates three base scenarios with demographic variants (teen and adult), which drive a user simulator (Claude Sonnet 5) through guided multi-turn conversations with the target model. A judge model (GPT-5.6 Luna) then determines whether the target behavior occurred in the conversation. The pipeline incorporates scenario quality checks, simulator realism ratings, judge repeatability tests, and human expert validation to assess measurement reliability.","core_discovery":"Across 48,540 simulated conversations covering 809 metrics from 26 benchmarks, ten frontier LLMs passed positive metrics (desirable behavior present) 68.7% of the time but passed negative metrics (harmful behavior absent) only 53.1% of the time. This asymmetry was largest in the psychological domain (24.2 percentage points) and most pronounced in subareas such as Learning & Skill Development, where models promoted learning behaviors in 45.9% of conversations but avoided undermining learning in only 16.3%. Models also differed far more on harm avoidance (range 29.8% to 68.4%) than on benefit promotion (59.2% to 75.1%). In teen contexts, models less often supported independent reasoning and more often displayed dependence-building behaviors than in adult contexts. External expert review found 83.3% agreement with the automated LLM judge (Cohen's kappa = 0.667), though inter-reviewer agreement on some metrics was low (Krippendorff's alpha as low as 0.121).","pith_inferences":["If the behavioral patterns observed in simulation translate to real-world interactions, users of current frontier AI systems are likely experiencing frequent subtle harms related to cognitive offloading, decision substitution, and emotional dependency, even when the models appear helpful.","The wide range in negative-metric performance across models suggests harm avoidance is a tractable engineering target; it may be possible to improve through targeted training without sacrificing helpfulness.","ImpactBench's modular open-submission design could serve as a template for regulatory frameworks requiring independent pre-deployment evaluation of AI impact on human flourishing, analogous to clinical trials for medical interventions.","The low inter-reviewer agreement on some metrics (e.g., scaffolded learning, alpha = 0.121) indicates a need for deeper conceptual work before those criteria are used for high-stakes decisions."],"forward_implications":["Evaluations of AI systems should separately measure promotion of beneficial behaviors and avoidance of harmful ones, because these are distinct dimensions of model performance that can diverge.","Model rankings in human-impact benchmarks depend heavily on which metrics are included and how they are weighted; aggregate scores alone are misleading and should be reported alongside benchmark-level and metric-level results.","Developers should prioritize improving harm avoidance, especially in learning, autonomy, and emotional-boundary contexts, where current models exhibit the largest gaps.","Age-adapted evaluation reveals that models may pose unique risks to adolescent users, warranting targeted safety measures beyond those for adult users.","The open, modular design of ImpactBench enables diverse stakeholders to contribute metrics, making the evaluation ecosystem adaptable to evolving evidence and values."],"supporting_citations":[{"why":"Used to argue that most benchmarks lack construct validity and that ImpactBench addresses this gap through explicit validation and reliability audits.","marker":"[9]"},{"why":"Frames the need for an evaluation science with real-world validity, which ImpactBench aims to instantiate.","marker":"[131]"},{"why":"Provides the methodological basis for simulating psychological risks in multi-turn conversations, adapted for ImpactBench.","marker":"[5]"},{"why":"Source benchmark for child safety metrics, adapted into ImpactBench's metrics and scenarios.","marker":"[59]"},{"why":"Source benchmark for humane technology principles, used as a reference for developing behavioral criteria.","marker":"[14]"},{"why":"Provides the holistic flourishing framework that ImpactBench extends with multi-turn simulation and open submission.","marker":"[45]"},{"why":"Supports the psychometric framework for evaluating measurement reliability and validity of LLM-based assessments.","marker":"[108]"}],"fun_headline_variants":["AI models: 69% helpful, but only 53% harm-free","ImpactBench reveals AI's help vs harm gap of 16 points","Frontier LLMs good at helping, bad at not harming","In 48K chats, AI avoids harm just 53% of the time","AI boosts wellbeing but fails to avoid harm in half of chats"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The automated LLM judge (GPT-5.6 Luna) reliably identifies the presence or absence of target behaviors in conversations, and the human validation on 18 metrics (297 ratings) generalizes to the full suite of 809 metrics, despite low inter-reviewer agreement on some dimensions.","fun_headline_variants_meta":{"raw":{"variants":["AI models: 69% helpful, but only 53% harm-free","ImpactBench reveals AI's help vs harm gap of 16 points","Frontier LLMs good at helping, bad at not harming","In 48K chats, AI avoids harm just 53% of the time","AI boosts wellbeing but fails to avoid harm in half of chats"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":8.3e-05,"raw_usage":{"cost":8.255478e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":8.255478e-05,"upstream_inference_prompt_cost":4.615338e-05,"upstream_inference_completions_cost":3.64014e-05},"total_tokens":1432,"prompt_tokens":1027,"completion_tokens":405,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":310}},"tokens_in":1027,"tokens_out":405,"cost_source":"openrouter_usage","duration_ms":4793,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":310,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-16T18:10:55.513030+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"falsifier":"Re-running the full evaluation with human expert judges instead of the LLM judge on a random sample of 1,000 conversations and finding systematic disagreement would falsify the claim that the automated judge provides reliable scores. Alternatively, a real-world longitudinal study showing no correlation between ImpactBench pass rates and actual user well-being outcomes would undermine the benchmark's predictive validity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Used to argue that most benchmarks lack construct validity and that ImpactBench addresses this gap through explicit validation and reliability audits."},{"cited_title":"Toward an Evaluation Science for Generative AI Systems","cited_arxiv_id":"2503.05336","evidence_quote":"Frames the need for an evaluation science with real-world validity, which ImpactBench aims to instantiate."},{"cited_title":"arXiv preprint arXiv:2511.08880 , month = nov, year =","cited_arxiv_id":null,"evidence_quote":"Provides the methodological basis for simulating psychological risks in multi-turn conversations, adapted for ImpactBench."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source benchmark for child safety metrics, adapted into ImpactBench's metrics and scenarios."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source benchmark for humane technology principles, used as a reference for developing behavioral criteria."},{"cited_title":"Measuring AI Alignment with Human Flourishing","cited_arxiv_id":"2507.07787","evidence_quote":"Provides the holistic flourishing framework that ImpactBench extends with multi-turn simulation and open submission."},{"cited_title":"A psychometric framework for evaluating and shaping personality traits in large language models.Nature Machine Intelligence, 7(12):1954–1968, December 2025","cited_arxiv_id":null,"evidence_quote":"Supports the psychometric framework for evaluating measurement reliability and validity of LLM-based assessments."}],"review_version":1}