{"id":"c9ab4113-ee7b-4914-9bd7-78f8d8c8985a","arxiv_id":"2412.16022","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A practitioner-focused guide that distills existing AI ethics literature into actionable Do's and Don'ts for each stage of LLM research projects.","lead":"This paper presents a practical guide, the LLM Ethics Whitepaper, that condenses published research on ethical issues in large language model development into Do's and Don'ts for each project stage. It is aimed at NLP practitioners who want concrete first steps rather than abstract policy frameworks.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core deliverable (Do's/Don'ts) rests on an unvalidated, partly ad hoc literature selection; a recall check against an independent seed set would settle whether the 'comprehensive directory' claim withstands scrutiny.","rationale":"The reader's weakest_assumption—that the partially ad hoc literature selection in §2 is the load-bearing premise—is the concern that survives scrutiny. The Do's and Don'ts are the entire deliverable, and they are explicitly said to be 'directly drawn from our extensive literature review'; if the selection is biased or incomplete, practitioners following the guide inherit that bias. Section 9's honest limitation statement (English-centric, Western perspective, text-only) narrows the settings where the premise holds but does not resolve the methodological gap: no screening data, no versioned protocol, and no measurable completeness. I find no stronger objection: the paper makes no empirical or formal claims, so correctness reduces to the review's representativeness and the accuracy of its synthesis. Independent support exists—the whitepaper is open and living, the Do's/Don'ts are framed as starting points, and several toolkits cited are canonical—so the concern is not fatal. A CONDITIONAL verdict is right: require either a softened comprehensiveness claim or a reproducible, versioned review protocol with inclusion/exclusion logs and a recall target. My concrete test—measuring recall against an independent seed set drawn from the paper's own cited surveys—would settle whether the concern actually lands; if recall is high, the guide's directory claim stands and the verdict can move to ACCEPT.","tokens_in":16683,"tokens_out":2608,"duration_ms":19393,"concrete_test":"Construct an independent reference set of roughly 40 LLM-ethics resources drawn from the bibliographies of the three most-cited LLM-ethics surveys published 2023–2024 (e.g., Weidinger et al. 2022, Gallegos et al. 2024, Solaiman et al. 2024), restricted to resources meeting the paper's own inclusion scope (practical toolkits, guidelines, frameworks, and ethics resources for LLMs). Run the §2 ACL Anthology and Semantic Scholar searches verbatim, replicate the manual screening against this set, and report recall plus the number of missing resources that the paper's own scope language would clearly cover. If recall is materially below ~80%, the 'comprehensive directory' claim and the abstract's gap-filling claim should be softened to 'a curated starting point', which would convert this CONDITIONAL into an ACCEPT.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that it distills a 'thorough literature review' into a practical guide filling a gap in LLM-ethics resources (Abstract; §1). The load-bearing premise is that the review protocol in §2 yields a representative and accurate distillation. This is insecure for three reasons: (1) The protocol is partially described—ACL Anthology search strings and one Semantic Scholar query are given, but the number of hits, screening criteria, inclusion/exclusion counts, and inter-reviewer agreement are all omitted, so the 'directory' component cannot be audited. (2) Section 2 then admits that resources familiar to the authors were added 'ad hoc' and that many papers were removed for subjective reasons ('limited relevance', 'too narrow'), making the final set an unmeasured mixture of systematic search and expert curation. (3) The claimed gap—'no work that integrates these resources into a single practical guide'—is a strong existence claim, yet no falsifiable search beyond two convenience sources is reported. The paper itself concedes in §9 that the resource is English-centric, largely Western, and text-only, which narrows the settings where the premise holds but does not repair the method. None of this makes the guide useless; it makes the comprehensiveness claim unverifiable as written. The decisive, testable weakness is recall: if a meaningful fraction of in-scope LLM-ethics resources are absent, practitioners following the Do's/Don'ts inherit an unmeasured selection bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a practical guide to ethical research with large language models, structured around a simplified project lifecycle (project start, data compilation, data preparation, model development, evaluation, deployment). The authors report a literature review of the ACL Anthology and Semantic Scholar, supplemented by expert-selected resources, and distill the findings into concrete Do's and Don'ts plus a list of useful toolkits for each lifecycle stage. The paper also points to a longer companion whitepaper (Ungless et al., 2024) that contains the full citations and detailed discussion. The stated contribution is a practitioner-facing resource that fills a gap between broad AI frameworks (NIST, EU AI Act) and association-level checklists (NeurIPS, ACL).","tokens_in":16934,"tokens_out":3986,"duration_ms":35078,"significance":"If the guide is adopted, it would provide NLP researchers with an actionable, lifecycle-structured reference that translates a large body of ethics literature into specific recommendations. The open and living format on GitHub, the explicit separation of Do's and Don'ts, and the curation of toolkits are genuine practical strengths. The paper is also honest about its limitations, acknowledging English-centricity, a largely Western perspective, and a text-only focus. However, the significance is contingent on the representativeness of the literature selection: the authors' claim of a 'comprehensive directory' is load-bearing because the authority of the Do's and Don'ts rests on the quality and coverage of the underlying review. As reported, the methodology does not allow a reader to verify that claim.","major_comments":[{"comment":"The protocol for the literature review is described with search strings but without the information needed to audit the resulting 'comprehensive directory': the number of hits retrieved, the screening criteria, the inclusion/exclusion counts, and the inter-reviewer agreement are all omitted. The subsequent admission that resources were added 'ad hoc' and removed for subjective reasons (e.g., 'limited relevance', 'too narrow') means the final set is an unmeasured mixture of systematic search and expert curation. Because the Do's and Don'ts are explicitly 'drawn from' this review, the representativeness of the selection is load-bearing for the guide's authority. I recommend either reporting a full PRISMA-style flow and screening protocol or reframing the deliverable as a curated, non-exhaustive resource rather than a 'comprehensive directory'.","section":"Section 2"},{"comment":"The claim that 'as yet there is no work that integrates these resources into a single practical guide' is a strong negative existence claim, yet the evidence reported is confined to two convenience sources (ACL Anthology and Semantic Scholar) and the authors' own familiarity. No recall check or comparison with an independent seed set of LLM-ethics resources is provided, so a reader cannot verify that the gap is not an artifact of the search strategy. Please either soften the claim to 'we are not aware of' or provide a more systematic basis for the gap claim.","section":"Abstract and Section 1"},{"comment":"The guide states that the Do's and Don'ts are applicable 'regardless of model language,' but many of the recommended toolkits and evaluation resources are English-specific, and the overall perspective is acknowledged as largely Western. The limitation is properly acknowledged in Section 9, but the main text (e.g., the abstract and Section 1) presents the guide without this scope restriction. This tension should be resolved by integrating the scope limitation into the framing rather than only in the concluding section.","section":"Section 9 versus Section 3.5"}],"minor_comments":[{"comment":"The sentence 'Sasha Luccioni and colleagues have in particular championed the accurate reporting of the carbon emissions of ML systems including LLMs' lacks a citation; please add a reference to enable follow-up.","section":"Section 3.4"},{"comment":"In the reference for Smith et al. (2022), 'Melissa Hall Melanie Kambadur' should be 'Melissa Hall, Melanie Kambadur'.","section":"References (Smith et al., 2022)"},{"comment":"The spelling 'Github' should be 'GitHub' in both locations.","section":"Section 1 and Section 9"},{"comment":"The Semantic Scholar query 'toolkit OR sheets OR guideline OR principles OR framework OR approach ethics OR ethical OR harms OR fair OR fairness OR risk AND \"language models\"' is ambiguous due to operator precedence; please clarify the intended Boolean grouping with parentheses.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The present paper is essentially a summary of the companion whitepaper (Ungless et al., 2024); the editor may wish to consider whether the primary contribution resides in the whitepaper and whether this overview paper adds sufficient standalone value. The pocket-guide format does serve a distinct purpose, but the relationship should be made clearer. The citation pattern includes several of the authors' own works, which is not inappropriate for a curated guide, yet the overlap with the companion whitepaper should be explicit so reviewers and readers can assess novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. It does what it claims: a pocket guide to ethical LLM research, organized by project lifecycle, with concrete Do's and Don'ts and toolkits per stage. The individual recommendations mostly trace to existing work—datasheets, model cards, ethics sheets, participatory design—so the novelty is the integration, not the ingredients. That integration is genuinely useful: I can see this being adopted in lab onboarding and teaching. The authors are transparent about method: two database searches plus ad hoc additions and subjective removals. That's where the soft spot is. The abstract's 'comprehensive directory' and 'no work that integrates these resources' are stronger claims than the method can support. No screening counts, no inclusion/exclusion numbers, no inter-reviewer agreement. So if you want to know whether they missed major relevant resources, you can't tell from the paper. A recall check against an independent seed set would fix that. The limitations section is honest: English-centric, Western, text-only, and explicitly disavows full coverage. So the actual overclaim is mainly in the abstract and Section 2 framing. Self-citation to the companion whitepaper is appropriate; it's the resource being summarized. All in all, it's a solid, usable guide with a method that deserves tightening. It should go to peer review: a serious referee can ask for the missing screening data or a softened comprehensiveness claim. I'd support acceptance after minor revision.","headline":"A genuinely useful lifecycle-organized ethics guide for LLM practitioners, with a methodology that needs one round of tightening to match its 'comprehensive' billing.","tokens_in":17498,"tokens_out":1843,"would_cite":false,"duration_ms":17646,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The authors claim no integrated, practical LLM ethics guide exists, and present one: a living whitepaper that turns a literature review into stage-by-stage Do's and Don'ts for NLP practitioners.","keywords":["LLM ethics","ethical guidelines","project lifecycle","Do's and Don'ts","NLP practitioners","literature review","responsible AI","ethics whitepaper"],"falsifier":"A traceability audit would settle it: compare every Do and Don't against the full set of papers returned by the two literature searches and the tutorial references. If a substantial share of recommendations cannot be traced to a cited source, or if major harm categories documented in the cited taxonomies (for example, non-English or non-text harms) are missing from the guide, the distillation claim fails.","tokens_in":16477,"feed_emoji":"📋","tokens_out":6740,"duration_ms":54039,"temperature":0.7,"pith_summary":"The paper's central claim is that nothing yet integrates the scattered ethics literature on large language models into a single practical guide aimed at the people actually building and evaluating them. To fill that gap, the authors present the LLM Ethics Whitepaper, an open, living resource, and this paper as its pocket-guide summary. The deliverable is a set of stage-by-stage Do's and Don'ts, each drawn from a literature review and paired with toolkits. The authors argue that these recommendations give computer scientists concrete first steps rather than abstract principles. If the guide works as intended, a practitioner can consult it throughout a project lifecycle and walk away knowing what to do and what to avoid at each stage.","feed_headline":"New pocket guide maps ethics onto every LLM project stage","feed_subtitle":"Distills the ethics literature into Do's and Don'ts for each step from data collection to deployment.","key_machinery":"The carrying device is the simplified project lifecycle, a six-stage diagram that structures both the whitepaper and this overview. It turns 'ethics' from a diffuse topic into a sequence of concrete decisions: what to do at project start, how to compile and prepare data, how to choose and develop a model, how to evaluate it, and how to deploy it. For each stage the paper pairs a set of Do's and Don'ts with a small set of tools — ethics sheets and internal audits at the start, datasheets and scraping guidance during data work, model cards and debiasing cautions during development, harm-evaluation taxonomies and red-teaming at evaluation, and release strategies and monitoring at deployment. The machinery works by making each recommendation point to a named, usable resource.","core_discovery":"The paper claims that existing ethical guidance is split between broad regulatory frameworks, which are hard to act on, and submission checklists, which are too thin, leaving LLM practitioners without an integrated, practical reference. It attempts to close that gap with the LLM Ethics Whitepaper, whose distilled content appears here as Do's and Don'ts organized by a simplified project lifecycle. Each lifecycle stage — project start, data compilation, data preparation, model development and selection, evaluation, deployment — comes with concrete recommendations and pointers to toolkits such as ethics sheets, datasheets, model cards, bias-test suites, and staged-release strategies. The authors state that the recommendations were drawn from a literature review of the ACL Anthology and Semantic Scholar, supplemented by the authors' expertise, and that the whitepaper cites the hundreds of underlying papers. The present paper is offered as the condensed pocket version.","pith_inferences":["If adopted widely, the stage-by-stage format could push ethics from a review-time formality toward a set of project artifacts that reviewers and auditors can check for.","The acknowledged English-centric and Western skew implies a testable prediction: practitioners working on non-English or multimodal systems will find fewer tools and recommendations directly applicable to their setting.","The paper's Do's and Don'ts could be operationalized further by linking each one to a verifiable output, effectively turning the guide into a lightweight audit checklist.","Because the companion whitepaper and this overview are versioned separately, the 'living' claim means the arXiv paper may not remain identical to the GitHub resource; readers should treat the online version as the current authority."],"forward_implications":["A researcher can use the paper as a reference during a project rather than as a post-hoc checklist, because each section maps to a distinct stage of work.","Following the Do's and Don'ts yields concrete artifacts: ethics sheets before starting, datasheets for new datasets, model cards for released models, and staged-release plans before deployment.","The guide is meant to be more actionable than broad AI frameworks and more detailed than association submission checklists, filling the middle space for LLM-specific work.","Because the resource is hosted as a living document on GitHub, the recommendations can be revised in response to practitioner feedback and future editions can extend beyond the current English, Western, text-only scope."],"supporting_citations":[{"why":"The companion LLM Ethics Whitepaper that this paper summarizes and introduces as the central resource.","marker":"Ungless et al. (2024)"},{"why":"Establishes the environmental and social risks of large language models that motivate the guide.","marker":"Bender et al. (2021)"},{"why":"Supplies a taxonomy of ethical and social risks that the guide's categories draw on.","marker":"Weidinger et al. (2021)"},{"why":"Provides Ethics Sheets for AI Tasks, the recommended first-step tool at project start.","marker":"Mohammad (2022)"},{"why":"Provides the internal algorithmic auditing framework recommended for oversight.","marker":"Raji et al. (2020)"},{"why":"Datasheets for Datasets, the recommended documentation practice for data compilation.","marker":"Gebru et al. (2020)"},{"why":"Model Cards for Model Reporting, the recommended documentation practice for model development and selection.","marker":"Mitchell et al. (2019)"},{"why":"Release Strategies, the basis for the deployment-stage recommendation to stage model releases.","marker":"Solaiman et al. (2019)"},{"why":"The EACL tutorial whose references were manually reviewed and folded into the whitepaper, part of the literature review method.","marker":"Benotti et al. (2023)"}],"fun_headline_variants":["LLM ethics pocket guide: Do's and Don'ts per stage","Stage-by-stage LLM ethics cheat sheet","From data to deployment: LLM ethics in Do's and Don'ts","Pocket guide transforms LLM ethics into actionable advice","One-page LLM ethics map for every project step"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the authors' literature selection and expert synthesis give a representative picture of LLM ethics, a premise the paper itself narrows by conceding its resources are English-centric, mostly Western, and text-only.","fun_headline_variants_meta":{"raw":{"variants":["LLM ethics pocket guide: Do's and Don'ts per stage","Stage-by-stage LLM ethics cheat sheet","From data to deployment: LLM ethics in Do's and Don'ts","Pocket guide transforms LLM ethics into actionable advice","One-page LLM ethics map for every project step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1525,"prompt_tokens":942,"completion_tokens":583,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":500}},"tokens_in":558,"tokens_out":583,"duration_ms":5048,"temperature":1.0,"reasoning_tokens":500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:52:28.657514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A traceability audit would settle it: compare every Do and Don't against the full set of papers returned by the two literature searches and the tutorial references. If a substantial share of recommendations cannot be traced to a cited source, or if major harm categories documented in the cited taxonomies (for example, non-English or non-text harms) are missing from the guide, the distillation claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a taxonomy of ethical and social risks that the guide's categories draw on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Ethics Sheets for AI Tasks, the recommended first-step tool at project start."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The EACL tutorial whose references were manually reviewed and folded into the whitepaper, part of the literature review method."}],"review_version":1}