{"id":"ad8502d0-a160-4fc1-b7fe-66c6a30e03d2","arxiv_id":"2508.14796","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A guide that enumerates direct and indirect stakeholder types in cybersecurity research, maps them to common research methods, and illustrates the process with four examples from the authors' own publications.","lead":"This preprint is a practical guide for cybersecurity researchers who must now include stakeholder-based ethics analyses in their submissions, following the USENIX Security 2026 requirement. It provides stakeholder taxonomies, method-to-stakeholder mappings, and four worked examples drawn from the authors' own papers.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 and the §4 examples classify direct stakeholders inconsistently with the paper's own §2.2 definition, undermining the guide's central promise of complete stakeholder identification.","rationale":"I examined the paper's central claim and its supporting artifacts. The reader's weakest assumption identified the validity of Table 2; I agree, and I find a more specific internal flaw: the table and examples violate the paper's own definition of 'direct stakeholder.' This is load-bearing because the guide's entire purpose is to help researchers enumerate stakeholders; if the foundational classification is inconsistent, the guidance cannot deliver the promised completeness. The concern is not about external consensus but about internal consistency. The paper has no formal verification or empirical validation; the four worked examples are self-authored and replicate the inconsistency. A simple analytic audit can settle the matter, so the paper should be conditionally accepted pending revision of the taxonomy to align with the definition, or a clear revision of the definition itself. I do not see a reason to reject outright; the paper is a useful starting point and the concern is fixable. Therefore I recommend keeping the CONDITIONAL verdict.","tokens_in":13941,"tokens_out":5971,"duration_ms":61693,"concrete_test":"Perform a line-by-line audit of Table 2 and the §4 examples against the §2.2 definition. For each entry labeled 'Typical Direct Stakeholder' or 'Direct Stakeholder,' answer: 'Did this stakeholder interact with or become explicitly involved in the research process?' If any entry fails (e.g., OSS maintainers in Repository Mining, original study authors in Systematic Review, adversaries in Example A), the mapping is internally inconsistent. As a second step, have two independent coders apply the guide to a set of 10 published security papers spanning at least 5 method clusters, and measure inter-coder agreement on direct vs. indirect classification. Low agreement would empirically confirm the ambiguity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that its method-to-stakeholder mapping enables researchers to identify relevant direct and indirect stakeholders with reasonable completeness. The load-bearing artifact is Table 2. However, Table 2 is not merely unvalidated; it is internally inconsistent with the definition in §2.2. §2.2 defines direct stakeholders as those who 'interact with, or are explicitly involved in, the research process (e.g., as participants or collaborators).' Yet Table 2 lists stakeholders who do not interact with or participate in the research as 'Typical Direct Stakeholders.' For example, 'Repository Mining, Corpus Analysis' lists 'OSS maintainers, contributors, downstream users' as direct. Maintainers of code mined from public repositories do not interact with the researchers or participate in the study; they are affected only through the research findings or publication, which matches the paper's own definition of indirect stakeholders. Similarly, 'Systematic Review, Replication' lists 'Original study authors' as direct, though they are not involved in the replication. The §4 examples repeat this: Example A classifies 'Adversaries' as direct stakeholders, while Example B—using overlapping methods (corpus analysis + tool evaluation)—lists 'Adversaries' as indirect, showing the classification is also arbitrary across examples. Because the guide's core output is this taxonomy, its inconsistency with the definition produces unreliable guidance: a researcher following §2.2 would place these groups differently from Table 2, and two researchers using the guide could reach contradictory completeness judgments. The guide therefore fails its stated goal of providing 'practical support' for identifying stakeholders with confidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is a practical guide to stakeholder-based ethics analysis for cybersecurity research, motivated by the USENIX Security 2026 requirement that submissions include a stakeholder analysis. It defines direct and indirect stakeholders, proposes a mapping from research method clusters to typical direct stakeholders (Table 2, claimed to be derived from the SIGSOFT Empirical Standards), and presents four worked examples drawn exclusively from the authors' own prior publications (Section 4). The paper's central claim is that a researcher following this guide can enumerate relevant stakeholders for a project and draft an ethics statement with reasonable completeness.","tokens_in":14141,"tokens_out":3809,"duration_ms":44473,"significance":"The guide addresses a timely and genuine need: many cybersecurity researchers are now required to perform stakeholder analysis but may lack a concrete procedure. The paper is clearly written, well structured, and offers useful baseline categories in Table 1. The worked examples, though self-referential, illustrate the intended workflow and surface realistic ethical concerns. If the method-to-stakeholder mapping were properly warranted and internally consistent, the guide would be a valuable community resource. However, the load-bearing mapping and the examples are not currently validated, and the central classification has an internal inconsistency with the paper's own definition of direct stakeholders.","major_comments":[{"comment":"The definition in §2.2 states that direct stakeholders 'interact with, or are explicitly involved in, the research process (e.g., as participants or collaborators).' Table 2, however, lists 'OSS maintainers, contributors, downstream users' as typical direct stakeholders for Repository Mining/Corpus Analysis, and 'Original study authors' as direct for Systematic Review/Replication. These actors do not interact with or participate in the research; they are affected through publication or use of results, which matches the paper's own indirect-stakeholder definition. Because Table 2 is the core deliverable enabling stakeholder identification, this inconsistency undermines the guide's central promise. The definition or the table must be revised and made mutually consistent.","section":"§2.2 and Table 2"},{"comment":"The paper asserts that the method clusters in Table 2 are 'derived from the SIGSOFT Empirical Standards' but provides no derivation. The reader cannot verify which standards categories map to which clusters, why these particular stakeholder sets are associated, or why the lists are complete. This makes the central mapping an unsubstantiated assertion. Please provide a traceable mapping from the standards' method categories to the clusters, or explicitly reframe the table as an untested heuristic. The current presentation overstates the evidence base.","section":"§3, Table 2"},{"comment":"The worked examples are explicitly restricted to papers whose authors are on the present author list, 'to mitigate concerns about bias or blame.' This means the examples cannot serve as validation of the framework's completeness or representativeness; they are self-selected illustrations from a single lab. Moreover, the classifications are internally inconsistent across examples: Table 3 lists Adversaries as a direct stakeholder in Examples A, C, and D, but as an indirect stakeholder in Example B, despite A, B, and D all combining corpus analysis with tool evaluation. The paper needs explicit criteria for direct/indirect assignment and at least one independent worked example outside the authors' own corpus.","section":"§4, Table 3, Examples A–D"}],"minor_comments":[{"comment":"The paper cites reference [19] for the 'USENIX Security 2026 Call for Papers,' but [19] is the USENIX Security 2025 CFP. The 2026 CFP is [3]. Please correct the citation.","section":"§5.1"},{"comment":"The text contains the typo 'e.g.,, IoT' in both examples; delete the extra comma.","section":"§4.1.2, §4.2.2"},{"comment":"The note says 'Citations in bold are included in the analysis in §4.' Ensure the bold formatting is applied consistently to all four papers analyzed in §4 ([32], [34], [28], [37]) or remove the note if formatting is not meaningfully rendered.","section":"Table 2"},{"comment":"The text says 'published in USENIX 2025'; referring to the venue as 'USENIX Security 2025' would be more precise and consistent with the reference list.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"This is a community-guide paper rather than a technical result, but it is potentially in scope if the journal publishes such resources. The main risks are the unsupported method-to-stakeholder mapping and the self-referential examples; both are fixable. The paper should not be accepted without a revised, internally consistent taxonomy and independent illustration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful, readable guide for a real and imminent requirement, and it does several things well. But the stress-test note has it right: the paper's own definition of direct stakeholders is contradicted by Table 2 and the worked examples, and that is a load-bearing inconsistency, not a stylistic quibble. The paper is still worth engaging, but it needs revision before it can serve as a community reference.\n\nWhat's new: Table 2's method-to-stakeholder mapping, even if imperfect, is a genuinely useful organizing device. The worked examples in §4 give concrete templates for writing an ethics statement. The synthesis of the Menlo Report and stakeholder theory is competent, and the authors are transparent about using their own lab's papers and about ChatGPT assistance.\n\nSoft spots, in order of seriousness. (1) Definition vs. practice. §2.2 says direct stakeholders are those who 'interact with, or are explicitly involved in, the research process.' But Table 2 lists OSS maintainers, contributors, and downstream users as direct stakeholders for repository mining; those people do not interact with the researchers. Example A classifies adversaries as direct, while Example B—using overlapping methods—lists them as indirect. This is exactly the kind of inconsistency that would confuse a researcher trying to follow the guide. It's fixable, but it's central. (2) The Table 2 mapping is asserted as 'derived from the SIGSOFT Empirical Standards' with no derivation shown. The clusters may be reasonable, but the stakeholder assignments read as judgment calls. (3) All four examples are from the authors' own lab; disclosed, but it limits independent confirmation. (4) Citation errors: 'Segal et al. [6]' doesn't match reference [6] (Kohno et al.), and §5.1 cites the 2025 USENIX CFP when the 2026 CFP is what's relevant. (5) The claim that 'top venues' require stakeholder analysis rests on a single CFP; the other CFPs cited don't say that.\n\nThe stress-test conclusion that the guide 'fails' is too strong. The stakeholder lists themselves are mostly sensible, and a researcher who ignores the direct/indirect labels would still get a decent starting point. But confidence is the paper's actual promise, and this inconsistency undercuts it.\n\nWho should read it: anyone preparing a USENIX Security 2026 submission, and people studying how ethics norms are institutionalizing in security research. I'd send it to a serious referee; the topic is timely, the writing is clear, and the flaws are correctable. I wouldn't cite it in my own work yet, but I'd point to it as a sign of the field's evolution.","headline":"A useful, clearly written guide for a real upcoming ethics requirement, but its central direct/indirect stakeholder distinction is internally inconsistent and the mapping needs justification before it can be trusted as a community reference.","tokens_in":14787,"tokens_out":3370,"would_cite":false,"duration_ms":37446,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that stakeholder-based ethics analysis for cybersecurity can be reduced to a two-step procedure: use a canonical stakeholder table, then use a research-method-to-stakeholder map, demonstrated on four real studies.","keywords":["stakeholder analysis","research ethics","cybersecurity","empirical methods","ethics statements","direct and indirect stakeholders","USENIX Security 2026","empirical standards"],"falsifier":"Take a cohort of published cybersecurity papers spanning Table 2's method clusters, apply the guide's tables to predict each paper's direct and indirect stakeholders, then compare those predictions with the harms, complaints, or retractions the papers later attract. If a substantial share of realized harms involves stakeholder categories absent from Table 1 or absent from that cluster's row in Table 2, the mapping is incomplete.","tokens_in":13700,"feed_emoji":"🧭","tokens_out":10461,"duration_ms":103587,"temperature":0.7,"pith_summary":"The paper is a practical guide, not an empirical study. It argues that the stakeholder-based ethics analysis now required by top cybersecurity venues, including USENIX Security 2026, can be started from a reusable taxonomy instead of a blank page. It defines direct and indirect stakeholders, maps common empirical research methods to the kinds of direct stakeholders those methods typically expose, and shows the mapping on four worked examples from the authors' own lab. If the mapping is sound, a research team can draft a reasonably complete stakeholder section quickly and reviewers gain a shared vocabulary for checking completeness.","feed_headline":"A two-table guide maps security research to the people it affects","feed_subtitle":"Pick your research method, look up its stakeholders, and draft the required ethics page from four worked examples.","key_machinery":"The load-bearing machinery is a pair of tables and a two-way distinction. Table 1 lists twelve stakeholder categories separated into direct and indirect. Table 2 groups common cybersecurity research methods—controlled experiments, interviews, case studies, repository mining and corpus analysis, tool evaluation and benchmarking, simulation, longitudinal and meta-science studies, systematic review and replication—and asserts the typical direct stakeholders for each cluster. The direct/indirect distinction separates stakeholders reached through short, observable causal links from those affected through diffuse or downstream effects. The Menlo Report's four principles supply the ethical baseline","core_discovery":"The guide's central claim is that stakeholder identification in cybersecurity research is a method-driven enumeration task. Step one is to use Table 1's canonical list of direct stakeholders (participants, system operators, software maintainers, data subjects, the research team) and indirect stakeholders (end users, vulnerable populations, adversaries, broader public, institutions). Step two is to use Table 2, whose rows are clusters of empirical research methods derived from the empirical standards cited in the paper, to see which stakeholders a project's method most likely exposes. The paper further claims that problem domain determines who is exposed, while research method determines how","pith_inferences":["My inference: Table 2 would be strongest if validated empirically—for each method cluster, one could collect post-publication harm reports, retractions, or public controversies and check whether Table 2 would have named the affected parties.","My inference: the paper leaves implicit how a multi-method study should weigh competing harms across clusters; a practical extension would be a prioritization rule for merging conflicting stakeholder exposures.","My inference: because all four worked examples are drawn from the authors' own research, transferability to human-subjects-heavy or large-scale measurement studies outside that setting is untested but directly testable by applying the tables to other published papers."],"forward_implications":["A research team preparing a USENIX Security 2026 ethics statement can begin with Table 1 as a checklist and use Table 2 to focus on the stakeholders its research method most likely exposes.","Because a single study can fall into several method clusters, the guide implies stakeholder lists should be unioned across clusters rather than read as mutually exclusive.","Reviewers and IRB reviewers can use the same two tables to test whether an ethics statement's coverage is plausible, instead of relying on intuition about who might be harmed.","The worked examples show that changing the research method while keeping the problem domain can change the exposed stakeholder set, so ethics analysis must track methodology, not just topic."],"supporting_citations":[{"why":"Establishes the formal requirement this guide responds to: USENIX Security 2026 mandates a stakeholder-based ethics analysis page.","marker":"[3]"},{"why":"Supplies the Menlo Report's four ethical principles, the baseline the paper uses to justify stakeholder identification.","marker":"[4]"},{"why":"Provides the contemporary security-research ethics framing in which stakeholder identification is the common starting point across ethical frameworks.","marker":"[6]"},{"why":"Provides the requirements-engineering definition of stakeholders and the identification process the paper adapts to research ethics.","marker":"[12]"},{"why":"Supplies the classic stakeholder definition that grounds the direct/indirect stakeholder distinction.","marker":"[13]"},{"why":"Source from which Table 2's research-method clusters are derived.","marker":"[27]"}],"fun_headline_variants":["Stakeholder analysis: a method-driven checklist for security research","Map your security research method to the people it affects","New guide turns stakeholder ethics into a lookup table","Cybersecurity ethics: a practical stakeholder mapping guide"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that Table 2's mapping from research methods to typical direct stakeholders is complete and correct, a claim the paper asserts from the empirical standards without showing its derivation; a secondary premise is that the four worked examples, all from the authors' own lab, are representative enough to demonstrate the procedure.","fun_headline_variants_meta":{"raw":{"variants":["Stakeholder analysis: a method-driven checklist for security research","Map your security research method to the people it affects","New guide turns stakeholder ethics into a lookup table","Cybersecurity ethics: a practical stakeholder mapping guide"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":7.9e-05,"raw_usage":{"total_tokens":708,"prompt_tokens":599,"completion_tokens":109,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":343,"completion_tokens_details":{"reasoning_tokens":46}},"tokens_in":343,"tokens_out":109,"duration_ms":2176,"temperature":1.0,"reasoning_tokens":46,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:16:13.371499+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a cohort of published cybersecurity papers spanning Table 2's method clusters, apply the guide's tables to predict each paper's direct and indirect stakeholders, then compare those predictions with the harms, complaints, or retractions the papers later attract. If a substantial share of realized harms involves stakeholder categories absent from Table 1 or absent from that cluster's row in Table 2, the mapping is incomplete.","supporting_citations":[],"review_version":1}