{"id":"678cbba2-4860-4556-bf78-6377b83deca2","arxiv_id":"2506.10234","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Seven experienced developers produced different Seccomp sandboxes for the same program, struggled to map syscalls to code, and often built over-privileged or fragile policies.","lead":"Seven experienced developers each tried to sandbox the same small C program using Linux's Seccomp tool, and they all ended up with different sandbox designs. The study documents where they struggled, why they made different choices, and what they said would make sandboxing easier.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The over-privileging claim (Sec. 4.1.2) relies on an unstated minimal syscall set; if any of the 15 'not needed' syscalls are actually required at runtime, that finding weakens.","rationale":"The reader's weakest assumption concerns external validity: whether seven self-selected, all-male, compensated developers are representative of developers in general. That is a legitimate limitation, but the paper explicitly disclaims generalizability in Sec. 3.3, so it does not threaten the central descriptive claims about the studied group. A more acute, internally located problem is the unsupported ground-truth syscall set used to label P5 and P7 as over-privileged. This claim is concrete, technical, and falsifiable: if the minimal set is different from what the authors assume, a specific reported finding (that strace/ChatGPT-driven policies are over-privileged) weakens. It does not invalidate the broader usability narrative, but it should be verified or softened. Since the paper already needs conditional revisions (noted by the reader for the novelty overclaim and data availability), this additional concern reinforces CONDITIONAL rather than changing it.","tokens_in":18384,"tokens_out":9537,"duration_ms":106490,"concrete_test":"Independently derive the minimal syscall set for the watchdog program using a delta-debugging harness: run the program to cover all reachable paths (normal loop, log rotation, signal handling, and error exits), then iteratively deny each syscall outside the common core (openat, write, close, connect, socket, rename, clock_nanosleep, newfstatat, lseek) and check whether the program still works correctly. Compare the resulting minimal set with the paper's list of 15 alleged 'not needed' syscalls in Sec. 4.1.2. If any of ioctl, mmap, access, or execv is required for a successful run, the over-privileging claim and the characterization of P5/P7's sandboxes as ineffective must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"A central security-relevant result is that P5 and P7 produced over-privileged sandboxes because they trusted strace output, allowing 15 syscalls 'not needed by the watchdog program,' such as ioctl, mmap, access, and execv. This classification is never justified: the paper does not report how the authors determined that these syscalls are unnecessary, nor does it present the ground-truth minimal syscall set for the watchdog program. Glibc and the dynamic loader can invoke syscalls like mmap or access in edge cases (e.g., stdio buffer allocation, file permission checks, or malloc large-chunk allocation) that may not appear in a short strace run but are still reachable in the program's execution. If even a subset of these 15 syscalls is required for any correct execution path, the conclusion that strace-based workflows lead to 'somewhat ineffective' sandboxes is overstated. The paper's qualitative usability findings remain plausible, but this specific empirical claim needs better support.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a qualitative usability study of the Linux Seccomp sandboxing API. Seven developers with prior Seccomp experience were asked to sandbox a 115-line C watchdog program under observation, using a think-aloud protocol, a System Usability Scale questionnaire, and a participatory design exercise. The paper finds that participants arrived at very different sandbox implementations, struggled to map syscalls to program functionality, commonly over-privileged their sandboxes (notably by trusting strace output), and one participant could not produce a working filter. The authors conclude that Seccomp's flexibility and low-level interface create usability challenges and argue for higher-level abstractions and design patterns.","tokens_in":18536,"tokens_out":6973,"duration_ms":78622,"significance":"The paper is a useful and timely contribution to the small but growing literature on usable security APIs. Its strengths include a transparent study protocol (Appendix B), direct participant quotes, a detailed description of participants' implementations (Table 2), and what appears to be the first empirical usability study of Seccomp. The qualitative findings are plausible and contribute new evidence about why sandboxing adoption is low. The paper is appropriately cautious about generalizability, but a key security-relevant claim—that two participants' sandboxes were over-privileged by allowing unnecessary syscalls—currently lacks a documented ground-truth analysis and needs strengthening before publication.","major_comments":[{"comment":"The claim that P5 and P7's sandboxes were over-privileged and allowed '15 syscalls that were not needed by the watchdog program' is not supported by a reported minimal-syscall analysis. The paper does not state how the authors determined that ioctl, mmap, access, and execv are unnecessary, nor does it report the test conditions (libc version, compile flags, network reachability, signal handling, log-rotation paths) under which a ground-truth syscall set was obtained. Because the watchdog program uses glibc I/O and allocation functions whose syscall usage can vary at runtime (e.g., mmap for large malloc chunks or access for permission checks), a single strace run is insufficient to establish that these syscalls are never required in a correct execution. Without this analysis, the conclusion that strace-based workflows lead to 'somewhat ineffective' sandboxes is overstated; I recommend either adding a systematic enumeration of required syscalls under varied execution conditions or tempering the security-effectiveness claim.","section":"Section 4.1.2, Table 2"},{"comment":"The study did not record participants' screens, yet Table 2 reports counts such as 'Times documentation referred to' and 'Attempts running the code', and Figure 1 constructs a detailed timeline of participant steps. The threats-to-validity section does not explain how these data were captured reliably without screen recording, nor what information may have been missed or misattributed. Please describe the real-time note-taking and verification process used during the sessions and discuss the potential impact of this limitation on the process-level findings.","section":"Section 3.3 and Appendix B.2"},{"comment":"The interpretation of P4's failure as a Seccomp usability problem is somewhat confounded by P4's choice to work with raw BPF filters, which is a particularly low-level interface, rather than the libseccomp wrapper that other participants used. While raw BPF is part of the Seccomp ecosystem, the authors should clarify whether the task instructions encouraged any particular API and discuss how much of P4's struggle is attributable to Seccomp versus to the specific programming interface P4 selected.","section":"Section 4.1 (P4 case)"}],"minor_comments":[{"comment":"The session length and compensation are described inconsistently: Section 3.1 says 1.5 hours and 'approximately $250', while Appendix C says '2 hours' and '£200'. Align these details.","section":"Section 3.1 vs Appendix C"},{"comment":"The typo 'SMPT_ACT_ALLOW' should be 'SCMP_ACT_ALLOW'.","section":"Section 4.2.3"},{"comment":"The SUS responses are presented only as individual item ratings (Figure 2); reporting the aggregate SUS score would allow comparison with other usability benchmarks.","section":"Section 4.4"},{"comment":"The authors' prior work [2] is cited for background; please make the self-citation more explicit in the text.","section":"Section 2.2"},{"comment":"The discussion of P3's 'time' syscall and VDSO should clarify whether VDSO syscalls are subject to seccomp filtering and whether omitting them from the filter is safe.","section":"Section 4.3.2"},{"comment":"The list of essential syscalls includes 'clock_nanosleep' but the program calls sleep(); clarify whether this is the libc wrapper behavior and whether variants such as 'nanosleep' were also observed.","section":"Section 4.1.2"}],"recommendation":"major_revision","confidential_remarks":"The over-privileging claim in Section 4.1.2 is the main technical weakness; the rest of the qualitative analysis is sound and clearly presented. I would recommend requiring a documented ground-truth syscall analysis before acceptance, as the security-effectiveness conclusions currently rest on an unstated minimal syscall set. The paper's methodological transparency and rich qualitative data make it a valuable contribution once this point is addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new thing here is empirical: seven developers with prior Seccomp experience, given the same 115-line watchdog program, produced seven different sandboxes. Most struggled to map syscalls to program behavior, one never got a working filter, and two over-privileged theirs by trusting strace output (one via ChatGPT). That is a concrete, useful observation for the sandboxing-usability literature, and the paper reports it transparently: full protocol in the appendix, participant quotes, and a threats-to-validity section that openly says the sample is tiny and non-generalizable.\n\nThe qualitative work is careful. The authors coded against ISO-9241, combined think-aloud, SUS, and participatory design, and they do not overclaim generalization. The finding that developers work progressively, assume their filters are wrong until proven working, and prefer localizing sandbox logic near the code feels credible.\n\nThe soft spots are real but fixable. The novelty claim is too broad: the abstract and introduction call this 'the only usability study examining how developers utilize sandboxing mechanisms,' but the paper's own related work cites Schreuders et al. on SELinux and AppArmor usability and Dodier-Lazaro's interviews with expert users. 'First usability study of Seccomp' may be defensible; the broader phrasing should be walked back.\n\nThe over-privileging result depends on a minimal syscall set that is never made explicit. The paper asserts that P5 and P7 allowed fifteen syscalls 'not needed' by the program, including mmap and ioctl, but does not show how that ground truth was determined. For a C program using stdio and dynamic linking, some of those may be reachable outside a short strace run. That is a fair concern. The finding would be stronger with the minimal set published and its derivation documented. Even if that classification is imperfect, the core usability claims rest on the transcripts, not on that one detail.\n\nThe all-male, self-selected, seven-person sample is a limitation the authors acknowledge. That is a limitation, not a fatal flaw, for an exploratory qualitative study.\n\nThis paper is for researchers working on sandbox usability, security-API design, or syscall-filter tooling. It deserves a serious referee: the method is disciplined, the contribution is small but real, and the over-claims are correctable. I would recommend conditional acceptance, asking for the novelty wording to be softened and the syscall ground-truth analysis to be documented. If they do that, it is a solid SOUPS-style paper.","headline":"First empirical Seccomp usability study with honest methods; the novelty claim is too broad and the minimal-syscall metric needs grounding, but the core finding — divergent sandbox designs from experienced developers — holds.","tokens_in":19067,"tokens_out":3616,"would_cite":true,"duration_ms":39491,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Even experienced Seccomp developers cannot reliably sandbox a simple program.","keywords":["seccomp usability","Linux sandboxing","system call filtering","usable security","security APIs","developer study","least privilege","sandbox design"],"falsifier":"Run the same 115-line watchdog sandboxing task with a larger random sample of experienced Seccomp developers in a routine, unobserved work setting; if most of them independently produced similar, correctly-scoped sandboxes and could name where each allowed syscall is used, the paper's central claim would fail.","tokens_in":18161,"feed_emoji":"🧩","tokens_out":4235,"duration_ms":48981,"temperature":0.7,"pith_summary":"This paper reports a usability study of Seccomp, Linux's system-call sandboxing mechanism, and it tries to explain why so few applications are sandboxed despite Seccomp's power. Seven developers who had used Seccomp before were given one hour to sandbox a 115-line watchdog program with no dynamic code. Every participant produced a different sandbox design; six produced working sandboxes, one did not, and two over-privileged their sandbox by trusting strace output (and, in one case, ChatGPT) and allowing unnecessary syscalls. The central difficulty was mapping the program's functionality to the syscalls it makes, and participants repeatedly expressed confusion about where syscalls came from. The paper concludes that Seccomp's flexibility, built on the complex and architecture-dependent syscall interface, puts too much burden on developers and that abstraction, automation, and function-level sandboxing would help.","feed_headline":"Seven Seccomp experts build seven different sandboxes","feed_subtitle":"A usability trial shows even experienced developers struggle to map syscalls and over-privilege their sandboxes.","key_machinery":"The paper's central object is Seccomp's filter model: developers initialize a seccomp context, add rules that allow or deny individual syscalls (optionally checking argument values), and load the filter at some point in the program. The study uses a 115-line C watchdog as the common task and traces participants' rule-building against strace output, source reading, and documentation. The load-bearing mechanism is the mapping from program behavior to syscall names—participants had to figure out which syscalls the program made, where, and which were setup-only—and the paper shows this mapping, not the BPF syntax, is what makes Seccomp hard.","core_discovery":"The paper's central claim is that Seccomp is flexible to a fault: for a simple, statically-behaving program, experienced developers do not converge on a sandbox, and the main source of difficulty is not writing BPF rules but deciding which syscalls the program actually needs and where they occur. The seven participants all stated the goal of least privilege, yet their allowed syscall sets, filter placement, use of argument checking, and code structure differed completely. P2 was the only participant to restrict syscall arguments. P5 outsourced filter generation to ChatGPT; P5 and P7 allowed unneeded syscalls such as execv and mmap because they trusted strace output. P4 could not finish. The paper takes this divergence and confusion as evidence that the syscall interface is a poor place to impose a sandbox policy for ordinary developers, and that a more abstract, automated, or function-local model would make sandboxing something developers can do reliably.","pith_inferences":["Going beyond the paper, the finding that some developers blindly trusted strace output suggests that automated Seccomp filter generators should be evaluated not only for functional correctness but also for over-privilege relative to the program's actual needs.","One testable extension is to compare over-privilege rates in LLM-generated Seccomp policies against expert-written policies on the same program, using the watchdog task as a benchmark.","The function-local sandboxing idea proposed by one participant could be prototyped as a library or language extension that derives each function's required syscalls automatically, then tested with a larger and more diverse developer sample.","The paper's recruitment through security mailing lists and freelancing platforms may have selected for unusually security-conscious developers, which would make the observed struggles a lower bound for the general developer population."],"forward_implications":["Even simple programs produce divergent sandbox designs, so without standard patterns the security of a sandboxed program depends heavily on the individual developer's approach.","Developers' difficulty mapping syscalls to program functionality leads to over-privileged sandboxes, which weakens the least-privilege guarantee that motivates sandboxing in the first place.","Abstraction layers, automation, and function-local sandbox rules align with the mental models participants described and could make sandboxing more reliable.","The system call interface itself is a questionable enforcement point for security policy, as participants could not always tell which syscalls were needed, which were setup-only, and which were architecture variations.","Without verification tools or standard design patterns, developers cannot easily tell whether their Seccomp policy is correct, and several participants reported low confidence even in working solutions."],"supporting_citations":[{"why":"Establishes the motivating gap: fewer than 1% of Debian and Fedora packages directly use sandboxing mechanisms like Seccomp.","marker":"[2]"},{"why":"Systematic review of sandboxing that highlights usability and validation as under-researched areas.","marker":"[20]"},{"why":"Closest prior usability work, examining SELinux, AppArmor and FBAC-LSM and identifying complexity as a barrier.","marker":"[30]"},{"why":"Provides the technical argument that interposing on the system call interface creates practical pitfalls, supporting the paper's critique of Seccomp's design point.","marker":"[9]"},{"why":"Controlled usability study of cryptographic APIs showing developers believe insecure solutions are secure, supplying the methodological template for studying security APIs.","marker":"[1]"},{"why":"Kernel maintainer opposition to per-system-call filtering as a maintenance nightmare, cited to show that Seccomp's complexity is recognized beyond the study.","marker":"[35]"}],"fun_headline_variants":["Seven Seccomp pros, seven sandboxes, zero consensus","Seccomp's real hurdle: picking the syscalls, not writing the filter","Why Seccomp sandboxing is so inconsistent: a usability trial","Seven experts can't agree on Seccomp rules, even for one app","Seccomp's flexibility is its flaw: no two experts sandbox alike"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study assumes that the behavior of seven self-selected, all-male, compensated developers working for one observed hour on a contrived watchdog task tells us how developers in general would use Seccomp in real production settings.","fun_headline_variants_meta":{"raw":{"variants":["Seven Seccomp pros, seven sandboxes, zero consensus","Seccomp's real hurdle: picking the syscalls, not writing the filter","Why Seccomp sandboxing is so inconsistent: a usability trial","Seven experts can't agree on Seccomp rules, even for one app","Seccomp's flexibility is its flaw: no two experts sandbox alike"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000809,"raw_usage":{"total_tokens":3485,"prompt_tokens":818,"completion_tokens":2667,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":2568}},"tokens_in":434,"tokens_out":2667,"duration_ms":25035,"temperature":1.0,"reasoning_tokens":2568,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:31:14.699916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 115-line watchdog sandboxing task with a larger random sample of experienced Seccomp developers in a routine, unobserved work setting; if most of them independently produced similar, correctly-scoped sandboxes and could name where each allowed syscall is used, the paper's central claim would fail.","supporting_citations":[{"cited_title":"Sandboxing adoption in open source ecosystems","cited_arxiv_id":null,"evidence_quote":"Establishes the motivating gap: fewer than 1% of Debian and Fedora packages directly use sandboxing mechanisms like Seccomp."},{"cited_title":"A systematic analysis of the science of sandboxing","cited_arxiv_id":null,"evidence_quote":"Systematic review of sandboxing that highlights usability and validation as under-researched areas."},{"cited_title":"Towards usable application-oriented access controls: qualitative results from a usability study of SELinux, AppArmor and FBAC-LSM","cited_arxiv_id":null,"evidence_quote":"Closest prior usability work, examining SELinux, AppArmor and FBAC-LSM and identifying complexity as a barrier."},{"cited_title":"Traps and pitfalls: Practical problems in system call interposition based security tools","cited_arxiv_id":null,"evidence_quote":"Provides the technical argument that interposing on the system call interface creates practical pitfalls, supporting the paper's critique of Seccomp's design point."},{"cited_title":"Com- paring the usability of cryptographic APIs","cited_arxiv_id":null,"evidence_quote":"Controlled usability study of cryptographic APIs showing developers believe insecure solutions are secure, supplying the methodological template for studying security APIs."},{"cited_title":"Enable ftrace-based system call filtering, 2011","cited_arxiv_id":null,"evidence_quote":"Kernel maintainer opposition to per-system-call filtering as a maintenance nightmare, cited to show that Seccomp's complexity is recognized beyond the study."}],"review_version":1}