{"id":"6886bfb5-3995-4144-b2e4-14d40d0a09d8","arxiv_id":"2608.12853","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Python bytecode is a visible, analyzable, and crash-reachable package artifact that source-only security workflows miss, and current decompilers cannot turn bytecode-level crashes back into ordinary source.","lead":"This paper measures how often Python packages ship compiled bytecode instead of or alongside readable source, and shows that this bytecode can crash analysis tools and the Python interpreter itself. It argues that security scanners should inspect bytecode directly, because source-only checks miss a real and reachable attack surface.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ3's 'at least 91.7% post-ingestion execution' rests on stack-prefix labels (Table 16) that may count teardown/construction crashes as execution; without a phase-instrument check the headline lower bound is unverified.","rationale":"The reader's weakest assumption is the same one I would defend as load-bearing: Table 16's context labels do the work of separating documented-unsafe marshal behavior from the claimed deeper interpreter-execution result. I do not see a reason to reject the paper; RQ1/RQ2 counts are internally consistent, the L4 definition is explicitly emission-only, RQ4 is explicitly tool-bounded in the body, and the artifact release is a genuine reproducibility asset. The problem is specifically that RQ3's most policy-relevant sentence, the 91.7% lower bound, is not supported by a direct phase measurement. A phase-instrument rerun is cheap relative to the campaigns and would settle it. I therefore keep the reader's CONDITIONAL verdict; no change to verdict is needed, but the condition should be stated as 'the execution-boundary split must be validated by phase instrumentation or manual triage before the abstract's 91.7% sentence is relied on.'","tokens_in":30472,"tokens_out":4871,"duration_ms":51143,"concrete_test":"Take the 1,009 retained crash groups and re-run each under a minimally instrumented CPython build that records two events: successful entry into _PyEval_EvalFrameDefault for the mutated code object, and completion or failure of code-object construction before that entry. Then recompute the loading-vs-execution split using only groups whose crash occurs after the eval-frame entry. If this fraction is substantially below 91.7%, revise the abstract and contribution to say 'crash-reachable during ingestion or post-ingestion processing' rather than asserting execution beyond ingestion for the large majority.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RQ3's headline lower bound—'at least 91.7% of groups reach execution beyond the documented-unsafe ingestion boundary'—is assigned in Section 5.3.2 purely from normalized stack prefixes. Table 16 classifies Object runtime, Frame eval., GC/final., and Instrument. as post-ingestion execution contexts, and only Code object plus Unknown as ingestion/unattributable. The arithmetic is conservative only about the 59 Unknown groups; it is not conservative about the four 'execution' labels. A crash during code-object construction, quickening, or deallocation can surface the same frames without the mutated instruction stream ever reaching _PyEval_EvalFrameDefault. For example, Table 17 itself lists _PyCode_Quicken and _PyCode_New frames, and GC/final. crashes could occur when a partially built code object is torn down after a marshal failure. With no sanitizer builds and no per-group root-cause phase check, the claim that hardening must extend beyond marshal rests on unvalidated labels. If the true post-ingestion fraction is materially lower, the abstract and Section 6.2 overstate the runtime-hardening implication.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a four-phase empirical study of Python bytecode as a security artifact. Phase 1 inventories bytecode in a package-balanced sample of PyPI wheels and source distributions (1,034,843 artifacts; 7,388 with .pyc files; 228,578 .pyc files; 28,193 artifact-local source-less .pyc files). Phase 2 measures version-aware analyzability with marshal, dis, Decompyle++, and PyLingual, finding that 204,901 of 204,904 in-scope CPython 3.8-3.14 .pyc files reach L4 (at least one decompiler emits source), with 50 tool robustness failures collapsing to 6 signatures. Phase 3 fuzzes CPython 3.8-3.14 with mutated .pyc seeds over seven 24-hour single-worker honggfuzz campaigns, yielding 12,404 crash inputs, 1,009 stack-deduplicated groups, 261 groups with potential memory-corruption symptoms, and a claim that at least 91.7% of groups reach post-ingestion execution. Phase 4 attempts to reproduce the 1,009 findings from source recovered by PyLingual and Decompyle++, reporting that none reproduce and that decompilers show 11 additional robustness signatures. The paper repeatedly and explicitly scopes its claims: L4 means source emission, not semantic equivalence; RQ3 findings are lower-bound reachability evidence, not exploitability; RQ4 is tool-bounded. The artifact release includes PycLens and manifests, with crash-triggering inputs gated pending disclosure.","tokens_in":30682,"tokens_out":11124,"duration_ms":98496,"significance":"If the central claims survive review, the paper makes a solid empirical contribution: it demonstrates at scale that executable bytecode is present in PyPI artifacts beyond source-visible content, that current decompilers can emit source for most modern bytecode while failing on a non-trivial subset, and that malformed bytecode can crash CPython in many distinct paths. The explicit separation of source emission from semantic equivalence, the distinction between executing and load-but-do-not-execute consumers, and the public tooling are strengths. The paper's main limitation is that its headline 'at least 91.7% post-ingestion execution' and the unqualified 'None reproduce from ordinary Python source' are stronger than the evidence supports; both are addressed by concrete, feasible fixes.","major_comments":[{"comment":"The 'at least 91.7% of groups reach execution beyond the documented-unsafe ingestion boundary' claim is not established by the presented evidence. The runtime-context labels are assigned from normalized stack prefixes, and the paper provides no criterion for distinguishing a crash during code-object construction, quickening, or deallocation from a crash after the mutated instruction stream begins executing. The representative CPython 3.13 group with stack prefix _Py_GetBaseOpcode/_PyCode_Quicken/_PyCode_New in Table 17 is itself described as a loading-time crash, yet Table 16 assigns only 12 of 198 CPython 3.13 groups to 'Code object'; if similar frames can appear in groups labeled 'Object runtime' or 'GC/final.', the 91.7% lower bound is inflated. Because the abstract and Section 6.2 use this number to argue that hardening must extend beyond marshal, the classification must be validated. I request a phase-instrumented replay of the 1,009 groups that records whether _PyEval_EvalFrameDefault (or the equivalent evaluation entry point) was entered for the mutated code object before the fault, or, if that is infeasible, a rewording of the claim to describe the stack-prefix distribution rather than post-ingestion execution.","section":"§5.3.2 and Table 16"},{"comment":"The sentence 'None reproduce from ordinary Python source' overstates the tool-bounded negative result. The experiment shows that two selected decompilers, PyLingual and Decompyle++, did not emit source that compiles and reproduces any of the 1,009 findings; §5.4.1 itself labels the categories 'tool-bounded.' Because the inputs are bytecode mutations that the CPython source compiler does not produce, the result conflates a representational gap with decompiler limitations on malformed inputs. The abstract and conclusion should be reworded to 'none reproduce through the evaluated source-recovery workflows,' and Section 1's claim that behavior 'need not match source-level behavior' should be qualified accordingly. A stronger design would also run the original pre-mutation seed source as a control to confirm that the crash is not source-reachable, although even such a control would not rule out other source-level programs.","section":"Abstract and §5.4"}],"minor_comments":[{"comment":"The 'Plausible cause' categories are inferred from path shape, but the path-shape heuristics are not described; please state how 'accidental import cache' is distinguished from 'bundled environment cache' and 'bare compiled-only module.'","section":"§5.1.3, Table 7"},{"comment":"The citation [12] after 'Honggfuzz retained 12,404 crash-triggering inputs' appears to reference Klees et al. on fuzz testing evaluation, not the honggfuzz tool; either cite the honggfuzz reference [9] or move [12] to the sentence about point estimates.","section":"§5.3.1"},{"comment":"The phrase 'without conflating those states with bytecode/interpreter mismatch' is ambiguous; clarify whether this means version mismatch between bytecode tag and interpreter or some other mismatch.","section":"§4.4"},{"comment":"The 'Translation failure' category is described as necessarily zero in RQ2 because RQ2 does not exercise decompile-compile-rerun; consider moving this explanation to the table caption or a footnote for readability.","section":"§5.2.2, Table 11"},{"comment":"The sentence 'bytecode can be consumed directly by Python runtimes, making untrusted or malformed bytecode part of the interpreter-facing attack surface' is a reasonable claim, but the sentence could note that the paper does not establish exploitability, aligning with Table 1; consider adding 'in terms of robustness' for precision.","section":"Section 1, paragraph 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodical and unusually careful in its scoping; the main risk is that the abstract and Section 6.2 state the 91.7% execution-boundary claim and the RQ4 negative result more strongly than the current evidence supports. Both can be fixed within the manuscript's scope by instrumentation or rewording. The single-campaign, no-sanitizer fuzzing design is a limitation but acceptable for an existence claim, provided the 91.7% label is corrected. I would not reject on the basis of these issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first PyPI-scale bytecode study I know of, and it deserves a serious referee. The measurement machinery is pinned, the denominators are explicit, and the authors repeatedly separate source emission from semantic equivalence and crash findings from exploitability. The central claim that source-only inspection misses executable content at scale is well supported.\n\nWhat is new: exposure counts across 1,034,843 artifacts, 28,193 artifact-local source-less .pyc files, version-aware analyzability of 204,904 files with a 99.999% L4 emission rate, 1,009 stack-deduplicated crash groups from bytecode fuzzing, and the source-reproduction negative result. RQ1 and RQ2 are the strongest parts. The L4 caveat is stated honestly, and the failure taxonomy across marshal, dis, Decompyle++, and PyLingual is genuinely useful for people building package-scanning pipelines. The pinned tool commits and structured manifests are the right level of reproducibility.\n\nWhere I would push back: the \"at least 91.7% reach execution beyond ingestion\" figure is load-bearing and not fully earned. It rests entirely on stack-prefix labels in Table 16 plus the treatment of the 59 Unknown groups. The arithmetic is conservative about Unknown, but it is not conservative about the four \"execution\" labels. A crash during code-object construction, quickening, or teardown can show frames like _PyCode_Quicken or GC/finalization without the mutated instruction stream ever reaching _PyEval_EvalFrameDefault. Without sanitizer builds or per-group phase instrumentation, the lower bound is inflated. The existence claim survives: bytecode fuzzing clearly reaches beyond marshal, and the 2.5% loading-localized share is compelling. But the honest statement is \"the large majority appear to reach execution, with the exact split unvalidated.\"\n\nTwo smaller points. RQ4's \"None reproduce from ordinary Python source\" is tool-bounded; the body says this, but the abstract should too. And one 24-hour single-worker campaign per CPython version is fine for existence, but the 1,009 count and the 91.7% ratio are point estimates; the authors mostly acknowledge this, though the abstract reads as stable. Gated triggering inputs are defensible, but reviewers will want hashes or a disclosure timeline.\n\nWho is this for: package-security scanner designers, CPython core triage, and decompiler maintainers. It is a solid empirical paper that should go to peer review with revision. I would cite the RQ1/RQ2 numbers in my own work.","headline":"A careful, genuinely new PyPI-scale measurement of Python bytecode exposure and analyzability; the RQ3 post-ingestion execution lower bound is less certain than the abstract implies.","tokens_in":31249,"tokens_out":2169,"would_cite":true,"duration_ms":23449,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Python bytecode in PyPI packages opens a measurable gap between what source-only scans inspect, what CPython executes, and what recovered source can represent.","keywords":["Python bytecode","PyPI package security","software supply chain","source-less .pyc files","decompilation analyzability","CPython fuzzing","runtime robustness","inspection-execution gap"],"falsifier":"Rebuild the seven CPython fuzzing targets with address and memory sanitizers, re-run the campaigns, and manually root-cause each of the 1,009 stack-deduplicated crash groups. If a large share of the groups now assigned to post-ingestion execution turn out to crash inside code-object construction, deallocation, or finalization before any instruction of the mutated stream executes, the at-least-91.7% execution-boundary claim is not supported. The mirror test for the source-reproduction result is simpler: a single one of the 1,009 findings that compiles from recovered decompiler source and reproduces the same behavior class under the matching CPython version would overturn the claim that none reproduce.","tokens_in":30232,"feed_emoji":"🐍","tokens_out":12304,"duration_ms":101834,"temperature":0.7,"pith_summary":"The paper tries to establish that Python bytecode is a first-class security artifact: it ships inside PyPI packages, is executed directly by CPython, and is largely invisible to source-only security inspection. Across 1,034,843 collected wheels and source distributions, the study finds 7,388 artifacts containing 228,578 .pyc files, of which 28,193 lack any matching source file within the same artifact. Version-aware decompilers emit source for 204,901 of 204,904 modern-CPython files, so the hidden bytecode is analyzable in principle, yet the analysis tools themselves hang, throw uncaught exceptions, or die by native signal, yielding 17 distinct robustness signatures. Fuzzing mutated bytecode against CPython 3.8 through 3.14 produces 1,009 stack-deduplicated crash groups, at least 91.7% of them reaching interpreter execution beyond the documented-unsafe ingestion boundary, and none of them reproducible from ordinary Python source. The paper concludes that package-security workflows must inventory bytecode directly, analyze it with version-aware tools, triage runtime findings at the bytecode level, and stop collapsing bytecode evidence into source-level vulnerability claims.","feed_headline":"28,193 hidden .pyc files slip past source-only Python scans","feed_subtitle":"A PyPI-wide study shows CPython runs that hidden bytecode and crashes in ways no recovered source reproduces.","key_machinery":"The machinery is a staged empirical pipeline held together by two distinctions. The first is the five-level analyzability ladder, L0 (not loadable) through L4 (source emitted), where L4 records only that at least one selected decompiler produced source, never that the source is semantically equivalent; the ladder lets the study separate loading, disassembly, decompilation, and faithfulness as independent failure points. The second is the ingestion-boundary split used to interpret fuzzing results: the marshal deserializer is documented as unsafe for untrusted data, so crash groups whose reproduced stacks are confined to code-object loading (25 groups) restate a known non-guarantee, while groups whose stack prefixes pass through object-runtime, frame-evaluation, garbage-collection/finalization, or instrumentation frames (925 groups, at least 91.7%) are counted as reaching post-ingestion interpreter execution. The fuzzing stage distinguishes this study from source-level campaigns: each CPython version's own unittest suite is compiled into version-correct bytecode seeds, and the serialized .pyc payloads, not the source text, are mutated and executed under the matching instrumented interpreter.","core_discovery":"The central claim is that bytecode introduces a measurable gap between what package-security workflows inspect, what Python runtimes execute, and what source-level artifacts can faithfully represent. The evidence is staged across four research questions. Bytecode exposure is an ecosystem-scale fact: 7,388 of 1,034,843 PyPI artifacts contain .pyc files, and 12.33% of all observed .pyc files are artifact-local source-less, meaning a source-only scan never learns that they need recovery in the first place. Analyzability is high but version-dependent: with CPython-version-matched environments, PyLingual emits source for 204,901 of 204,904 in-scope files, placing all but three at level L4, defined as successful decompiler emission rather than verified functional equivalence. Robustness is a separate axis: observed PyPI bytecode produces 50 managed-code failure tool results, adversarial mutated bytecode drives decompilers into native process failures, and bytecode fuzzing across every evaluated CPython release yields 1,009 stack-deduplicated crash groups dominated by pointer-dereference symptoms, with 261 groups showing potential memory-corruption characteristics. Finally, the gap is behavioral rather than merely representational: none of the 1,009 findings is reproduced when recovered source is recompiled and rerun under the matching interpreter.","pith_inferences":["The distribution-artifact snapshot likely understates bytecode exposure in the wild: installation creates __pycache__ caches locally, and container images and private indexes fall outside the PyPI denominator, so deployment-time bytecode is probably more prevalent than the 0.71% artifact-level rate measured here.","The crash-reachability result suggests a concrete hardening criterion for CPython: fail on malformed bytecode through predictable, structured exceptions rather than native terminations, with invariant checks on instruction streams, constant pools, exception tables, and quickening metadata rather than only at the marshal loader.","A malicious package could already exploit the anti-analysis primitive without writing a crasher: a .pyc that hangs or aborts a scanner's decompiler defeats source-recovery triage while remaining executable, and the paper's own RQ2 data show such failure signatures occur on unmodified PyPI bytecode today.","The paper deliberately measures source emission, not semantic equivalence; a natural extension is to decompile-compile-rerun a sample of the 204,901 level-L4 files to quantify how often emitted source preserves behavior, which would sharpen the faithfulness side of the source gap."],"forward_implications":["Package scanners should inventory .pyc files explicitly, record their bytecode versions, and track whether a matching source file exists in the same artifact: in the collected corpus, 28,193 executable .pyc files would otherwise escape source-only inspection.","Bytecode analysis needs version-aware environments and tool diversity: a single decompiler would misclassify analyzable files as opaque and hide the 50 tool-robustness failures observed on unmodified PyPI bytecode.","Malformed bytecode is a distinct CPython robustness boundary: every evaluated release from 3.8 to 3.14 is crash-reachable through mutated .pyc inputs, with 261 stack-deduplicated groups showing potential memory-corruption symptoms that warrant prioritized triage.","Bytecode-level findings should be triaged separately from source-level vulnerabilities: none of the 1,009 crash groups reproduced from recovered source, so collapsing bytecode evidence into source-level claims would misattribute the behavior.","Decompilers and disassemblers that process untrusted bytecode as data are part of the attack surface: adversarial .pyc inputs drove the evaluated tools into native-signal terminations through a pure data channel, giving an attacker an availability or anti-analysis primitive without any code-execution capability."],"supporting_citations":[{"why":"Defines the documented-unsafe ingestion boundary: marshal is stated as not secure against malicious data, the reference point for the post-ingestion execution split.","marker":"[29]"},{"why":"Supplies the code-object and bytecode-format background and the reference disassembly path used for the L2 level and the dis robustness failures.","marker":"[27]"},{"why":"One of the two decompilers whose source emission defines L3-L4 analyzability and whose RQ4 failures bound the source-reproduction result.","marker":"[4]"},{"why":"The decompiler that emits source for 204,901 of 204,904 in-scope files, carrying the RQ2 L4 result.","marker":"[20]"},{"why":"Prior Python runtime fuzzing that exercises the source-to-runtime path; the bytecode-seed design is explicitly contrasted with it as a distinct input surface.","marker":"[16]"},{"why":"Supplies the coverage-guided greybox fuzzing methodology underlying the bytecode-mutation campaigns.","marker":"[2]"},{"why":"The coverage-guided native fuzzer used to run the seven version-matched campaigns and collect crash and timeout inputs.","marker":"[9]"},{"why":"Fuzzing evaluation study that supports scoping the RQ3 counts as lower-bound existence evidence rather than stable rates.","marker":"[12]"},{"why":"Decompiler study showing source recovery can fail syntactically and semantically, motivating the separation of emission from functional equivalence behind L4.","marker":"[11]"}],"fun_headline_variants":["28,193 source-less .pyc files evade source-only Python checks","Bytecode fuzzing: 1,009 crashes Python source never reproduces","261 bytecode crash groups hint at memory-corruption risks","Source-only Python security ignores a bytecode execution gap","Python bytecode: source scans miss what CPython actually runs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise behind the claim that at least 91.7% of crash groups reach interpreter execution beyond the unsafe-ingestion boundary is that a crash group's reproduced stack prefix reliably shows where the interpreter was when it died; if many groups attributed to object-runtime, frame-evaluation, garbage-collection, or instrumentation contexts actually crash during code-object construction, deallocation, or interpreter shutdown, before the mutated instruction stream begins executing, that lower bound is inflated.","fun_headline_variants_meta":{"raw":{"variants":["28,193 source-less .pyc files evade source-only Python checks","Bytecode fuzzing: 1,009 crashes Python source never reproduces","261 bytecode crash groups hint at memory-corruption risks","Source-only Python security ignores a bytecode execution gap","Python bytecode: source scans miss what CPython actually runs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001395,"raw_usage":{"total_tokens":5727,"prompt_tokens":1113,"completion_tokens":4614,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":729,"completion_tokens_details":{"reasoning_tokens":4525}},"tokens_in":729,"tokens_out":4614,"duration_ms":33847,"temperature":1.0,"reasoning_tokens":4525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:56:29.121114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the seven CPython fuzzing targets with address and memory sanitizers, re-run the campaigns, and manually root-cause each of the 1,009 stack-deduplicated crash groups. If a large share of the groups now assigned to post-ingestion execution turn out to crash inside code-object construction, deallocation, or finalization before any instruction of the mutated stream executes, the at-least-91.7% execution-boundary claim is not supported. The mirror test for the source-reproduction result is simpler: a single one of the 1,009 findings that compiles from recovered decompiler source and reproduces the same behavior class under the matching CPython version would overturn the claim that none reproduce.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the documented-unsafe ingestion boundary: marshal is stated as not secure against malicious data, the reference point for the post-ingestion execution split."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the code-object and bytecode-format background and the reference disassembly path used for the L2 level and the dis robustness failures."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the two decompilers whose source emission defines L3-L4 analyzability and whose RQ4 failures bound the source-reproduction result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The decompiler that emits source for 204,901 of 204,904 in-scope files, carrying the RQ2 L4 result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior Python runtime fuzzing that exercises the source-to-runtime path; the bytecode-seed design is explicitly contrasted with it as a distinct input surface."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the coverage-guided greybox fuzzing methodology underlying the bytecode-mutation campaigns."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The coverage-guided native fuzzer used to run the seven version-matched campaigns and collect crash and timeout inputs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Decompiler study showing source recovery can fail syntactically and semantically, motivating the separation of emission from functional equivalence behind L4."}],"review_version":1}