{"id":"144b6ca8-c4b3-45c5-b671-3377ccf0d9d3","arxiv_id":"2611.16800","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A code-level audit of X's open-source recommender pipeline finds 13 transparency dead-ends and shows that ranking and filtering components are in the least inspectable tiers, worsened by Grok LLM integration.","lead":"This paper proposes a code-level audit methodology for recommender systems and applies it to X's open-source repository, identifying 13 transparency dead-ends where the public code depends on missing models, services, or configuration. The study shows that the components deciding which posts are filtered and ranked fall into the two least inspectable tiers, a pattern intensified by the 2025 integration of the proprietary Grok language model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The reachability check assumes the registered pipelines match production, but the paper already acknowledges this limitation (b215); the more load-bearing concern is that the dead-ends inventory conflates 'absent from the release' with 'absent from the production system' without independent…","rationale":"The reader's weakest_assumption correctly identifies that the reachability check may not match production (b215). However, the paper explicitly acknowledges this limitation (Section 8). The more subtle concern is that the dead-ends inventory conflates missing implementations with missing parameters/models, and the paper's argument that the release is insufficient for auditing depends on treating all 13 as equivalent. The paper's own classification (Class II, III, IV) already distinguishes these, but the headline count and the discussion in Section 6 treat them uniformly. This is not a fatal flaw—the paper is transparent about what each dead-end means—but it is a load-bearing concern because the central claim (that the release has 13 transparency dead-ends on the live serving path) could be misinterpreted as 13 points where auditing is impossible, when in fact some are architectural separations that are standard in distributed systems. The concrete test would clarify the severity distribution. The verdict should remain UNCHANGED because the paper's methodology and findings are sound; the concern is about framing and interpretation, not correctness. The reader's verdict of ACCEPT is appropriate.","tokens_in":25316,"tokens_out":850,"duration_ms":6949,"concrete_test":"For each of the 13 dead-ends in Table 3, determine whether the missing artifact is (i) an implementation that could be replaced without changing the interface (e.g., the evergreen table producer), or (ii) a parameter or model whose absence makes the component's behavior indeterminate (e.g., ranking weights). If the majority of dead-ends fall into category (i), then the headline claim that the release 'does not show what X's recommender does' is overstated, because the interface alone may suffice for many audit questions. Re-classify each dead-end as 'interface sufficient' or 'behavior indeterminate' and recompute the count of critical gaps.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical contribution is the dead-ends inventory (Table 3), which lists 13 components that 'the public code depends on but the release does not contain.' The methodology identifies these by static reachability from registered pipelines. However, the paper cannot verify that these pipelines are the ones X actually deploys (b215). More critically, even if the pipelines match, the dead-ends are defined as dependencies that are absent from the release, not as dependencies that are absent from the production system. For example, the 'Gold, Gray, and Legacy verification flags' are listed as Class IV dead-ends because the upstream producer is not in the release, but the code shows these flags are copied from upstream service responses (C.1.2). The release does not contain the producer, but the flags themselves are present and used. The dead-ends inventory thus conflates two distinct gaps: (a) components whose implementation is missing but whose interface is present (Class III), and (b) components that are referenced by name only with no interface (Class IV). The paper's classification scheme already distinguishes these, but the headline count of '13 transparency dead-ends' treats them uniformly as gaps. The load-bearing question is whether the absence of the implementation (e.g., the producer of the evergreen table) is a transparency dead-end in the same sense as the absence of the ranking model weights. The paper's argument that the release 'does not show what X's recommender does' depends on treating all 13 as equivalent failures, but some are architectural separations of concerns (interface vs. implementation) that are standard in distributed systems, not necessarily transparency failures. The concrete test would be to check whether the missing artifacts are truly opaque or whether the interface alone suffices for meaningful auditing in each case.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes a code-level audit methodology for recommender systems, applies it to X's 2025 open-source repository, and identifies 13 transparency dead-ends where the public code depends on components not included in the release. The methodology involves a backwards-walk traversal from the user-facing timeline, a reachability check to separate live from orphaned code, and a four-tier inspectability classification. The authors find that the components deciding which posts are filtered and ranked fall into the two least inspectable tiers (Class III Interface only and Class IV Opaque), a pattern intensified by the integration of the proprietary Grok LLM into moderation, annotation, and ranking. They also identify 36 parameter references in orphaned 2023 code that would appear as dead-ends without the reachability check. The paper discusses implications for DSA compliance, systemic risk auditing, and the need for additional artifacts (model cards, parameter dumps, reachability manifests) for meaningful auditing.","tokens_in":25699,"tokens_out":2308,"duration_ms":15970,"significance":"If the methodology and findings hold, this paper makes a significant contribution by providing a systematic, generalizable method for code-level auditing of recommender systems, a domain where such audits are extremely rare. The empirical application to X's repository is timely given the DSA's transparency requirements and the industry trend toward hybrid code/LLM recommender architectures. The dead-ends inventory provides concrete evidence of transparency gaps that can inform regulatory data access requests. The paper also contributes a conceptual framework (four-tier inspectability classification) that could be adopted by other auditors and regulators. The methodology is designed to be platform-agnostic and applicable under DSA data access provisions, increasing its potential impact.","major_comments":[{"comment":"The central empirical claim is the identification of 13 'transparency dead-ends on the live serving path' (Table 3). However, the methodology's reachability check (§3.2, step 3) determines which code is on the live serving path by following pipelines registered in the repository. As the authors acknowledge (§8, b215), this is a static analysis that may not match what X actually deploys. The paper does not provide independent verification that the registered pipelines correspond to production pipelines. This is load-bearing because the classification of components as dead-ends depends on the premise that the code paths traced are actually served. Without corroborating evidence (e.g., from network observations, documentation, or regulatory disclosures), the inventory conflates 'absent from the release' with 'absent from the production system'. The authors should either provide additional evidence that the registered pipelines match production, or reframe the dead-ends as 'dead-ends in the release relative to the registered pipelines' and discuss the implications of this gap more explicitly.","section":"§3.2, §6, Table 3","authors_acknowledge":"Our reachability check is static as well: it follows the pipelines that the release registers, which need not match what X deploys.","authors_acknowledge_p":0.95},{"comment":"The dead-ends inventory (Table 3) lists 'Gold, Gray, and Legacy verification flags' as Class IV dead-ends because the upstream producer is not in the release. However, as noted in §C.1.2, the code shows these flags are copied from upstream service responses. The flags themselves are present and used in the code; what is missing is the producer of those flags. This is a different kind of gap from, say, the missing ranking model weights (Navi, Phoenix). The paper's classification scheme distinguishes Class III (interface present, implementation missing) from Class IV (referenced by name only), but the headline count of '13 transparency dead-ends' treats them uniformly. The authors should clarify whether the count conflates different types of gaps and whether the practical audit implications differ between a missing implementation (Class III) and a missing producer of a value that is otherwise present and used (Class IV for the flags). This does not undermine the overall finding but affects how the inventory should be interpreted.","section":"§6, Table 3, §C.1.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The discussion of the political inclination label (§5.1.1) is careful to note that the code shows it is 'used only for metrics tracking' and does not affect recommendations. However, the paper then raises concerns about GDPR and DSA implications based on the possibility that logged counts could feed advertising, profiling, or training of unreleased models. While the authors frame this as an open question requiring further artifacts, the presentation could be read as implying that the label is more consequential than the code evidence supports. The authors should more sharply distinguish between what the code shows (the label is not used for ranking) and what cannot be ruled out (downstream use of logs). The current framing is balanced but could be tightened to avoid giving the impression that the code itself reveals a privacy violation.","section":"§5.1.1, §C.4.3, §C.9.1","authors_acknowledge":"The release supports the claim that the label does not act on ranking, but whether the logged counts feed advertising, profiling, or the training of unreleased models cannot be established from the code.","authors_acknowledge_p":0.9},{"comment":"The limitations section is admirably thorough, but several of the acknowledged limitations are so fundamental that they call into question the strength of the central claims. Specifically: (1) the reachability check may not match production (b215); (2) complete coverage is unrealistic (b216); (3) the repository is a snapshot that may not reflect current production (b217-b218); (4) a newer repository exists but was not analyzed (b219). These limitations collectively mean that the paper's findings are about a specific commit of a repository that may not correspond to what X currently runs. The authors should consider whether the conclusions need to be tempered to reflect that the audit is of the release, not of the production system. The current framing (e.g., '13 transparency dead-ends on the live serving path') implies a stronger connection to production than the limitations warrant. I recommend rewording to consistently refer to 'dead-ends in the released code relative to the registered pipelines' and adding a sentence in the conclusion that explicitly restates this scope.","section":"§8, b215, b216, b217, b218, b219","authors_acknowledge":"Our reachability check is static as well: it follows the pipelines that the release registers, which need not match what X deploys.","authors_acknowledge_p":0.93}],"minor_comments":[{"comment":"The abstract states 'We further show that the release keeps 2023 code that the 2025 pipeline no longer reaches, with 36 parameter references that would pass for dead-ends without a reachability check.' This is a nice finding, but the number 36 appears only in the abstract and conclusion; it would be helpful to have a table or explicit list in §6.2 showing which parameters are orphaned.","section":"§1, abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The four-tier inspectability classification is a useful contribution, but the boundary between Class II and Class III could be clarified with additional examples. For instance, the 'Grok annotations' are listed as Class II/III in Table 2 because the logic is visible but the model is external. It would help to have a decision rule for when a component is Class II vs. Class III (e.g., is the presence of a default value the distinguishing factor?).","section":"§3.3, Table 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 4 lists the 13 Grok annotation features. It would be helpful to indicate which of these are used by the ranking model (as inputs) versus only by filters or logging. The text mentions that the political inclination label is metrics-only, but the table does not distinguish usage.","section":"§C.4.1, Table 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The discussion of monetized amplification is informative, but the claim that 'the explicit 2023 boosts show that the paid badge was wired to reach' could be misinterpreted. The 2023 boosts were for Blue-verified authors, which at that time included both legacy verified and paid subscribers. The paper should clarify that the 2023 code does not distinguish between paid and legacy verification for the boost, so the boost applied to all verified accounts, not just paid ones.","section":"§5.1.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The discussion of required artifacts (model cards, parameter dumps, reachability manifest) is valuable. However, the paper could more explicitly connect each artifact to the specific dead-end it would resolve. Table 3 already does this partially, but a cross-reference in §7 would strengthen the argument.","section":"§7, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper states that code audits are rare and cites only one study (Weber and Kosterich 2017). While this is accurate for recommender systems, there is a growing literature on code-level audits of AI systems more broadly (e.g., audits of facial recognition, hiring algorithms). The authors might consider citing a few examples from adjacent domains to contextualize the novelty of their approach.","section":"§2.4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The appendix reference for orphaned pipelines is thorough, but the paper would benefit from a diagram or table summarizing which pipelines are live vs. orphaned in the 2025 release. Figure 1 shows the high-level architecture but does not indicate which components are orphaned.","section":"§C.9.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The sentence labeled b254 ('Black-Box Access is Insufficient for Rigorous AI Audits') appears to be a citation title rather than a claim of the paper. The paper should ensure that all citations are properly formatted and that quoted titles are clearly attributed.","section":"§1, b254","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper notes that X published a separate repository in January 2026 that was not analyzed. Given that the paper is submitted in November 2026, this is a significant limitation. The authors should explain why the newer repository was not incorporated, even as a comparison point. The current justification (analysis began before the new release) is reasonable but could be expanded.","section":"§8, b219","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The reference to 'screenshot engagement action definition' is interesting but the connection to the main argument is unclear. Consider moving to a footnote or removing if not essential.","section":"§C.6.5","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper does something genuinely useful: it proposes a backwards-walk code audit methodology with a four-tier inspectability classification, applies it to X's 2025 open-source repository, and produces a concrete inventory of 13 transparency dead-ends. The methodology is clearly described, the code claims are pinned to a specific commit and archived in Software Heritage, and the authors are upfront about limitations. The finding that the components deciding what users see fall into the two least inspectable tiers (Class III Interface only, Class IV Opaque) is well-supported and important, especially given the Grok integration. The separation of live from orphaned code (36 parameter references in unreachable 2023 pipelines) is a nice methodological contribution that prevents false positives.\n\nThe soft spots are mostly acknowledged by the authors. The load-bearing assumption is that the registered pipelines match production (b215). The stress-test note raises a fair point: the dead-ends inventory treats 'absent from the release' as equivalent to 'absent from the production system,' but some dead-ends are architectural separations of concern (interface vs. implementation) that are standard in distributed systems. For example, the Gold/Gray/Legacy verification flags are listed as Class IV dead-ends because the upstream producer is not in the release, but the flags themselves are present and used. The paper's classification already distinguishes Class III from Class IV, but the headline count of 13 dead-ends lumps them together. This is a nuance, not a flaw—the authors are transparent about what each dead-end means—but it means the inventory should be read as a map of where provenance breaks down in the release, not necessarily as 13 equivalent transparency failures. The paper's argument does not depend on treating all 13 as equivalent; it depends on the pattern that the most consequential components (ranking models, Grok annotations, evergreen selection) are in the least inspectable tiers, and that pattern holds.\n\nWho this is for: researchers and regulators working on DSA compliance, algorithm auditing methodology, and platform transparency. It deserves a serious referee. The methodology is generalizable, the empirical contribution is solid, and the limitations are honestly stated. I would accept peer review and would cite this if I were working on code-level auditing or DSA transparency.\n\nRecommendation: Accept with minor revisions. The authors should add a sentence or two in Section 6 clarifying that the dead-ends inventory includes both missing implementations and missing interfaces, and that the audit significance varies by case. The stress-test concern is already partially addressed by the classification scheme, but making it explicit would strengthen the paper.","headline":"A well-executed code audit that delivers a clear methodology and a concrete dead-ends inventory; the central finding is real but the stress-test concern about conflating interface/implementation gaps is a fair nuance, not a flaw.","tokens_in":26274,"tokens_out":631,"duration_ms":7736},"pith_extraction":null,"created_at":"2026-11-13T12:27:42.595441+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}