{"id":"17381a5d-4301-4aec-a91e-e5299978a4c4","arxiv_id":"2608.10115","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Outer Limits lets independent researchers run matched content experiments inside the existing Old Reddit interface without publishing experimental content or write actions to Reddit.","lead":"Researchers built Outer Limits, a browser extension that swaps researcher-written posts and comments into real Reddit pages while keeping that content and participant actions from ever reaching Reddit. A 219-person fidelity study found the swapped-in posts were rated about as realistic as genuine posts within the tested bounds.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Containment of write interactions is asserted but not verified; plausible bypasses via keyboard shortcuts or DOM changes could let participant actions reach Reddit, undermining the central ethical boundary.","rationale":"The reader's weakest_assumption identifies exactly the concern that I find most load-bearing: the containment guarantee is the technical and ethical core of the contribution, and it is asserted rather than demonstrated. My independent reading of Sec. 4.3, Sec. 7.3, and Sec. 7.4 supports this. The paper's own limitations sections acknowledge that researchers must 'document and test which write interactions are intercepted' and that DOM changes require updates, which is tantamount to admitting that such testing was not provided and that the interception is fragile. I further specify a concrete, currently unaddressed bypass: keyboard shortcuts like 'a'/'z' for voting and Ctrl+Enter for comment submission are not mentioned in the containment description, and Reddit's own keydown handlers would plausibly process these events unless explicitly blocked. This is a testable, concrete weakness in the central claim. Because this concern is about missing verification rather than an identified failure, the appropriate outcome is conditional acceptance pending evidence, not rejection. The reader's CONDITIONAL verdict is therefore unchanged: the paper is a promising methodological contribution whose core containment claim needs independent verification before the system is used in consequential studies. I do not see an internal inconsistency that would warrant REJECT or UNVERDICTED, and the lack of machine-checked proofs or shipped reproducible code further supports the conditional stance.","tokens_in":8802,"tokens_out":3967,"duration_ms":42119,"concrete_test":"Run an automated browser session on Old Reddit with the Outer Limits extension loaded, capturing all network requests via Chrome DevTools Protocol or a local proxy. Perform each configured write action through every available UI path: (1) mouse click on upvote and downvote; (2) keyboard shortcuts 'a' and 'z'; (3) comment submission via Ctrl+Enter; (4) comment submission via the on-screen comment button; (5) comment edit via the edit button; (6) programmatic .click() and form submission as used by autofill or assistive technologies. After each action, assert that no request to reddit.com carries a vote payload, comment text, or any constructed content. If any reddit.com request appears with such a payload, containment is incomplete and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution (Sec. 7.1) and ethical justification (Sec. 7.3) depend on the claim that configured write interactions never reach Reddit. Sec. 4.3 describes intercepting vote clicks before Reddit's handler and capturing comment submissions without generating write requests, but provides no network-level trace, automated test suite, or code-level verification. Crucially, Old Reddit supports keyboard shortcuts: pressing 'a' or 'z' to vote and Ctrl+Enter to submit a comment. These actions are not described as intercepted, and a global keydown listener in Reddit's own code could receive them before the extension's capture-phase listener, causing a real vote or comment. Similarly, comment edits, programmatic form submissions, and browser autofill may bypass the described click-based interception. The paper itself flags this gap: Sec. 7.3 instructs future researchers to 'document and test which write interactions are intercepted,' and Sec. 7.4 concedes that DOM changes to Reddit 'can require updates and renewed fidelity assessment,' implying no such testing was provided. Without evidence that every write path (including keyboard shortcuts and non-click UI paths) is blocked in every page state, a single bypass would let participant actions reach Reddit and violate the containment promise that distinguishes this approach from a field experiment and underpins its ethical boundary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Outer Limits, a Chrome-extension-based experimental platform that renders researcher-specified content inside the existing Old Reddit interface while claiming to contain all participant write interactions (votes, comment submissions, edits) locally. The authors report a 219-participant perceptual-fidelity study using ART ANOVAs and exploratory TOSTs with d=±0.50 bounds, and illustrate the system with a factorial case study. The central contribution is the claimed combination of precise content control, an existing platform interface, and containment of experimental content and actions.","tokens_in":9018,"tokens_out":4252,"duration_ms":39088,"significance":"If the containment claim is substantiated, this is a meaningful methodological advance for independent social-media researchers: it enables matched, randomized content comparisons with realistic interface context without requiring platform cooperation or risking spillover to host communities. The paper is commendably transparent about its limitations (e.g., narrow validation materials, post hoc equivalence bounds), and the decision to release the software open-source is a strength. The validation design is reasonable for its purpose, with an a priori power analysis and a 2x2 factorial structure. However, the missing verification of write-interaction containment is a serious gap that must be addressed before the central claim can be accepted.","major_comments":[{"comment":"The containment guarantee is not verified for all input paths. Section 4.3 describes intercepting vote clicks and capturing comment submissions, but Old Reddit also supports keyboard shortcuts (e.g., 'a'/'z' to vote, Ctrl+Enter to submit a comment), and other mechanisms such as programmatic form submission or autofill could bypass the described click-based handlers. The paper itself concedes in §7.3 that researchers should 'document and test which write interactions are intercepted' and in §7.4 that DOM changes 'can require updates and renewed fidelity assessment,' which indicates that no such testing is currently provided. Because the claim that no configured write action reaches Reddit is the basis for the ethical boundary and for distinguishing this approach from a field experiment, the authors need to provide network-level evidence (e.g., request traces) or automated tests covering all input paths, or explicitly disable these paths (e.g., prevent default on keyboard shortcuts) and narrow the claim accordingly.","section":"§4.3, §7.3, §7.4"},{"comment":"The equivalence conclusion relies on bounds (d=±0.50) that were selected after data collection, as the paper acknowledges in §7.4. Although the TOSTs are labelled exploratory, the abstract and §5.1.1 present them as supporting statistical equivalence 'within these bounds.' Post hoc selection of equivalence margins weakens the inferential strength of the claim; the authors should pre-specify the margin for future studies or report a sensitivity analysis across a range of bounds. As it stands, the validation supports only a permissive equivalence claim.","section":"§5.1.1, §7.4"},{"comment":"The rule by which six participants were excluded for failing content-recall checks is not described: the paper does not specify the checks, the passing threshold, or whether exclusion was applied blind to condition. This is a researcher degree of freedom that matters in a study whose headline results are null/equivalence findings. Please report the full exclusion protocol and, ideally, results with the six participants retained.","section":"§5.0.3, §5.1.1"}],"minor_comments":[{"comment":"The confidence item is described as rated from 'not confident at all' to 'very confident' without specifying the number of scale points; specify whether it used the same 7-point scale.","section":"§5.0.2"},{"comment":"In the manipulated condition, the paper says Outer Limits 'replaced their title, text, image, and comments'; for the text-only post, no image was replaced. Please clarify the replacement set per stimulus.","section":"§5.0.1"},{"comment":"The listing address is omitted for anonymous review, and the software repository address omission is appropriate, but please state in the camera-ready version where readers should look for the code.","section":"§4.2"},{"comment":"Use '±0.50' instead of 'plus-minus 0.50' for clarity.","section":"Abstract and §5.1.1"},{"comment":"The table caption says 'Eq. denotes...' but the meaning is clear; consider defining it in the table header instead of the caption.","section":"Table 2"},{"comment":"'Containmentis both atechnicalandan ethicalboundary' has missing spaces.","section":"§7.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim about containment is likely to attract scrutiny from reviewers and from the platform/community side; the open acknowledgment of untested write paths in §7.3 is appropriate but should be addressed before publication. The post hoc equivalence margin is a further point that should be tightened. The work is otherwise a solid methodological contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution, and it should get a serious referee. The design-space positioning is the right kind of work: it identifies a gap that no single existing tool fills. Outer Limits lets a researcher specify exact post and comment content, render it inside Old Reddit, log interactions, and claim those interactions never reach Reddit. Prior work either controls content without platform context or uses the platform without content control. That combination is new and worth having.\n\nWhat the paper does well: the validation is proportional to the claim. 219 participants, a power analysis, ART ANOVAs plus TOSTs, qualitative coding with Krippendorff's alpha, and honest reporting that the equivalence margin was chosen after data collection. The authors also disclose the scope: two posts, one subreddit, Prolific sample, Old Reddit desktop only. They do not oversell the fidelity result. The illustrative factorial study is framed as a demonstration, not a substantive result.\n\nWhere it is soft: the containment claim is load-bearing, and the paper gives architecture but no verification. A click-interception listener and capturing comment text do not show that every write path is blocked. Old Reddit supports keyboard shortcuts; programmatic form submission and autofill are other plausible paths. The paper itself admits the gap by telling future researchers to 'document and test which write interactions are intercepted.' That sentence belongs in the current paper's methods, not only in future work. I would not block publication on the suspicion that the architecture is wrong — it may well work — but I would require either a network-level trace showing no write requests to Reddit in a test session, or an explicit carve-out stating which interaction paths are verified.\n\nThe post hoc equivalence margin is a smaller issue but real. d = ±0.50 is a wide bound, and choosing it after seeing the data weakens the equivalence claim. The paper is transparent about this, and the TOSTs are exploratory, but the abstract gives them headline weight. The absence of released code and data is also a gap, though the repository is omitted for anonymous review; as a referee I would want the artifacts to check containment before any consequential study builds on this.\n\nBottom line: this paper is for the CSCW/CHI audience — people who want to run controlled experiments near real platforms without platform cooperation. It deserves peer review and likely publication after containment verification and a modest reframing of the fidelity claim. I would cite it for the design-space framing and would bring it to a reading group.","headline":"A genuinely useful methodological brick — matched content on a real platform with containment — but the containment guarantee is asserted, not demonstrated, and the fidelity claim rests on a post hoc equivalence margin.","tokens_in":9558,"tokens_out":2061,"would_cite":true,"duration_ms":19859,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Outer Limits is a browser-based system that runs matched, researcher-specified content experiments inside the real Reddit interface while ensuring no constructed post or participant action ever reaches Reddit.","keywords":["social media experiments","controlled content manipulation","browser extension","Reddit","perceptual fidelity","experimental realism","equivalence testing","containment"],"falsifier":"Open the browser's network inspector while a participant uses Outer Limits and perform every supported action, including voting, submitting a comment, editing it, and using keyboard shortcuts; any outbound request to a Reddit write endpoint means containment failed. For fidelity, a preregistered replication with posts drawn from several subreddits that finds a Post Type effect beyond the $d=\\pm0.50$ bound would falsify the generality of the perceptual-equivalence claim.","tokens_in":8591,"feed_emoji":"🧪","tokens_out":6946,"duration_ms":67015,"temperature":0.7,"pith_summary":"Outer Limits is a browser-based system for running controlled social-media experiments inside the existing Old Reddit interface. The paper claims it gives independent researchers something existing alternatives do not: researcher-specified, matched content presented in the real platform interface, with containment that keeps constructed posts and participants' votes and comments from ever reaching Reddit. A 219-participant perceptual-fidelity study found no significant differences between manipulated and authentic posts, and exploratory equivalence tests met the $d=\\pm0.50$ criterion, supporting the claim that altered posts look realistic in this setting. If the claim holds, researchers can test causal content mechanisms with platform context intact, without platform cooperation or community spillover.","feed_headline":"Reddit experiments run in-browser, never reaching Reddit","feed_subtitle":"A browser extension swaps in researcher-written posts and blocks votes and comments from leaving the page.","key_machinery":"The active component is a content-replacement and event-interception loop in a browser extension. On a target page, the extension requests the participant's assigned condition from a study backend and writes the returned title, body, image, displayed author, timestamp, and vote count into the DOM together with the comment thread; surrounding page content still loads from Reddit. For containment, the extension intercepts vote clicks before Reddit's event handler, updates the displayed state locally, and captures comment submissions as study records without issuing write requests to Reddit. The validating machinery is a $2\\times2$ between-subjects design analyzed with aligned-rank-transform ANOVAs and two-one-sided-tests equivalence bounds at $d=\\pm0.50$.","core_discovery":"The central claim is that Outer Limits combines three properties that previous approaches provide only separately: precise control over experimental content, the existing platform interface, and containment of experimental content and interactions from the host community. Concretely, the extension fetches a condition-specific set of post and comment fields from a study backend, replaces the corresponding elements in the Reddit page DOM, logs designated interactions, and intercepts votes, comment submissions, and comment edits before Reddit's own handlers run, so the interface responds but no corresponding write request reaches Reddit. The validation claim is that, in a 219-participant between-subjects study on two posts from one subreddit, ART ANOVAs found no Post Type, Participant Awareness, or interaction effects and exploratory TOSTs met the $d=\\pm0.50$ equivalence criterion for the tested contrasts and setting. The paper is explicit that this supports perceptual equivalence within these bounds, not zero difference or full experimental realism.","pith_inferences":["If containment holds at the network level, the same architecture could be audited to certify that no outbound write ever occurs, turning the ethical boundary into a testable property rather than a design intention.","The method's largest open question is whether self-reported fidelity translates into the behavioral realism needed for causal estimates; a direct comparison of the same manipulation inside Outer Limits versus a reconstructed simulation would test whether the host interface changes effect sizes.","Containment verification would need to cover edge cases the paper does not test, such as keyboard-triggered votes, browser autofill, and Reddit's DOM changing after deployment; automated regression tests on the interception layer would make this practical.","In principle the approach transfers to any DOM-modifiable platform with a stable interface, but each transfer would require renewed fidelity and ethical review because the platform's conventions and write paths differ."],"forward_implications":["Researchers can run randomized, matched conditions on live Old Reddit pages without publishing any constructed post to the community.","Participant votes and comments produce no corresponding Reddit write requests, so experimental actions stay bounded to the study backend and the participant's browser.","Studies involving sensitive content, such as vaccine-related messages, can be tested in platform context without posting that content to real communities.","Because conditions are read from the backend at runtime, factorial designs varying post frame, comment frame, and comment stance can be deployed without rebuilding the extension.","Each new deployment still requires its own fidelity assessment; the validation covers the specific tested posts, outcomes, equivalence bounds, and sample."],"supporting_citations":[{"why":"Defines the credibility and experimental-realism standard that the perceptual-fidelity validation is designed to meet.","marker":"[2]"},{"why":"Represents a reconstructed simulation that provides control but lacks the social embeddedness Outer Limits preserves.","marker":"[11]"},{"why":"Surveys simulation software and its limits, motivating the decision to retain the host interface.","marker":"[16]"},{"why":"The platform-run emotional-contagion experiment that required internal access, the contrast motivating independent intervention.","marker":"[18]"},{"why":"Supplies the two-one-sided-tests equivalence method used to assess perceptual fidelity.","marker":"[19]"},{"why":"Client-side reranking browser-extension work that operates without platform cooperation but cannot control post wording; Outer Limits extends that route.","marker":"[23,24]"},{"why":"An unauthorized live-platform experiment that motivates the containment requirement.","marker":"[25]"}],"fun_headline_variants":["In-browser Reddit experiments with zero write leakage","Rewrite Reddit locally, run experiments, block all writes","Controlled Reddit content experiments without touching the server","Reddit study system: local content, full containment","Equivalence-tested Reddit experiment tool, 219 users"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire argument rests on the assumption that every participant write action, including votes, comment submissions, and comment edits, is intercepted before Reddit's own code executes in every page state the participant can reach; the paper describes this mechanism but does not supply network-level verification or automated tests covering edge cases.","fun_headline_variants_meta":{"raw":{"variants":["In-browser Reddit experiments with zero write leakage","Rewrite Reddit locally, run experiments, block all writes","Controlled Reddit content experiments without touching the server","Reddit study system: local content, full containment","Equivalence-tested Reddit experiment tool, 219 users"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1369,"prompt_tokens":901,"completion_tokens":468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":389}},"tokens_in":517,"tokens_out":468,"duration_ms":5160,"temperature":1.0,"reasoning_tokens":389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:11:01.549534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Open the browser's network inspector while a participant uses Outer Limits and perform every supported action, including voting, submitting a comment, editing it, and using keyboard shortcuts; any outbound request to a Reddit write endpoint means containment failed. For fidelity, a preregistered replication with posts drawn from several subreddits that finds a Post Type effect beyond the $d=\\pm0.50$ bound would falsify the generality of the perceptual-equivalence claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the credibility and experimental-realism standard that the perceptual-fidelity validation is designed to meet."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"An unauthorized live-platform experiment that motivates the containment requirement."}],"review_version":1}