{"id":"f4f0b4c6-0daf-4d14-8f89-945ac5c4a790","arxiv_id":"2505.21604","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The Public Discourse Sandbox is a Django-based research platform with human-AI and AI-AI accounts, IRB-based experiment management, and open-source deployment.","lead":"This paper describes the Public Discourse Sandbox, an open-source web platform for studying conversations between humans and AI agents in a controlled, consent-based environment. Its authors position it as a safer alternative to running AI experiments on live social media, with a hosted version provided for researchers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AI accounts are described as currently limited to creating posts and hashtags, while the System Events Flow and use cases require replies, likes, and reposts; Future Work lists these as unimplemented. The central human-AI/AI-AI discourse capability is thus internally inconsistent and unverified.","rationale":"The reader's weakest assumption is that the described feature set is actually implemented and stable, which is close to this concern. I sharpen it: the paper internally contradicts itself about whether AI accounts can reply, comment, like, and repost, with Future Work explicitly deferring those actions. Since the central claim is that PDS enables human-AI and AI-AI discourse research, this capability is load-bearing. The concern is concrete and testable by code inspection or an integration test, and it does not depend on external evaluation norms. I do not recommend rejection: a system paper may legitimately describe a partially implemented platform if scoped honestly, and the contradiction could be resolved by a correction or clarification. But the current text overstates readiness, so the reader's conditional verdict stands: acceptance should require verification that threaded AI participation is actually implemented, or revision of the claims to match the currently available posting-only functionality.","tokens_in":10416,"tokens_out":6129,"duration_ms":63660,"concrete_test":"Inspect the GitHub repository (public-discourse-sandbox, current master or the state referenced by the paper) and run a minimal integration test: deploy with docker compose, create a researcher account, two internal AI accounts, and a human account; have the human create a post and a comment; wait for Celery task completion; then query the database and logs for AI-generated comments, likes, and reposts. Independently, inspect the agent task source to see which endpoints it calls: if only create-post is invoked and no comment/repost/like endpoints exist or are called, the contradiction resolves in favor of 'not implemented' and the central claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, PDS must let AI accounts participate in threaded discourse with humans and with each other. The paper's own text is contradictory on this point. Section 'AI Account Deployment' says: 'Each type of AI account has the following actions available: create posts with up to 280 characters and create hashtags.' Section 'Future Work' then explicitly lists as planned: 'like, undo like, create a comment, like a comment, repost posts, repost comments, follow other users, see the content of followed accounts, and report other users.' Yet Section 'System Events Flow' describes agents 'liking the post, reposting it under their own account, or replying to the post' as part of the current event flow, and the abstract and use cases claim human-AI and AI-AI discourse research. If the deployment section and Future Work reflect the current state, then AI accounts cannot comment, like, or repost, so the hosted version cannot support the conversational interactions that are the platform's stated purpose. If the event-flow description is accurate, the feature list is wrong. Either way, the paper does not establish that the core discourse capability exists. This is not an evaluation-quality concern only; it is an internal inconsistency about the main functionality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Public Discourse Sandbox (PDS), a Django-based web platform intended to support controlled research on human-AI and AI-AI digital discourse. It describes the platform's design, account types, research-participation and IRB-oriented consent workflows, AI account deployment via internal (hosted) and future external (API-based) mechanisms, a Docker-based technical architecture with Celery task queues, and two illustrative use cases (experiments and training). The authors provide a hosted live instance and open-source code on GitHub. The paper presents no empirical evaluation, performance measurements, security audit, or user study; several features are explicitly described as work in progress or future work.","tokens_in":10586,"tokens_out":2648,"duration_ms":29378,"significance":"If the platform delivers what is claimed, it would address a genuine and timely need: an ethical, consent-preserving alternative to in-the-wild experiments on commercial social media, with the ability to study human-AI and AI-AI interactions. The open-sourced code and hosted instance are concrete assets that lower the barrier for non-technical researchers, and the explicit integration of IRB consent and research-participation agreements is a strength. However, the paper's central claim that PDS 'serves as' a functional research platform is currently supported only by architectural description and use-case narratives, not by evidence of deployment, usability, or reliability. The internal inconsistency about AI account capabilities further weakens confidence in the current state of the system.","major_comments":[{"comment":"There is a load-bearing internal contradiction about what AI accounts can currently do. The 'AI Account Deployment' section states that 'Each type of AI account has the following actions available: create posts with up to 280 characters and create hashtags,' and the 'Future Work' section lists 'like, undo like, create a comment, like a comment, repost posts...' as planned additions. Yet the 'System Events Flow' section describes agents 'liking the post, reposting it under their own account, or replying to the post' as part of the current event flow, and the abstract and use cases claim support for human-AI and AI-AI discourse. If the deployment section and Future Work reflect the current state, then AI accounts cannot reply, like, or repost, which would prevent the threaded conversational interactions that are the platform's stated purpose. If the event-flow description is accurate, then the feature list is wrong. The paper must reconcile these statements and clarify which actions are actually implemented in the hosted version.","section":"AI Account Deployment; System Events Flow; Future Work"},{"comment":"The central claim that PDS is a usable platform for discourse research is not supported by any evaluation. There is no user study, no performance benchmark, no reliability or scalability measurement, no security audit, and no end-to-end demonstration of an experiment with human participants and AI accounts. The 'Future Work' section states 'We will test the features and design of the sandbox,' implying that testing has not yet been done. The paper should include at least a minimal evaluation of the hosted instance: a small cohort of users, latency and throughput of agent event processing, error rates, and a concrete walkthrough of an experiment scenario. Without such evidence, the claim that the platform 'can be of great use' for non-technical researchers is unverified.","section":"Implementation - Potential Use Cases; Future Work"},{"comment":"The paper repeatedly calls the platform 'safe and secure,' but the only content moderation mechanism described is a single Profanity Check library, and the only security measure described is mandatory two-factor authentication. A profanity classifier is insufficient to 'identify and remove inappropriate, harmful, or illegal content,' especially in a research context where hostile or inflammatory content may be intentionally studied. Similarly, 'secure' storage and access claims are stated without a threat model, data-security controls, or an audit. Since the ethics argument in the Introduction depends on PDS being a safe alternative to public social media, this gap is significant and should be addressed, either by describing additional moderation and security measures or by softening the claim.","section":"Public Discourse Sandbox Design; Platform Rules"},{"comment":"The description of experiment visibility is internally inconsistent. The 'Researcher Account' section says 'Currently, the sandbox only allows private experiments,' but the 'Regular User Account' section states that a user who creates an account without an invitation 'can only see the content and actions of users who are part of public experiments.' These statements cannot both be true. This inconsistency, like the one about AI account actions, makes it difficult to determine the actual behavior of the hosted platform and should be clarified.","section":"Researcher Account; Regular User Account"}],"minor_comments":[{"comment":"The reference to 'Adam Wiggins. 2017. The Twelwe-Factor App' contains a typo; it should be 'Twelve-Factor App'.","section":"References"},{"comment":"The sentence 'Reddit, have recently experience the same problem' in the Related Work section has subject-verb agreement and tense errors; it should be 'Reddit has recently experienced the same problem.'","section":"Introduction"},{"comment":"The sentence 'First, we plan on enabling users to select the type of recommendation algorithms other than the time-based, which is already included in the current version of the sandbox' is awkwardly phrased; consider rewriting for clarity.","section":"Public Discourse Sandbox Design"},{"comment":"The Figure 2 caption states 'Some features are still a work in progress, as described in the paper.' Since several described features are in fact future work, the figure should distinguish implemented features from planned ones, or the text should clearly label current capabilities.","section":"Figure 2"},{"comment":"The citation for the Profanity Check library gives 'Python Software Foundation' as the author, but the library is a third-party PyPI package; please cite the actual maintainer or repository.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper reads more like a systems/position paper than an evaluated platform paper. The core idea is reasonable and the open-source artifact is a useful contribution, but the internal contradictions about implemented AI-account actions and public/private experiments, together with the complete absence of evaluation data, prevent acceptance in a serious journal at this stage. The authors should be encouraged to reconcile the feature descriptions and provide a minimal empirical validation of the hosted platform. The lack of evaluation is perhaps more readily forgiven in a workshop or demonstration track, but the manuscript's framing as a full research contribution appears to be aimed at a journal audience."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper describes an open-source Django-based sandbox for running human-AI and AI-AI discourse experiments under IRB oversight. The motivation is real: doing this on public social media is ethically fraught and technically messy, and the authors survey the existing platforms (Truman, Deliberate Lab, OASIS, Chirper) accurately. The artifact is also genuinely open: code on GitHub, a hosted instance, and a thoughtful permissions model for researchers, moderators, and participants. For a niche community that needs a controlled Twitter-like environment, this is a plausible contribution.\n\nThe problem is a load-bearing internal inconsistency about whether AI accounts can actually do what the paper claims. In the AI Account Deployment section, the authors say AI accounts can only create posts and hashtags. Future Work then lists \"like, undo like, create a comment, repost posts, follow other users\" as planned additions. But the System Events Flow section describes agents \"liking the post, reposting it under their own account, or replying to the post\" as part of the current event flow. You cannot have all three be true. If the deployment section is accurate, the platform's core human-AI/AI-AI discourse capability does not exist yet. If the event-flow description is accurate, the feature list is wrong. Either way, the paper does not currently establish that its central claim holds.\n\nBeyond that, there is no evaluation data, no user study, no performance benchmark, and no security audit. The content moderation relies entirely on a profanity-check library, which is not robust enough to catch harmful or manipulative content that a moderation claim would imply. Several features are explicitly work in progress. This is less a system paper than a demo announcement, and the paper itself concedes as much.\n\nWho gets value from this? Researchers who want a deployment guide would be better served by the GitHub repository. The survey of related work is decent, so someone skimming the field might cite it. But I would not base my own work on the platform until the authors clarify the current AI capabilities and provide a reproducible setup with a commit hash.\n\nFor peer review: the topic deserves referee time, and the artifact is real enough to warrant a serious look, but this version needs major revision. I'd send it to a workshop or short-paper venue with a request to resolve the contradiction and show that the core interaction loop actually runs. As is, I wouldn't accept it for a full conference.","headline":"Useful sandbox idea for controlled human-AI discourse research, but the paper doesn't establish that the core AI conversation features currently work.","tokens_in":11174,"tokens_out":2815,"would_cite":false,"duration_ms":31419,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A sandbox platform gives researchers a consent-based arena for studying human and AI discourse.","keywords":["public discourse sandbox","human-AI interaction","AI agents","social media simulation","research platform","LLM bots","informed consent","open-source software"],"falsifier":"Run the claimed end-to-end workflow on the released code and hosted instance: create a private experiment, apply as researcher, invite a participant, deploy two internal AI accounts, have the human post, and check that each agent autonomously likes, reposts, or replies within a bounded time and that the full interaction exports as a dataset; if any step is missing or unstable, the central claim of a usable research platform fails.","tokens_in":10162,"feed_emoji":"💬","tokens_out":6004,"duration_ms":58745,"temperature":0.7,"pith_summary":"Social media data is hard to obtain and ethically risky to experiment on, so this paper introduces the Public Discourse Sandbox (PDS): a social-media-like web platform built specifically for research. PDS lets researchers create private, institutionally approved experiments, invite human participants who have consented, deploy AI accounts that post and reply like users, and export the resulting discourse for analysis. A hosted live version and the open-source code are released so that technical and non-technical researchers can use or adapt the platform. If the platform works as described, it would give the research community a controlled, repeatable alternative to running bot or discourse experiments in the wild. The paper is establishing the existence and design of this tool, not yet reporting results from experiments run on it.","feed_headline":"A sandbox platform runs consent-based human-AI discourse experiments","feed_subtitle":"Open-source and hosted, it lets researchers study AI behavior without exposing real social media users.","key_machinery":"The load-bearing object is the sandbox itself: a containerized web application that reproduces familiar social-media actions, such as posts up to 280 characters, likes, comments, reposts, hashtags, and follows, inside a research framework. The mechanism that gives the platform its experimental value is the event-driven agent loop. A post or reply by any human or internal AI account emits an event to every selected AI agent; each agent runs its response logic in parallel, and the response logic is determined by the researcher through a personification prompt plus a configurable LLM inference endpoint. The research workflow around it carries the ethical claim: account creation is required, two-factor authentication is mandatory, users agree to platform rules and a research participation agreement, and each experiment is private and tied to an uploaded institutional review approval.","core_discovery":"The central claim is that a working sandbox now exists where human and AI participants can interact in a controlled, consent-based social-media-like environment, and where researchers control the experiment. The authors describe the full workflow: a researcher creates a private experiment, uploads an institutional review form, defines participant roles, invites human participants, and deploys AI accounts that are configured with a persona prompt and an LLM inference endpoint. When a human user posts or replies, the event triggers selected AI agents, each of which independently decides to like, repost, or reply according to its prompt. Interaction data is stored in an isolated database per experiment and can be exported, while a profanity filter and moderation roles enforce content rules. The authors also state the intended uses: studying how AI personas affect discourse, training moderators to spot problematic behavior, and developing digital twins before deployment.","pith_inferences":["If the sandbox becomes widely adopted, different labs could run the same standardized scenarios on the same codebase, making discourse experiments more directly comparable than studies run on different platforms.","The platform's consent-by-design model could serve as a template for opt-in experiment modes on real social networks, where users grant permission for research participation instead of being unknowingly enrolled.","A testable extension would be to deploy the same LLM persona in the sandbox and on a public platform to measure how much of bot behavior is shaped by platform affordances rather than the model itself.","The deliberate absence of recommendation algorithms means early results about discourse dynamics may not transfer directly to algorithmically curated feeds; adding selectable ranking algorithms would test which effects depend on feed curation."],"forward_implications":["Researchers can study AI influence on discourse without exposing unaware social media users, because every human participant is invited and consented.","AI behavior can be varied systematically through prompts, retrieval-augmented generation, and fine-tuning, making it possible to compare how different AI personas affect conversation dynamics.","Because each experiment's data is isolated and exportable, labs can produce datasets with clear provenance for reuse and secondary analysis.","The hosted version lowers the barrier for researchers without deep technical expertise, while the open-source release lets others modify the platform for different designs.","Future external AI accounts and a public API would let researchers connect their own advanced bots and would move the platform closer to realistic social-media conditions."],"supporting_citations":[{"why":"Supplies the precedent of a mock social-media platform built for controlled experiments; PDS extends that model to AI participants.","marker":"DiFranzo and Bazarova 2018"},{"why":"Shows LLM-driven agents can exhibit believable social behavior in a simulated environment, motivating AI-account deployment in PDS.","marker":"Park et al. (2023)"},{"why":"Supplies a comparison platform for running human and LLM discussion experiments online.","marker":"People+AI Research (PAIR) Initiative 2024"},{"why":"Demonstrates LLM-agent social simulation at large scale, informing the sandbox's goal of scalable, customizable AI interactions.","marker":"Yang et al. 2024"},{"why":"Grounds the need for research-grade controls by showing an AI-AI platform that lacks research approval, consent, and open-source code.","marker":"Chirper 2025"},{"why":"Provides the direct predecessor framework for human and LLM-bot communication that PDS is designed to improve upon.","marker":"Radivojevic, Clark, and Brenner 2024"}],"fun_headline_variants":["Consent-based sandbox tests human-AI discourse","Controlled sandbox simulates social media for AI studies","Open-source platform for human-AI conversation experiments","PDS: a safe space to study AI and human discourse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the platform's described features are actually implemented and stable enough for real experiments, since the paper itself flags work-in-progress features and lists external AI accounts, recommendation algorithms, and a public API as future work.","fun_headline_variants_meta":{"raw":{"variants":["Consent-based sandbox tests human-AI discourse","Controlled sandbox simulates social media for AI studies","Open-source platform for human-AI conversation experiments","PDS: a safe space to study AI and human discourse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1236,"prompt_tokens":924,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":247}},"tokens_in":540,"tokens_out":312,"duration_ms":4031,"temperature":1.0,"reasoning_tokens":247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:25:45.034243+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the claimed end-to-end workflow on the released code and hosted instance: create a private experiment, apply as researcher, invite a participant, deploy two internal AI accounts, have the human post, and check that each agent autonomously likes, reposts, or replies within a bounded time and that the full interaction exports as a dataset; if any step is missing or unstable, the central claim of a usable research platform fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the precedent of a mock social-media platform built for controlled experiments; PDS extends that model to AI participants."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a comparison platform for running human and LLM discussion experiments online."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Grounds the need for research-grade controls by showing an AI-AI platform that lacks research approval, consent, and open-source code."}],"review_version":1}