Pith. sign in

REVIEW 4 major objections 5 minor 42 references

Public Discourse Sandbox: Facilitating Human and AI Digital Communication Research

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A sandbox platform gives researchers a consent-based arena for studying human and AI discourse.

desk verdict Useful sandbox idea for controlled human-AI discourse research, but the paper doesn't establish that the core AI conversation features currently work. read the letter →

arxiv 2505.21604 v1 pith:YPGE6NOV submitted 2025-05-27 cs.CY cs.AIcs.HC

Kristina Radivojevic , Caleb Reinking , Shaun Whitfield , Paul Brenner This is my paper
classification cs.CYcs.AIcs.HC
keywords publicdiscoursesandboxhuman-AIinteractionAIagentssocialmediasimulationresearchplatformLLMbotsinformedconsentopen-sourcesoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Social media data is hard to obtain and ethically risky to experiment on, so this paper introduces the Public Discourse Sandbox (PDS): a social-media-like web platform built specifically for research. PDS lets researchers create private, institutionally approved experiments, invite human participants who have consented, deploy AI accounts that post and reply like users, and export the resulting discourse for analysis. A hosted live version and the open-source code are released so that technical and non-technical researchers can use or adapt the platform. If the platform works as described, it would give the research community a controlled, repeatable alternative to running bot or discourse experiments in the wild. The paper is establishing the existence and design of this tool, not yet reporting results from experiments run on it.

What carries the argument

The load-bearing object is the sandbox itself: a containerized web application that reproduces familiar social-media actions, such as posts up to 280 characters, likes, comments, reposts, hashtags, and follows, inside a research framework. The mechanism that gives the platform its experimental value is the event-driven agent loop. A post or reply by any human or internal AI account emits an event to every selected AI agent; each agent runs its response logic in parallel, and the response logic is determined by the researcher through a personification prompt plus a configurable LLM inference endpoint. The research workflow around it carries the ethical claim: account creation is required, two-factor authentication is mandatory, users agree to platform rules and a research participation agreement, and each experiment is private and tied to an uploaded institutional review approval.

What would settle it

Run the claimed end-to-end workflow on the released code and hosted instance: create a private experiment, apply as researcher, invite a participant, deploy two internal AI accounts, have the human post, and check that each agent autonomously likes, reposts, or replies within a bounded time and that the full interaction exports as a dataset; if any step is missing or unstable, the central claim of a usable research platform fails.

Watch

Extended reading notes

Core claim

The central claim is that a working sandbox now exists where human and AI participants can interact in a controlled, consent-based social-media-like environment, and where researchers control the experiment. The authors describe the full workflow: a researcher creates a private experiment, uploads an institutional review form, defines participant roles, invites human participants, and deploys AI accounts that are configured with a persona prompt and an LLM inference endpoint. When a human user posts or replies, the event triggers selected AI agents, each of which independently decides to like, repost, or reply according to its prompt. Interaction data is stored in an isolated database per experiment and can be exported, while a profanity filter and moderation roles enforce content rules. The authors also state the intended uses: studying how AI personas affect discourse, training moderators to spot problematic behavior, and developing digital twins before deployment.

Load-bearing premise

The load-bearing premise is that the platform's described features are actually implemented and stable enough for real experiments, since the paper itself flags work-in-progress features and lists external AI accounts, recommendation algorithms, and a public API as future work.

Editorial extensions

If this is right

  • Researchers can study AI influence on discourse without exposing unaware social media users, because every human participant is invited and consented.
  • AI behavior can be varied systematically through prompts, retrieval-augmented generation, and fine-tuning, making it possible to compare how different AI personas affect conversation dynamics.
  • Because each experiment's data is isolated and exportable, labs can produce datasets with clear provenance for reuse and secondary analysis.
  • The hosted version lowers the barrier for researchers without deep technical expertise, while the open-source release lets others modify the platform for different designs.
  • Future external AI accounts and a public API would let researchers connect their own advanced bots and would move the platform closer to realistic social-media conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sandbox becomes widely adopted, different labs could run the same standardized scenarios on the same codebase, making discourse experiments more directly comparable than studies run on different platforms.
  • The platform's consent-by-design model could serve as a template for opt-in experiment modes on real social networks, where users grant permission for research participation instead of being unknowingly enrolled.
  • A testable extension would be to deploy the same LLM persona in the sandbox and on a public platform to measure how much of bot behavior is shaped by platform affordances rather than the model itself.
  • The deliberate absence of recommendation algorithms means early results about discourse dynamics may not transfer directly to algorithmically curated feeds; adding selectable ranking algorithms would test which effects depend on feed curation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the Public Discourse Sandbox (PDS), a Django-based web platform intended to support controlled research on human-AI and AI-AI digital discourse. It describes the platform's design, account types, research-participation and IRB-oriented consent workflows, AI account deployment via internal (hosted) and future external (API-based) mechanisms, a Docker-based technical architecture with Celery task queues, and two illustrative use cases (experiments and training). The authors provide a hosted live instance and open-source code on GitHub. The paper presents no empirical evaluation, performance measurements, security audit, or user study; several features are explicitly described as work in progress or future work.

Significance. If the platform delivers what is claimed, it would address a genuine and timely need: an ethical, consent-preserving alternative to in-the-wild experiments on commercial social media, with the ability to study human-AI and AI-AI interactions. The open-sourced code and hosted instance are concrete assets that lower the barrier for non-technical researchers, and the explicit integration of IRB consent and research-participation agreements is a strength. However, the paper's central claim that PDS 'serves as' a functional research platform is currently supported only by architectural description and use-case narratives, not by evidence of deployment, usability, or reliability. The internal inconsistency about AI account capabilities further weakens confidence in the current state of the system.

major comments (4)
  1. [AI Account Deployment; System Events Flow; Future Work] There is a load-bearing internal contradiction about what AI accounts can currently do. The 'AI Account Deployment' section states that 'Each type of AI account has the following actions available: create posts with up to 280 characters and create hashtags,' and the 'Future Work' section lists 'like, undo like, create a comment, like a comment, repost posts...' as planned additions. Yet the 'System Events Flow' section describes agents 'liking the post, reposting it under their own account, or replying to the post' as part of the current event flow, and the abstract and use cases claim support for human-AI and AI-AI discourse. If the deployment section and Future Work reflect the current state, then AI accounts cannot reply, like, or repost, which would prevent the threaded conversational interactions that are the platform's stated purpose. If the event-flow description is accurate, then the feature list is wrong. The paper must reconcile these statements and clarify which actions are actually implemented in the hosted version.
  2. [Implementation - Potential Use Cases; Future Work] The central claim that PDS is a usable platform for discourse research is not supported by any evaluation. There is no user study, no performance benchmark, no reliability or scalability measurement, no security audit, and no end-to-end demonstration of an experiment with human participants and AI accounts. The 'Future Work' section states 'We will test the features and design of the sandbox,' implying that testing has not yet been done. The paper should include at least a minimal evaluation of the hosted instance: a small cohort of users, latency and throughput of agent event processing, error rates, and a concrete walkthrough of an experiment scenario. Without such evidence, the claim that the platform 'can be of great use' for non-technical researchers is unverified.
  3. [Public Discourse Sandbox Design; Platform Rules] The paper repeatedly calls the platform 'safe and secure,' but the only content moderation mechanism described is a single Profanity Check library, and the only security measure described is mandatory two-factor authentication. A profanity classifier is insufficient to 'identify and remove inappropriate, harmful, or illegal content,' especially in a research context where hostile or inflammatory content may be intentionally studied. Similarly, 'secure' storage and access claims are stated without a threat model, data-security controls, or an audit. Since the ethics argument in the Introduction depends on PDS being a safe alternative to public social media, this gap is significant and should be addressed, either by describing additional moderation and security measures or by softening the claim.
  4. [Researcher Account; Regular User Account] The description of experiment visibility is internally inconsistent. The 'Researcher Account' section says 'Currently, the sandbox only allows private experiments,' but the 'Regular User Account' section states that a user who creates an account without an invitation 'can only see the content and actions of users who are part of public experiments.' These statements cannot both be true. This inconsistency, like the one about AI account actions, makes it difficult to determine the actual behavior of the hosted platform and should be clarified.
minor comments (5)
  1. [References] The reference to 'Adam Wiggins. 2017. The Twelwe-Factor App' contains a typo; it should be 'Twelve-Factor App'.
  2. [Introduction] The sentence 'Reddit, have recently experience the same problem' in the Related Work section has subject-verb agreement and tense errors; it should be 'Reddit has recently experienced the same problem.'
  3. [Public Discourse Sandbox Design] The sentence 'First, we plan on enabling users to select the type of recommendation algorithms other than the time-based, which is already included in the current version of the sandbox' is awkwardly phrased; consider rewriting for clarity.
  4. [Figure 2] The Figure 2 caption states 'Some features are still a work in progress, as described in the paper.' Since several described features are in fact future work, the figure should distinguish implemented features from planned ones, or the text should clearly label current capabilities.
  5. [References] The citation for the Profanity Check library gives 'Python Software Foundation' as the author, but the library is a third-party PyPI package; please cite the actual maintainer or repository.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: PDS is a software systems contribution with no fitted inputs, predictions, or self-citation-dependent derivation.

full rationale

The paper's central claim is the introduction of an open-source discourse research platform. There is no mathematical derivation, no fitted parameter, and no prediction: the contribution is a software artifact whose description is self-contained against external comparisons with Truman Platform, OASIS, Chirper, and Deliberate Lab. The two self-citations (Radivojevic, Clark, and Brenner 2024 on 'LLMs Among Us' and Radivojevic et al. 2024 on social media bot policies) supply background motivation and related work only; neither is load-bearing for the platform's claimed functionality, and the paper does not invoke any uniqueness theorem or ansatz from prior work. The paper itself flags incompleteness: the Figure 2 caption states 'Some features are still a work in progress,' the AI Account Deployment section limits internal AI accounts to 'create posts with up to 280 characters and create hashtags,' and Future Work lists 'like, undo like, create a comment ... repost posts ... follow other users' as planned for AI accounts. This is an internal consistency and completeness limitation relevant to correctness, not an instance of circularity, because no claim is reduced by construction to its own input. No circular steps were found.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters or invented entities appear; the paper is a software contribution. The two domain assumptions above are load-bearing for the platform's research utility: the sandbox must emulate real social media meaningfully, and LLM personas must approximate human discourse participation well enough for experiments to be informative.

assumptions (2)
  • domain assumption A Django-based reproduction of X/Twitter functionality is a valid substrate for studying real digital discourse.
    The paper assumes that the sandbox's emulation of social media features such as posts, likes, and follows yields findings transferable to real platforms. This is stated in the Design section and in the potential use cases.
  • domain assumption LLM-based AI accounts with persona prompts meaningfully simulate human participant behavior in discourse research.
    The AI account deployment relies on LLM inference and personification prompts, but the paper provides no validation of behavioral fidelity. The usefulness of experiments depends on this approximation being adequate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Public Discourse Sandbox: Facilitating Human and AI Digital Communication Research." pith.science (2026). https://pith.science/paper/YPGE6NOV

@misc{pith2026250521604,
  author       = {Pith},
  title        = {Pith review of: Public Discourse Sandbox: Facilitating Human and AI Digital Communication Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YPGE6NOV}},
  note         = {Machine review of arXiv:2505.21604}
}
read the original abstract

Social media serves as a primary communication and information dissemination platform for major global events, entertainment, and niche or topically focused community discussions. Therefore, it represents a valuable resource for researchers who aim to understand numerous questions. However, obtaining data can be difficult, expensive, and often unreliable due to the presence of bots, fake accounts, and manipulated content. Additionally, there are ethical concerns if researchers decide to conduct an online experiment without explicitly notifying social media users about their intent. There is a need for more controlled and scalable mechanisms to evaluate the impacts of digital discussion interventions on audiences. We introduce the Public Discourse Sandbox (PDS), which serves as a digital discourse research platform for human-AI as well as AI-AI discourse research, testing, and training. PDS provides a safe and secure space for research experiments that are not viable on public, commercial social media platforms. Its main purpose is to enable the understanding of AI behaviors and the impacts of customized AI participants via techniques such as prompt engineering, retrieval-augmented generation (RAG), and fine-tuning. We provide a hosted live version of the sandbox to support researchers as well as the open-sourced code on GitHub for community collaboration and contribution.

Figures

Figures reproduced from arXiv: 2505.21604 by the authors.

Figure 1
Figure 1. Example of scientific research or training event workflow leveraging PDS. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Account types and their respective permitted actions in the PDS. Some features are still a work in progress, as [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Technical Architecture of the PDS. filters: all, likes, comments, reposts, and follows. Posts can be created on the Home or Explore page in the post box or by clicking the post button, which opens a post creation dialog box. AI Account Deployment AI accounts, such as AI agents and digital twins, can be en￾rolled in the experiment by the researcher as a part of the ex￾periment. PDS supports two types of AI accounts: … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: PDS System Events Flow. Implementation - Potential Use Cases PDS features and functions were selected and designed based on two primary use cases, internal to our research in￾stitution and in collaboration with other research and train￾ing organizations. Experiments an…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 36 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Adam Wiggins . 2017. The Twelwe-Factor App. https://www.12factor.net/. Accessed: 2025-04-16

    work page 2017

  4. [4]

    Arias Jim \'e nez, B.; Rodr \' guez-Hidalgo, C.; Mier-Sanmart \' n, C.; and Coronel-Salas, G. 2022. Use of chatbots for news verification. In Communication and Applied Technologies: Proceedings of ICOMTA 2022, 133--143. Springer

    work page 2022

  5. [5]

    Ask Solem and Contributors . 2023. Celery - Distributed Task Queue. https://docs.celeryq.dev/en/stable/getting-started/introduction.html. Accessed: 2025-04-16

    work page 2023

  6. [6]

    W.; Poliak, A.; Dredze, M.; Leas, E

    Ayers, J. W.; Poliak, A.; Dredze, M.; Leas, E. C.; Zhu, Z.; Kelley, J. B.; Faix, D. J.; Goodman, A. M.; Longhurst, C. A.; Hogarth, M.; et al. 2023. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA internal medicine, 183(6): 589--596

    work page 2023

  7. [7]

    T.; and Mercea, D

    Bastos, M. T.; and Mercea, D. 2019. The Brexit botnet and user-generated hyperpartisan news. Social science computer review, 37(1): 38--54

    work page 2019

  8. [8]

    Calderaro, A. 2018. Social media and politics. The SAGE handbook of political sociology, 2: 781--795

    work page 2018

Show all 42 references
  1. [9]

    Chirper . 2025. Chirper AI - AI Life Simulation. https://chirper.ai/. Accessed: 2025-04-16

    2025

  2. [10]

    Cookiecutter Django . 2025. Cookiecutter Django's Documentation. https://github.com/cookiecutter/cookiecutter-django. Accessed: 2025-04-16

    2025

  3. [11]

    De Choudhury, M.; Sundaram, H.; John, A.; and Seligmann, D. D. 2010. Analyzing the dynamics of communication in online social networks. Handbook of social network technologies and applications, 59--94

    2010

  4. [12]

    DiFranzo, D.; and Bazarova, N. 2018. The Truman Platform: Social Media Simulation for Experimental Research. In ICSWM Workshop" Bridging the Lab and the Field. https://socialmedialab. cornell. edu/the-truman-platform

    2018

  5. [13]

    Felt, M. 2016. Social media and the social sciences: How researchers employ Big Data analytics. Big data & society, 3(1): 2053951716645828

    2016

  6. [14]

    Grady, C. 2015. Institutional review boards: Purpose and challenges. Chest, 148(5): 1148--1155

    2015

  7. [15]

    A.; Epstein, D

    Himelein-Wachowiak, M.; Giorgi, S.; Devoto, A.; Rahman, M.; Ungar, L.; Schwartz, H. A.; Epstein, D. H.; Leggio, L.; and Curtis, B. 2021. Bots and misinformation spread on social media: implications for COVID-19. Journal of medical Internet research, 23(5): e26933

    2021

  8. [16]

    N.; and Kollanyi, B

    Howard, P. N.; and Kollanyi, B. 2016. Bots,\# strongerin, and\# brexit: Computational propaganda during the uk-eu referendum. arXiv preprint arXiv:1606.06356

    2016 arXiv

  9. [17]

    Hu, T.; Liakopoulos, D.; Wei, X.; Marculescu, R.; and Yadwadkar, N. J. 2025. Simulating Rumor Spreading in Social Networks using LLM Agents. arXiv preprint arXiv:2502.01450

    2025 arXiv

  10. [18]

    Jiang, H.; Cheng, Y.; Yang, J.; and Gao, S. 2022. AI-powered chatbot communication with customers: Dialogic interactions, satisfaction, engagement, and customer behavior. Computers in Human Behavior, 134: 107329

    2022

  11. [19]

    Kaggle . 2025. Kaggle Datasets. https://www.kaggle.com/datasets. Accessed: 2025-04-16

    2025

  12. [20]

    Y.; Rana, N.; Majmudar, U.; et al

    Kaul, A.; Chaudhri, V.; Cherian, D.; Freberg, K.; Mishra, S.; Kumar, R.; Pridmore, J.; Lee, S. Y.; Rana, N.; Majmudar, U.; et al. 2015. Social media: The new mantra for managing reputation. Vikalpa, 40(4): 455--491

    2015

  13. [21]

    Krishnan, C.; Gupta, A.; Gupta, A.; and Singh, G. 2022. Impact of artificial intelligence-based chatbots on customer engagement and business growth. In Deep learning for social media data analytics, 195--210. Springer

    2022

  14. [22]

    M.; Norris, D

    Kruse, L. M.; Norris, D. R.; and Flinchum, J. R. 2018. Social media as a public sphere? Politics on social media. The Sociological Quarterly, 59(1): 62--84

    2018

  15. [23]

    Lauren Stewart . 2025. Social Media Research: Analysis of Social Media Data. https://atlasti.com/research-hub/social-media-research. Accessed: 2025-04-16

    2025

  16. [24]

    H.; and Yan Chan, W

    Leung, C. H.; and Yan Chan, W. T. 2020. Retail chatbots: The challenges and opportunities of conversational commerce. Journal of Digital & Social Media Marketing, 8(1): 68--84

    2020

  17. [25]

    MCP . 2025. Model Context Protocol. https://modelcontextprotocol.io/introduction. Accessed: 2025-04-16

    2025

  18. [26]

    M nsted, B.; Sapie \.z y \'n ski, P.; Ferrara, E.; and Lehmann, S. 2017. Evidence of complex contagion of information in social media: An experiment using Twitter bots. PloS one, 12(9): e0184148

    2017

  19. [27]

    Nosrati, S.; Sabzali, M.; Heidari, A.; Sarfi, T.; and Sabbar, S. 2020. Chatbots, counselling, and discontents of the digital life. Journal of Cyberspace Studies, 4(2): 153--172

    2020

  20. [28]

    S.; O'Brien, J.; Cai, C

    Park, J. S.; O'Brien, J.; Cai, C. J.; Morris, M. R.; Liang, P.; and Bernstein, M. S. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, 1--22

    2023

  21. [29]

    People+AI Research (PAIR) Initiative . 2024. Deliberate Lab. https://github.com/PAIR-code/deliberate-lab. Accessed: 2025-04-16

    2024

  22. [30]

    Prabowo, R.; Thelwall, M.; Hellsten, I.; and Scharnhorst, A. 2008. Evolving debates in online communication: a graph analytical approach. Internet Research, 18(5): 520--540

    2008

  23. [31]

    Python Software Foundation . 2025 a . Beautiful Soup. https://pypi.org/project/beautifulsoup4/. Accessed: 2025-04-16

    2025

  24. [32]

    Python Software Foundation . 2025 b . Profanity Check. https://pypi.org/project/profanity-check/. Accessed: 2025-04-16

    2025

  25. [33]

    Radivojevic, K.; Clark, N.; and Brenner, P. 2024. Llms among us: Generative ai participating in digital discourse. In Proceedings of the AAAI Symposium Series, volume 3, 209--218

    2024

  26. [34]

    Radivojevic, K.; McAleer, C.; Conley, C.; Kennedy, C.; and Brenner, P. 2024. Social Media Bot Policies: Evaluating Passive and Active Enforcement. arXiv preprint arXiv:2409.18931

    2024 arXiv

  27. [35]

    Software Freedom Conservancy . 2025. Selenium. https://www.selenium.dev/. Accessed: 2025-04-16

    2025

  28. [36]

    Stella, M.; Ferrara, E.; and De Domenico, M. 2018. Bots increase exposure to negative and inflammatory content in online social systems. Proceedings of the National Academy of Sciences, 115(49): 12435--12440

    2018

  29. [37]

    Stocking, G.; and Sumida, N. 2018. Social media bots draw public’s attention and concern. Pew Research Center

    2018

  30. [38]

    Suarez-Lledo, V.; and Alvarez-Galvez, J. 2022. Assessing the role of social bots during the COVID-19 pandemic: infodemic, disagreement, and criticism. Journal of Medical Internet Research, 24(8): e36085

    2022

  31. [39]

    Twitter - Developer Platform . 2025. Twitter API v2: Early Access. https://developer.x.com/en/docs/x-api/early-access. Accessed: 2025-04-16

    2025

  32. [40]

    N.; Wisniewski, H.; Halamka, J

    Vaidyam, A. N.; Wisniewski, H.; Halamka, J. D.; Kashavan, M. S.; and Torous, J. B. 2019. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. The Canadian Journal of Psychiatry, 64(7): 456--464

    2019

  33. [41]

    Vivian Ho . 2025. Reddit slams ‘unethical experiment’ that deployed secret AI bots in forum. https://www.washingtonpost.com/technology/2025/04/30/reddit-ai-bot-university-zurich/. Accessed: 2025-05-15

    2025

  34. [42]

    Yang, Z.; Zhang, Z.; Zheng, Z.; Jiang, Y.; Gan, Z.; Wang, Z.; Ling, Z.; Chen, J.; Ma, M.; Dong, B.; et al. 2024. Oasis: Open agents social interaction simulations on one million agents. arXiv preprint arXiv:2411.11581

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.