{"id":"092f8903-6ec9-4f04-85e1-3d2b9068ffd2","arxiv_id":"2501.19245","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SHARPIE is a modular framework that wraps RL environments and algorithms to support human-AI interaction experiments through a web interface and standardized logging.","lead":"SHARPIE is a new modular software platform for running experiments that mix reinforcement-learning agents and human participants. It wraps existing RL libraries and adds a web interface, logging, and deployment tools, aiming to become a standard for human-AI interaction experiments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The standardization claim depends on the wrapper-plus-web-channel abstraction being sufficient for diverse human-RL interactions, but the paper provides no API specification, no compatibility details for Gymnasium+ variants, and no end-to-end validation; the claim is currently unsupported.","rationale":"The reader's weakest assumption identifies exactly the load-bearing condition: that wrapping Gymnasium environments and adding web communication is enough to cover the diverse interaction modes the paper claims to support. I agree. The concern is load-bearing because the paper's stated contribution is the generic interface and the standardization ambition itself, so the minimal evidence is a demonstration that one non-trivial interaction mode can be implemented through the public abstraction without per-use-case core engineering. The paper provides no such demonstration, no API-level specification, and no handling details for the 'Gymnasium+' variants listed in Appendix A. This is not an internal inconsistency or a violation of consensus; it is an unsupported empirical claim about usability and generality. The paper is early-stage, the code is public, and the architecture is plausible, so rejection is not warranted. The reader's CONDITIONAL verdict remains appropriate, but the condition should be explicit: provide an end-to-end runnable example of at least one listed use case before assessing the generic-standard claim. I do not change the verdict.","tokens_in":9489,"tokens_out":3058,"duration_ms":33222,"concrete_test":"Clone the SHARPIE repository and, using only its documented public API, launch one full end-to-end experiment for the action-delegation use case: a human in a web UI hands control to and from an RL agent in Mountain Car, with learning, logs, and session data captured. If this requires writing new core abstractions or modifying the library, the wrapper-plus-web-channel abstraction is insufficient even for a listed use case; if it succeeds without core changes, it supplies the missing existence proof. Separately, inspect the repository for handling of multi-objective (vector reward) and multi-agent (vector action) spaces to verify the 'Gymnasium+' compatibility claim in Appendix A.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SHARPIE provides a generic interface that will standardize human-RL experiments. The load-bearing condition is in the 'SHARPIE Framework' section: 'it can encapsulate any existing environment that follows the conventional Gymnasium API ... which encompasses most of the existing RL platforms,' plus web-based communication channels. For the claim to hold, wrapping Gymnasium-style environments and adding a web channel must be sufficient to support the paper's own use cases: reward annotation, exploration, teaching, action delegation, task specification, human-AI teaming, utility elicitation, and shared decision support. This condition is not established. No formal interface or protocol for these interaction modes is defined in the paper; the only concreteness is the generic 'reset, step, render' calls and short symbolic communication 'initially'. Human-in-the-loop experiments impose timing, asynchrony, pausing, and multi-modal input requirements that the step-based Gymnasium API does not address; wrapping an environment does not create those capabilities. Appendix A also lists many 'Gymnasium+' platforms with vectorial rewards or vectorial actions, but the paper does not specify whether or how SHARPIE's wrapper handles those altered APIs. The paper itself states the library is 'currently under active development' and communication modalities 'will be expanded'; so the current artifact does not yet substantiate the generic-standard claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes SHARPIE, a Python-based web framework intended to support experiments in which humans interact with reinforcement learning (RL) agents. It motivates the platform with eight use cases spanning reward annotation, exploration, teaching, action delegation, task specification, human-AI teaming, utility elicitation, and shared decision support. The envisioned architecture combines a wrapper for RL environments and algorithm libraries, a web-based participant interface, logging utilities, and deployment support for cloud and recruitment platforms. The paper positions SHARPIE as a generic interface that could standardize human-RL interaction experiments, analogous to the role of the Gymnasium API for fully simulated RL. The text also reviews related RL, behavioral-experiment, and multi-agent platforms and includes an appendix that compares popular RL packages and their APIs. No implementation, interface specification, runnable example, or experimental validation is provided.","tokens_in":9746,"tokens_out":4362,"duration_ms":44222,"significance":"SHARPIE addresses a genuine gap: there is no widely adopted platform for controlled experiments that combine human participants with RL agents across many interaction modes. The use-case table is a useful structuring of the problem space, and the decision to build on existing RL libraries and to include logging and deployment utilities is pragmatically motivated. If the framework were implemented and validated, it could reduce the engineering burden of human-in-the-loop RL research and improve comparability across studies. The connection to cognitive-science questions (theory of mind, trust, communication grounding) is a strength. However, the paper's central contribution is currently a design proposal: the generic interface that underpins the standardization claim is not defined, and no evidence is presented that the framework actually works for any of the listed use cases.","major_comments":[{"comment":"The central claim that SHARPIE provides a 'generic interface for human-RL interactions' is never made concrete. The paper does not specify the interface: there is no description of the methods exposed by the environment wrapper, the message schemas or data types exchanged over communication channels, the lifecycle of a human-in-the-loop episode, or the semantics of actions such as delegation, feedback, and task specification. Figure 1 shows blocks but no protocols. Without this specification, the assertion that any Gymnasium-style environment can be encapsulated and that the eight motivating use cases are supported is not checkable.","section":"SHARPIE Framework"},{"comment":"The paper's own statements undermine the present-tense framework claim. It says the library 'is currently under active development,' that only some of the listed use cases 'will be implemented initially,' and that communication modalities 'will be expanded upon.' These are prospective statements about a plan, not a description of an existing artifact. As written, the paper cannot support the claim that SHARPIE 'provides' a generic interface or that it 'aims to standardize' the field; those claims require at least a specification of the current interface and a statement of which use cases are operational.","section":"Motivating Use Cases; SHARPIE Framework"},{"comment":"The compatibility claim that the wrapper 'can encapsulate any existing environment that follows the conventional Gymnasium API' is weakened by the paper's own Appendix A, which classifies several platforms as 'Gymnasium+' with 'minor alterations' such as vectorial rewards (multi-objective) or vectorial actions (multi-agent). The paper does not state whether SHARPIE's reset/step/render wrapper preserves these altered APIs. Since the multi-objective use cases (7 and 8) and multi-agent use cases (3, 4, and 6) depend on exactly those extensions, the claim that SHARPIE encompasses 'most existing RL platforms' is incomplete without a concrete compatibility statement.","section":"Appendix A; SHARPIE Framework"},{"comment":"The manuscript contains no empirical or demonstrative validation: no end-to-end example with code, no screenshots, no benchmark results, and no usability or performance measurements. For a framework paper, at least one fully worked use-case walkthrough or a small pilot experiment is necessary to substantiate the claim that SHARPIE 'streamlines' and 'empowers' research. Without such evidence, the reader cannot distinguish a viable architecture from a collection of intended features.","section":"General evaluation"}],"minor_comments":[{"comment":"The phrase 'at worse' should be 'at worst' in the discussion of restarting experiments.","section":"SHARPIE Framework"},{"comment":"The comparison with Interactive Gym is necessarily vague because both systems are under development, but the paper would benefit from listing concrete architectural differences (e.g., support for non-game interaction modalities, explicit communication channels, multi-stakeholder use cases) rather than the general statement that SHARPIE is 'more generic and extendable.'","section":"Related Work"},{"comment":"The term 'Gymnasium+' is introduced only in the table footnote; it should be defined and used consistently in the main text, especially since the wrapper compatibility argument depends on it.","section":"Appendix A"},{"comment":"The reference to Sutton and Barto (1998) uses the first edition; if the intended citation is the widely used second edition, it should be updated to 2018.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like a position or 'blue sky' abstract rather than a systems paper. For a journal venue, the authors should either reframe the contribution as a design proposal and add a concrete interface specification, or implement a substantial subset of the framework and demonstrate it on at least one of the eight use cases. The current version does not contain enough technical detail for a reader to build or reuse the claimed abstraction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper is a framework proposal, not a validated system. The code is public, the architecture is sensible, but the paper explicitly states the library is under active development and only some use cases will be implemented initially. Treat it as a design statement with a roadmap, not as an evaluated artifact.\n\nWhat's actually new: it combines Gymnasium-style wrapping, a web-based participant UI, logging, deployment, and multi-agent/multi-objective support in one package. That combination is real, though the distance from Interactive Gym is modest, and the authors are candid about that. The use-case table is the best part: it gives a concrete sense of the interaction modes they care about, from reward annotation to shared decision support.\n\nIt does well on related work: fair and current. They acknowledge Interactive Gym, HIPPO-Gym, Uni-RLHF, and they don't pretend SHARPIE is already a standard. The 'aims to standardize' phrasing is a goal, not a claim of achieved status.\n\nSoft spots, in order of seriousness. First, the central abstraction is underspecified. They say any Gymnasium-style environment can be wrapped and that web channels suffice for rich human-AI interaction, but the paper gives no API specification, no protocol for the interaction modes, and no treatment of Gymnasium+ variants (vectorial rewards/actions). Wrapping reset/step/render does not create asynchrony, pausing, or multimodal input. This is a real gap, and the stress-test note is right to flag it. Second, there is zero empirical validation: no screenshots, no runnable example, no benchmark, no pilot study. For a framework paper, that is a serious omission, though not fatal if the code is available and the design is sound. I did not inspect the code, so I cannot verify whether the implementation substantiates the design. Third, the standardization claim is premature. It may be a useful goal, but without a demonstration of at least two different use cases running on the same generic interface, it remains a hope.\n\nWho is this for? Researchers in human-in-the-loop RL who want to set up experiments without building custom interfaces. They would get value from a working SHARPIE, but from this paper alone they get a roadmap, not yet a tool.\n\nShould a serious editor send this to review? Yes, but with the expectation of major revision. The idea is worth discussing, and the authors have done enough homework to deserve referee time. My own verdict is conditional: I'd want the API specified and at least one end-to-end use case demonstrated before believing the generic-standard claim.\n\nRecommendation: engage, but ask for a validation section and an explicit interface definition before publication.","headline":"A plausible early-stage framework proposal with honest limitations, but the 'generic standard' claim is underspecified and unvalidated.","tokens_in":10287,"tokens_out":2322,"would_cite":false,"duration_ms":21853,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SHARPIE is a modular platform that wraps standard reinforcement-learning environments into web-based experiments where humans and AI agents can interact, aiming to become the standard interface for human-in-the-loop RL research.","keywords":["reinforcement learning","human-AI interaction","experiment platform","modular framework","web interface","human feedback","multi-agent systems"],"falsifier":"Set up the paper's 'human-AI teaming' use case using only SHARPIE's public API and prebuilt UI components, with bidirectional communication of intentions in a simple-tag environment. If implementing the intention channel requires modifying the environment source or writing new backend middleware beyond configuration, then the claim of a generic interface for human-RL interactions fails for that mode.","tokens_in":9353,"feed_emoji":"🤖","tokens_out":8021,"duration_ms":62061,"temperature":0.7,"pith_summary":"The paper proposes SHARPIE, a modular software framework that lets researchers run experiments in which humans interact with reinforcement learning agents. The core claim is that a single generic interface, built by wrapping the widely used Gymnasium environment API and adding a web-based participant front-end, can support a broad range of interaction patterns including human feedback, demonstrations, action delegation, preference elicitation, and shared decision-making. A sympathetic reading is that the framework aims to become for human-in-the-loop RL what the Gymnasium API became for simulated RL: a common standard that makes experiments easier to build, compare, and reproduce. The framework is under active development, so the paper's contribution is the architecture and a set of motivating use cases rather than a completed evaluation.","feed_headline":"SHARPIE aims to standardize human-AI RL experiments","feed_subtitle":"Wrapping the standard RL environment API, it lets researchers set up human-agent interaction studies in a browser.","key_machinery":"The central mechanism is the generic interface for human-RL interactions. Concretely, SHARPIE wraps environments that implement the conventional Gymnasium API (reset, step, render) so that the environment can be driven either by an RL algorithm or by a human participant through a browser-based front-end. Around that wrapper, the framework provides pluggable communication channels for coordination and teaching, a preference-elicitation module for ranking trajectories and policies, and utilities for logging and deployment to cloud servers. The wrapper is what does the work: it converts a rich human-agent interaction into the step/reset loop that RL algorithms already expect, so existing environments and algorithms can be reused without modification.","core_discovery":"The paper's central discovery is that the missing piece in current reinforcement-learning tooling is not another environment or algorithm library, but a generic interaction layer between humans and agents. SHARPIE supplies that layer by encapsulating any environment that follows the conventional Gymnasium API (reset, step, render) and connecting it to a multimodal web interface for participants. The authors argue that this design covers a wide range of human-AI interaction scenarios, including interactive reward specification, learning from human feedback, action delegation, preference elicitation, user modeling, and human-AI teaming, and that it lays the foundation for a standard in human-in-the-loop RL, just as Gymnasium standardized fully simulated environments.","pith_inferences":["Editorial inference: if the standard is adopted, SHARPIE could become a common substrate for cognitive-science studies of human-AI collaboration, allowing researchers to reuse one interaction layer to investigate trust, theory of mind, and language grounding.","Editorial inference: the reliance on the Gymnasium API means any interaction pattern that cannot be expressed as discrete reset/step cycles, such as continuous real-time coordination or free-form dialogue, will require custom extensions; the framework's generality may be bounded by that wrapper.","Editorial inference: a direct stress test would be to implement all eight use cases from the paper on the current public API; any use case that needs new backend middleware or environment patches shows where the generic interface stops."],"forward_implications":["Researchers can set up human-RL experiments using existing Gymnasium-compatible environments without building custom interfaces, lowering the barrier to studying human-AI interaction.","A shared interaction layer makes results from different labs more directly comparable, since the experimental setup would share the same infrastructure.","The platform can serve as a simulator for human-robot interaction studies before physical deployment, as the paper notes.","Because the wrapper extends to multi-agent and multi-objective environments, the same framework can host studies from a single human with one agent to multiple stakeholders sharing a decision."],"supporting_citations":[{"why":"Defines the Gymnasium (originally OpenAI Gym) API that SHARPIE wraps; the framework's genericity claim depends on this interface being the de facto standard.","marker":"Brockman 2016"},{"why":"Maintains the Gymnasium standard and lists compatible environments, supporting the paper's assertion that most RL platforms follow this API.","marker":"Towers et al. 2024"},{"why":"HIPPO-Gym is the closest prior human-in-the-loop RL framework; the paper uses it as the comparison point that is limited to single-agent teaching.","marker":"Taylor et al. 2023"},{"why":"Interactive Gym is the most closely related ongoing framework; SHARPIE's claim to be more generic is defined against it.","marker":"McDonald 2024"},{"why":"Establishes the learning-from-human-preferences paradigm that SHARPIE aims to support through its feedback channels.","marker":"Christiano et al. 2017"}],"fun_headline_variants":["SHARPIE standardizes human-in-the-loop RL workflows","Generic API for human-RL interaction experiments","Web-based RL framework for human-AI teaming studies","SHARPIE: modular toolkit for human-AI RL","Gymnasium-style standard for human-in-the-loop RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that wrapping any environment that follows the standard reset–step–render API is enough to cover the range of human-AI interaction modes it promises; if real studies require deep per-use-case engineering, the generic standard will not materialize.","fun_headline_variants_meta":{"raw":{"variants":["SHARPIE standardizes human-in-the-loop RL workflows","Generic API for human-RL interaction experiments","Web-based RL framework for human-AI teaming studies","SHARPIE: modular toolkit for human-AI RL","Gymnasium-style standard for human-in-the-loop RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1707,"prompt_tokens":830,"completion_tokens":877,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":799}},"tokens_in":446,"tokens_out":877,"duration_ms":7487,"temperature":1.0,"reasoning_tokens":799,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T20:48:27.023282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Set up the paper's 'human-AI teaming' use case using only SHARPIE's public API and prebuilt UI components, with bidirectional communication of intentions in a simple-tag environment. If implementing the intention channel requires modifying the environment source or writing new backend middleware beyond configuration, then the claim of a generic interface for human-RL interactions fails for that mode.","supporting_citations":[{"cited_title":"E.; Nissen, N.; Wang, Y.; and Navidi, N","cited_arxiv_id":null,"evidence_quote":"HIPPO-Gym is the closest prior human-in-the-loop RL framework; the paper uses it as the comparison point that is limited to single-agent teaching."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Interactive Gym is the most closely related ongoing framework; SHARPIE's claim to be more generic is defined against it."}],"review_version":1}