{"id":"7c0c110f-16a7-48e2-bb9c-f9bcead28b94","arxiv_id":"2504.17705","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"LUIDA unifies experiment implementation, parallel execution, recruitment, and data collection on a commercial metaverse platform, shown usable by VR researchers and able to replicate prior results with ~200 public participants per study.","lead":"LUIDA is a framework built on the commercial Cluster metaverse that automatically sets up connected virtual rooms, supplies ready templates, and handles recruitment plus data logging for VR experiments. Researchers can run studies with hundreds of public participants in days instead of weeks of custom coding and separate tools.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Replication match with self-selected Cluster users does not securely validate experimental integrity if participant pools differ from original lab studies.","rationale":"The reader's weakest assumption pinpoints the exact point where the replication evidence is least secure for the central validation claim. No stronger internal inconsistency, missing derivation, or statistical circularity is visible in the given material; the concern is therefore confirmatory rather than corrective.","tokens_in":1764,"tokens_out":322,"duration_ms":35316,"concrete_test":"Re-analyze the three replication datasets for demographic and behavioral covariates (age, VR experience, session duration, dropout rate) and test whether any covariate differs significantly from the original studies' reported samples; if a covariate shift exceeds 20% and correlates with outcome deviation >10% of the original effect size, the integrity-validation claim is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim rests on three replications each recruiting ~200 public Cluster users and yielding results that 'closely matched' the originals. This match is taken to confirm that LUIDA preserves experimental integrity across domains. The supporting assumption is that any population differences (self-selection, motivation, demographics, or platform familiarity) between commercial-metaverse volunteers and the controlled-lab participants of the source studies are irrelevant to the measured outcomes. If those differences systematically affect task performance, attention, or response patterns, the observed match could arise from compensating biases rather than from LUIDA's fidelity. No evidence is supplied that the two populations are comparable on the relevant variables or that sensitivity analyses were performed.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces LUIDA, a unified infrastructure built on the commercial metaverse platform Cluster that integrates experiment implementation templates, automatic allocation of interconnected virtual environments, participant recruitment, execution, and data collection for VR/HCI studies. It reports a usability evaluation with VR researchers yielding SUS 73.75 and NASA-TLX 24.11, plus three replication studies each recruiting ~200 self-selected public Cluster users within one week, with results claimed to closely match the original laboratory studies and thereby validate experimental integrity across domains.","tokens_in":1903,"tokens_out":416,"duration_ms":46396,"significance":"If the replication outcomes prove robust, LUIDA would represent a practical advance for HCI/VR research by enabling rapid, large-scale data collection with minimal development expertise and standardized protocols, potentially improving efficiency and reproducibility. The reported usability scores from researchers and the scale of the three replication studies (~200 participants each) are concrete strengths that demonstrate feasibility on a commercial platform.","major_comments":[{"comment":"§ on replicated experiments (the second evaluation study): The claim that results from the three replications with public Cluster users 'closely matched' the originals validates experimental integrity is load-bearing for the central contribution, yet the manuscript provides no details on the statistical tests performed, the quantitative criteria used to define a 'close match,' how self-selection was handled, or any analysis comparing the self-selected public users to the original controlled-lab participant pools on variables such as demographics, VR familiarity, motivation, or attention. Without such evidence or sensitivity checks, population differences could produce compensating biases that mimic fidelity rather than confirm it.","section":"replicated experiments section"}],"minor_comments":[{"comment":"Abstract: The summary of the replication results would be strengthened by briefly noting the statistical approach or matching criteria used, consistent with the level of detail already given for the SUS and NASA-TLX scores.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback and for recognizing the potential of LUIDA to improve efficiency and reproducibility in VR/HCI research. We address the major comment on the replicated experiments section below and describe the changes we will make to strengthen the manuscript.","responses":[{"response":"We agree that the current description of the replication results is insufficiently detailed to fully support the claim of experimental integrity. In the revised manuscript we will expand this section to report the specific statistical tests performed on each key dependent variable (including p-values, effect sizes, and confidence intervals), define quantitative criteria for a 'close match' (e.g., non-significant differences combined with equivalence-test bounds or effect-size thresholds), describe the self-selection mitigation steps used (pre-screening questions, attention checks, and exclusion criteria), and include any available demographic or VR-experience comparisons between the public Cluster sample and the original laboratory pools. Where direct comparisons are not possible because certain variables were not collected in the source studies, we will explicitly note this limitation and present sensitivity analyses on the measures that are available. These additions will be placed in a new subsection with tables summarizing the statistical outcomes.","revision_made":"yes","referee_comment":"[replicated experiments section] § on replicated experiments (the second evaluation study): The claim that results from the three replications with public Cluster users 'closely matched' the originals validates experimental integrity is load-bearing for the central contribution, yet the manuscript provides no details on the statistical tests performed, the quantitative criteria used to define a 'close match,' how self-selection was handled, or any analysis comparing the self-selected public users to the original controlled-lab participant pools on variables such as demographics, VR familiarity, motivation, or attention. Without such evidence or sensitivity checks, population differences could produce compensating biases that mimic fidelity rather than confirm it."}],"tokens_in":1390,"tokens_out":395,"duration_ms":67328,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"LUIDA is a practical system that unifies the workflow for VR experiments on the Cluster metaverse platform. It handles setup with adaptable templates, runs parallel sessions automatically, recruits from the public user base, and collects data all in one place. The paper shows this works with reasonable researcher feedback and some replication success.","headline":"LUIDA gives a workable end-to-end pipeline for running larger VR experiments on the Cluster platform, with usable researcher scores and replication matches that are promising but rest on unexamined population differences.","tokens_in":2457,"tokens_out":147,"would_cite":false,"duration_ms":34507,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"Metaverse VR experimentation platform with no RS machinery","alignment":"orthogonal","rationale":"The paper presents LUIDA, a practical HCI/VR tool for unified recruitment, parallel execution, low-code templates, and data collection on commercial metaverse platforms. Its central claims concern usability (SUS/NASA-TLX), participant throughput (~200 per experiment), and replication fidelity across perception/cognition/UI domains. None of this intersects RS forcing chains, J-cost, φ-ladders, 8-tick periodicity, or parameter-free constant derivations. Domain is cs.HC; RS has no theorems about metaverse tooling or crowdsourced VR validity.","tokens_in":56168,"confidence":"high","tokens_out":152,"duration_ms":8245,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"LUIDA unifies fragmented VR research tasks into one metaverse platform that recruits hundreds of public participants and yields results matching original lab studies.","keywords":["metaverse","virtual reality","online experiments","human-computer interaction","experimental reproducibility","digital assessments","VR research workflows"],"falsifier":"A new experiment run through LUIDA that produces statistically different outcomes from its matched original laboratory study on the same task would show the platform does not preserve experimental integrity.","tokens_in":2648,"feed_emoji":"🌐","tokens_out":602,"duration_ms":39126,"temperature":0.7,"pith_summary":"The paper presents LUIDA as a single metaverse framework that combines experiment building, participant recruitment, running, and data gathering to reduce the usual split across separate tools. Developers who tried the prototype found it usable with moderate effort and noted smoother workflows than standard lab setups. Three separate replications each drew around 200 public users in roughly a week and produced outcomes close to the source studies, indicating that the platform can support valid experiments in different research areas.","feed_headline":"Metaverse platform runs VR experiments with 200 users in one week","feed_subtitle":"Replicated studies match prior lab results, showing one unified system can handle recruitment and data collection while keeping validity.","key_machinery":"LUIDA, the Large-scale Unified Infrastructure for Digital Assessments, which merges system implementation, recruitment, execution, and data collection inside one commercial metaverse environment.","core_discovery":"LUIDA is a metaverse-based framework that automatically sets up linked virtual spaces for running multiple experiments at once and supplies ready templates that researchers can adjust for different VR topics without needing deep platform-building skills. Tests showed that the approach lets researchers move from idea to completed data collection more directly than before, and the public-user replications preserved the key findings of earlier controlled work.","pith_inferences":["The same integration approach could be adapted to other virtual-world systems beyond the one tested here.","Studies that need broad or hard-to-reach participant groups become easier to run at scale.","Longer-term use might reveal whether data quality stays stable as the user community changes over time."],"forward_implications":["Researchers gain a single workflow instead of juggling separate tools for building, recruiting, running, and collecting data.","Hundreds of participants can be reached in days rather than weeks or months.","Replicated studies across varied VR topics continue to match earlier lab findings.","A standardized, open version of the platform could raise reproducibility by giving every team the same protocol."],"fun_headline_variants":["LUIDA automates linked virtual spaces for multiple VR experiments","Templates enable easy metaverse VR studies without deep development skills","Public user tests replicate prior VR results using unified infrastructure","Researchers report streamlined workflows with LUIDA on commercial platform"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Results from self-selected public users of a commercial metaverse platform remain close enough to results from controlled laboratory participants to count as validation of experimental integrity.","fun_headline_variants_meta":{"raw":{"variants":["LUIDA automates linked virtual spaces for multiple VR experiments","Templates enable easy metaverse VR studies without deep development skills","Public user tests replicate prior VR results using unified infrastructure","Researchers report streamlined workflows with LUIDA on commercial platform"]},"model":"grok-4.3","cost_usd":0.007525,"raw_usage":{"total_tokens":3381,"prompt_tokens":688,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":75253000,"prompt_tokens_details":{"text_tokens":688,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2628,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":688,"tokens_out":65,"duration_ms":39610,"temperature":1.0,"reasoning_tokens":2628,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-22T18:11:44.948903+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A new experiment run through LUIDA that produces statistically different outcomes from its matched original laboratory study on the same task would show the platform does not preserve experimental integrity.","supporting_citations":[],"review_version":1}