{"id":"c5ddb534-30e0-4d46-9c39-04a05ca3c4e7","arxiv_id":"1908.02409","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An AR app spanning synchronous and asynchronous, colocated and remote collaboration showed that users prefer colocated synchronous building but participate most when content is location-independent.","lead":"Blocks is a smartphone app for co-creating 3D block structures in augmented reality that persist in the physical world for others to view and edit later. In tests with over 160 people, users most enjoyed building together in the same room, but they used the app most when access was not tied to a single location.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Field-study evidence for 'location-independent increases activity' is confounded by day and site and lacks a statistical test; the Day 3 within-site comparison is the decisive check.","rationale":"The paper is a legitimate HCI contribution: a working, deployed collaborative AR system with server logs and three studies totaling more than 160 participants. The lab study finding that colocated-synchronous collaboration is preferred for fun and engagement is a within-subject ranking result and is the strongest evidence in the paper. The central weak point is the field-study basis for the second half of the headline claim—'most active when not restricted by time or place.' The reader identified this as the weakest assumption, and I agree in substance, though the Day-3 design is not as fully confounded as the reader's summary implies: both conditions occurred at the cafeteria on Day 3, which offers a cleaner within-day, within-site test. Because the paper only reports descriptive counts and no statistical test, and because the activity metric flips if blocks per session is used, the design recommendation in §9.1 is conditional on a re-analysis that the authors have not provided. This is addressable and does not invalidate the system or the lab results, so the appropriate verdict remains conditional rather than acceptance or rejection.","tokens_in":21385,"tokens_out":10350,"duration_ms":117178,"concrete_test":"Request the timestamped server logs and reconstruct per-user session counts in four cells: Day1-lobby-dependent, Day2-lobby-independent, Day3-morning-cafeteria-dependent, Day3-afternoon-cafeteria-independent. Fit a mixed-effects Poisson/negative binomial model with condition as fixed effect and random intercepts for participant and day/site; at minimum run a within-Day-3 comparison of morning vs. afternoon. If the location-independent afternoon does not show more sessions per user than the location-dependent morning, the aggregate difference is not attributable to location-independence. Report blocks-added and active minutes per session as secondary activity metrics to disambiguate 'most active.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central design recommendation in §9.1—that location-independent AR experiences lead to more usage—rests on the field study's aggregate session counts (§6.1, §7.1, Table 2): 91 sessions from 32 users in location-dependent vs. 131 sessions from 36 users in location-independent. The comparison is not randomized or statistically modeled. Day 1 (lobby) was location-dependent, Day 2 (lobby) was location-independent, and Day 3 (cafeteria) was split morning dependent / afternoon independent. Condition is therefore entangled with calendar day, physical site, and time of day. A novelty or foot-traffic effect on Day 2, or a morning/afternoon effect on Day 3, could produce the observed difference without any causal effect of location-independence. The paper reports no inferential test for these counts, unlike the lab study's ANOVAs. Moreover, the 'activity' claim depends on the chosen metric: location-dependent sessions had more blocks per session (58 vs. 35), so per-user total blocks are similar across conditions; only the session count favors location-independence. The authors' own Limitations (§10) address task-order variance and participant diversity but not this field confound. The 5-day extended deployment is consistent with high engagement but does not manipulate the location dimension, so it cannot rescue the causal claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Blocks is a mobile AR application for collaborative creation of persistent block-based structures, presented as the first tool to cover all combinations of synchronous/asynchronous collaboration with colocated/remote users, and with AR structures that are either tied to a physical location or available anywhere. The paper reports a 24-participant lab study comparing colocated-sync, remote-sync, and async creation; a 68-participant three-day field study comparing location-dependent and location-independent deployments; and a 70-participant five-day extended deployment. The main reported findings are that participants prefer synchronous colocated collaboration and that participation, measured by session counts, is higher when structures are location-independent. The paper contributes design implications centered on location-independent, colocated synchronous experiences.","tokens_in":21588,"tokens_out":5032,"duration_ms":55464,"significance":"The contribution is an early, systematic empirical exploration of two collaborative dimensions in a single persistent AR creation tool. The lab study is methodologically stronger than typical system papers: within-subject counterbalanced conditions, ANOVA and Friedman tests with Holm-Bonferroni corrections, and server-log plus survey measures. The field and extended deployments provide realistic usage data and rich qualitative observations. If the field comparison were causally identified, the location-independence engagement claim would be an important design guideline. As it stands, the system and the lab preference results are likely to interest the IMWUT/UbiComp community, but the headline activity claim requires additional support or a tempered framing.","major_comments":[{"comment":"The claim that location-independent scenarios produce more activity is not supported by a controlled or statistically modeled comparison. The location-dependent condition ran on Day 1 (lobby) and Day 3 morning (cafeteria), while the location-independent condition ran on Day 2 (lobby) and Day 3 afternoon (cafeteria). Session counts (91 vs. 131) are thus entangled with calendar day, physical site, and time of day; novelty, foot traffic, or diurnal patterns could explain the difference. No inferential test is reported for these counts, in contrast to the lab study. This is load-bearing because §9.1 and §11 generalize to a recommendation to prefer location-independent experiences for participation. The paper should at minimum analyze the Day 3 within-site morning/afternoon split, model day and site as random effects, or substantially temper the causal wording; the Limitations section (§10) currently does not address this confound.","section":"§6.1, §7.1, Table 2, Fig. 14"},{"comment":"The activity finding is metric-dependent. In Table 2, location-dependent sessions have substantially more blocks per session (58 vs. 35) and similar total OurWorld blocks to location-independent (1,722 vs. 1,936); only the number of user sessions is higher. The abstract's phrase 'most active' therefore conflates session frequency with total contribution. The authors should report and test per-user and per-session productivity measures, and state explicitly which dimension of activity the design recommendation targets.","section":"§7.1, Table 2"},{"comment":"The five-day extended deployment is presented as consistent with earlier findings, but it does not manipulate the location dimension; participants could experience structures both at the installation and anywhere. It therefore cannot provide independent evidence for the location-independence effect. This is not itself a flaw, but it should not be cited in support of the causal claim about location-independence increasing participation.","section":"§8"},{"comment":"The design recommendation for a location-independent, colocated synchronous experience is a conjunctive extrapolation that is not directly tested. The lab study varied the location of people across synchronous and asynchronous conditions but did not vary whether structures were location-dependent or independent; the field study varied structure persistence across locations but did not manipulate whether collaborators were colocated or remote. The paper should present the combined recommendation as an inference from two separate studies, not as an empirically validated combination.","section":"§9.1, §5.3.3, §7.1"}],"minor_comments":[{"comment":"The sentence 'On Day 2, we deployed the location-independent scenario was deployed' contains a duplicated verb phrase; it should likely read 'On Day 2, we deployed the location-independent scenario, where...'.","section":"§6.1"},{"comment":"Plotting cumulative sessions and average blocks against time of day for conditions that ran on different days may visually suggest a within-day comparison; adding day and site annotations would improve clarity.","section":"Fig. 14, Fig. 15"},{"comment":"The post-hoc comparison for blocks added in the async condition versus the colocated-sync condition reports adjusted p = 0.139; the text should state clearly that this difference is not significant after the Holm-Bonferroni correction, since the current phrasing 'post-hoc analyses revealed' could be misread.","section":"§5.2.2"},{"comment":"The phrase 'space and time' is used for two distinct dimensions: the location of the people and the location of the AR structures. Consider distinguishing these dimensions more explicitly in Table 1 and the surrounding text to avoid conceptual ambiguity.","section":"§2.2, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and likely to be of interest to the IMWUT readership. The central risk is the field-study confound: the location-independence engagement claim currently rests on a non-randomized, day/site-entangled comparison without a statistical test. If the authors cannot supply a within-site reanalysis or a mixed-effects model, they should reframe the claim as observational rather than causal. There are no concerns about novelty disclosure or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: the lab preference result is credible, but the field-study activity claim is weaker than the abstract suggests. The paper is worth engaging, though.\n\nWhat's genuinely good: Blocks is a real system that spans the full space/time grid, and the 24-person within-subject lab study is carefully run. The finding that people prefer colocated synchronous collaboration is statistically supported, with appropriate nonparametric checks and Holm-Bonferroni adjustments. The design implications around separating consumption from creation and supporting location-independent access are thoughtful, and the deployment data are rich.\n\nThe soft spot is exactly where the stress test lands. The claim that location-independent structures drive more activity rests on a field comparison where condition is entangled with calendar day and physical site: Day 1 in the lobby was dependent, Day 2 in the lobby was independent, and Day 3 in the cafeteria was split morning/afternoon. There is no inferential test on the session counts, and alternative explanations (novelty on Day 2, foot traffic, time of day) are just as plausible. The metric issue makes it worse: location-dependent sessions had more blocks per session (58 vs. 35), so per-user total blocks are roughly similar; only session count favors independence. The abstract and §9.1 present this as a finding, but it's a descriptive observation with a confound. The Limitations section mentions task order and participant diversity but not this, which is a real gap.\n\nThat said, the lab study and the system contribution carry the paper. The 5-day extended deployment is consistent but doesn't manipulate the location dimension, so it can't rescue the causal language. The narrow participant pool is acknowledged and is a genuine limitation for generalization.\n\nWho should read it: people designing collaborative AR or persistent social VR systems, and HCI researchers working on space-time taxonomies. It deserves a serious referee—the system is real, the lab data are careful, and the field data are useful even if the causal reading needs fixing. My recommendation: send it to review, and in review push for either a re-analysis that isolates day/site or restrained wording that presents the activity difference as a descriptive pattern with alternative explanations. I'd cite the system and the lab study.","headline":"Solid system and a careful lab study, but the headline field-study claim about location-independence driving activity is confounded by day and site and needs to be re-analyzed or substantially softened.","tokens_in":22119,"tokens_out":1789,"would_cite":true,"duration_ms":17881,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Blocks, a mobile AR app, demonstrates that non-experts can co-create persistent block structures across every space-time collaboration mode, and the studies show colocated synchronous building is the most enjoyable while location-free…","keywords":["augmented reality","collaborative AR","persistence","mobile AR","content creation","synchronous and asynchronous collaboration","field deployment","location-based AR"],"falsifier":"Re-run the field study as a within-site crossover that alternates location-dependent and location-independent conditions on the same day and same poster, then compare sessions per user and repeat-use rates; if the advantage of location independence disappears or tracks the day or site, the paper's central design recommendation is not supported.","tokens_in":21184,"feed_emoji":"🧱","tokens_out":7648,"duration_ms":81524,"temperature":0.7,"pith_summary":"The paper tries to establish that collaborative creation of persistent augmented-reality content is not reserved for experts: Blocks lets anyone place colored cubes, leaves those cubes in place for others to find and edit, and works whether collaborators are together or apart and whether they build at the same time or different times. It matters because today's AR is mostly content consumption, and a persistent, user-created AR world would change how people share information in physical spaces. The empirical core is a 24-person lab study plus a 68-person field deployment and a 70-person extended deployment. The paper reports that participants enjoy colocated synchronous collaboration most, but are most active when structures are not bound to a particular location, and reads this as evidence for designing colocated, synchronous, location-independent AR experiences.","feed_headline":"Colocated sync wins on enjoyment; untethered AR wins on usage","feed_subtitle":"Real-time in-person building rated best; freely accessible structures drew the most repeat sessions.","key_machinery":"The key mechanism is Blocks' persistence architecture: a cloud database stores every block's position, size, color, and anonymous owner, so structures survive across app sessions and device changes. Image markers anchor location-dependent structures to a fixed spot and re-calibrate them on re-detection, while world tracking places blocks on real surfaces. Live awareness comes from shared cursors, recent-change highlighting, and an online-user counter, and the split between a personal sandbox and a shared OurWorld lets newcomers practice before contributing.","core_discovery":"The central claim is that a single mobile AR tool can cover all four combinations of space and time, and that the two dimensions pull in different directions. In the lab, colocated synchronous collaboration ranked highest for fun, engagement, and awareness of a partner's actions, while asynchronous collaboration produced more blocks and more color variety. In the field, the location-independent condition drew more sessions per user and more repeat use, while the location-dependent condition drew more blocks per session. The paper concludes that the most promising design configuration combines the enjoyment of colocated synchronous work with the participation boost of location-independent access.","pith_inferences":["Because the field comparison ran location-dependent and location-independent conditions on different days and at different sites, a cleaner test would hold the site and day constant while toggling location dependence; this would tell whether the session-count advantage is really about AR location design and not about calendar or novelty effects.","If the patterns generalize, persistent collaborative AR could follow the same arc as Web 2.0, shifting end users from viewers of expert-made content to producers of an embedded digital layer over the physical world, with all the ownership, moderation, and curation problems that shift will bring.","The contrast between the lab's enjoyment result and the field's participation result suggests there may be two distinct audiences or two distinct motivational routes, one driven by live social presence and one by flexible access; a useful next study would measure both dimensions within the same participants over time.","The voxel grammar is simple enough that Blocks' persistence model could transfer to other persistent AR applications such as collaborative annotations, accessible signage, or navigation aids, where the shared object is not a sculpture but a community-maintained layer of information."],"forward_implications":["Designers who want both positive experience and high participation should consider a configuration that is location-independent at the level of access while keeping synchronous colocated creation as the core activity.","Persistent structures reliably bring users back: repeat-use rates reached 74 percent in the field study and 67 percent in the extended deployment.","Open-ended creation goals increase output, with pairs adding roughly 465 blocks in open-ended sessions versus 260 for robots and 306 for tables.","Asynchronous collaboration changes how people build: it yields more blocks and more colors, less deletion of a partner's work, and more planning time, suggesting it suits independent creative contributions rather than tight coordination.","Persistent public editing creates governance needs, since participants asked future creators not to alter their work and noted that space can become cluttered over time, motivating permissions, moderation, and expiry controls."],"supporting_citations":[{"why":"Supplies the space/time design dimensions the paper uses to structure all studies.","marker":"[8]"},{"why":"Provides the low-floor, high-ceiling, wide-walls design principle behind Blocks' creation tools.","marker":"[42]"},{"why":"Offers the voxel-based building model Blocks adapts for mobile AR.","marker":"[33]"},{"why":"Contributes collaborative-writing outcome metrics and participation-balance analysis methods.","marker":"[36]"},{"why":"Supports using word-count-style measures, here block counts, for collaborative output.","marker":"[31]"},{"why":"Inspires the real-time shared cursors and syncing model for collaboration.","marker":"[13]"},{"why":"Provides the Inclusion of Other in the Self scale used to measure partner closeness.","marker":"[3]"},{"why":"Supplies the thematic analysis method for open-ended survey responses.","marker":"[15]"},{"why":"Motivates the legitimate-peripheral-participation onboarding of new users in the shared world.","marker":"[27]"}],"fun_headline_variants":["AR building: colocated sync tops fun, but free access tops usage","Split in AR co-creation: proximity boosts enjoyment, freedom boosts participation","Blocks: sync and colocation please users; untethered access gets more use","AR collaboration: fun prefers sync and colocation, use prefers anytime, anywhere","Different pulls in AR: colocated sync for fun, untethered for repeat use"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that location-independent structures increase engagement rests on field data in which the location-dependent and location-independent conditions ran on different days and in different physical sites, so day, site, or novelty effects could account for the higher session counts instead of the AR location design.","fun_headline_variants_meta":{"raw":{"variants":["AR building: colocated sync tops fun, but free access tops usage","Split in AR co-creation: proximity boosts enjoyment, freedom boosts participation","Blocks: sync and colocation please users; untethered access gets more use","AR collaboration: fun prefers sync and colocation, use prefers anytime, anywhere","Different pulls in AR: colocated sync for fun, untethered for repeat use"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000842,"raw_usage":{"total_tokens":3589,"prompt_tokens":789,"completion_tokens":2800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":405,"completion_tokens_details":{"reasoning_tokens":2698}},"tokens_in":405,"tokens_out":2800,"duration_ms":17745,"temperature":1.0,"reasoning_tokens":2698,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:44:16.734220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the field study as a within-site crossover that alternates location-dependent and location-independent conditions on the same day and same poster, then compare sessions per user and repeat-use rates; if the advantage of location independence disappears or tracks the day or site, the paper's central design recommendation is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the space/time design dimensions the paper uses to structure all studies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers the voxel-based building model Blocks adapts for mobile AR."},{"cited_title":"Olson, Dakuo Wang, Gary M","cited_arxiv_id":null,"evidence_quote":"Contributes collaborative-writing outcome metrics and participation-balance analysis methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports using word-count-style measures, here block counts, for collaborative output."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the real-time shared cursors and syncing model for collaboration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the thematic analysis method for open-ended survey responses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the legitimate-peripheral-participation onboarding of new users in the shared world."}],"review_version":1}