{"id":"90b4a2e0-b1f8-47eb-8c9c-a76443ec023b","arxiv_id":"2505.24102","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Social and organizational barriers, from stakeholder disagreement to expiring grants, explain why civic visual analytics tools rarely survive past the prototype stage.","lead":"Digital tools built to help towns gather and understand public input often fail to survive beyond their pilot phase. Drawing on a decade of real deployments, this paper names the social and organizational reasons and what to do about them.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The deployments reported are short-term pilots, not long-term integrations, so the conclusion's causal attribution to sociotechnical barriers is under-supported.","rationale":"The reader's weakest_assumption concerns external validity: whether two US municipalities support generalization to the civic domain. I identify a more direct internal problem: the evidence is not about long-term integration at all. The strongest_claim requires that the authors experienced sustained-adoption attempts and could observe why they failed. Their deployments lasted days to months (Sec 2.2), and the paper never reports the termination decision or any follow-up. The paper is honest about this gap in §3.2.4, where it calls for longitudinal studies. Thus the central claim is a plausible hypothesis rather than a finding. This does not change the verdict from CONDITIONAL—the paper is explicitly a reflection/position piece, and its recommendations are reasonable—but it sharpens the condition: external longitudinal evidence is needed not merely for generalization; it is needed to confirm the causal attribution itself. I credit the authors for not hiding the absence of process data and for grounding the discussed systems in peer-reviewed publications, which makes the gap a limitation of the argument, not a sign of bad faith.","tokens_in":10880,"tokens_out":3225,"duration_ms":37620,"concrete_test":"Conduct a structured post-mortem with the municipal partners (Amherst town officials and San Diego planning group) using a fixed coding scheme that assigns each project's non-continuation to categories: funding end, developer availability, technical maintenance, workflow mismatch, organizational resistance, stakeholder misalignment, and other. If the majority of projects are coded as funding/technical with no mention of organizational hierarchy or resistance, the paper's central claim would be substantially weakened. If the partners independently name workflow integration and hierarchy as primary, the claim gains internal support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 describes deployments that are short and episodic: CommunityClick used four times in 2018-2019, CommunityClick-Virtual in 2021, CommunityCrit for four weeks in 2017, and CommunityPulse in 2019. No period of continued use is reported, no termination decisions are documented, and no process data explain why each project ended. The Conclusion states: 'Through our experiences, we have identified key barriers, including organizational hierarchies, stakeholder misalignment, and resistance to new technologies, that hinder sustained adoption even when participatory design methods are employed.' This inference assumes that discontinuation was caused by sociotechnical barriers, but the paper provides no evidence on that causal question. Section 3.2.4 explicitly concedes that longitudinal studies are needed, acknowledging that the authors' own data do not cover long-term integration. Alternative causes—grant termination, loss of developer staffing, technical debt, shifting municipal priorities—are mentioned in §3.2.2 but are not weighed against the stated barriers. Without such a comparison, the central claim that sociotechnical factors dominate is an interpretation rather than an observed finding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a position/reflection piece from two authors who have designed, developed, and deployed visual analytics systems for civic engagement over roughly a decade. It summarizes four deployments: CommunityClick and CommunityClick-Virtual in Amherst, Massachusetts, and CommunityCrit and CommunityPulse in San Diego, California. The paper organizes reported challenges into design challenges (cross-sector partnerships, mismatched stakeholder needs, equity and inclusivity, explainable AI) and adoption challenges (visual literacy, infrastructure and funding, workflow integration, technology and AI concerns, data privacy). It concludes that sociotechnical barriers—organizational hierarchies, stakeholder misalignment, and resistance to new technology—hinder sustained adoption even when participatory design is used, and it calls for deeper integration strategies, equitable engagement, and sustainable implementation frameworks.","tokens_in":10914,"tokens_out":2772,"duration_ms":31903,"significance":"If the paper's central claim is accepted, its value is as an experience-based synthesis: it names concrete, recurring barriers in civic visual analytics and offers a set of practical directions for future research. The paper is honest that its evidence comes from the authors' own deployments, and it grounds many specific challenges in prior peer-reviewed deployment papers (e.g., JKW21, JHSM21, MJN18), which is a real strength. Its recommendations—complementary rather than replacement integration, visual literacy education, hybrid digital/analog engagement, and longitudinal study—are sensible and actionable. However, the paper's title and conclusion promise findings about long-term integration, and the evidence reported here is episodic and short-term; the causal weight placed on sociotechnical barriers over alternative explanations is not directly supported by the presented data.","major_comments":[{"comment":"The Conclusion states that 'through our experiences, we have identified key barriers, including organizational hierarchies, stakeholder misalignment, and resistance to new technologies, that hinder sustained adoption,' but the deployment evidence in §2.2 consists of CommunityClick used four times in 2018–2019, CommunityClick-Virtual in 2021, CommunityCrit for four weeks in 2017, and CommunityPulse in 2019. No period of continued use, no termination decisions, and no process data are reported, so the causal attribution to sociotechnical barriers is under-supported. Section 3.2.2 itself acknowledges alternative causes such as grant termination, outdated technology, and lack of ongoing funding; these are not weighed against the stated sociotechnical barriers. I recommend reframing the conclusion as hypotheses or lessons learned from the authors' experience, or adding comparative or longitudinal evidence that distinguishes among these causes.","section":"§2.2 and §4 (Conclusion)"},{"comment":"The paper explicitly concedes, in the final sentence of §3.2.4, that 'it is imperative to conduct longitudinal studies to identify nuanced challenges and potential solutions that can address long-term adoption challenges.' This concession directly undercuts the paper's title and its central claim about long-term integration. The deployment record in §2.2 documents pilots, not sustained integration. The authors should either narrow the paper's scope to early-stage adoption lessons or present new longitudinal data; as written, the title overclaims what the body supports.","section":"§3.2.4"},{"comment":"The abstract and discussion present the barriers and recommendations as applying to 'civic contexts' generally, but all reported deployments occurred in two US municipalities (Amherst, Massachusetts, and San Diego, California) with volunteer participants. The paper does not discuss how these cases might be unrepresentative of other civic settings, other countries, or non-volunteer participation structures. This generalization is a load-bearing premise for the paper's synthesis; I recommend adding a limitations paragraph or comparative examples to justify the transferability of the barrier list and recommendations.","section":"Abstract and §3"}],"minor_comments":[{"comment":"The sentence 'we draw from over a decade of individual and collaborative experiences on designing, developing, and deploying building visual analytics systems' contains a redundant word, 'building,' and later the text says 'addressing these challenge' where the plural 'challenges' is needed.","section":"§1 (Introduction)"},{"comment":"The UD Co-Spaces system is introduced as designed and developed, but §2.2 does not report any deployment of it; its role in supporting the deployment-based claims is unclear and should be clarified or omitted.","section":"§2.1"},{"comment":"Reference [JSM] is missing a publication year and venue, and the reference list entry is incomplete; several other references, such as [pol19], also do not follow the consistent formatting used elsewhere.","section":"References"},{"comment":"The statement that differential privacy is 'widely considered a gold standard' is presented without an explicit citation; a supporting reference would strengthen this claim.","section":"§3.2.5"},{"comment":"The claim that introducing novel systems 'can be detrimental' to established workflows is general; grounding this in the specific Amherst or San Diego experiences would make the argument more concrete and persuasive.","section":"§3.2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly a workshop position paper, and I have evaluated it on that basis. My major concerns are about the mismatch between the title/conclusion and the short-term, two-municipality evidence base. If the editors consider the position-paper format sufficient, a revision that reframes the claims as experience-based hypotheses and adds a limitations paragraph would address most of my concerns. I do not see signs of fabrication or misleading reporting of the deployments themselves."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reflective position paper from authors who have genuinely deployed civic visual analytics systems, and that experience is the paper's value. But the conclusion's causal claim—that sociotechnical barriers prevented sustained adoption—goes beyond what the reported deployments can support.\n\nWhat's new: not the individual barriers. Each one appears in the cited literature, including the authors' own earlier papers (MNC19, JKW21, MJN18, JHSM21). The contribution is the consolidation and the concrete grounding. The design/adoption split works, and the sections on equity and accessibility, and on integrating systems as complements rather than replacements, are practical and specific. The paper ships no artifacts, but it rests on peer-reviewed deployments, so the experience base is real.\n\nWhere it gets soft: the deployment record is short and episodic. CommunityClick was used four times in 2018-2019; CommunityClick-Virtual in 2021; CommunityCrit ran four weeks in 2017; CommunityPulse in 2019. There is no report of continued use, no termination data, no process evidence about why each project ended. So the conclusion's framing ('we have identified key barriers... that hinder sustained adoption') turns an interpretation into a finding. Alternative causes—grant cycles, developer attrition, technical debt, shifting municipal priorities—are mentioned in §3.2.2 but not weighed against the listed barriers. To its credit, §3.2.4 explicitly calls for longitudinal studies, which is an admission that the current data don't answer the long-term question. That tension should be resolved by softening the conclusion, not by adding more speculation.\n\nAlso, the paper generalizes from two US municipalities with volunteer participants. That's fine for a workshop paper if framed as experience, but the abstract and conclusion state claims about 'the civic domain' broadly. A couple of typos ('deploying building visual analytics systems', 'this challenge') should be fixed.\n\nBottom line: as a workshop position paper this deserves peer review and publication with minor revisions. As a generalizable empirical claim it overreaches. I'd send it to referees and ask them to push for a clear separation between reported experience and hypotheses.","headline":"A useful reflective synthesis from real deployments that overreaches when it turns 'our barriers' into 'the barriers'—still worth refereeing.","tokens_in":11602,"tokens_out":3442,"would_cite":true,"duration_ms":31790,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A decade of civic visual analytics deployments shows sustained adoption is blocked less by technical limits than by organizational hierarchies, stakeholder misalignment, and resistance to new technology — even when the public co-designs…","keywords":["civic visual analytics","long-term adoption","sociotechnical barriers","participatory design","community engagement","workflow integration","digital divide","explainable AI"],"falsifier":"Track a cohort of civic visual analytics deployments over five years, recording whether each is co-owned by the municipality, budgeted beyond the grant, and integrated into official workflows. If systems with strong official ownership and workflow integration survive at rates no higher than grant-funded research pilots, the paper's claim that these sociotechnical factors decide sustained adoption would be contradicted.","tokens_in":10513,"feed_emoji":"🏛️","tokens_out":4754,"duration_ms":45403,"temperature":0.7,"pith_summary":"The paper argues that the reason civic visual analytics systems rarely outlive their research pilot is primarily sociotechnical, not technical: organizational hierarchies, misaligned goals across stakeholders, and resistance to new technology undermine sustained adoption even when the public participates in designing the systems. Its evidence is a decade of real deployments, including town hall feedback tools in Amherst, Massachusetts, and a street-redesign consultation in San Diego, California. If the authors are right, lasting civic technology depends on embedding visual analytics as a complement to official workflows rather than a replacement, on equitable engagement that reaches underserved residents, and on implementation and funding frameworks that extend beyond grant cycles. The value of the claim is that it redirects design effort from the dashboard itself to the organizational conditions around it.","feed_headline":"People, not pixels, kill civic data tools","feed_subtitle":"A decade of town hall deployments shows participatory design alone can't keep civic visual analytics alive.","key_machinery":"The load-bearing mechanism is the authors' collective deployment history, which functions as a longitudinal field record. CommunityClick augmented physical town hall meetings with anonymous iClicker voting aggregated into a visual analytics dashboard; CommunityClick-Virtual extended this to online and hybrid meetings with multimodal feedback; CommunityCrit gathered public input on a San Diego street redesign via microtasks; and CommunityPulse provided the visual analysis layer over that input. These deployments ground the paper's two-level taxonomy of barriers, design and adoption, and motivate its central remedy: position visual analytics as a complementary, auxiliary layer inside existing civic decision workflows so officials retain agency and can adopt the technology at their own pace.","core_discovery":"The central claim is that the transition from research prototype to long-term civic tool fails on social and organizational ground even when participatory design methods are used. The authors draw on three systems they built and deployed — CommunityClick and CommunityClick-Virtual for real-time anonymous feedback in physical and virtual town hall meetings, and CommunityCrit with CommunityPulse for collecting and analyzing public input on urban design — and identify barriers at two levels. Design challenges include building cross-sector partnerships, reconciling the conflicting priorities of policymakers, analysts, and the public, ensuring equity and digital access, and embedding explainable AI without overwhelming non-experts. Adoption challenges include low visual literacy, the absence of infrastructure and ongoing funding, disruption of established civic workflows, public concern about technology and AI, and data privacy. Because even co-designed, field-tested systems did not achieve sustained use, the authors conclude that integration strategies, equitable stakeholder engagement, and sustainable implementation frameworks, not further technical refinement, are what bridge research and practice.","pith_inferences":["A testable implication the authors do not state: deployments where a public agency co-owns and budgets for the system should show markedly higher survival rates than grant-funded pilots launched by external research teams.","The same sociotechnical lens plausibly extends beyond civic spaces to other institutional settings — journalism, public health, open-data offices — where stakeholders have conflicting goals and entrenched workflows.","The paper's evidence implies that training policymakers in visual literacy may matter as much as any visualization feature, an intervention that could be studied as its own design material.","If the pattern holds, evaluation criteria for civic visual analytics should shift from prototype usability metrics to multi-year adoption metrics, such as whether the tool is still in use after the research team departs."],"forward_implications":["Participatory design alone does not guarantee that a civic visual analytics system will be adopted; organizational buy-in and workflow fit decide its fate.","Civic visual analytics should be introduced as a complement or fail-safe to existing processes, letting officials keep their established methods while gradually experimenting.","Without funding models that outlast the research grant — via open sourcing, agency ownership, or recurring municipal budgets — systems decay as technology ages and needs change.","Equity requires low-technology access routes, such as offline companion support, QR-free sharing, and designs that accommodate low digital literacy, disability, and language differences.","Longitudinal studies are needed to see whether AI-enhanced visual analytics erode or rebuild public trust in democratic processes over time."],"supporting_citations":[{"why":"Documents the CommunityClick deployments in Amherst town halls and the resistance and workflow concerns they surfaced; primary evidence for adoption barriers.","marker":"[JKW∗21]"},{"why":"Describes CommunityClick-Virtual, adopted by the Town of Amherst for hybrid meetings; source of equity and access observations about QR codes and multimodal access.","marker":"[JSM]"},{"why":"Describes CommunityCrit, the San Diego microtask engagement whose input fed CommunityPulse; one of the two field cases.","marker":"[MJN∗18]"},{"why":"Presents CommunityPulse, the visual analytics layer for community input; evidence for stakeholder priority mismatches and visual literacy limits.","marker":"[JHSM21]"},{"why":"The civic data deluge study; establishes the underlying complexity of analyzing large-scale community input that motivates the systems.","marker":"[MNC∗19]"},{"why":"Provides the open civic design framework the paper draws on for structured participation and sustainability arguments.","marker":"[RDM21]"},{"why":"Supplies the wicked-problems framing used to characterize why civic technology design resists simple solutions.","marker":"[RW73]"}],"fun_headline_variants":["Even co-design can't keep civic tools alive","Social barriers outlast civic data prototypes","Civic analytics fail on people, not pixels","Participatory design isn't enough for civic tech","Long-term civic analytics need more than usability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions rest on the assumption that a handful of volunteer deployments in two US municipalities — town hall meetings in Amherst, Massachusetts, and a street-redesign consultation in San Diego, California — represent the civic domain broadly enough for the barriers and remedies to transfer elsewhere.","fun_headline_variants_meta":{"raw":{"variants":["Even co-design can't keep civic tools alive","Social barriers outlast civic data prototypes","Civic analytics fail on people, not pixels","Participatory design isn't enough for civic tech","Long-term civic analytics need more than usability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1157,"prompt_tokens":827,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":261}},"tokens_in":443,"tokens_out":330,"duration_ms":4266,"temperature":1.0,"reasoning_tokens":261,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:36:14.411959+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track a cohort of civic visual analytics deployments over five years, recording whether each is co-owned by the municipality, budgeted beyond the grant, and integrated into official workflows. If systems with strong official ownership and workflow integration survive at rates no higher than grant-funded research pilots, the paper's claim that these sociotechnical factors decide sustained adoption would be contradicted.","supporting_citations":[],"review_version":1}