Pith. sign in

REVIEW 4 major objections 6 minor 74 references

Revibing Code from Papers: Reimplementing HCI Artifacts

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Agentic coding tools can rebuild interactive HCI systems directly from their published papers, reaching a mean best-run revibeability of 94 percent across ten systems.

desk verdict The qualitative claim is solid; the 94% headline is best-of-three and measures fidelity to the paper text, not to the original artifact. read the letter →

arxiv 2608.00450 v2 pith:TBVTMAIH submitted 2026-08-01 cs.HC cs.SE

Eytan Adar , Yoonjoo Lee , Nina Lei , Q. Vera Liao , Weirui Peng This is my paper · ORCID
classification cs.HCcs.SE
keywords ReimplementationOpenScienceVibe-codingRevibeabilityAgenticprogrammingHCIevaluationStrongbaselineReproducibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that modern agentic coding tools can reimplement interactive software systems directly from their published papers, with no access to the original code, and that the rebuilt systems can be good enough to serve as strong baselines in A/B evaluations. The authors define 'revibing' as building a working system from the paper text alone, and 'revibeability' as a scored measure of how well the rebuilt system passes a generated test rubric. Across ten recent technical HCI papers with no public implementations, they report a mean best-run revibeability of 0.939 (minimum 0.8, maximum 1.0) after two rounds of feedback, with most runs completing in under thirty minutes and under fifteen dollars. The significance is that the shortage of released code, long a barrier to replication and to fair baseline comparison, could become far less binding if paper text is enough to reconstruct the artifact.

What carries the argument

The load-bearing mechanism is the automatically generated, system-specific test rubric, produced by an agentic coding tool from the paper's markdown text and figures. Each rubric test specifies the steps a user should perform, what should happen, and the difference between full success, partial success, and failure. The rubric does three jobs: it defines what counts as a faithful reimplementation, it gives a human tester a uniform instrument for grading the rebuilt system, and it serves as a correction signal when the graded rubric is fed back to the agentic tool with a request to fix reported issues. The revibeability metric is then the weighted ratio of successful tests described in the paper.

What would settle it

Have the original authors of the seven non-interviewed systems independently grade the revibed systems against a rubric they write themselves; if the mean score falls substantially below 0.94, the self-generated rubric overstates fidelity.

Watch

Extended reading notes

Core claim

The central claim is that the practical floor for reimplementing interactive HCI systems, set by vibe-coding directly from a paper, is high enough for strong-baseline use. The paper introduces the verb 'revibe' for such a reimplementation and the metric 'revibeability,' computed as a weighted score over a system-specific rubric of manual interaction tests, with weight 2 for full success, 1 for partial success, and 0 for failure. In a study of ten systems drawn from recent technical HCI papers that lacked public code, the authors report that after an unaided build and two aided revision cycles, the best run per system averaged 0.939 revibeability, with two systems reaching 1.0 and a minimum of 0.8. Scores were monotonic across runs in almost all cases, and the few decreases were all below 0.02. Author interviews with three of the original teams broadly confirmed that the rebuilt systems captured the intended interactive features, but also exposed gaps such as missed drag-and-drop behavior and loss of notebook-native integration in one system.

Load-bearing premise

The assumption that the automatically generated revibeability rubric captures the key interactive features the original authors intended; the paper validates this against author interviews for only three of the ten systems.

Editorial extensions

If this is right

  • Technical HCI papers that do not release code could still be compared against in A/B evaluations, because revibed artifacts can serve as strong baselines.
  • A revibeability score could become a publishable badge or reviewing signal, giving authors a reason to check that their paper text is sufficient for reimplementation.
  • Authors could write papers with reimplementation in mind, providing interaction rubrics or more detailed feature descriptions, lowering the cost of reproduction for everyone.
  • The same rubric could be reused over time to benchmark improvements in agentic coding tools, turning specific HCI systems into long-lived reimplementation benchmarks.
  • If revibing becomes routine, the community may expect baseline comparisons more often, and reviewers may need norms to avoid rejecting non-revibeable work for the wrong reasons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 94% figure is a best-run average and masks substantial variance: one of the three agentic environments averaged only 0.67 mean revibeability, so the floor is not uniform across tools.
  • Because the rubric is generated by the same model family that builds the system, the scores may contain a self-confirmation component; using a different model or original authors to write the rubric would test whether the 94% holds.
  • The paper samples papers from 2020 to 2024 that mostly target web or notebook interfaces; older, hardware-linked, or thick-client systems may not revibe nearly as well, so the estimate should not be extrapolated to all technical HCI artifacts.
  • A direct testable extension is to have the original authors of all ten systems grade the revibes against their own intended features; if the gap between author grades and rubric grades is large, the metric needs to be recalibrated before it is used as a baseline-comparison badge.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces "revibing"—reimplementing interactive HCI systems directly from paper text using agentic coding tools—and proposes a "revibeability" metric based on a system-specific test rubric generated from the paper. Ten UIST systems are reimplemented with Claude Code, Cursor, and Gemini; after two aided feedback rounds, the authors report a mean best-run revibeability of 0.939 (min 0.8, max 1.0). The paper also interviews the original authors of three systems and analyzes rubric construction. The authors argue that revibed artifacts can serve as strong baselines in A/B evaluations and that this capability may change how technical HCI artifacts are produced and evaluated.

Significance. If the metric and the empirical claims were validated, the paper would address a real problem: most HCI artifacts are unavailable, and reimplementation is usually impractical. The pipeline is concrete, the prompts are reported in detail, and the qualitative failure analysis (over-implementation, environment dependencies, rubric ambiguities) is thoughtful and useful. The interviews with original authors are a valuable addition. However, the central quantitative claim is not yet established: the headline score is a best-run statistic, the rubric instrument has unaddressed validity and reliability concerns, and internal evidence shows that scores depend strongly on the rubric generator. The contribution is promising but currently stronger as a qualitative feasibility study than as a validated metric.

major comments (4)
  1. [Abstract; Section 6] The headline "94% revibeability" is the mean of the best run per system (0.939), not the mean performance of a typical revibe. Per-agent means are Cursor 0.912, Claude 0.87, and Gemini 0.67, so the number describes a best-of-three selection. Since the abstract and Section 7.2 use this number to argue that revibed systems can serve as strong baselines, the claim overstates what a researcher should expect from a single agentic run; the paper should report per-agent averages prominently and either justify the best-run convention or reframe the claim to describe an upper-bound capability.
  2. [Sections 4.2, 6.1, 6.2] The construct validity of the revibeability instrument is not established. The rubric is generated by Claude Code and graded by the authors, and author interviews were conducted for only 3 of 10 systems. Those interviews reveal concrete failures: no revibe fully implemented mage's drag-and-drop, the B2 author said the revibes would not provide the intended experience because reactive cells relied on Jupyter infrastructure, and the Spellburst author recalled an important internal feature not described in the paper. Thus revibeability measures fidelity to the paper text as interpreted by Claude Code, not author-intended functionality or suitability as a strong baseline. The paper should either validate rubrics against original authors for substantially more systems or explicitly narrow the definition and all downstream claims.
  3. [Section 6.2; Appendix A.4] The metric is not invariant to the rubric generator. Claude-produced rubrics average 35.1 tests versus Gemini's 10.25, and Figure 18 shows best-run score correlates r=-0.74 with rubric length. The bootstrap in Figure 19 resamples within a single rubric and therefore cannot bound this cross-instrument variance. Because the same model family (Claude) generates the rubrics and many of the revibes, a rubric biased toward easily specifiable or agent-suggested features would inflate scores for all tools; the paper needs sensitivity analyses across rubric generators or evidence that different rubrics yield similar rankings before claiming that revibeability is a stable property of the paper-agent pair.
  4. [Sections 5 and 6] All rubric grading was performed by the authors without inter-rater calibration, blind assessment, or reported reliability statistics. Since the paper proposes revibeability as a reusable measurement for the community, at least a small inter-rater reliability study (e.g., two raters independently scoring a subset of tests) is needed to demonstrate that the scores are not idiosyncratic to the authors' interpretations of success, partial success, and failure.
minor comments (6)
  1. [Abstract] The abstract contains the sentence fragment "The results are encouraging. In many cases producing code suitable for strong baseline use."; this should be rewritten as one or more complete sentences.
  2. [Section 4.2] The claim that Claude rubrics are "most detailed" is supported only by test count and file size; reporting a qualitative measure of whether the additional tests cover key features would make the argument stronger.
  3. [Sections 4.3 and 4.4] The notation for graded rubrics alternates between "rubric 1" and "rubric1"; please use a consistent convention.
  4. [Section 5] The system selection description says the final 10 were "selected roughly randomly" but then states that non-LLM-centered systems were oversampled; the algorithm for choosing the 10 from the 20 should be clarified.
  5. [Appendix A.4] Figure 19's bootstrap confidence intervals are plotted but not interpreted in the text; a sentence summarizing what the intervals imply for the stability of the scores would help readers.
  6. [Section 6.1] In the mage discussion, the quoted author comment contains an ellipsis and stray formatting that should be cleaned up.

Circularity Check

2 steps flagged · score 5.0 of 10

The 94% aided revibeability figure is partly forced by a rubric-feedback loop, and the rubric is generated from the same paper text used to build the revibes.

  1. fitted input called prediction [Section 4.4 (Aided Revibe), Section 6 (headline result), Equation (1) in Section 3]
    "The graded rubric was provided back to the agentic tool to perform an aided revibe. ... The specific prompt was: “I have completed an evaluation of the system and tested its features. Please fix any reported issues in the attached rubric.” ... With a fixed number of these feedback prompts (N=2), we demonstrate that we can achieve 94% revibeability (mean, 80% min, and 100% max) on the tested systems."

    Revibeability is defined by the rubric tests in Section 3. The aided protocol hands the graded rubric, with all failures marked, back to the agent as repair instructions, then re-scores the result with the same rubric. The final score is therefore a measure of how well the agent complied with the very instrument used to grade it. The 94% figure is an optimization outcome relative to the evaluation criteria, not an independent estimate of fidelity to the original system. The high aided score is largely forced by the feedback loop by construction.

  2. self definitional [Section 4.2 (Rubric Development) and Section 6.2 (Rubric Evaluation)]
    "Although we evaluated the generation of rubrics with multiple models, we ultimately opted to use rubrics produced by Claude Code (Sonnet 4.6). This produced the most detailed rubric (e.g., 38 tests in one case produced by Claude versus 6 produced by Gemini). ... One possible complication in our approach is that it might introduce a confound: Claude Code may identify the same set of interactive features in the paper when creating the rubric as when it produces the revibe. This may lead to it performing better because it also generated the test."

    The rubric and the revibes are both generated from the same paper text, and for a plurality of revibes, by the same model family (Claude). The rubric prompt defines success as meeting “the description of the system as intended by the authors,” but the only source of that description is the paper, which is also the sole input to the revibe agent. A high score therefore confirms consistency between two outputs of one source and one model family, not verification against an external artifact. The metric cannot detect features the paper omits, as the Spellburst author interview illustrated.

full rationale

The paper's core idea, reimplementing HCI systems from paper text, is not circular in the derivation sense: the rubric is an externalized specification, and the revibe code must actually run and pass interactive checks. However, the headline 94% figure is partly self-confirmatory. The aided loop feeds the graded rubric, including exact failures, back to the agent and then re-grades with the same rubric, so the final score measures compliance with the evaluation instrument rather than an independent property of the artifact. A second self-referential element is that both rubric and revibe derive from the same paper text, with Claude Code generating the rubrics and also producing several of the highest-scoring revibes; the paper itself flags this confound in Section 6.2. The rebuttal that Cursor scores well on Claude rubrics is informative but does not resolve the shared-source issue, and the Spellburst author interview shows that features absent from the paper are invisible to the metric. The r=-0.74 correlation between rubric length and maximum score further shows that revibeability is not invariant to how the rubric is constructed; the bootstrap in Appendix A.4 resamples within a single rubric and cannot bound this cross-instrument variance. At the same time, the paper is transparent about the aided protocol, reports unaided scores alongside aided ones, and includes three author interviews, which limit but do not eliminate the circularity. No load-bearing self-citation chain or imported uniqueness theorem is present; the only author-overlapping reference, PromptPaint in the rubric corpus, is not load-bearing. Score 5 reflects partial circularity in the central claim rather than a fully tautological derivation.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical entities are postulated. The 'revibe' and 'revibeability' are new concepts, not entities; the only hand-chosen constants are the metric weights.

free parameters (1)
  • revibeability weights = w_s=2, w_p=1, w_f=0
    Hand-chosen weights in the aggregated revibeability metric (Section 3). The score is not fitted to data, but the numerical results depend on this choice.
assumptions (4)
  • domain assumption The LLM-generated rubric identifies all key interactive features of the original artifact
    The whole evaluation rests on the rubric. Only 3 of 10 systems were checked against original authors, and the rubric generator is the same model family as one builder (Section 4.2, 6.2).
  • domain assumption The paper text and figures contain sufficient information to reimplement the system
    This is the revibe premise; the authors note that ambiguous screenshots and undocumented prompts create failures (Section 6.1).
  • domain assumption The assessors' grading of success, partial success, and failure is reliable
    The authors state no calibration was performed and only one grader per system was used (Section 5).
  • domain assumption The three agentic tools with default models represent the current state of agentic programming
    Results vary strongly across tools (means 0.67 to 0.91), so generalization to other tools or future models is uncertain (Section 6).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revibing Code from Papers: Reimplementing HCI Artifacts." pith.science (2026). https://pith.science/paper/TBVTMAIH

@misc{pith2026260800450,
  author       = {Pith},
  title        = {Pith review of: Revibing Code from Papers: Reimplementing HCI Artifacts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TBVTMAIH}},
  note         = {Machine review of arXiv:2608.00450}
}
read the original abstract

Software artifacts for most technical HCI research projects are unavailable. The lack of access to these imposes limits on academic knowledge production. It is difficult to: extend or reuse research artifacts; use strong baselines in evaluating follow-up work; and perform replication or reproducibility research. In this work, we demonstrate the potential of new agentic AI technologies to revibe interactive software: reimplement systems directly from research papers. To measure the success of the approach, we describe a revibeability metric. By revibing recent research papers from UIST, and interviewing their original authors, we demonstrate the plausibility (and limitations) of revibed system. The results are encouraging. In many cases producing code suitable for strong baseline use. We argue that this may represent a fundamental shift in how we produce, use, and evaluate research artifacts in the technical HCI community.

Figures

Figures reproduced from arXiv: 2608.00450 by the authors.

Figure 1
Figure 1. Screenshots of two revibed interfaces (Spellburst [1] and Rescribe [42]) produced by three different agentic environments (Claude Code, Cursor, and Gemini) Abstract Software artifacts for most technical HCI research projects are un￾available. The lack of access to these imposes limits on academic knowledge production. It is difficult to: extend or reuse research artifacts; use strong baselines in evaluating follow-u… view at source ↗
Figure 2
Figure 2. A figure of the revibe procedure implemented in this paper. The system ‘deconstructs’ the paper into markdown and figures (1). From this, we construct a revibeability rubric for the paper using an automated prompt (2) and create an initial implementation (3). Debugging and feature checks (4 & 5) produce a working implementation. This can then be graded (6) through the rubric. Repeatedly grading and asking the agenti… view at source ↗
Figure 4
Figure 4. Revibeability performance for Spellburst. Red/Yel￾low/Blue correspond to failure, partial success, and full suc￾cess fo each test (respectively) U A1 A2 Cla u d e U A1 A2 C ursor TL-01 TL-02 TL-03 TL-04 TL-05 TL-06 TR-01 TR-02 TR-03 TR-04 TR-05 DP-01 DP-02 DP-03 DP-04 DP-05 DP-06 DP-07 DP-08 DP-09 RC-01 RC-02 RN-01 RN-02 RN-03 RN-04 RN-05 PR-01 PR-02 PR-03 PR-04 PR-05 CI-01 CI-02 CI-03 EC-01 EC-02 EC-03 EC-04 EC-05 … view at source ↗
Figures from the paper (13 more)
Figure 3
Figure 3. Figure 3: Summary of revibeability scores in our experiments (calculated as the aggregated revibeability score). The color of the lines/dots represents the vibe-coding environment. The size of the dots, from smallest to largest, indicates the revibeability score for each run (af…
Figure 6
Figure 6. Figure 6: Revibeability performance for mage. U A1 A2 Cla u d e U A1 A2 C ursor T-01 T-02 T-03 T-04 T-05 T-06 T-07 T-08 T-09 T-10 T-11 T-12 T-13 T-14 T-15 T-16 T-17 T-18 T-19 T-20 T-21 T-22 T-23 T-24 T-25 T-26 U A1 A2 G e mini GenAssistrubric outcomes [PITH_FULL_IMAGE:figures/f…
Figure 7
Figure 7. Figure 7: Revibeability performance for GenAssist. Appendix). Each row represents a graded rubric (columns are indi￾vidual tests). A red cell indicates a failure, yellow is partial success, and blue is full success. The results are grouped by agentic system, with the first row s…
Figure 8
Figure 8. Figure 8: A comparison of revibeability rubric test counts and file size. Each dot represents a system. the number of tests in each rubric, the number of steps in each test, and the length of the rubric (as a proxy for detail) [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Revibeability performance for CoLadder. U A1 A2 Cla u d e T-1.1 T-1.2 T-1.3 T-2.1 T-2.2 T-2.3 T-2.4 T-3.1 T-3.2 T-3.3 T-3.4 T-3.5 T-3.6 T-3.7 T-4.1 T-4.2 T-4.3 T-4.4 T-5.1 T-5.2 T-5.3 T-6.1 T-6.2 T-7.1 T-7.2 T-8.1 T-9.1 T-10.1 T-10.2 T-10.3 T-11.1 T-11.2 T-11.3 T-12.1 …
Figure 10
Figure 10. Figure 10: Revibeability performance for B2 U A1 A2 Cla u d e TEST-01 TEST-02 TEST-03 TEST-04 TEST-05 TEST-06 TEST-07 TEST-08 TEST-09 TEST-10 TEST-11 TEST-12 TEST-13 TEST-14 TEST-15 TEST-16 TEST-17 TEST-18 TEST-19 TEST-20 TEST-21 TEST-22 TEST-23 TEST-24 TEST-25 TEST-26 TEST-27 T…
Figure 11
Figure 11. Figure 11: Revibeability performance for EvalGen U A1 A2 Cla u d e U A1 A2 C ursor CHART-001 CHART-002 CHART-003 TREE-001 TREE-002 TREE-003 TREE-004 TREE-005 TREE-006 TABLE-001 TABLE-002 TABLE-003 TABLE-004 QA-COLD-001 QA-COLD-002 QA-ANA-001 QA-ANA-002 QA-ANA-003 QA-VIS-001 QA-V…
Figure 12
Figure 12. Figure 12: Revibeability performance Vizability [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Revibeability performance for Sketch-n-Sketch. U A1 A2 Cla u d e TEST-001 TEST-002 TEST-003 TEST-004 TEST-005 TEST-006 TEST-007 TEST-008 TEST-009 TEST-010 TEST-011 TEST-012 TEST-013 TEST-014 TEST-015 TEST-016 TEST-017 TEST-018 TEST-019 TEST-020 TEST-021 U A1 A2 G e mi…
Figure 14
Figure 14. Figure 14: Revibeability performance for XCreation T-1.1 T-1.2 T-1.3 T-2.1 T-2.2 T-2.3 T-2.4 T-3.1 T-3.2 T-3.3 T-3.4 T-3.5 T-3.6 T-3.7 T-4.1 T-4.2 T-4.3 T-4.4 T-5.1 T-5.2 T-5.3 T-6.1 T-6.2 T-7.1 T-7.2 T-8.1 T-9.1 T-10.1 T-10.2 T-10.3 T-11.1 T-11.2 T-11.3 T-12.1 T-12.2 T-12.3 T-1…
Figure 16
Figure 16. Figure 16: Comparative revibeability performance for [PITH_FULL_IMAGE:figures/full_fig_p014_16.png]
Figure 18
Figure 18. Figure 18: Correlation between best revibeability score and [PITH_FULL_IMAGE:figures/full_fig_p015_18.png]
Figure 19
Figure 19. Figure 19: 95% CI calculated on the last run for each system [PITH_FULL_IMAGE:figures/full_fig_p015_19.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 40 canonical work pages

  1. [2]

    Association for Computing Machinery. 2020. Artifact Review and Badging - Current. Retrieved March 21, 2026 from https://www.acm.org/publications/ policies/artifact-review-and-badging-current

    work page 2020

  2. [3]

    Warriar, and Sebastian Deterding

    Nick Ballou, Vivek R. Warriar, and Sebastian Deterding. 2021. Are You Open? A Content Analysis of Transparency and Openness Guidelines in HCI Journals. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan)(CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 176, 10 pages. doi:10.1145/341176...

    arXiv 2021

  3. [4]

    Tim Baumgärtner and Iryna Gurevych. 2026. SciCoQA: Quality Assurance for Scientific Paper–Code Alignment.arXiv preprint arXiv:2601.12910(2026)

    arXiv 2026

  4. [5]

    Seems Complicated and Unachievable

    Maria Belen Saavedra Rios, Youngsoon Takei, Blade Hicks, Lydia Stamato, Wei Wu, and Jasmine Jones. 2025. "Seems Complicated and Unachievable": A Collab- orative Autoethnography of Design Replication in Undergraduate Research. In Proceedings of the 7th Annual Symposium on HCI Education. 1–11

    work page 2025

  5. [6]

    Drew Breunig. 2026. whenwords: An Open Source Library Without Code. https: //github.com/dbreunig/whenwords

    work page 2026

  6. [8]

    Pengcheng Chen, Ning Yan, Zihan Zhao, Yixiao Lin, Huaibo Chen, Yue Hu, Qinbo Bai, Xiang Li, and Masood S Mortazavi. 2025. Deep-Reproducer: From Paper Understanding to Code Generation. InNeurIPS 2025 Fourth Workshop on Deep Learning for Code. UIST ’26, November 02–05, 2026, Detroit, MI, USA Adar et al

    work page 2025

  7. [11]

    Johanna Cohoon and James Howison. 2021. Norms and open systems in open science.Information & Culture56, 2 (2021), 115–137

    work page 2021

  8. [12]

    Cursor. 2026. Browser. Retrieved March 14, 2026 from https://cursor.com/docs/ agent/tools/browser

    work page 2026

Show all 74 references
  1. [14]

    Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, et al . 2025. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. arXiv preprint arXiv:2512.12730(2025)

    2025

  2. [15]

    Alan Dix. 2010. Human–computer interaction: A stable discipline, a nascent science, and the growth of the long tail.Interacting with computers22, 1 (2010), 13–27

    2010

  3. [16]

    Florian Echtler and Maximilian Häußler. 2018. Open Source, Open Science, and the Replication Crisis in HCI. InExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada)(CHI EA ’18). Association for Computing Machinery, New York, NY...

    2018

  4. [17]

    Nicholas Edwards, Yukyung Lee, Yujun Audrey Mao, Yulu Qin, Sebastian Schuster, and Najoung Kim. 2025. RExBench: Can coding agents autonomously implement AI research extensions?arXiv preprint arXiv:2506.22598(2025)

    2025 arXiv

  5. [18]

    James Fogarty. 2017. Code and contribution in interactive systems research. In Workshop HCITools: Strategies and Best Practices for Designing, Evaluating and Sharing Technical HCI Toolkits at CHI. 1–4

    2017

  6. [19]

    Yuyao Ge, Lingrui Mei, Zenghao Duan, Tianhao Li, Yujia Zheng, Yiwei Wang, Lexin Wang, Jiayu Yao, Tianyu Liu, Yujun Cai, et al. 2025. A survey of vibe coding with large language models.arXiv preprint arXiv:2510.12399(2025)

    2025

  7. [20]

    Google Antigravity. 2026. Browser. Retrieved March 14, 2026 from https: //antigravity.google/docs/browser

    2026

  8. [22]

    Saul Greenberg and Bill Buxton. 2008. Usability evaluation considered harmful (some of the time). InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Florence, Italy)(CHI ’08). Association for Computing Machinery, New York, NY, USA, 111–120. doi:10.114...

    2008

  9. [23]

    Brian Hempel, Justin Lubin, and Ravi Chugh. 2019. Sketch-n-Sketch: Output- Directed Programming for SVG. InProceedings of the 32nd Annual ACM Sym- posium on User Interface Software and Technology(New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, ...

    2019

  10. [24]

    2008.Eating the IT elephant: Moving from greenfield development to brownfield

    Richard Hopkins and Kevin Jenkins. 2008.Eating the IT elephant: Moving from greenfield development to brownfield. Addison-Wesley Professional

    2008

  11. [25]

    Kasper Hornbæk, Søren S Sander, Javier Andrés Bargas-Avila, and Jakob Grue Si- monsen. 2014. Is once enough? On the extent and content of replications in human-computer interaction. InProceedings of the SIGCHI conference on human factors in computing systems. 3523–3532

    2014

  12. [26]

    Chuxuan Hu, Liyun Zhang, Yeji Lim, Aum Wadhwani, Austin Peters, and Daniel Kang. 2025. REPRO-BENCH: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?. InFindings of the Association for Computational Linguistics: ACL 2025. 23616–23626

    2025

  13. [27]

    Tianyu Hua, Harper Hua, Violet Xiang, Benjamin Klieger, Sang T Truong, Weixin Liang, Fan-Yun Sun, and Nick Haber. 2025. Researchcodebench: Benchmarking llms on implementing novel machine learning research code.arXiv preprint arXiv:2506.02314(2025)

    2025 arXiv

  14. [28]

    Ziheng Huang, Sebastian Gutierrez, Hemanth Kamana, and Stephen Macneil

  15. [30]

    Tobias Isenberg. 2024. The state of reproducibility stamps for visualization research papers. In2024 IEEE Evaluation and Beyond-Methodological Approaches for Visualization (BELIV). IEEE, 97–105

    2024

  16. [31]

    Peter Jansen, Samiah Hassan, and Pragnya Narasimha. 2025. CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents.arXiv preprint arXiv:2512.01089(2025)

    2025 arXiv

  17. [32]

    Hyeonsu B Kang, Tongshuang Wu, Joseph Chee Chang, and Aniket Kittur. 2023. Synergi: A Mixed-Initiative System for Scholarly Synthesis and Sensemaking. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology(San Francisco, CA, USA)(UIST ’23). As...

    2023

  18. [35]

    Sam Lau and Philip J Guo. 2025. The Design Space of LLM-Based AI Coding Assistants: An Analysis of 90 Systems in Academia and Industry. In2025 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 300–313

    2025

  19. [36]

    David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation Strategies for HCI Toolkit Research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada)(CHI ’18). Association for ...

    2018

  20. [37]

    Yujie Luo, Zhuoyun Yu, Xuehai Wang, Yuqi Zhu, Ningyu Zhang, Lanning Wei, Lun Du, Da Zheng, and Huajun Chen. 2025. Executable Knowledge Graphs for Replicating AI Research.arXiv preprint arXiv:2510.17795(2025)

    2025 arXiv

  21. [38]

    Khalid Majrashi. 2023. Inter-platform consistency inspection method.Interna- tional Journal of Technology and Human Interaction (IJTHI)19, 1 (2023), 1–20

    2023

  22. [39]

    Chunyu Miao, Henry Peng Zou, Yangning Li, Yankai Chen, Yibo Wang, Fangxin Wang, Yifan Li, Wooseong Yang, Bowei He, Xinni Zhang, et al. 2025. Recode-h: A benchmark for research code development with interactive human feedback. arXiv preprint arXiv:2510.06186(2025)

    2025

  23. [40]

    Dan R. Olsen. 2007. Evaluating user interface systems research. InProceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (Newport, Rhode Island, USA)(UIST ’07). Association for Computing Machinery, New York, NY, USA, 251–258. doi:10.1145/1294211.1294256

    2007

  24. [41]

    Jonas Oppenlaender and Simo Hosio. 2025. Keeping score: A quantitative analysis of how the CHI community appreciates its milestones. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–17

    2025

  25. [43]

    Asrar Qassem, Renee Bryce, and Khalid Alkhaldi. 2025. A Survey on Visual GUI Testing for Automatic Test Case Generation: Tools, AI Techniques, and Emerging Trends (S). 108–113. doi:10.18293/SEKE2025-108

    2025 doi

  26. [44]

    Rudolf Ramler, Georg Buchgeher, and Claus Klammer. 2018. Adapting automated test generation to GUI testing of industry applications.Information and Software Technology93 (2018), 248–263

    2018

  27. [45]

    Kavous Salehzadeh Niksirat, Lahari Goswami, Pooja SB Rao, James Tyler, Alessan- dro Silacci, Sadiq Aliyu, Annika Aebli, Chat Wacharamanotham, and Mauro Cherubini. 2023. Changes in research ethics, openness, and transparency in empirical studies between CHI 2017 and CHI 2022. I...

    2023

  28. [46]

    Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. 2025. Paper2code: Automating code generation from scientific papers in machine learning.arXiv preprint arXiv:2504.17192(2025)

    2025

  29. [47]

    Zamfirescu-Pereira, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo

    Shreya Shankar, J.D. Zamfirescu-Pereira, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo. 2024. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software...

    2024

  30. [48]

    Chenglei Si, Tatsunori Hashimoto, and Diyi Yang. 2025. The ideation-execution gap: Execution outcomes of llm-generated versus human research ideas.arXiv Revibing Code from Papers: Reimplementing HCI Artifacts UIST ’26, November 02–05, 2026, Detroit, MI, USA preprint arXiv:2506...

    2025 arXiv

  31. [49]

    Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. 2024. Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark.arXiv preprint arXiv:2409.11363(2024)

    2024 arXiv

  32. [50]

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. 2025. PaperBench: Evaluating AI’s Ability to Replicate AI Research.arXiv preprint arXiv:2504.01848(2025)

    2025 arXiv

  33. [51]

    Uzma Haque Syeda, Laura South, Justin Raynor, Liudas Panavas, David Saffo, Tommy Morriss, Cody Dunne, and Michelle A Borkin. 2024. Vis repligogy: Towards a culture of facilitating replication studies in visualization pedagogy and research. (2024)

    2024

  34. [52]

    Jean Vanderdonckt and Radu-Daniel Vatavu. 2025. Context is Key for Repro- ducibility of Empirical Studies in Human-Computer Interaction. InProceedings of the 3rd ACM Conference on Reproducibility and Replicability. 41–50

    2025

  35. [53]

    Chat Wacharamanotham, Lukas Eisenring, Steve Haroz, and Florian Echtler. 2020. Transparency of CHI Research Artifacts: Results of a Self-Reported Survey. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA)(CHI ’20). Association f...

    2020

  36. [54]

    Simon Willison. Marc. Not all AI-assisted programming is vibe coding (but vibe coding rocks). Retrieved March 14, 2026 from https://simonwillison.net/2025/ Mar/19/vibe-coding/

    2026

  37. [55]

    Max Wilson, Wendy Mackay, Ed Chi, Michael Bernstein, and Jeffrey Nichols

  38. [56]

    Wilson, Ed H

    Max L. Wilson, Ed H. Chi, Stuart Reeves, and David Coyle. 2014. RepliCHI: the workshop II. InCHI ’14 Extended Abstracts on Human Factors in Computing Systems(Toronto, Ontario, Canada)(CHI EA ’14). Association for Computing Machinery, New York, NY, USA, 33–36. doi:10.1145/25592...

    2014

  39. [57]

    Wilson, Wendy Mackay, Ed Chi, Michael Bernstein, Dan Russell, and Harold Thimbleby

    Max L. Wilson, Wendy Mackay, Ed Chi, Michael Bernstein, Dan Russell, and Harold Thimbleby. 2011. RepliCHI - CHI should be replicating and validating results more: discuss. InCHI ’11 Extended Abstracts on Human Factors in Comput- ing Systems(Vancouver, BC, Canada)(CHI EA ’11). ...

    2011

  40. [58]

    Max L. L. Wilson, Paul Resnick, David Coyle, and Ed H. Chi. 2013. RepliCHI: the workshop. InCHI ’13 Extended Abstracts on Human Factors in Computing Systems (Paris, France)(CHI EA ’13). Association for Computing Machinery, New York, NY, USA, 3159–3162. doi:10.1145/2468356.2479636

    2013

  41. [59]

    Hellerstein, and Arvind Satyanarayan

    Yifan Wu, Joseph M. Hellerstein, and Arvind Satyanarayan. 2020. B2: Bridging Code and Interactive Visualization in Computational Notebooks. InProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA)(UIST ’20). Association for ...

    2020

  42. [60]

    Liwenhan Xie, Chengbo Zheng, Haijun Xia, Huamin Qu, and Chen Zhu-Tian

  43. [61]

    Yiqing Xu and Leo Yang Yang. 2026. Scaling Reproducibility: An AI-Assisted Workflow for Large-Scale Reanalysis.arXiv preprint arXiv:2602.16733(2026)

    2026 arXiv

  44. [62]

    Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, Juntong Ni, George Michalopoulos, et al. 2025. Lmr-bench: Evaluat- ing llm agent’s ability on reproducing language modeling research. InProceedings of the 2025 Conference on Empirical M...

    2025

  45. [63]

    Zihan Yan, Chunxu Yang, Qihao Liang, and Xiang ’Anthony’ Chen. 2023. XCre- ation: A Graph-based Crossmodal Generative Creativity Support Tool. InProceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA)(UIST ’23). Associa...

    2023

  46. [64]

    Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian LV Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J Qu, et al. 2025. ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?arXiv preprint arXiv:2510.24591(2025)

    2025

  47. [65]

    Ryan Yen, Jiawen Stefanie Zhu, Sangho Suh, Haijun Xia, and Jian Zhao. 2024. CoLadder: Manipulating Code Generation via Multi-Level Blocks. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA)(UIST ’24). Association for ...

    2024

  48. [66]

    Xuanle Zhao, Zilin Sang, Yuxuan Li, Qi Shi, Weilun Zhao, Shuo Wang, Duzhen Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025. Autoreproduce: Automatic ai experiment reproduction with paper lineage.arXiv preprint arXiv:2505.20662 (2025)

    2025 arXiv

  49. [67]

    Yuheng Zhao, Xueli Shu, Liwen Fan, Lin Gao, Yu Zhang, and Siming Chen

  50. [72]

    Identify a child sketch node that has a parent node and at least one descendant node

  51. [73]

    Click the`×`close button in the node's header

  52. [74]

    **Expected Result:** The deleted node is removed from the canvas

    Observe: (a) whether the node is removed, (b) what happens to its descendants, and (c) what happens to the edge connecting parent to descendants. **Expected Result:** The deleted node is removed from the canvas. Its descendants are not deleted - they remain. The descendants ar...

    2026

  53. [75]

    Create a root sketch node via`+ New Sketch`

  54. [76]

    Open the code editor and paste in a simple p5.js concentric circles sketch

  55. [77]

    Verify the output renders (concentric circles visible)

  56. [78]

    Verify the change renders instantly

    From the root node, use Duplicate to create a child; open its code editor and change a numeric parameter (e.g., number of circles). Verify the change renders instantly

  57. [79]

    Verify it shows a different color from Step 4's branch

    From the root node, create a second branch using Duplicate; change a color parameter in this branch. Verify it shows a different color from Step 4's branch

  58. [80]

    Pretend the circles are the orbit of planets of our solar system. Add some fake planets and a sun

    From the root node, create a Modify operator with the prompt: *"Pretend the circles are the orbit of planets of our solar system. Add some fake planets and a sun."* Click Generate. Verify a new animated sketch with planets is created

  59. [81]

    Generate

    On the generated planet sketch, type a partial prompt in a new Modify node and use autocomplete to select a suggestion. Generate. Verify a new child sketch is created

  60. [82]

    Verify real-time output changes

    On a subsequent sketch in the topographic/noise branch, use the semantic parameter sliders to adjust parameters. Verify real-time output changes

  61. [83]

    Verify a new combined sketch is generated with a merge comment in its code

    Select two sketches from different branches and Merge them. Verify a new combined sketch is generated with a merge comment in its code

  62. [84]

    Use the Extract operator on one sketch to isolate a visual property

  63. [85]

    text:",

    Review the overall canvas: verify the auto-layout shows a hierarchical tree from left to right matching the structure in Figure 2. **Expected Result:** By the end of the session, the canvas shows a tree of interconnected sketch and operator nodes. The session mirrors the workf...

    2026

  64. [2012]

    InCHI ’12 Extended Abstracts on Human Factors in Computing Systems(Austin, Texas, USA)(CHI EA ’12)

    RepliCHI SIG: from a panel to a new submission venue for replication. InCHI ’12 Extended Abstracts on Human Factors in Computing Systems(Austin, Texas, USA)(CHI EA ’12). Association for Computing Machinery, New York, NY, USA, 1185–1188. doi:10.1145/2212776.2212419

  65. [2023]

    InAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology(San Francisco, CA, USA)(UIST ’23 Adjunct)

    Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents. InAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology(San Francisco, CA, USA)(UIST ’23 Adjunct). Association for Computing Machinery, New York, N...

  66. [2026]

    ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agent.IEEE Transactions on Visualization and Computer Graphics32, 1 (2026), 451–461. doi:10. 1109/TVCG.2025.3642628 A Appendix A.1 Systems For our rubric experiments, the following papers were used:B2[59], Cells, Genera...

    2026

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.