{"id":"88ce992f-2869-4289-9314-17e683d9b86f","arxiv_id":"2608.00450","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Agentic coding tools reimplemented ten UIST systems from paper text alone, achieving high rubric-based revibeability scores with the best of three agents.","lead":"AI coding agents can rebuild interactive software tools directly from a research paper, with no original code available. The authors rebuilt ten UIST systems this way and report that the best version of each passed most feature checks, suggesting a new way to make HCI research reproducible.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 94% revibeability figure is not established: scores correlate r=-0.74 with the length of a Claude-generated rubric (Fig. 18), the same model family builds rubrics and revibes, and author interviews for 3 of 10 systems (mage, B2) show rubric-passing revibes missing author-intended functionality.","rationale":"I read the paper as a feasibility study whose central quantitative claim is the 94% best-run mean revibeability after two aided rounds (0.939 per-system best run). For this number to license the strong-baseline conclusion, the rubric must measure author-intended functionality reliably and independently of the tool that generated both the rubric and many of the revibes; that condition is the least secure part of the argument. The paper honestly reports much of the countervailing evidence itself: Figure 18 (r=-0.74, p<.05) links scores to rubric length; Section 6.2 reports large generator differences; Section 6.1 documents mage's drag-and-drop gap, B2's Jupyter-dependence, and Spellburst's undocumented internal feature; Section 7.1 concedes the rubrics can be improved with author feedback. These are not hidden flaws; they are reported limitations, and I am weighing them as the manuscript instructs. My concern sharpens the reader's weakest assumption by adding that the metric may not even be invariant across rubric generators, independent of the self-generation loop, and that author-validation counterexamples (mage, B2) show rubric-passing revibes missing author-intended design goals. The paper still has real value: the pipeline and prompts are concrete and reproducible via the supplement, per-run scores are shown rather than hidden, the cost and time data are useful, and the qualitative failure analysis is candid. The qualitative finding that agentic tools can reconstruct many UI features from paper text is credible and supported. I therefore do not recommend rejection; the reader's CONDITIONAL verdict stands, and the conditions should include (1) cross-rubric regrading using the existing Gemini rubrics, (2) inter-rater reliability on rubric grading, and (3) author rubric review for all 10 systems before the 94% figure is cited as evidence.","tokens_in":26163,"tokens_out":15441,"duration_ms":136596,"concrete_test":"Regrade each final (post-aided-2) revibe of the 10 systems against the Gemini-generated rubric already collected for the same paper in Section 6.2, using the same grading protocol and weights, and recompute the best-run mean and per-tool ordering. If the mean moves by more than about 0.1 from the reported 0.939, or if the per-tool ranking changes, the headline figure is an artifact of rubric construction rather than a stable property of the revibes. A complementary gold-standard check is to have the original authors of all 10 systems mark each Claude-rubric test as core/non-core and list missing key features, then recompute scores restricted to core tests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the headline claim that revibed systems can serve as strong baselines, the revibeability instrument—the Claude-generated rubric—must measure author-intended functionality. That condition is the least secure, and three pieces of internal evidence weaken it. First, the score depends on rubric construction: Figure 18 shows best-run score correlates r=-0.74 with rubric test count, and Claude rubrics average 35.1 tests versus Gemini's 10.25 (Section 6.2), so revibeability is not invariant to which rubric generator was used; the bootstrap in Figure 19 resamples within a single rubric and cannot bound this cross-instrument variance. Second, the rubric and a plurality of the revibes come from the same model family. Cursor's strong scores on Claude rubrics are fair evidence against a narrow self-testing effect, but a rubric biased toward easily specifiable or agent-suggested features would inflate all tools equally, so this does not resolve the confound. Third, the rubric is bounded by what the paper text describes: the Spellburst author recalled an important internal feature that was never described in the paper, so the metric cannot see its absence. Author input covers only 3 of 10 systems, and those interviews show the concrete failure mode: no revibe fully implemented mage's drag-and-drop, and B2's author said the revibes would not provide the intended experience because reactive cells relied on Jupyter infrastructure. The metric is therefore at best 'fidelity to the paper's description,' not fidelity to the original artifact, which shifts the burden of proof onto the rubric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces \"revibing\"—reimplementing interactive HCI systems directly from paper text using agentic coding tools—and proposes a \"revibeability\" metric based on a system-specific test rubric generated from the paper. Ten UIST systems are reimplemented with Claude Code, Cursor, and Gemini; after two aided feedback rounds, the authors report a mean best-run revibeability of 0.939 (min 0.8, max 1.0). The paper also interviews the original authors of three systems and analyzes rubric construction. The authors argue that revibed artifacts can serve as strong baselines in A/B evaluations and that this capability may change how technical HCI artifacts are produced and evaluated.","tokens_in":26469,"tokens_out":3735,"duration_ms":34745,"significance":"If the metric and the empirical claims were validated, the paper would address a real problem: most HCI artifacts are unavailable, and reimplementation is usually impractical. The pipeline is concrete, the prompts are reported in detail, and the qualitative failure analysis (over-implementation, environment dependencies, rubric ambiguities) is thoughtful and useful. The interviews with original authors are a valuable addition. However, the central quantitative claim is not yet established: the headline score is a best-run statistic, the rubric instrument has unaddressed validity and reliability concerns, and internal evidence shows that scores depend strongly on the rubric generator. The contribution is promising but currently stronger as a qualitative feasibility study than as a validated metric.","major_comments":[{"comment":"The headline \"94% revibeability\" is the mean of the best run per system (0.939), not the mean performance of a typical revibe. Per-agent means are Cursor 0.912, Claude 0.87, and Gemini 0.67, so the number describes a best-of-three selection. Since the abstract and Section 7.2 use this number to argue that revibed systems can serve as strong baselines, the claim overstates what a researcher should expect from a single agentic run; the paper should report per-agent averages prominently and either justify the best-run convention or reframe the claim to describe an upper-bound capability.","section":"Abstract; Section 6"},{"comment":"The construct validity of the revibeability instrument is not established. The rubric is generated by Claude Code and graded by the authors, and author interviews were conducted for only 3 of 10 systems. Those interviews reveal concrete failures: no revibe fully implemented mage's drag-and-drop, the B2 author said the revibes would not provide the intended experience because reactive cells relied on Jupyter infrastructure, and the Spellburst author recalled an important internal feature not described in the paper. Thus revibeability measures fidelity to the paper text as interpreted by Claude Code, not author-intended functionality or suitability as a strong baseline. The paper should either validate rubrics against original authors for substantially more systems or explicitly narrow the definition and all downstream claims.","section":"Sections 4.2, 6.1, 6.2"},{"comment":"The metric is not invariant to the rubric generator. Claude-produced rubrics average 35.1 tests versus Gemini's 10.25, and Figure 18 shows best-run score correlates r=-0.74 with rubric length. The bootstrap in Figure 19 resamples within a single rubric and therefore cannot bound this cross-instrument variance. Because the same model family (Claude) generates the rubrics and many of the revibes, a rubric biased toward easily specifiable or agent-suggested features would inflate scores for all tools; the paper needs sensitivity analyses across rubric generators or evidence that different rubrics yield similar rankings before claiming that revibeability is a stable property of the paper-agent pair.","section":"Section 6.2; Appendix A.4"},{"comment":"All rubric grading was performed by the authors without inter-rater calibration, blind assessment, or reported reliability statistics. Since the paper proposes revibeability as a reusable measurement for the community, at least a small inter-rater reliability study (e.g., two raters independently scoring a subset of tests) is needed to demonstrate that the scores are not idiosyncratic to the authors' interpretations of success, partial success, and failure.","section":"Sections 5 and 6"}],"minor_comments":[{"comment":"The abstract contains the sentence fragment \"The results are encouraging. In many cases producing code suitable for strong baseline use.\"; this should be rewritten as one or more complete sentences.","section":"Abstract"},{"comment":"The claim that Claude rubrics are \"most detailed\" is supported only by test count and file size; reporting a qualitative measure of whether the additional tests cover key features would make the argument stronger.","section":"Section 4.2"},{"comment":"The notation for graded rubrics alternates between \"rubric 1\" and \"rubric1\"; please use a consistent convention.","section":"Sections 4.3 and 4.4"},{"comment":"The system selection description says the final 10 were \"selected roughly randomly\" but then states that non-LLM-centered systems were oversampled; the algorithm for choosing the 10 from the 20 should be clarified.","section":"Section 5"},{"comment":"Figure 19's bootstrap confidence intervals are plotted but not interpreted in the text; a sentence summarizing what the intervals imply for the stability of the scores would help readers.","section":"Appendix A.4"},{"comment":"In the mage discussion, the quoted author comment contains an ellipsis and stray formatting that should be cleaned up.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"This paper fits UIST's scope and the qualitative contribution is real, but the headline quantitative claim needs substantial revision before publication. I would not recommend rejection: the deficiencies are addressable by reporting non-best-run statistics, adding rubric-validation evidence, and softening the strong-baseline claim to match what the instrument actually measures. The authors should also be encouraged to make their graded rubrics and implementation logs publicly available, since those are the strongest evidence in the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The core demonstration is real: agentic coding tools can reimplement interactive HCI systems from papers well enough to be useful in many cases, and the qualitative failure analysis plus author interviews make that credible. The 94% number is not what it looks like, though. It is the best of three agents per system after two feedback rounds, not typical performance. Per-agent means: Cursor 0.91, Claude 0.87, Gemini 0.67. Tell a colleague 'agentic tools hit 94%' and they'll overestimate a normal run.\n\nThe new and useful part is the revibe/revibeability framing and the protocol. The rubric prompt is concrete, the cost and time data are practical, and they ship code and rubrics. The appendix rubric example (TEST-S05 for Spellburst) shows exactly what graders did. The author interviews, even with only 3 of 10 systems, are the most informative bit: they show rubric-satisfying revibes missing the intended experience (mage's drag-and-drop absent, B2's reactive cells lost without Jupyter). The paper does not bury these.\n\nSoft spots, in order of severity. First, the 94% headline needs a qualifier. The abstract says 'we demonstrate that we can achieve 94% revibeability' without saying it is the best run. That is misleading for a metric presented as a floor; the paper's own 'floor, not a ceiling' remark is about the process, not the score, but readers will trip on it. Second, rubric validity: Claude generates rubrics, the same team grades without inter-rater calibration, and the rubric only captures what the paper text describes. The Spellburst author recalled an important internal feature the paper never mentions; the rubric cannot see its absence. Cursor's strong scores on Claude rubrics is partial evidence against a narrow self-testing effect, but not against a bias toward easily specifiable features. Third, the correlation between rubric length and max score (r=-0.74, Fig. 18) means the score depends on rubric construction; the bootstrap (Fig. 19) resamples within one rubric and cannot bound cross-rubric variance. Minor, but worth a sentence in revision.\n\nMath and data are simple and adequate for a feasibility study. Citations look fine. The qualitative conclusion—that reimplementation from papers is now plausible with agentic tools—holds up. The quantitative headline does not, as stated.\n\nThis paper deserves a serious referee. The revision should report per-run and per-agent means in the abstract, add inter-rater reliability or explain its absence, and either validate rubrics with authors more broadly or rename the metric 'fidelity to the paper description.' Send it to review.","headline":"The qualitative claim is solid; the 94% headline is best-of-three and measures fidelity to the paper text, not to the original artifact.","tokens_in":27012,"tokens_out":4633,"would_cite":true,"duration_ms":41456,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agentic coding tools can rebuild interactive HCI systems directly from their published papers, reaching a mean best-run revibeability of 94 percent across ten systems.","keywords":["Reimplementation","Open Science","Vibe-coding","Revibeability","Agentic programming","HCI evaluation","Strong baseline","Reproducibility"],"falsifier":"Have the original authors of the seven non-interviewed systems independently grade the revibed systems against a rubric they write themselves; if the mean score falls substantially below 0.94, the self-generated rubric overstates fidelity.","tokens_in":25967,"feed_emoji":"🤖","tokens_out":7233,"duration_ms":57669,"temperature":0.7,"pith_summary":"This paper argues that modern agentic coding tools can reimplement interactive software systems directly from their published papers, with no access to the original code, and that the rebuilt systems can be good enough to serve as strong baselines in A/B evaluations. The authors define 'revibing' as building a working system from the paper text alone, and 'revibeability' as a scored measure of how well the rebuilt system passes a generated test rubric. Across ten recent technical HCI papers with no public implementations, they report a mean best-run revibeability of 0.939 (minimum 0.8, maximum 1.0) after two rounds of feedback, with most runs completing in under thirty minutes and under fifteen dollars. The significance is that the shortage of released code, long a barrier to replication and to fair baseline comparison, could become far less binding if paper text is enough to reconstruct the artifact.","feed_headline":"Agentic AI rebuilds HCI demos from papers at 94% mean fidelity","feed_subtitle":"Ten unpublished systems were recreated from text alone, enough to serve as strong baselines in A/B tests.","key_machinery":"The load-bearing mechanism is the automatically generated, system-specific test rubric, produced by an agentic coding tool from the paper's markdown text and figures. Each rubric test specifies the steps a user should perform, what should happen, and the difference between full success, partial success, and failure. The rubric does three jobs: it defines what counts as a faithful reimplementation, it gives a human tester a uniform instrument for grading the rebuilt system, and it serves as a correction signal when the graded rubric is fed back to the agentic tool with a request to fix reported issues. The revibeability metric is then the weighted ratio of successful tests described in the paper.","core_discovery":"The central claim is that the practical floor for reimplementing interactive HCI systems, set by vibe-coding directly from a paper, is high enough for strong-baseline use. The paper introduces the verb 'revibe' for such a reimplementation and the metric 'revibeability,' computed as a weighted score over a system-specific rubric of manual interaction tests, with weight 2 for full success, 1 for partial success, and 0 for failure. In a study of ten systems drawn from recent technical HCI papers that lacked public code, the authors report that after an unaided build and two aided revision cycles, the best run per system averaged 0.939 revibeability, with two systems reaching 1.0 and a minimum of 0.8. Scores were monotonic across runs in almost all cases, and the few decreases were all below 0.02. Author interviews with three of the original teams broadly confirmed that the rebuilt systems captured the intended interactive features, but also exposed gaps such as missed drag-and-drop behavior and loss of notebook-native integration in one system.","pith_inferences":["The 94% figure is a best-run average and masks substantial variance: one of the three agentic environments averaged only 0.67 mean revibeability, so the floor is not uniform across tools.","Because the rubric is generated by the same model family that builds the system, the scores may contain a self-confirmation component; using a different model or original authors to write the rubric would test whether the 94% holds.","The paper samples papers from 2020 to 2024 that mostly target web or notebook interfaces; older, hardware-linked, or thick-client systems may not revibe nearly as well, so the estimate should not be extrapolated to all technical HCI artifacts.","A direct testable extension is to have the original authors of all ten systems grade the revibes against their own intended features; if the gap between author grades and rubric grades is large, the metric needs to be recalibrated before it is used as a baseline-comparison badge."],"forward_implications":["Technical HCI papers that do not release code could still be compared against in A/B evaluations, because revibed artifacts can serve as strong baselines.","A revibeability score could become a publishable badge or reviewing signal, giving authors a reason to check that their paper text is sufficient for reimplementation.","Authors could write papers with reimplementation in mind, providing interaction rubrics or more detailed feature descriptions, lowering the cost of reproduction for everyone.","The same rubric could be reused over time to benchmark improvements in agentic coding tools, turning specific HCI systems into long-lived reimplementation benchmarks.","If revibing becomes routine, the community may expect baseline comparisons more often, and reviewers may need norms to avoid rejecting non-revibeable work for the wrong reasons."],"supporting_citations":[{"why":"Supplies the Spellburst system, a primary test case whose node-based interface and error-handling features define a large part of the rubric tests.","marker":"[1]"},{"why":"Supplies the mage system, whose drag-and-drop behavior was the clearest failure across all environments and whose original authors were interviewed.","marker":"[33]"},{"why":"Supplies the Rescribe system, an algorithm-heavy artifact that is tested only through its interface in the current rubric.","marker":"[42]"},{"why":"Supplies the B2 system, whose author interview highlighted the lost notebook-native integration in the revibes.","marker":"[59]"},{"why":"Supplies the EvalGen system, where agents ignored published prompts, providing evidence about prompt-adherence limits.","marker":"[47]"},{"why":"Supplies the Vizability system, one of two systems that achieved a perfect revibeability score.","marker":"[21]"},{"why":"Supplies the XCreation system, the other perfect-score test case.","marker":"[63]"},{"why":"Defines the vibe-coding paradigm from which revibe-coding is derived, supplying the conceptual grounding for the reimplementation approach.","marker":"[54]"}],"fun_headline_variants":["AI revibes UIST demos from papers, hitting 94% fidelity","Rebuilding HCI demos from text alone: 94% fidelity","Agentic AI recreates unpublished demos from paper text","Revibe success: ten demos restored from papers at 94%","From PDF to prototype: AI revibes HCI systems at 94%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that the automatically generated revibeability rubric captures the key interactive features the original authors intended; the paper validates this against author interviews for only three of the ten systems.","fun_headline_variants_meta":{"raw":{"variants":["AI revibes UIST demos from papers, hitting 94% fidelity","Rebuilding HCI demos from text alone: 94% fidelity","Agentic AI recreates unpublished demos from paper text","Revibe success: ten demos restored from papers at 94%","From PDF to prototype: AI revibes HCI systems at 94%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000394,"raw_usage":{"total_tokens":2051,"prompt_tokens":911,"completion_tokens":1140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":1043}},"tokens_in":527,"tokens_out":1140,"duration_ms":7576,"temperature":1.0,"reasoning_tokens":1043,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:18:54.459133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have the original authors of the seven non-interviewed systems independently grade the revibed systems against a rubric they write themselves; if the mean score falls substantially below 0.94, the self-generated rubric overstates fidelity.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the vibe-coding paradigm from which revibe-coding is derived, supplying the conceptual grounding for the reimplementation approach."}],"review_version":2}