{"id":"b81785e2-4e26-4f6d-bb9a-f698497bfba4","arxiv_id":"2607.09900","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AfterVibe extracts natural-language specs from vibe-coding trajectories, validates them via blind regeneration scored by a three-tier verifier, and reaches mean scores of 5.06–5.74/6 on 72 industrial tasks.","lead":"AfterVibe recovers abstract natural-language specifications from AI coding conversations and checks them by having a blind agent rebuild the code from the spec alone. As AI-generated code outpaces review, such specs could become the primary human-readable artifact and source of record.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The regeneration score can be inflated by shared LLM priors and residual leakage rather than by faithful capture of developer intent.","rationale":"The reader correctly flags the three-tier verifier as the load-bearing assumption. The paper’s own leakage analysis (Table 3), diversity numbers (chrF 0.89), and negative controls (Table 4) are careful and partially mitigate the concern, which is why I do not move the verdict to REJECT. They do not, however, close the shared-prior / residual-leakage channel: all LLM components use a single frontier model, GT alignment is binary and conversation-conditioned, and flex salvage is deliberately permissive. A cross-family re-evaluation is the cleanest falsifier. Until that (or an equivalent human-oracle sample) is shown, CONDITIONAL remains the right call; the concern is the same one the reader identified, only sharpened to a concrete experimental check.","tokens_in":24104,"tokens_out":561,"duration_ms":5596,"concrete_test":"Re-run the 68 graded tasks with a different-family regenerator and a different-family GT-alignment/VC grader (e.g., Claude-family regenerator + GPT-family judges, or open-weight models), holding the AfterVibe specs fixed; if mean regen_score falls by >0.5 or the AfterVibe–human gap shrinks below 0.4, the original scores substantially reflect shared-model idiosyncrasies rather than intent capture.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that recovered specs are strong because blind regenerations score 5.06–5.74/6.0—rests on the three-tier verifier being a faithful oracle of functional equivalence modulo the conversation (Sections 3.3–3.4, 4.3–4.7). Two concrete weaknesses remain. First, the ground-truth alignment grader (weight 1) and the VC extractor are themselves LLM agents that share the same frontier model family as the regenerator; they can reward regenerations that match common coding-agent conventions even when those conventions were never stated in the trajectory. Second, standard specs still leak 38.6% of distinctive code tokens (Table 3), and the flex-test tier can salvage interface mismatches, so a high score can be achieved by partial surface reproduction plus environment-grounded defaults rather than by the abstract intent the authors claim. The negative controls (RQ5) show VCs discriminate wrong code, but they do not show that high scores require the recovered intent rather than shared model priors. If this holds, the headline numbers overstate how much of developer intent the specs actually capture.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"AfterVibe proposes retrospective recovery of natural-language specifications from vibe-coding sessions (conversation trajectory T plus landed code change C). Specs are structured into Intent/Rationale, Essential Design Decisions, Undiscoverable Facts, and a declarative requirements checklist, then validated by a regeneration test: a blind agent reimplements from the spec alone and is scored by a three-tier verifier (flexible tests, independently extracted verification conditions, ground-truth alignment) into a 0–6 regeneration score. On 72 curated industrial monorepo tasks, baseline mean score is 5.06/6.0 (rising to 5.74 best-of-K after per-task strengthening), outperforming human-authored review summaries (4.23), with supporting analyses of abstraction–leakage trade-offs, regeneration diversity, and negative-control grader validity. The authors argue that such specs could become the primary human-review artifact and source of record as AI-generated code outpaces conventional review.","tokens_in":24408,"tokens_out":1576,"duration_ms":23629,"significance":"If the regeneration scores genuinely reflect capture of developer intent rather than shared model priors plus residual leakage, the paper offers a practical bridge from vibe coding to reviewable artifacts without requiring upfront formalization. Strengths include evaluation on real industrial sessions (not synthetic benchmarks), independence of verification conditions from the distilled spec, sharp negative-control discrimination for VCs (100%→0–1.4%), leakage and diversity analyses that partially substantiate abstractness, weight-sensitivity checks, and release of the full prompt suite. The regeneration-strength test and trajectory-as-spec-source framing are novel relative to prior code-to-spec and commit-message work. The industrial grounding and multi-RQ design make the contribution more than a prompt demo.","major_comments":[{"comment":"Sections 3.3–3.4 and 4.3–4.7: the central claim that recovered specs are “strong” rests on the three-tier verifier being a faithful oracle of functional equivalence modulo the conversation. Flex tests and VCs are useful, and RQ5 shows VCs discriminate wrong/pre-change/partial code, but ground-truth alignment is an LLM agent from the same frontier family as the regenerator, and flex salvage can rewrite interfaces. Shared coding-agent priors can therefore reward regenerations that match common conventions never stated in T. Negative controls do not isolate this confound. Please add either (a) a cross-model regeneration/grading experiment (different model family for regenerator vs. graders), or (b) a human-rated sample of high-score regenerations judged for intent fidelity, and revise claims that equate high regen_score with “captured developer intent” accordingly.","section":"§3.3–3.4, §4.3–4.7, RQ5"},{"comment":"Section 3.2 and the problem formulation: distillation is conditioned on both T and the final unified diff C, so the pipeline is post-hoc reverse-engineering of a landed change guided by conversation, not pure recovery of intent from the conversation alone. The paper sometimes frames the contribution as recovering “developer intent” from the trajectory (Abstract, Contributions, Vision). That overstates what the method does. Please restate consistently as “post-hoc abstract specification of the landed change, informed by the session,” and clarify which claims depend on C being available at extraction time (most of them).","section":"§3.1–3.2, Abstract, Contributions"},{"comment":"Table 3 / RQ3: standard specs still leak 38.6% of distinctive code tokens, and the score gap vs. constrained (4.81) and diff-preserving (5.55) variants is partly driven by GT alignment, which rewards surface match. The diversity result (chrF ~0.89 for both-pass pairs) is helpful but does not fully separate “abstract intent” from “partial code echo + environmental grounding.” Either report regeneration scores with a leakage-controlled or identifier-scrubbed ablation, or qualify the “abstract by design” claim to match the measured leakage.","section":"§4.5, Table 3, RQ4"},{"comment":"Section 4.1 and Threats: the 72-task set is a single-company monorepo sample over a 7-day window, filtered by ground-truth grader pass, and all pipeline stages use one frontier LLM. The headline numbers (5.06 / 5.74) are therefore not yet evidence of broad effectiveness. External-validity discussion is present but understates how load-bearing this is for the industrial-effectiveness claim in the contributions. At minimum, report failure modes and score distributions more fully, and temper “real-world effectiveness” language to the studied setting.","section":"§4.1, §5, Contributions"}],"minor_comments":[{"comment":"Table 1: AfterVibe+strengthening reports only mean regen score (5.74) with dashes for per-grader rates; either fill the per-grader best-of-K rates or explain why they are omitted so the comparison with baseline is complete.","section":"Table 1"},{"comment":"Section 4.4 / Table 2: best-of-K is monotonically non-decreasing by construction; state this more prominently next to the +0.68 delta so readers do not read 5.74 as a typical single-round outcome.","section":"§4.4"},{"comment":"Equation (1): τ is introduced as a pass threshold for RegenTest, but evaluation primarily reports continuous regen_score and uses τ=6.0 only for strengthening entry; align the formal definition with the metrics actually used.","section":"§3.4, Eq. (1)"},{"comment":"Figure 4 and §3.3: tier ordering in the figure (flex, VC, GT) differs from the appendix prompt labels (GT as Tier 1); a single consistent numbering would reduce confusion.","section":"Fig. 4, Appendix A.2"},{"comment":"Related work is solid but could more sharply contrast with commit-message generation and code summarization on the regeneration-sufficiency criterion already stated in §6.1.","section":"§6.1"},{"comment":"Minor polish: “AfterVibeuses” / missing spaces after tool names appear in the abstract and intro; fix typesetting of tool names throughout.","section":"Abstract, §1"}],"recommendation":"major_revision","confidential_remarks":"The work is a strong systems/SE contribution with real industrial data and careful multi-RQ design; I would not reject. The main risk is overclaiming “intent recovery” and “strong specs” on an LLM-as-judge stack that shares priors with the regenerator. If the authors add a cross-model or human-intent check and tighten framing, this is close to a solid accept for a software-engineering venue. Scope fit is good for cs.SE; the vision section is speculative and should stay clearly labeled as future work rather than a result."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is the first system that treats a full vibe-coding trajectory (conversation + landed diff) as the source for a post-hoc natural-language spec, then checks that spec by whether a blind agent can rebuild something the original session would accept. On 72 real monorepo tasks they get mean regen 5.06/6, 5.74 after per-task strengthening, beating the developer’s own review summary (4.23). That is a concrete, usable result, not a slogan.\n\nWhat is new and done well. Retrospective recovery from agent trajectories is distinct from formal-spec extraction from code, commit-message generation, or trajectory auditing for failures. The regeneration test is a sensible operationalization of “strong.” They separate distillation from regeneration (no shared context), hold VCs fixed while refining the spec, run negative controls (VCs go 100% → ~0–1% on pre-change/wrong code), measure leakage and regeneration diversity, and show the human-summary baseline is weaker for the same reason a changelog is not a reimplementation guide. The abstraction–strength trade-off and the environmental-grounding hypothesis are argued with data, not hand-waving. Prompts are released; the experimental design is inspectable.\n\nSoft spots, in proportion. The load-bearing assumption is that the three-tier verifier tracks functional equivalence modulo the conversation. Shared model family between regenerator, VC extractor, and GT-alignment grader can still reward common agent conventions; standard specs still leak ~39% distinctive tokens; flex tests are softer on wrong code than VCs. The paper does not fully close that gap—negative controls show discrimination, not that high scores require the recovered intent rather than priors plus salvage. Proprietary data and single-frontier-LLM evaluation limit external validity. The “specs as source of record” vision is stated as future work, not demonstrated. None of these sink the central empirical claim; they bound how far you can push it.\n\nWho it is for: people building AI coding agents, review tooling, or requirements recovery. Worth a serious referee. I would engage with it and expect revision on the oracle-validity discussion and multi-model checks, not a rewrite of the idea.","headline":"Clean new problem (post-hoc NL specs from agent trajectories) with careful industrial evaluation; the regeneration numbers are real enough to take seriously, residual LLM-judge risk is managed but not gone.","tokens_in":24991,"tokens_out":556,"would_cite":true,"duration_ms":11465,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Specs recovered after vibe coding can regenerate the original change and may replace code as the review artifact.","keywords":["vibe coding","specification recovery","regeneration test","agent trajectories","code review","AI-assisted development","natural-language specs","spec strengthening"],"falsifier":"On the same 72 tasks, show that regenerations scoring high under the three-tier pipeline still diverge from the original on independently written acceptance tests or human review of behavioral intent, or that the human-authored summary baseline closes the gap once leakage and weighting are controlled.","tokens_in":24975,"feed_emoji":"📝","tokens_out":599,"duration_ms":5277,"temperature":0.7,"pith_summary":"After a developer and an AI assistant finish a vibe-coding session, the conversation and the resulting code change still contain the developer's intent, but that intent is ephemeral and hard to review. AfterVibe recovers a short natural-language specification from the session and checks it by a regeneration test: a second, blind agent that never saw the original code or chat is asked to re-implement the change from the specification alone. The new code is scored against the original by flexible tests, independent verification conditions, and a holistic alignment check. On 72 real industrial tasks the recovered specifications reach a mean score of 5.06 out of 6, rise to 5.74 after iterative strengthening, beat the human-written review summaries already present in the workflow, and still leave room for diverse implementations. The paper therefore argues that the durable thing left after a vibe-coding session can be an abstract, re-implementable specification rather than the voluminous generated code itself.","feed_headline":"Post-vibe specs regenerate code at 5.06/6 and can be the review artifact","feed_subtitle":"Blind agents rebuild 72 industrial changes from recovered intent, beating human summaries.","key_machinery":"The regeneration test: a blind agent re-implements the change from the recovered specification alone, and a three-tier verifier (flexible tests extracted from the session, independently extracted verification conditions, and ground-truth alignment) scores functional equivalence; high score means the specification is strong, low score triggers iterative strengthening.","core_discovery":"Natural-language specifications distilled post-hoc from a vibe-coding trajectory (conversation plus code change) are both abstract and strong: a blind agent regenerating from the specification alone achieves a mean regeneration score of 5.06/6.0 on 72 real industrial tasks, rising to 5.74 after per-task iterative refinement, while independent regenerations remain diverse in structure and the specifications outperform the human-authored review summaries already written for the same changes.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["AfterVibe recovers specs that regenerate code at 5.06/6 mean","Blind agents rebuild 72 industrial tasks from post-vibe specs","Recovered specs score 5.06/6 and refine to 5.74 while staying abstract","Post-hoc specs outperform human summaries as review artifacts","Vibe-session specs enable diverse regenerations constrained only by intent"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The three-tier verifier is treated as a faithful measure of whether two implementations match the developer's intent, so a high regeneration score really means the recovered specification captured that intent rather than merely satisfying the judges.","fun_headline_variants_meta":{"raw":{"variants":["AfterVibe recovers specs that regenerate code at 5.06/6 mean","Blind agents rebuild 72 industrial tasks from post-vibe specs","Recovered specs score 5.06/6 and refine to 5.74 while staying abstract","Post-hoc specs outperform human summaries as review artifacts","Vibe-session specs enable diverse regenerations constrained only by intent"]},"model":"grok-4.5","effort":"low","cost_usd":0.005886,"raw_usage":{"total_tokens":1591,"prompt_tokens":819,"num_sources_used":0,"completion_tokens":81,"cost_in_usd_ticks":58860000,"prompt_tokens_details":{"text_tokens":819,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":691,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":819,"tokens_out":81,"duration_ms":6000,"temperature":1.0,"reasoning_tokens":691,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T14:42:52.352345+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same 72 tasks, show that regenerations scoring high under the three-tier pipeline still diverge from the original on independently written acceptance tests or human review of behavioral intent, or that the human-authored summary baseline closes the gap once leakage and weighting are controlled.","supporting_citations":[],"review_version":1}