{"id":"617f9d64-ad95-4d85-9cdc-371914afe6ef","arxiv_id":"2602.02235","paper_version":3,"verdict":"REJECT","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"ArtifactCopilot automates artifact evaluation end-to-end with graph-based execution, claiming 85.42% agreement with human badge outcomes on 48 software engineering artifacts.","lead":"This paper proposes ArtifactCopilot, an LLM-agent system that reads a paper's PDF and README, builds a dependency graph of evaluation commands, executes them in Docker, and recommends an artifact-evaluation badge. The headline result—85.42% agreement with human-assigned badges on 48 artifacts—is not verifiable from the manuscript, and the abstract describes a different system and different results than the body.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BCR metric is degenerate: all 48 artifacts have positive functional labels, so condition (2) is always true and 85.42% measures command completion, not badge agreement.","rationale":"The central claim is the 85.42% BCR as 'high evaluation accuracy' in matching human AE outcomes. For this to hold, BCR must be a meaningful measure of agreement. The paper's own definitions prevent this. Section 2 restricts the dataset to papers awarded Functional or Reusable badges and treats Reusable as functionally positive, so all 48 artifacts have positive functional labels. Under the BCR definition in §5.1, the second conjunct (outcome consistency with official badge) is therefore always true. BCR collapses to the proportion of artifacts where all GCS commands were executed successfully — a command-completion rate, not a badge-consistency rate. The metric cannot detect false successes, and a trivial system that always predicts 'functional' would be scored as matching every label as long as it executes the GCS. This is the most load-bearing concern because it invalidates the primary quantitative support for the paper's contribution. The reader's weakest_assumption about ground-truth reliability is related but less direct: even if the official badges and GCS are correct, the metric structure prevents any negative comparison. A concrete test is to compute the label distribution and, if all positive, run the system on known non-functional artifacts to measure false positives. The abstract/body inconsistency (ArtifactGuide vs ArtifactCopilot, 60 vs 48 artifacts, 70.56% vs 85.42%) and the missing Data Availability section further undermine verifiability, but the metric degeneracy is the sharper, internal flaw. Thus the REJECT verdict stands, though confidence should remain low because the underlying system might still be capable; the reported evidence does not demonstrate it.","tokens_in":18093,"tokens_out":7692,"duration_ms":67665,"concrete_test":"Extract the functional badge labels for all 48 artifacts from the dataset description in §2. Since the sampling frame includes only papers awarded Functional or Reusable badges, every label is positive. Confirm that condition (2) of the BCR definition is trivially true by computing the label distribution. If all are positive, recompute the reported BCR without condition (2) and verify it is unchanged. Then run ArtifactCopilot on a control set of 10 artifacts from the same venues that did NOT receive a functional badge (e.g., Available-only or rejected); report how often the system declares functional success. If it declares success on any negative example, those would have counted as matches under the current BCR, proving the metric does not measure badge agreement. Report sensitivity and false-positive rate; if specificity is undefined because no negative examples exist, the 85.42% figu","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §5.1, Badge Consistency Rate (BCR) is defined as the fraction of artifacts for which (1) all guideline-specified commands in the GCS are executed successfully and (2) the resulting evaluation outcome is consistent with the official functional-level badge outcome. However, §2 restricts the dataset to papers awarded Artifacts Functional or Artifacts Reusable badges, and explicitly treats Reusable as functionally positive. Consequently, every one of the 48 artifacts has a positive functional label. Condition (2) is therefore satisfied a priori for all artifacts, so BCR reduces to the GCS command-completion rate. The reported 85.42% measures whether the agent executed the author-extracted command set, not whether its badge decision matches human judgments. The metric cannot penalize false positives: an agent that executes all GCS commands and always predicts 'functional' will match every label even if a human reviewer would reject the artifact for incorrect outputs. Thus the headline claim that ArtifactCopilot 'matches human artifact evaluation outcomes' is not supported by the BCR as defined. The comparison with baselines may still show relative gains in command-coverage, but the absolute accuracy figure is not a measure of badge consistency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an end-to-end agent framework, ArtifactCopilot, that takes a paper PDF or a downloaded repository and produces an artifact-evaluation badge recommendation. The framework builds an AE Graph from README/documentation, uses a Planning Agent to construct and normalize a Docker-based execution environment, and uses an Evaluation Agent to execute commands, recover from errors, and determine a functional badge outcome. Evaluation on 48 artifacts from four SE venues reports an 85.42% Badge Consistency Rate (BCR), 0.11 human interventions per artifact, $0.091 average cost, and ablations showing large drops when the AE Graph, environment normalization, or hierarchical two-agent decomposition are removed. The abstract additionally advertises a separate scoring rubric called ArtifactGuide, a 70.56% agreement result over 60 artifacts, and a user study with 8 researchers; none of these appear in the body of the paper.","tokens_in":18397,"tokens_out":6092,"duration_ms":58927,"significance":"If the empirical claims were reliable, this would be a useful step toward semi-automated artifact evaluation: the AE Graph representation, environment normalization, and hierarchical planning/execution decomposition are sensible responses to real failure modes, and the ablation study and per-artifact cost analysis are informative. The paper also contributes a real-world dataset of 48 artifacts and a documented failure analysis. However, the headline metric does not measure what it claims, the abstract and body report different systems and different results, and a claimed user study is absent from the body. The current evidence does not support the central claim that ArtifactCopilot 'matches human artifact evaluation outcomes.'","major_comments":[{"comment":"The abstract advertises ArtifactGuide, a structured scoring rubric, and reports 70.56% three-run mean exact badge agreement over 60 artifacts, plus a controlled user study with 8 experienced researchers. The body describes ArtifactCopilot with an AE Graph, reports 85.42% BCR on 48 artifacts, and contains no ArtifactGuide, no 60-artifact dataset, no three-run results, and no user-study section. These are different systems, different datasets, and different metrics; a reader cannot determine which result is the paper's actual claim.","section":"Abstract vs. §1, §5.2, Table 4"},{"comment":"BCR is degenerate because the dataset contains no negative labels. Section 2 restricts the dataset to papers awarded Artifacts Functional or Artifacts Reusable badges and explicitly treats Reusable as functionally positive, so all 48 artifacts have a positive functional label. Consequently, BCR condition (2), 'the resulting evaluation outcome is consistent with the functional-level badge outcome,' is automatically true whenever the agent returns any positive outcome. An agent that executes all GCS commands and always predicts 'functional' cannot be penalized for false positives, even if a human reviewer would reject the artifact for incorrect outputs. The reported 85.42% therefore measures GCS command completion plus positive prediction, not agreement with human badge decisions.","section":"§2 and §5.1 (BCR definition)"},{"comment":"The Golden Command Set is manually extracted by the authors from README files and is treated as the operational definition of successful execution, while official badges are used only as positive labels. The paper itself documents that badge criteria differ across venues and years (§2), and the GCS may omit commands that human reviewers actually ran or include commands without verifying their output semantics. The BCR is thus an agreement with an author-constructed command inventory, not with the official human evaluation process. The metric should be presented as a command-completion rate unless the GCS is independently validated against reviewer behavior.","section":"§2 and §5.1 (ground truth)"},{"comment":"The abstract claims that a controlled user study with 8 experienced researchers shows ArtifactCopilot reports improve reviewer confidence, help locate evidence, and clarify evaluation scope. No user study appears in the body: there is no protocol, no participant description, no measured outcomes, and no statistical analysis. This is a load-bearing missing piece of evidence and must either be added or removed from the abstract.","section":"Abstract vs. full text (user study)"},{"comment":"The 12-artifact exploration set was used to identify challenges and inform system design, and it is then included in the headline 48-artifact BCR of 85.42%. The paper discloses this and reports a separate 86.11% on the 36 held-out artifacts, but the abstract and conclusion use the pooled figure. Since the design was shaped by the exploration set, the held-out-only result is the appropriate headline; reporting the pooled number as the main result overstates the evidence for the framework's general performance.","section":"§3, §5.1, §5.2 (exploration/validation contamination)"}],"minor_comments":[{"comment":"Section 4.4.2 says each command is retried up to five times, while Section 6.3 says the retry threshold is k=3. Please reconcile the two numbers.","section":"§4.4.2 vs. §6.3"},{"comment":"The blocking-detection threshold of three minutes appears only in Section 6.3; Section 4.4.2 describes only 'sustained low CPU utilization across intervals.' Specify the threshold where the mechanism is introduced.","section":"§4.4.2 and §6.3"},{"comment":"Claude Code has no LCR or RASR entries. Clarify whether these metrics were not measured for this baseline or are not applicable.","section":"Table 4"},{"comment":"The LLM+Scripts baseline has zero interventions and $0.002 cost, but its BCR is 20.83%. A sentence explaining why the baseline is not trivially improved by adding a feedback loop would help the reader interpret the comparison.","section":"§5.1 baselines"}],"recommendation":"reject","confidential_remarks":"The abstract/body mismatch is severe: the abstract describes a different system, different dataset size, different success rate, and a user study that does not exist in the body. The BCR metric is, as defined, a command-completion rate rather than a badge-consistency rate because no negative examples exist. These are load-bearing issues that cannot be fixed by local edits. If the authors can re-run the evaluation with a metric that includes artifacts that failed official evaluation and can align the abstract with the body, a resubmission might be worth considering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things before anything else. The abstract and the body describe different papers, and the headline metric doesn't measure what it claims.\n\nThe abstract reports ArtifactGuide, 60 artifacts, 70.56% agreement, and a user study. The body is about ArtifactCopilot, no ArtifactGuide, 48 artifacts, 85.42% BCR, and no user study. That is not a small copyedit; it is a versioning/integrity problem the authors need to resolve before anyone can trust which result is real.\n\nThe core direction is worth taking seriously. Treating artifact evaluation as a state-aware workflow, representing READMEs as a dependency graph, normalizing host/container execution, and separating a planning agent from an evaluation agent are sensible and well-motivated by the preliminary study. The ablation is informative, and dataset construction is documented carefully, including manual Golden Command Set extraction and inter-annotator agreement. The cost and intervention numbers are plausibly useful.\n\nBut the main result is not supported by the metric. The dataset restricts to artifacts awarded Functional or Reusable badges and treats Reusable as functionally positive, so every artifact has a positive functional label. Under BCR, condition (2)—outcome consistent with the official badge—is trivially satisfiable by any agent that labels everything functional. The reported 85.42% therefore measures how often the agent successfully ran the extracted commands, not how often its badge decision matched a human reviewer. It cannot catch false positives. The baseline comparison may show a real gap in command coverage, but the paper's central claim about matching human evaluation outcomes is not established.\n\nThe paper also reports 85.42% on a set that includes the 12 exploration artifacts used to inform system design. It discloses this and reports a validation-only number (86.11% on 36), which is honest, but the headline table still mixes them.\n\nSmaller issues: the Data Availability section is referenced but missing; the retry threshold and blocking interval are acknowledged as pilot-selected, so the free-parameter handling is reasonable but should be more explicit.\n\nWho is this for? Researchers working on automated reproducibility and LLM agents for SE will want to know this paper exists, but only after substantial revision. The right next step is a major overhaul: fix the abstract/body mismatch, redefine the metric so it actually compares badge decisions—ideally on a dataset that includes negative badge outcomes—and present the current results as command-completion rates.\n\nRecommendation: send it to peer review. The problems are serious but identifiable, and the underlying direction deserves expert scrutiny. I would not accept it in anything like its current form.","headline":"The abstract and body describe different systems and different results, and the headline BCR metric cannot actually measure badge agreement; the engineering work is real, but the paper needs major restructuring before it is publishable.","tokens_in":18842,"tokens_out":3441,"would_cite":false,"duration_ms":34850,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM agent can reproduce human artifact-evaluation outcomes for 85% of real software artifacts, the paper claims, by modeling the evaluation as a state-aware workflow rather than a one-shot script.","keywords":["artifact evaluation","LLM agents","software engineering","reproducibility","AE Graph","automation","dependency modeling","badge consistency"],"falsifier":"An independent run of the same system on the same 48 artifacts, with a fresh set of human reviewers documenting every command they execute and every repair they make, would produce a new set of ground-truth labels; if the agent's agreement with those labels is substantially below 85.42%—or if the official badges disagree with the authors' functional labels—the claim does not survive.","tokens_in":17990,"feed_emoji":"🤖","tokens_out":6145,"duration_ms":53956,"temperature":0.7,"pith_summary":"Artifact evaluation is a standard but labor-intensive part of software engineering conferences, and this paper tries to show it can be automated end-to-end by a large language model agent. The proposed system, ArtifactCopilot, treats evaluation as a state-aware workflow: it converts README documentation into a dependency-aware AE Graph, normalizes containers so all commands run from the host, and uses two cooperating agents to plan and execute with automatic error recovery. Evaluated on 48 real artifacts from four SE conferences, the agent matches human-assigned badges in 85.42% of cases, needs only 0.11 human interventions per artifact, and costs about $0.09 per artifact—far outperforming a static script-generation baseline and a general-purpose interactive coding agent. If this holds, automated pre-screening could absorb the growing submission volume while leaving final judgment to human reviewers.","feed_headline":"LLM agent replicates human artifact review 85% of the time","feed_subtitle":"READMEs become dependency graphs, cutting evaluation cost to $0.09 per artifact with near-zero human help.","key_machinery":"The AE Graph is the load-bearing representation: a directed graph in which command nodes and artifact nodes are connected by sequential-execution edges and artifact input/output edges, with each node holding execution state. It is constructed by an LLM from aggregated README, INSTALL, and SETUP files, then validated and refined during execution. Its role is to make implicit task dependencies explicit so the agent can query global state, isolate the blast radius of a failure, and continue from a checkpoint instead of restarting. Working alongside it are environment normalization (host-driven command injection into Docker containers, with custom entrypoints replayed through a detached shell) a","core_discovery":"The paper's central discovery is that the barriers to automating artifact evaluation are representational and environmental, not just a lack of model capability. After manually executing twelve artifacts, the authors identify three bottlenecks: READMEs interleave commands with prose so execution state is implicit; docker run -it and similar commands silently shift the execution context between host and container; and errors require iterative diagnosis and repair. ArtifactCopilot answers these with (1) an AE Graph whose nodes are commands and artifact files and whose edges are execution order and data dependencies, built by an LLM from the documentation; (2) an environment-normalization stage","pith_inferences":["Editorial observation: the paper's abstract reports a different headline result (a system called ArtifactGuide, 70.56% on 60 artifacts) than the body (ArtifactCopilot, 85.42% on 48 artifacts). These numbers and systems need reconciliation before the stronger figure is taken at face value.","A natural next step is to close the failure modes the paper itself identifies: integrating static code-level dependency analysis into the AE Graph so implicit imports and data-validity checks are caught before execution.","The approach could be turned into an author-facing pre-submission check: if an artifact passes ArtifactCopilot's run, it has at least satisfied an explicit command-based baseline, which would likely reduce reviewer intervention counts community-wide.","Given that the dataset excludes commercial software, private data, specialized hardware, and repositories larger than 10GB, the true scope is narrower than 'all artifacts'; measuring generalization to other venues and domains is the immediate empirical test."],"forward_implications":["If the 85.42% figure holds, artifact evaluation can be pre-screened automatically at scale, reserving human review for borderline or high-risk cases.","The system's structured reports give reviewers a concrete map of which commands ran, failed, or were repaired, potentially making human evaluation faster and more consistent.","The AE Graph exposes underspecification in artifacts, so running the agent can give authors a concrete checklist of missing dependencies or ambiguous instructions before submission.","At roughly $0.09 per artifact, automated evaluation is cheap enough to run multiple times per artifact, enabling risk-based sampling rather than exhaustive manual checks.","The ablation results imply that any future automation system needs explicit dependency modeling, environment normalization, and task decomposition: skipping any one costs a large accuracy drop."],"fun_headline_variants":["LLM agent hits 70.56% badge agreement on real artifacts","Agent-based review boosts badge accuracy by up to 28 points","ArtifactCopilot automates artifact review with graph-based evidence","Structured rubric lifts LLM artifact review above baseline prompts","Agent completes every run with review reports, unmatched by others"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 85.42% agreement is measured against a ground truth built from the authors' manually extracted Golden Command Set and official conference badge outcomes, both treated as correct and complete—if official badges vary in strictness across venues and years, or the Golden Command Set omits commands that human reviewers actually ran, the headline number overstates automation quality.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent hits 70.56% badge agreement on real artifacts","Agent-based review boosts badge accuracy by up to 28 points","ArtifactCopilot automates artifact review with graph-based evidence","Structured rubric lifts LLM artifact review above baseline prompts","Agent completes every run with review reports, unmatched by others"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1474,"prompt_tokens":787,"completion_tokens":687,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":601}},"tokens_in":531,"tokens_out":687,"duration_ms":6178,"temperature":1.0,"reasoning_tokens":601,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:23:58.676801+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent run of the same system on the same 48 artifacts, with a fresh set of human reviewers documenting every command they execute and every repair they make, would produce a new set of ground-truth labels; if the agent's agreement with those labels is substantially below 85.42%—or if the official badges disagree with the authors' functional labels—the claim does not survive.","supporting_citations":[],"review_version":1}