{"id":"91f7a5a3-ae54-40eb-874c-d8f9dca76f56","arxiv_id":"2505.21069","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM-driven agent, CXXCrafter, automatically builds 587 of 752 C/C++ open-source projects (78%), beating default build commands (39%) and bare LLMs (32 to 38%).","lead":"CXXCrafter is an automated tool that uses a large language model to write and fix Docker build scripts, and it successfully builds 78% of 752 popular C/C++ projects. It turns a task that took human experts 153 hours for 100 projects into an autonomous pipeline costing about 41 cents per successful build.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 78% result rests on a success criterion validated mainly by an unspecified artifact diff and 22/24 unit tests; a fresh, independently judged rebuild sample would settle whether 'success' actually yields the primary build artifact.","rationale":"The reader's weakest_assumption already identifies the success criteria as the critical point, and I find this is indeed the most load-bearing assumption. The 78% claim is a rate, so the definition of the numerator controls the conclusion. The paper provides more validation than a bare assertion: an LLM discriminator evaluated on 249 manually confirmed successes, a 75-project artifact-diff sentence, and unit tests on 24 projects. Those checks support the metric, but the diff methodology is omitted and the unit tests cover only 32% of Top100 successes and none of the Awesome-CPP successes. Neither the reader nor I found an internal inconsistency in Table 5: the arithmetic is coherent and the ablations are sensible. The concern is therefore about verifiability of the measurement, not about the design's coherence. A targeted reproducibility check on a sample of claimed successes would settle it. Because the reader already conditioned acceptance on this issue, I recommend keeping the CONDITIONAL verdict rather than upgrading or rejecting.","tokens_in":23558,"tokens_out":6370,"duration_ms":80088,"concrete_test":"Select 20 Top100 projects that Table 5 counts as CXXCrafter successes, recover their pinned commits and generated Dockerfiles from the released Zenodo artifact, and rerun them in clean Ubuntu 22.04 containers. Independent success requires an artifact-level check: the primary binary/library named in the project's README exists, is executable/loadable, and passes a smoke test (--version/--help or a project test target), not just a log containing 'Building CXX object'. Separately, rerun the claimed manual-versus-automated diff for those 20 projects using an explicit procedure, such as comparing the file list and content hashes of relevant binaries and libraries. If fewer than 18/20 pass, or the 'completely consistent' diff cannot be reproduced, the 78% success rate should be treated as unverified; if 18/20 or more pass, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is a measurement: CXXCrafter builds 587/752 projects, 78.10%, near the manual 86/100 on Top100. That number stands or falls with the Metrics of Success in Section 5: a build counts only if a human inspector confirms (static) that the Dockerfile contains build commands targeting the primary components and (dynamic) that logs show compilation progress such as 'Building CXX object' and no errors. Section 6 then validates the metric with (i) unit tests on only 24 of the 75 Top100 successes, 22 passing, and (ii) a single sentence: a 'diff tool' compared automated and manual artifacts and found them 'completely consistent' for all 75. The diff procedure is not described: what was diffed (binaries, file lists, logs?), what 'consistent' means, and whether the comparison tolerates nondeterministic build metadata. Since the same team defines and applies the success criteria, the artifact diff is the only check that breaks the loop from 'the log looked like a build' to 'a build actually happened'. If the criterion accepts a partial cmake/make run that produces many object files but not the documented primary executable, every count in Table 5—especially the 512 Awesome-CPP successes, which get no artifact-level validation—is systematically inflated. The claim is plausible and internally coherent, but as written the 78% rate is not externally verifiable from the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper first reports an empirical study of building 100 popular C/C++ projects: only 21 succeed with default build commands, and human experts reach 86/100 after classifying 384 build errors. It then presents CXXCrafter, an LLM-based agent with parser, generator, and executor modules that produces Dockerfiles and iteratively repairs them from build logs. The evaluation claims 587/752 successful builds (78.10%), split as 75/100 on Top100 and 512/652 on Awesome-CPP, outperforming default build commands (39.01%) and bare LLMs (19.81--38.43%). The paper also reports ablations, multiple LLM backends, a case study of three projects built by CXXCrafter but not manually, cost and efficiency data, cross-version experiments, and a Java migration result.","tokens_in":23863,"tokens_out":6861,"duration_ms":75866,"significance":"If the headline measurement is correct, this is a valuable contribution: it is among the first LLM-agent systems for C/C++ build automation, with a large 752-project evaluation, four baselines, four LLM backends, parser and interaction ablations, unit-test spot checks, cost data, and a released implementation. The error taxonomy of 384 errors across 79 projects is itself a useful empirical resource. The main caveat is that the central 78% number rests on success criteria applied by the authors and validated by an underspecified artifact-diff claim, so the paper's principal claim is not yet externally verifiable as written.","major_comments":[{"comment":"The headline claim of 587/752 successful builds is defined by criteria applied by the authors: static inspection of whether the Dockerfile contains build commands for primary components, plus dynamic inspection of whether logs show compilation progress and no errors. The only checks against actual build outcomes are unit tests run on 24 of the 75 Top100 successes (22 passing) and a single sentence stating that a diff tool found automated and manual artifacts completely consistent. The diff procedure is not specified: what artifacts were compared, at what granularity, what tolerance was allowed for nondeterministic build metadata, and whether the primary executable was confirmed present. Because none of the 512 Awesome-CPP successes receive artifact-level validation, the 78% rate is not externally verifiable from the paper. The authors should describe the diff methodology in detail, release per-project Dockerfiles and logs (or a well-defined sample), and ideally have an independent party re-judge a random sample under the stated criteria.","section":"Section 5, Metrics of Success and Section 6"},{"comment":"The prompts and the LLM discriminator criteria were refined through iterative experimentation on the Top100 dataset, and Top100 is also one of the two evaluation datasets, yielding the 75/100 result and the near-manual comparison (72 overlap, 3 CXXCrafter-only, 14 manual-only). This makes the abstract's claim that CXXCrafter nearly matches manual performance an in-sample measurement. The Awesome-CPP dataset mitigates the concern for the overall 78% rate because it is external, but the paper should either report a holdout split or quantify how many prompt/criteria revisions were made after seeing Top100 results, so readers can assess the degree of adaptation.","section":"Section 4.5 and Section 5"}],"minor_comments":[{"comment":"The discriminator validation paragraph reports that 249 of 400 build processes were classified as successful and that all 249 were manually verified as correct, but it does not report how many of the remaining 151 processes were true failures or whether the discriminator ever misclassified a failed build as successful; the claim that the discriminator is effective is therefore incomplete.","section":"Section 4.4"},{"comment":"The text says bare LLMs achieve '23 and 17 successful builds' on Top100, but Table 5 lists three bare-LLM rows with 23, 25, and 17 Top100 builds; the DeepSeek-v3 value is omitted from the sentence and should be aligned.","section":"Section 5.1, Table 5"},{"comment":"The figure legend is truncated in the manuscript ('CXXCrafter-w…[1]') and the reader must rely on the caption note to identify CXXCrafter-w/o-Interaction; the legend and labels should be made fully readable.","section":"Figure 5"},{"comment":"The description of the version-diversity experiment is unclear: 20 projects and 5 commits each cannot literally cover 'their entire repository commit histories,' and it is not stated whether the 20 projects come from Top100 or Awesome-CPP or how the commits were sampled.","section":"Section 6, Building Different Software Versions"},{"comment":"The statement that 'the authors have rich experiences in C/C++ related researches' and therefore construct-validity threats are limited is an assertion rather than an analysis; the paper should either provide a concrete construct-validity discussion or remove this sentence.","section":"Section 7, Threats to Validity"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits FSE's scope, and I see no novelty or citation concerns. The main risk is that the headline metric is validated by the authors' own success criteria plus an unspecified artifact diff; I would ask for a detailed diff methodology, artifact release, and preferably an independent sample check before accepting. The in-sample refinement of prompts on Top100 is a genuine but addressable concern that is mitigated by the external Awesome-CPP dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: CXXCrafter is a real advance. It's the first LLM-agent system aimed at C/C++ build automation, and it beats the obvious baselines by a large margin. The empirical study is genuinely useful: 384 build errors on 100 popular projects, with a clean taxonomy that will help anyone working on build tooling. The evaluation is more thorough than most systems papers: four baselines, parser ablation, interaction-loop ablation, four LLMs, cross-version builds, a Java migration, and per-build cost/time data. They also ship code and data, so the work is reproducible.\n\nThe headline number — 587/752, 78% — is internally consistent, and the Awesome-CPP portion (512/652) is outside the design loop, which does a lot to reduce the circularity concern. The reader's circularity score of 3 is fair: the prompts and discriminator were tuned on Top100, and Top100 is in the test set, but the main result doesn't depend on Top100 alone.\n\nThe soft spots are real but not fatal. The success criterion is self-authored: a Dockerfile must contain build commands and the logs must show compilation progress. The paper validates this with unit tests on 24 Top100 successes (22 pass) and a one-sentence claim that a diff tool found artifacts 'completely consistent' with manual builds. That diff analysis is not described: what was diffed, how consistency was judged, whether nondeterministic build metadata was tolerated. The unit-test sample is small and doesn't cover Awesome-CPP at all. The stress-test note is right that this is the weakest link. An independent rebuild of, say, a random 30-50 projects with a check for the primary executable would settle it.\n\nOne smaller thing: the 'completely consistent' wording is doing too much work. If the procedure is straightforward, a paragraph in the appendix fixes it.\n\nOverall: this is a serious piece of work, the limitations are stated honestly (Threats to Validity admits popularity bias, network flakiness, LLM variance), and the architecture—parser/generator/executor with an LLM discriminator—is clearly explained. I'd send it to review. The right outcome is probably conditional acceptance with a request for artifact-diff details and an externally checked sample.","headline":"A valuable first LLM-agent system for C/C++ build automation with a plausible 78% success rate, but the success metric needs an externally audited artifact check before the number is fully trustable.","tokens_in":24426,"tokens_out":2235,"would_cite":true,"duration_ms":23738,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CXXCrafter, an LLM-based agent, automatically builds 587 of 752 popular C/C++ open-source projects in the paper's evaluation.","keywords":["LLM agent","C/C++ build automation","Dockerfile generation","dependency resolution","build failure taxonomy","open source software","large language models"],"falsifier":"Rebuild the released Dockerfiles for the claimed successful projects in clean containers and run each produced binary through the project's own tests plus an independent diff against manual builds; the paper's unit-test check covers only 24 of 75 Top100 projects, so the remaining builds are the open question.","tokens_in":23357,"feed_emoji":"🛠️","tokens_out":8566,"duration_ms":87276,"temperature":0.7,"pith_summary":"An LLM-driven agent can automate most C/C++ project builds. The paper reports that CXXCrafter successfully builds 587 of 752 projects (78%), compared with 39% for default build commands and 31.65% for a bare GPT-4o prompt, while human experts built 86 of the 100-project subset. The paper first documents why building is hard: 384 errors across 79 projects, costing 153 man-hours, with missing libraries the dominant cause. Its claimed result matters because building C/C++ software from source is the bottleneck for tasks that need compiled binaries or intermediate representations, such as fuzzing, static analysis, and vulnerability reproduction. If the 78% rate holds, an automated agent can prepare the majority of popular repositories for those downstream analyses without manual setup.","feed_headline":"LLM agent builds 78% of 752 C/C++ projects","feed_subtitle":"Agent loop beats default build commands (39%) and bare GPT-4o (32%) while nearing manual success.","key_machinery":"The load-bearing mechanism is the generator-executor feedback loop wrapped around an LLM. The Parser module pre-extracts environment details, dependencies, and build-relevant documentation via retrieval; the Generator turns those into a candidate Dockerfile using structured prompt templates; the Executor runs the Dockerfile in a clean container and returns captured logs. The loop repeats up to ten times, so each error message becomes the prompt for the next patch. A second piece does the verification: an LLM-based discriminator checks both that the Dockerfile contains real build instructions and that the execution log shows compiling progress, which is what lets the paper count genuine builds rather than silent no-op failures.","core_discovery":"On the paper's own terms, the central claim is that the obstacle to automated C/C++ building is not raw compilation but the decisions around it: which build system to trust, which dependencies to install, which OS version to choose, and how to tell a real build from a silent no-op. CXXCrafter addresses this with a three-module agent loop. A Parser extracts environment details, dependency lists, and build-relevant documentation; a Generator writes a Dockerfile from nested prompt templates; an Executor runs the Dockerfile in a clean container and returns error logs. The Generator and Executor iterate up to ten times, so each failure message becomes the input for the next repair. The paper reports 75/100 and 512/652 successes on its two datasets, three projects that manual builders could not finish, artifact equivalence with manual builds via diff, and a cost of $0.41 and 875 seconds per successful build.","pith_inferences":["The 587 generated Dockerfiles themselves form a reusable corpus: they could be mined as retrieval examples for unseen repositories or distilled into a cheaper builder, which the paper does not do.","If the success rate holds for less popular projects, the bottleneck shifts from whether a build is possible to whether the produced artifact is trustworthy, so independent artifact verification would become the limiting step.","A natural next test the paper does not run is using the loop in pull-request CI, where the ten-step budget and clean-container model fit but the time and cost constraints are stricter."],"forward_implications":["Downstream program-analysis pipelines can treat building the project as an automatable step, enabling fuzzing, vulnerability reproduction, and IR-based static analysis across large open-source corpora.","The reported $0.41 and roughly 15 minutes per successful build make the approach affordable to run at corpus scale rather than only on hand-picked projects.","Success across old commits (81/100) and consecutive commits (96/100) suggests the agent adapts to version drift instead of memorizing a single snapshot.","The Java adaptation experiment (57/76, 75%) indicates the agent architecture is not C/C++-specific and may transfer to other language ecosystems."],"supporting_citations":[{"why":"Supplies the Java-build feasibility study and default-command methodology that the empirical study adapts to C/C++ projects.","marker":"[16]"},{"why":"Provides the Awesome-CPP list of 652 projects that makes up most of the 752-project evaluation dataset.","marker":"[7]"},{"why":"CCScanner is the static dependency extractor the Parser module uses to identify third-party libraries before building.","marker":"[45]"},{"why":"CppBuild is the prior C/C++ build automation tool the paper positions against, and it is limited to three build systems.","marker":"[15]"},{"why":"The released implementation, prompts, logs, and diff artifacts back the reproducibility of the reported 78% result.","marker":"[56]"}],"fun_headline_variants":["LLM agent builds 78% of C/C++ projects automatically","CXXCrafter: LLM agent cracks C/C++ build automation","Agent learns from build errors to reach 78% success","Docker-based LLM agent automates C/C++ builds at 78%","CXXCrafter: 78% build success with LLM agent loop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 78% rate rests on the paper's own checklist for success: build commands present and logs showing the compiler actually ran, rather than on a fully described independent check of the produced binaries.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent builds 78% of C/C++ projects automatically","CXXCrafter: LLM agent cracks C/C++ build automation","Agent learns from build errors to reach 78% success","Docker-based LLM agent automates C/C++ builds at 78%","CXXCrafter: 78% build success with LLM agent loop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000792,"raw_usage":{"total_tokens":3539,"prompt_tokens":1041,"completion_tokens":2498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":2403}},"tokens_in":657,"tokens_out":2498,"duration_ms":18707,"temperature":1.0,"reasoning_tokens":2403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:37:00.571141+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the released Dockerfiles for the claimed successful projects in clean containers and run each produced binary through the project's own tests plus an independent diff against manual builds; the paper's unit-test check covers only 24 of 75 Top100 projects, so the remaining builds are the open question.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Java-build feasibility study and default-command methodology that the empirical study adapts to C/C++ projects."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Awesome-CPP list of 652 projects that makes up most of the 752-project evaluation dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CCScanner is the static dependency extractor the Parser module uses to identify third-party libraries before building."},{"cited_title":"2021.CppBuild: Large-Scale, Automatic Build System for Open Source C++ Repositories","cited_arxiv_id":null,"evidence_quote":"CppBuild is the prior C/C++ build automation tool the paper positions against, and it is limited to three build systems."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The released implementation, prompts, logs, and diff artifacts back the reproducibility of the reported 78% result."}],"review_version":1}