{"id":"c4bba38a-1f81-4eb9-b2f1-ac2f159064bf","arxiv_id":"2607.18642","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A generator that produces oracle-verified, deliberately messy code in five languages with orthogonal difficulty labels and a contamination-resistant re-minting protocol.","lead":"Spaghetti Architect mints deliberately messy but still correct programs in five languages, each verified against a reference result and labelled with separate 'problem size' and 'code messiness' scores. It gives AI code-benchmark builders a fresh, contamination-resistant supply of test items.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixture-only oracle checks don't establish the claimed universal 'semantics are identical'; off-fixture divergence would invalidate the incidental axis and refactoring gate.","rationale":"The paper's real deliverable is a regenerable, labelled dataset generator, and its most load-bearing assumption is semantic equivalence between clean IR and messy renderings. The incidental axis is defined as 'presentation at fixed semantics' and the refactoring task's semantic_ok is an equivalence gate; both inherit the validator's fixture-only checking. The paper never explicitly discloses that equivalence is only verified on shipped fixtures, so Section 2.1's 'Provable correctness' and Section 2.2's 'semantics are identical' overstate the evidence. I agree with the reader's identification of this as the weakest assumption. A practical differential fuzz would settle whether any transform actually diverges. I credit the paper for open-sourcing the generator, the byte-deterministic golden-snapshot design, the canary, and for candidly disclosing the inert standard rung, the weak Buse-Weimer correlation (rho=0.39), the single-programme-family caveats, and the uncontrolled two-week gap in the annotation ablation. Those disclosures do not cover the equivalence gap, which is why the conditional verdict should stand. The contamination-tier caveats (raw completions make current literal values recoverable, Tier B/C shapes are public) are also honestly disclosed and do not undermine the mint-after-cutoff protocol as a re-mint mechanism. The two-week ablation gap is a real confound but secondary to the artifact's core validity, so I do not move the verdict.","tokens_in":26273,"tokens_out":5052,"duration_ms":61609,"concrete_test":"Differential-fuzz the released generator: for each of the 11 SPAGH_* transforms and each of the 5 backends, draw at least 10,000 random valid IR instances (vary collection sizes 0/1/2/large, duplicate and negative values, boundary comparator operands, empty and non-empty maps), render both the clean reference and the max-profile version, execute both, and compare outputs. Any mismatch falsifies the 'fixed semantics' claim for that transform; zero mismatches across a large edge-case-heavy sample would not prove equivalence but would materially raise confidence. Separately, apply the same differential check to the refactoring grader's semantic_ok predicate to see whether any accepted rewrite diverges on random off-fixture inputs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim 'semantics are identical' (Section 2.2) and 'each instance is correct by construction' (Abstract) are supported only by compiling and running each emitted program on the shipped input fixtures and comparing to the oracle (Section 2.1). This is finite testing, not equivalence. The incidental-messiness axis is defined as 'presentation at fixed semantics' (Section 2.3), and the refactoring semantic_ok gate credits any rewrite that matches the oracle on those same fixtures. Both presuppose that every SPAGH_* transform preserves IR semantics for all valid inputs, but no proof is given. A concrete place where this could fail is the always-on try/except fallback in Listing 2: an unexpected exception on an off-fixture valid input silently sets is_found=False rather than computing the oracle value. Similarly, a model rewrite that diverges only off-fixture would be falsely graded semantic_ok. The paper never states this boundary; Section 2.1's 'Provable correctness' overstates what the validator establishes. No counterexample is offered, and the transforms are simple enough that the claim may be true, but the gap is load-bearing and unaddressed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Spaghetti Architect, an open-source generator that synthesizes code datasets from a clean, language-agnostic JSON IR. An anti-optimization transpiler renders each IR into deliberately redundant, fully flattened programs in Python, JavaScript, Go, Java, and C++, at multiple 'messiness' levels defined by strictly nested sets of SPAGH_* anti-pattern transforms. A validator compiles, runs, and compares each emitted program against a reference oracle on shipped input fixtures. The dataset labels each instance on two orthogonal axes—intrinsic problem size (N/L/W/T) and incidental presentation (the messiness knob)—and ships a contamination protocol with a public dev split, a private held-out seed, a canary GUID, and A/B/C regeneration tiers. Reference baselines on four LLMs show comprehension exact match rising with model scale, intrinsic scale collapsing arithmetic-aggregation accuracy, and an annotation ablation in which the generator's own comments differentially help weaker models on refactoring. Construct-validity evidence from static complexity and readability metrics is reported with explicit caveats.","tokens_in":26579,"tokens_out":6254,"duration_ms":66337,"significance":"If the central guarantees held, Spaghetti Architect would be a genuinely useful resource: a regenerable, fully labelled, contamination-controlled, multi-language code dataset with a known-optimal reference and two independently controllable difficulty axes—precisely what mined corpora cannot supply. The paper is unusually honest: it reports the inert messiness rung, the weak readability signal (ρ=0.39), the unstable rung-separation statistic, the uncontrolled time gap in the annotation ablation, and the partly tautological nature of the complexity-metric agreement. The artifact is open source, deterministic, and includes non-learned baselines, a canary, and a datasheet. However, the central 'correct by construction' and 'semantics are identical' claims are currently supported only by finite fixture-level oracle checks, not by a semantic-equivalence argument. This gap is load-bearing for the incidental-messiness axis and the refactoring equivalence gate, and must be addressed before the resource's core labelling claims are valid.","major_comments":[{"comment":"The claims 'correct by construction' (Abstract) and 'The semantics are identical' (§2.2) are stronger than what the validator establishes. §2.1 says every target is 'compiled, run, and checked against the oracle'; this is a finite check on the shipped fixtures, not a proof of behavior preservation over all valid inputs. The always-on try/except fallback in Listing 2 silently converts any unexpected exception into `is_found = False`; a model rewrite that diverges only off-fixture would also be falsely graded `semantic_ok` by the refactoring gate. Because the incidental axis is defined as 'presentation at fixed semantics' (§2.3), and because refactoring equivalence is graded by the same oracle, the labels and the gate both presuppose universal equivalence. Please either (a) prove—or at least carefully argue from a precise semantic model of the IR and each SPAGH_* transform—that every trans","section":"Abstract, §2.1, §2.2, Listing 2"},{"comment":"The headline annotation-ablation result—that the generator's comments inflate the weakest model by an order of magnitude more than the strongest (−0.173 vs −0.017)—rests on a comparison with an uncontrolled two-week time gap between the annotated and unannotated arms, as the paper itself notes. The k=1 re-grading argument in the Table 6 caption is an indirect justification, not a contemporaneous paired replication. Since this finding is highlighted in the Abstract ('A benchmark's own commentary is not free'), the claim should either be strengthened by re-running the annotated arm at the same time as the unannotated one, or explicitly presented as suggestive rather than conclusive in the abstract and conclusions. The paper's own caveats are a credit, but they do not remove the confounding.","section":"§4, Table 6"},{"comment":"The construct-validity section is transparent, but the evidence is weaker than the phrase 'demonstrated metric-mover' in the Abstract suggests. The complexity-metric agreement is partly tautological (the knob injects the constructs that cyclomatic and cognitive complexity count), and the less tautological readability signal is ρ=0.39 with a CI clearly below the complexity metrics. The paper already states this, and it is appropriately scoped. The concern is that the resource's 'by-construction quality order' is, as it stands, a definitional ordering of transform inclusion, not an independently validated quality order; the paper's own plan for human calibration is the right remedy, but until it is run, the 'quality' wording in the Abstract and §1 should be softened further.","section":"§3, Tables 2–3"}],"minor_comments":[{"comment":"The relation between 'six transform sets' (clean, minimal, light, standard, heavy, max) and 'five distinct rungs' is confusing, especially since Table 3 reports 'five engine profiles' excluding clean. A small table listing which transforms are active at each rung, and which rungs are byte-identical, would help.","section":"§2.2"},{"comment":"The caption reports 'Per-cell n=15 items for W∈{8,64,128} and n=90 for W∈{16,32}' with 'each over k=8 draws'. This is puzzling: if n is the number of items, the W=16/32 cells have six times more items than W=8/64/128; if n includes draws, the wording is inconsistent. Please clarify the counting.","section":"§4, Table 7"},{"comment":"The phrase 'each item scores the fraction of its unordered candidate pairs ranked correctly (an exact complexity tie is credited 0.5)' is slightly self-contradictory: a tie is not 'ranked correctly'. Suggest 'correctly ordered, with a tie credited 0.5'.","section":"§4, Table 4 caption"},{"comment":"The comment 'manual index loop instead of ‘in‘' uses curly quotes around `in`; use straight formatting for code identifiers.","section":"Listing 2 comment"}],"recommendation":"major_revision","confidential_remarks":"The paper is one of the most transparent submissions I have reviewed, with limitations stated plainly and often quantified. The decisive issue is the gap between 'verified on fixtures' and 'semantics are identical / correct by construction'; this is fixable but requires either a real equivalence argument or a systematic weakening of the central labelling claims. The annotation-ablation time gap is a second concern that should be addressed before the headline result is taken as established. I see no grounds for rejection, but the manuscript is not ready in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid resource paper with one over-strong claim. Read it if you build or use code benchmarks; the generator itself — clean IR to deliberately messy programs in five languages, with a strictly-nested messiness ladder at fixed semantics and a re-mintable private split — is a real contribution. The shipped artifact is open source, deterministic, and the authors are unusually candid: they report the inert \"standard\" rung, the weak readability correlation (rho=0.39), the unstable rung-separation statistic, and the two-week gap in the annotation ablation. That honesty is genuine credit.\n\nThe soft spot is the one the stress test flags. \"Correct by construction\" and \"semantics are identical\" are not established by compiling and running on the shipped fixtures. That is finite testing, not equivalence. The always-on try/except fallback can silently convert an off-fixture exception into a wrong result, and a model rewrite that diverges only off-fixture would still pass the semantic gate. This matters because both the incidental-messiness axis and the refactoring equivalence gate depend on universal behavior preservation. The transforms are simple and the claim may well be true, but it needs to be bounded or proven. Reframing \"provable correctness\" to \"oracle-verified on shipped fixtures\" would fix it.\n\nThe other weaknesses are proportionate. The complexity-metric validation is partly tautological — the transforms inject exactly what cyclomatic complexity counts — and the paper says so. The readability signal is weak but reported honestly. The annotation ablation's time gap is uncontrolled, though the direction of the result is plausible and family-robust at the extremes. None of this sinks the resource.\n\nWho is this for? Anyone doing contamination-aware evaluation or needing paired clean/messy programs with known-optimal references. The protocol and the honesty are worth borrowing. It deserves a serious referee — conditional accept, with requests to tone down the correctness language, add an explicit limitation about fixture-based verification, and ideally re-run the ablation under a controlled time window.\n\nI'd bring it to a reading group, and I'd cite it if I were working in this area.","headline":"A genuinely useful generator with refreshingly honest reporting; the only load-bearing flaw is that \"correct by construction\" is fixture-checked, not proven — a reframing, not a fatal problem.","tokens_in":27075,"tokens_out":2433,"would_cite":true,"duration_ms":24397,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Spaghetti Architect mints oracle-verified code datasets with two independent difficulty axes and contamination-resistant re-minting.","keywords":["data-centric machine learning","code dataset generation","data contamination","by-construction labels","multi-language code","benchmark resource","dataset documentation"],"falsifier":"Take a released max-profile program (e.g., the manual-index membership check with opaque predicate and try/except fallback) and run it on inputs external to the shipped fixtures—empty lists, duplicate targets, non-list collections, or mismatched types—and check whether the output always equals the clean IR's result; a single divergence, or a randomized differential test across many such inputs, would falsify the 'semantics are identical' claim.","tokens_in":26094,"feed_emoji":"🍝","tokens_out":10007,"duration_ms":79058,"temperature":0.7,"pith_summary":"This paper presents Spaghetti Architect, a generator that creates code datasets for evaluating code models, with three properties mined corpora cannot offer. First, every instance is graded against a known-optimal clean reference—a simple JSON IR—so there is ground truth for correctness and quality. Second, difficulty is split into two orthogonal, independently controllable axes: intrinsic problem size and incidental presentation messiness. Third, the test split is re-minted from a private held-out seed, so scored items stay fresh after a model's training cutoff. The central claim is that this combination lets researchers attribute a model's failure to problem size versus surface mess, and detect memorization rather than assume it.","feed_headline":"Mint fresh code benchmarks with two labelled difficulty knobs","feed_subtitle":"Oracle-checked programs let you vary problem size and surface mess independently, then re-mint the test set fresh.","key_machinery":"The central mechanism is the anti-optimization transpiler pipeline: a Parser validating a typed JSON IR, a Planner selecting anti-pattern profiles from a data-driven database, five language Generators emitting idiomatic spaghetti code with always-on safety scaffolds, and a Validator that compiles, runs, and compares every output against a reference oracle. The two labelled axes—intrinsic size (N/L/W/T knobs) and incidental messiness (the nested clean⊂minimal⊂light⊂standard⊂heavy⊂max profile chain)—are carried through the IR and the profile database, making correctness and difficulty properties of the construction rather than of measurement.","core_discovery":"Spaghetti Architect is an anti-optimization transpiler: given a clean, language-agnostic JSON intermediate representation (IR) built from four operation primitives, it renders each operation into deliberately redundant, fully-flattened programs in Python, JavaScript, Go, Java, and C++. Every emitted program is compiled, executed, and checked against a reference oracle, so the released instances are correct by construction. Because the clean IR serves as a known-optimal reference and messiness is dialed by a strictly-nested set of anti-pattern profiles (clean, minimal, light, standard, heavy, max), each instance carries two orthogonal by-construction labels: intrinsic difficulty (problem size","pith_inferences":["The same generator could be pointed at other semantics (floats, strings, richer control flow) once cross-language oracle equivalence is handled, extending the controlled stimulus to a wider class of reasoning probes.","The two-axis design invites an adaptive-testing protocol: mint items inside a model's transition region (e.g., W between 8 and 64 for agg_stats) rather than past a ceiling every model fails, yielding sharper difficulty estimates.","The differential-annotation result is a general caution for synthetic benchmark builders: any self-documenting output (comments, docstrings, even descriptive variable names) can differentially inflate weak models and distort leaderboard resolution, not just in code.","The private-seed re-mint protocol could become a standard for other procedurally-generated benchmarks: making test items regenerable from a held-out seed is a general antidote to training-set leakage across modalities."],"forward_implications":["If the central claim holds, code-benchmark studies can finally separate 'model fails because the problem is bigger' from 'model fails because the code is messier,' a decomposition no static corpus offers.","The held-out re-mint protocol turns contamination from a hope into a measured quantity: public-split scores matching private fresh-minted scores within |Δ|≤0.012 makes memorization detectable rather than assumed.","The annotated-vs-unannotated ablation shows that a benchmark's own comments are not neutral: they inflate weak models more than strong ones, so any leaderboard built on annotated prompts overstates the ladder's resolution.","The intrinsic knob provides a tunable arithmetic-aggregation probe that floors even the strongest tested model by width W≥64, giving a controlled, regenerable probe for reasoning limits.","The construction offers a known-optimal reference for refactoring, enabling semantic-equivalence-gated quality scoring with a clean ceiling (0.83 Python) and a non-trivial floor, instead of grader-judged approximations."],"fun_headline_variants":["Anti-optimization transpiler mints verified code tests","Two axis difficulty labels, correctness by construction","Code benchmarks with independent intrinsic and incidental knobs","Re-mintable, oracle-checked code sets: size vs messiness","Fresh labelled code: problem size and mess dialed separately"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim assumes that running the oracle on the fixed input fixtures shipped with each instance is enough to guarantee that the deliberately messy renderings behave identically to the clean reference on every possible input—if an injected transform diverges on any input outside those fixtures, the by-construction correctness label and the semantic-equivalence gate lose their guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Anti-optimization transpiler mints verified code tests","Two axis difficulty labels, correctness by construction","Code benchmarks with independent intrinsic and incidental knobs","Re-mintable, oracle-checked code sets: size vs messiness","Fresh labelled code: problem size and mess dialed separately"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1528,"prompt_tokens":900,"completion_tokens":628,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":548}},"tokens_in":644,"tokens_out":628,"duration_ms":34150,"temperature":1.0,"reasoning_tokens":548,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:45:17.329293+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a released max-profile program (e.g., the manual-index membership check with opaque predicate and try/except fallback) and run it on inputs external to the shipped fixtures—empty lists, duplicate targets, non-list collections, or mismatched types—and check whether the output always equals the clean IR's result; a single divergence, or a randomized differential test across many such inputs, would falsify the 'semantics are identical' claim.","supporting_citations":[],"review_version":1}