Pith. sign in

REVIEW 4 major objections 8 references

Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment

T0 review · 4 major / 0 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM-based judging pipeline can label architectural quality at scale and turn that signal into much stronger code models.

desk verdict Solid data-curation result for SWE-bench; the architectural-quality half of the story is circular and unvalidated. read the letter →

arxiv 2606.14948 v2 pith:6XEYCQAU submitted 2026-06-12 cs.SE cs.AI

Kirill Vasilevski , Ximing Dong , Benjamin Rombaut , Milad Soltany , Ruochen Deng , Jiahuei Lin (Justina) , Arthur Leung , Dayi Lin
show 3 more authors
Boyuan Chen Shaowei Wang Ahmed E. Hassan
This is my paper
classification cs.SEcs.AI
keywords architecturalreasoningcodeLLMsLLM-as-judgesupervisedfine-tuningSWE-benchrepository-specificrubricssoftwarearchitectureagenticlabeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Real software work needs more than patches that pass tests: models must reason about module boundaries, dependency structure, and repository-specific conventions. That kind of architectural quality is expensive for humans to label and cannot be checked by unit tests alone. This paper argues that a strong language model can act as a scalable proxy for expert architectural judgment. It builds two judges: one that estimates how much codebase-specific architecture a task requires, and one that builds a source-grounded, repository-specific rubric and scores whether a candidate patch respects it. Filtering training trajectories with both judges, then fine-tuning mid-sized open models on the resulting few thousand examples, produces large gains on bug-fixing benchmarks, better architectural conformance of the patches themselves, and transfer to non-Python languages even though training was Python-only. The practical claim is that architectural reasoning is learnable when the training data is selected for structural quality rather than volume alone.

What carries the argument

The agentic judging pipeline of two complementary static judges: ACJ (Architecture Complexity Judge), which reverse-explores from the patch to score five axes of codebase-specific architectural demand and assign a Trivial-to-Expert verdict; and AQJ (Architecture Quality Judge), which first derives a repository- and issue-specific rubric from source evidence, then scores patch conformance into High / High-with-concerns / Acceptable / Low.

What would settle it

A blinded human study in which experienced repository maintainers independently rate a substantial sample of ACJ and AQJ labels (and the resulting filtered versus unfiltered training sets) and show that human agreement with the judges is low, or that human-filtered data does not reproduce the reported resolve-rate and conformance gains.

Watch

Extended reading notes

Core claim

Supervised fine-tuning of Qwen3-8B/14B/32B on 3,360 trajectories kept only when an Architecture Complexity Judge rates the task at least Trivial and an Architecture Quality Judge rates the patch at least Acceptable raises SWE-bench Verified resolve rates to 17.4–27.2%—up to 540% over the untuned base and up to 256% over full unfiltered fine-tuning—while lifting High and High-with-concerns architectural conformance shares to 84–94% and transferring gains to a multilingual SWE-bench suite.

Load-bearing premise

That a strong language model doing only static structural analysis is a trustworthy enough stand-in for expert architectural judgment that its labels can safely drive training data selection and architectural quality scoring.

Editorial extensions

If this is right

  • Training data for software agents can be filtered for architectural signal rather than only for test-pass rate or trajectory length.
  • Architectural patch quality becomes a measurable training and evaluation target even when no test suite exists.
  • Cross-language transfer of architectural reasoning is expected when the signal is structural (boundaries, layering, conventions) rather than syntax-specific.
  • Static code metrics alone will under-select high-value architectural examples relative to the agentic judges.
  • Smaller curated SFT sets can beat larger unfiltered sets for both resolve rate and structural conformance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the proxy holds, the same two-judge pattern could curate data for other hard-to-verify software qualities such as security boundaries or API stability contracts.
  • The large gap between agentic filtering and metric-based filtering suggests future work should treat static metrics as weak features inside a learned judge rather than as drop-in replacements.
  • Repositories that lack tests become newly usable for architectural SFT once AQJ can score patches from source structure alone.
  • Human-agreement studies on ACJ/AQJ would be the natural next measurement to decide how far the method can be trusted outside the reported model family and benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. The paper argues that architectural reasoning in code LLMs is bottlenecked by the cost and subjectivity of labeling, and proposes an agentic pipeline with two LLM judges—ACJ (task architectural complexity via five structural axes and patch-guided exploration) and AQJ (repository- and issue-specific, source-grounded rubrics for patch conformance)—to curate SFT data without execution. Filtering a 5,000-trajectory Python pool to 3,360 instances (ACJ ≥ Trivial, AQJ ≥ Acceptable) and fine-tuning Qwen3-8B/14B/32B yields SWE-bench Verified Pass@1 of 17.4–27.2% (large gains over untuned and FullData baselines), Multilingual transfer despite Python-only training, higher AQJ High+HWC shares, and better results than static-metric filtering. The authors conclude that architectural reasoning is learnable when training data are selected with explicit structural criteria.

Significance. If the resolve-rate results hold under independent scrutiny, the work is significant for repository-level SE agents: it shows that smaller, structurally curated SFT sets can beat full-data SFT by large margins on SWE-bench Verified and transfer across languages, with useful ablations against FullData and 157 static metrics (§5.3, Tables 1–3). Strengths include a clear two-judge design, detailed multi-stage judge procedures, open data/code links, specified training and OpenHands evaluation settings, and the collinearity analysis showing that ACJ/AQJ are not reducible to change-size heuristics. The architectural-quality half of the claim is less secure without independent validation of the judges, but the test-oracle gains alone would still matter for data-centric training of coding agents.

major comments (4)
  1. Table 1 and §5.1 report large shifts in High/HWC/Acc/Low architectural conformance after SFT, but AQJ both defines the training filter (D* requires q ≥ Acceptable; §3.3, §4) and supplies the post-training quality labels. The same unvalidated proxy therefore selects trajectories and then certifies that fine-tuned models improved on that proxy. Unlike resolved rate (unit tests), this half of the abstract’s “architectural patch quality” claim is not independently measured. At minimum, evaluate architectural quality with a held-out protocol: human expert ratings on a stratified sample, a different judge model/family than the one used for filtering, or a pre-registered external rubric not used in selection.
  2. §1 and §3 justify the entire pipeline by treating a strong LLM under static structural analysis as a scalable proxy for expert architectural evaluation, yet the manuscript reports no human agreement, inter-rater reliability, or calibration study for ACJ or AQJ (axes, rubrics, or final verdicts). Given that architectural quality is described as subjective and repository-specific, this premise is load-bearing for both data construction and the quality claims. A modest but rigorous study (e.g., expert labels on a few hundred instances with agreement metrics and error analysis of rejected-fix-path and primary-axis judgments) is needed before the labels can be trusted as training signal or evaluation.
  3. Experimental Settings and §3 never clearly name the judge model(s), temperature, tool/scaffold, or compute budget used for ACJ/AQJ, nor whether the same model family is used for both judges and for trajectory generation (MiniMax-M2.5). Without this, the pipeline is not reproducible and one cannot assess judge–student contamination or cost. Please specify the exact models, prompts/versions (beyond Appendix sketches), and any separation between labeling and student models.
  4. §4 sets τc = Trivial so ACJ effectively admits almost all non-doc-only instances; the 5,000→3,360 cut is then driven mainly by AQJ ≥ Acceptable. The paper’s hypothesis (§3) that valuable SFT data need diverse architectural complexity is therefore only weakly operationalized. Report the ACJ and AQJ label distributions before/after filtering, an ablation over stricter τc (e.g., Moderate+), and whether complexity diversity—not only quality filtering—drives the SWE-bench gains.

Circularity Check

2 steps flagged · score 5.0 of 10

SWE-bench resolve rates rest on independent unit tests, but the 84–94% High+HWC architectural-quality claim is scored by the same AQJ family used to select D*, so that half of the strongest claim is self-reinforcing.

  1. self definitional [§3.3 Filtering Thresholds; §4 Dataset; Abstract; §5.1 Table 1 AQJ columns]
    "We use ACJ ≥ Trivial and AQJ ≥ Acceptable as our threshold to select data from the training dataset, and ended up with 3,360 architecturally curated instances. ... our approach increases the proportion of architecturally conformant patches from 61–72% to 84–94% across model sizes. ... AQJ quality distribution in Table 1 further reveals that our approach consistently shifts patch quality toward higher architectural conformance"

    Architectural conformance is defined solely as AQJ’s High/HWC/Acc/Low labels. D* is filtered by AQJ ≥ Acceptable, then post-SFT “architectural patch quality” is again the AQJ label distribution. The claimed improvement is therefore improvement on the same unvalidated proxy that selected the training signal: training on AQJ-approved trajectories and reporting more AQJ-approved patches is self-definitional for the quality half of the claim, not an independent structural oracle.

  2. other [§1 Introduction; §5.3 RQ3; Abstract “consistent improvements in architectural patch quality”]
    "Such understanding is prohibitively expensive to label manually and impossible to verify through tests alone. We propose an agentic judging pipeline using a strong LLM as a scalable proxy for expert architectural evaluation ... Our agentic-based ACJ/AQJ scores cannot be replicated using static code quality and complexity metrics. ... no individual static metric is strongly predictive of the agentic judges’ assessments."

    The paper states architectural quality cannot be verified by tests and shows static metrics do not track AQJ, then still treats AQJ shifts as evidence of improved architectural reasoning. With no external human or metric validation of AQJ, the only remaining measure of the quality claim is the judge that built the training set—closing the loop rather than providing an independent check of the proxy.

full rationale

This is an empirical SFT paper, not a closed-form derivation. The load-bearing resolve-rate results (Table 1, SWE-bench Verified; Table 2 Multilingual) are scored by external unit-test oracles under Pass@1 and are further supported by FullData and static-metric ablations; those claims are not circular. The circularity is confined to the architectural-conformance half of the abstract/strongest claim. Architectural quality is operationally defined only by AQJ categorical verdicts (High / High_with_concerns / Acceptable / Low). The same AQJ (with threshold ≥ Acceptable) is used to construct the 3,360-instance SFT set D* and, after training, to report that High+HWC rose from 61–72% to 84–94%. No human agreement study, inter-judge reliability, or held-out architectural oracle is reported for AQJ (or ACJ). Thus “models produce more architecturally conformant patches” reduces to “models trained on AQJ-approved trajectories produce more AQJ-approved patches,” which is expected under the selection criterion rather than independently verified. FullData reaches comparable High+HWC shares, which further shows the quality metric does not uniquely validate the filter. Static-metric collinearity (§5.3) cannot substitute for AQJ, so there is no independent check. Score 5 reflects partial circularity on one packaged claim while the primary resolve-rate derivation remains externally grounded. No self-citation uniqueness chain, fitted-parameter-as-prediction, or renamed known theorem is load-bearing for the SWE-bench numbers.

Assumptions & free parameters 4 free parameters · 4 assumptions · 4 invented entities

The central empirical claim depends less on fitted physical constants than on methodological postulates: that LLM judges can stand in for architectural experts, that chosen categorical thresholds define ‘architecturally curated’ data, and that the invented ACJ/AQJ procedures capture design reasoning static metrics miss. Free parameters are mainly filter thresholds and training knobs; axioms are domain assumptions about architecture and LLM-as-judge validity; invented entities are the two judges and their axis/rubric machinery.

free parameters (4)
  • ACJ complexity threshold τc = ≥ Trivial
    Set to ≥ Trivial to keep diverse complexity; directly determines which of 5,000 trajectories enter D*. Hand-chosen categorical cutoff, not derived.
  • AQJ quality threshold τv = ≥ Acceptable
    Set to ≥ Acceptable; jointly with τc yields 3,360 instances. Choice controls selectivity of architectural conformance filter.
  • AQJ pattern confidence cutoff = 0.6
    Only architectural patterns with confidence ≥0.6 enter the rubric (§3.2.1 Step 2). Affects which structural axes exist for evaluation.
  • SFT learning rate and schedule = 5e-6 (cosine, 3 epochs)
    Initial LR 5e-6, min 1.25e-7, cosine, 3 epochs, batch 32/16—standard but claim-dependent training knobs for reported resolve rates.
assumptions (4)
  • domain assumption A strong LLM performing static structural analysis can act as a scalable proxy for expert architectural evaluation.
    Stated in abstract and §1 as the solution to the labeling bottleneck; underwrites both ACJ and AQJ without large human gold labels in the main evaluation.
  • domain assumption Architectural quality is repository-specific and cannot be captured by fixed universal rubrics or single static metrics.
    Motivates two-stage AQJ rubric generation from source evidence (§3.2); contrasts with Chidamber-Kemerer-style metrics discussed in §2.
  • ad hoc to paper An SFT example is valuable for architectural reasoning when task complexity is diverse and the patch conforms to repository architecture.
    Central hypothesis of §3; defines the joint ACJ∧AQJ selection rule for D*.
  • domain assumption Functional correctness via unit tests is independent of architectural conformance and insufficient alone for real-world patch quality.
    Opening motivation (§1); justifies measuring both resolved rate and AQJ distributions.
invented entities (4)
  • Architecture Complexity Judge (ACJ)
    purpose: Assign Trivial–Expert complexity via exploration, rejected fix paths, and five structural axes for data stratification.
    New operational procedure; no independent external standard beyond the paper’s prompts and axes.
  • Architecture Quality Judge (AQJ)
    purpose: Generate repository/issue-specific rubrics then score patch conformance (High…Low) with gap lists.
    Core invented evaluator; used for both training selection and reported architectural quality outcomes.
  • Five ACJ complexity axes (Scope, Dependency Chain Depth, Implicit Knowledge, Coordination Complexity, Insight Density)
    purpose: Decompose architectural complexity so a single holistic score does not conflate breadth and hidden-invariant cost.
    Paper-defined axes (Table 8); not standard SE metrics with prior external validation as a joint scale.
  • Source-grounded repository- and issue-specific architectural rubric U
    purpose: Replace fixed universal architecture checklists with evidence-backed primary/secondary/minor axes and prose anchors.
    Methodological object produced by RG(p,δ,R); quality depends entirely on the judge pipeline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment." pith.science (2026). https://pith.science/paper/6XEYCQAU

@misc{pith2026260614948,
  author       = {Pith},
  title        = {Pith review of: Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6XEYCQAU}},
  note         = {Machine review of arXiv:2606.14948}
}
read the original abstract

LLMs have substantially improved software engineering yet real-world development requires architectural understanding. Such understanding is prohibitively expensive to label manually and impossible to verify through tests alone. We propose an agentic judging pipeline using a strong LLM as a scalable proxy for expert architectural evaluation, comprising two judges: the Architecture Complexity Judge (ACJ), which estimates codebase-specific architectural understanding a task demands, and the Architecture Quality Judge (AQJ), which evaluates patch conformance to repository-specific architectural conventions via source-grounded rubrics. Fine-tuning Qwen3-8B/14B/32B on 3,360 curated instances achieves resolved rates of up to 27.2% on SWE-bench Verified - up to 540% over the base model and 256% over unfiltered fine-tuning. Meanwhile, the trained models achieve strong cross-language generalization and consistent improvements in architectural patch quality.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

8 extracted references · 1 linked inside Pith

  1. [1]

    None identified

    IEEE. SmartBear Software. 2006. Best practices for peer code review. Accessed 2026-05-22. Trae Research Team, Pengfei Gao, Zhao Tian, Xi- angxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, Yun Lin, Yingfei Xiong, Chao Peng, and Xia Liu. 2025. Trae agent: An llm-based agent for soft- ware engineering with tes...

    arXiv 2006

  2. [2]

    Read BOTH quality anchors (ideal and poor) before writing anything

  3. [3]

    Analyze the patch’s structural characteristics on this dimension

  4. [4]

    Write a prose assessment that describes: - What the patch does on this dimension (specific structural changes) - How the patch’s characteristics compare to the ideal anchor - How the patch’s characteristics compare to the poor anchor - Where on the spectrum between ideal and poor this patch falls - Any nuances, trade-offs, or context that affect the assessment

  5. [5]

    Cite specific evidence for every claim (file:line references)

  6. [6]

    Ignore bugs, syntax errors, runtime behavior

    Focus on structural conformance ONLY . Ignore bugs, syntax errors, runtime behavior

  7. [7]

    [primary | secondary | minor]

    Do NOT include scaffolding files in your assessment — they are out of scope. Required output format Return ONLY the following Y AML block. No prose before or after it. [axis_name]: priority: "[primary | secondary | minor]" assessment: | [Your prose assessment — multiple paragraphs as needed. Describe what the patch does structurally on this dimension, how...

  8. [8]

    The root cause of this issue is that convert_to_contiguous unconditionally converts allSequence types (includingtuple) intolist. This type change propagates downstream: list_data_collate treats the resulting list as a collection of individual elements to be stacked, causing a shape mismatch error when the tensors have different sizes. The high-scoring pat...

    2006

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.