Pith.

Artificial Intelligence · arXiv:2610.13356 v2 · submitted 9 Oct 2026

SWE-Critix: Post-Training Generalist Verifiers for Assessing Coding-Agent Trajectories

Jiajun Hu, Chun Yong Chong

The paper says

“Coding agents are increasingly deployed for end-to-end software engineering tasks, where a readily available oracle evaluator that defines the success criteria is often absent. This motivates verifications for coding agents that can reliably judge and explain their behavior.”

more of the abstract

However, verification is non-trivial as coding agents are increasingly capable of solving long-horizon software engineering tasks. Their execution trajectories can be extremely long, containing hundreds of iterative interactions with code repositories and external tools. Existing verifiers either rely on general-purpose LLMs without specialized training or train dedicated critics that predominantly focus on producing scalar rewards, highlighting the relatively limited exploration of training-based generalist verification. We introduce SWE-Critix, a post-training pipeline for training generalist verifiers of coding-agent trajectories. Given a task and its execution trajectory, the learned verifier predicts the completion label and generates a natural-language analysis that explains its judgment. Our pipeline combines supervised fine-tuning (SFT) with reinforcement learning (RL) and trains a series of 30B models on trajectories from CodeForge-Preview. On a held-out test set, SWE-Critix improves classification accuracy from 57% for the base Qwen3-30B-A3B reasoning model to 66% after SFT and 78% after RL. We release our models, datasets, and training code to facilitate further research.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 12 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. SWE-Critix builds a plausible pipeline for coding-agent verification with open-sourced models, but the 'generalist verifier' claim is unsubstantiated without baselines and CoT quality evaluation. This line is wrong

4 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Abstract, §I, Table I, §III, §IV.5

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The paper frames SWE-Critix as a 'generalist verifier' that 'generates a natural-language analysis that explains its judgment' (Abstract) and distinguishes it from scalar-reward critics (Table I). Yet the evaluation (§III, Table II) exclusively measures binary classification accuracy. The quality, correctness, coherence, and grounding of the generated chain-of-thought explanations are never assessed.

more of objection 1

The authors acknowledge this gap in §IV.5 ('We plan to conduct a qualitative analysis'), but without such evidence the central claim that the model is a 'generalist' verifier (beyond a binary classifier) is unsupported. The authors should include at least qualitative examples, a human evaluation, or automatic coherence metrics to substantiate that the CoTs are meaningful and non-trivial.

The paper says, at §III, Table II

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

No baseline verifier comparisons are included. The paper only compares its own three checkpoints (base, SFT, RL). To assess the practical value of SWE-Critix, the authors should compare against representative methods from the taxonomy in Table I, such as a training-free verifier (e.g., LLM-as-a-Verifier [8] or Agentless [7]) and a training-based scalar critic (e.g., SWE-RM [22]) on the same test set.

more of objection 2

This gap is noted in §IV.4 as future work, but the current conclusion of 'substantial improvement' is not positioned relative to existing alternatives.

The paper says, at §III, Table II

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The results report point estimates without confidence intervals or variance measures. Given that each model is evaluated only once (as stated), it is unclear whether the observed differences (e.g., the drop in accuracy on R2E-Gym from 60% to 52% after SFT) are statistically significant. The authors should compute bootstrap confidence intervals for accuracy and per-class metrics, or report standard errors, especially because many subsets are large (over 5,000 samples).

The paper says, at §II-A, §IV.1

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

All training and test trajectories come from a single source (CodeForge-Preview, generated by OpenHands with Qwen3-Coder-480B). The paper acknowledges this limitation in §IV.1 but does not provide any external validation. Since the verifier is claimed to be 'generalist', some evidence of robustness to distribution shift (e.g., trajectories from different agent scaffolds or base models) is needed, even on a small external set. This is a load-bearing issue for the generalizability claim.

6 minor objections

The paper says, at Figure 1

Pith says: minor objection 1 of 6

The figure contains labeled boxes 'Casual Summary' (twice); the intended word is likely 'Causal Summary'. This typo recurs in the figure and should be corrected.

The paper says, at Table II

Pith says: minor objection 2 of 6

In the row headers for the RL model, the abbreviation 'E-Critix RL' appears instead of 'SWE-Critix RL'. This is inconsistent and may confuse readers.

show 4 more

The paper says, at Equation (2)

Pith says: minor objection 3 of 6

The equation is missing parentheses in the expectation: it reads 'Ex∼DRL,y∼πθ(.|x) [r(x, y)]−βD KL...'. The maximization expression should be 'max_{πθ} [E_{x∼DRL, y∼πθ(.|x)} r(x,y)] - β D_KL(πθ || π_SFT)' for clarity.

The paper says, at §IV.5

Pith says: minor objection 4 of 6

The future work item 5 lists 'qualititive analysis'; the intended spelling is 'qualitative'.

The paper says, at References

Pith says: minor objection 5 of 6

Several reference entries lack consistent capitalization or punctuation (e.g., [10], [12], [15]). Please ensure uniform formatting throughout.

The paper says, at §I, Table I

Pith says: minor objection 6 of 6

The table places SWE-Critix in the 'Training-based Generalist' category, but the only generalist methods cited are Agentic Rubrics [10], ExecCritic [28], SWE-Review [29], Gandhi et al. [30], and Scout [31]. Some of these (e.g., Agentic Rubrics) are listed earlier as training-free; the categorization should be checked for consistency.

What would settle it

Select 500 test-set trajectories (250 correct, 250 wrong) and have two domain-expert annotators independently evaluate each generated CoT from π_RL on three binary criteria: (1) Is the CoT's final judgment consistent with the trajectory evidence? (2) Are the specific failure/ success factors identified in the CoT actually present in the trajectory? (3) Is the taxonomic analysis (steps 1–14, 2 in Figure 1) appropriately applied? Report inter-annotator agreement (Cohen's κ) and the fraction of convincing CoTs on each criterion. If the convincing fraction falls below 70% on criterion1, the 'generalist verifier' claim is not supported.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 12 Oct 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic disagrees 1 min 11 s · 13,317 tokens read, 7,864 written
  3. Editor wrote the verdict above 24 s · 14,232 tokens read, 2,636 written
  4. Referee major revision: 4 major objections, 6 minor 31 s · 13,700 tokens read, 3,296 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Another week, another agent evaluation paper. This one from Huawei and Monash proposes SWE-Critix: a pipeline (SFT then RL) that trains a 30B verifier to classify coding-agent trajectories as pass/fail while generating a chain-of-thought explanation. On paper it looks solid—clear engineering, open-sourced models, 119K trajectories, and a held-out test of 43K samples. They report a clean progression from 57% (base) to 66% (SFT) to 78% (RL) accuracy.

What's genuinely new is the specific combination: large-scale SFT with differentially-annotated CoTs from a teacher LLM, followed by outcom-based GRPO on 91% of data without CoTs. That fills a real gap in Table I's taxonomy. The training dynamics plots are helpful, and releasing weights is good practice.

But there are two soft spots that matter.

First, no baselines. The paper doesn't compare against any existing verifier (Agentless, SWE-RM, the training-free LLM-as-a-verifier line, or the prior training-based critics in the same taxonomy). Without a comparison, we have no idea whether 78% is good or medioccre. The authors list this as future work, which is honest but means the central result is uninterpretable right now.

Second, the CoT quality is never evaluated. The paper positions natural-language analysis as the key distinction from scalar critics, yet only measures binary accuracy. The RL reward ignores the CoT entirely. The only CoT supervision is 8.7% of the data via SFT. The authors again flag this as future work. So the 'generalist' half of the contribution is currently a promise.

Beyond these, the evaluation is light on statistical rigor (one run, no error bars) and the generalization to other agent pipelines/LLMs is untested—both acknowledged.

Despit these gaps, the paper is worth engaging with. The pipeline is well-designed and reproducible. For a researcher building verifiers for coding agents, this is a solid starting point that identifies where future work should go. I'd send it out for review—the reviewer can push for baselines and CoT analysis—and I'd be satisfied with a conditional accept after those are added. The authors seem like serious thinkers who know what they haven't done yet.

The strongest objection

The paper positions SWE-Critix as a 'generalist verifier' that 'generates a natural-language analysis that explains its judgment' (Abstract, §I conclusion). This is the stated distinction from prior scalar-reward critics (see Table I taxonomy). However, the evaluation (§III) exclusively measures binary classification accuracy (Table II). The quality, correctness, coherence, and grounding of the generated CoT explanations are never assessed. Moreover, the RL training objective (Equation 3, r = +1/–1 based only on the final <judgment>) provides no reward signal for explanation quality. The only supervision for CoT generation comes from the SFT stage, which uses 10,390 trajectories (~8.7% of training data) with teacher-generated CoTs from GPT-5.5 (§II-B). The SFT loss (Equation 1) minimizes negative log-likelihood over both the CoT and the final judgment, but the RL stage could drift the CoT distribution away from the SFT initialization since there's no stabiliztion for the explanation portion. The paper explicitly acknowledged: 'We plan to conduct a qualitative analysis of the model's reasoning traces...' (§IV.5, future work). But without this evaluation, the central claim of being a 'generlist' verifier (eyond binary classification) rests on trust alone. If the CoTs are mugle or hallucinated, the contribution collapses to training a binary classifier, which is already well-covered by existing scalar critics in Table I.

The reader's weakest_assumption focused on generalizability across agent scaffolds, base models, and issue sources – an external validity concern acknowledged in the paper's future work (§IV.1). I identify a different, more internal concern: the paper's central distinction as a 'generlist verifier' (vs. scalar critics) requires meaningful natural-language explanations, but explanation quality is never evaluated. The paper positions itself as providing both judgment and explanation (§1: 'not only determines whether the task has been sucessfully completed but als justifies the rationale'), and the taxonomy in Table I distinguishes 'Training-based Generalist' from 'Training-based Scalr' precisely on this dimnsion. However, the evaluation (§III) and Table II only report classification accuracy. The RL reward (Equation 3) is purely outcome-based, so CoT quality receives no learning signal during RL. The paper acknowledges this gap in §IV.5 (future work on qualitative analysis). So the load-bearing concern is not about external validity but about whether the paper's core contridution (the 'generalist' aspect) is supported by the evidence. The concrete test (human evaluation of CoT quality on a representative sample) would settle whether the explanation generation is actually meaningful. I recomend the verict reman unchanged (conditional) because the classification results are credible and reproducible, but this concern strengthens the need for additional evaluation before the claim of being a 'generlist verifier' can be fully accepted.

The referee's report

If the reported results hold, SWE-Critix demonstrates a scalable post-training recipe for coding-agent trajectory verification. The combination of SFT bootstrapping reasoning patterns and RL refining discriminative ability is methodologically sound. The training infrastructure and open release are valuable contributions. However, the central claim of a 'generalist verifier' that generates meaningful natural-language analyses is not validated: the evaluation only measures binary classification accuracy, and the quality of the generated CoTs is unassessed. This gap weakens the claimed novelty over existing scalar critics. The paper also lacks comparisons against baseline verifiers and confidence intervals, which lowers the evidential strength.

Pith's reading of the paper (a machine's summary, not the authors')

The paper introduces SWE-Critix, a post-training pipeline that trains a generalist verifier for coding-agent trajectories. The verifier not only decides whether a trajectory solved its task but also produces a natural-language analysis explaining the judgment. Starting from the Qwen3-30B-A3B reasoning model, the pipeline first applies supervised fine-tuning on 10,390 trajectories annotated with chain-of-thought rationales from a teacher model, then applies GRPO reinforcement learning on the remaining 108,250 trajectories using unit-test results as ground-truth reward signals. On the held-out test set classification accuracy rises from 57% (base) to 66% (SFT) to 78% (RL), and the model shifts from a strong positive bias toward balanced discrimination. The work targets the underexplored category of training-based, process-oriented generalist verification for autonomous coding agents.

What carries the argument. The central mechanism is a two-stage post-training pipeline. (1) SFT stage: a teacher LLM (GPT-5.5) compares paired successful and failed trajectories for the same issue and writes a structured chain-of-thought covering procedure summary, mistake taxonomy, and causal analysis. These CoTs are used as targets for supervised fine-tuning on 10,390 trajectories. (2) RL stage: on 108,250 additional trajectories without CoT annotations, the model is trained with GRPO (group size 8) using a rule-based outcome reward: +1 if the binary judgment matches the unit-test ground truth, −1 otherwise. The reward, together with a KL-divergence penalty against the SFT checkpoint, sharpens discriminative ability. The verifier input is the full trajectory and task, output is a CoT followed by a <judgment>YES/NO</judgment> tag.

Core claim. SWE-Critix establishes that a two-stage post-training recipe—first supervised fine-tuning (SFT) on a small set of differentially annotated chain-of-thought examples, then reinforcement learning (RL) with a rule-based outcome reward on a much larger unannotated set—can turn a general large language model into a process-level generalist verifier for coding-agent trajectories. The verifier classifies each trajectory as successful or failed and produces a structured natural-language rationale. On the combination of three benchmark suites (R2E-Gym, SWE-Rebench, SWE-Smith) from CodeForge-Preview, the Qwen3-30B-A3B model's accuracy improves from 57% (base) to 66% (SFT) to 78% (RL). The SFT stage primarily corrects the base model's strong positive bias and improves instruction following, while the RL stage recovers positive recall from 67% to 85% without substantially sacrificing negative recall.

The premise it rests on. All trajectories used for training and evaluation come from a single source (CodeForge-Preview) generated by one specific agent scaffold and one base model; the verifier's ability to judge trajectories from different agents, models, or issue collections is untested, so its generalizability remains an open question.

If the paper is right (Pith's inferences, not the authors')

The trained verifier can serve as an automated judge for coding-agent trajectories in scenarios where ground-truth tests are unavailable, providing both a success/failure label and an explanatory rationale.

The verifier's judgment and analysis could be used to identify specific failure modes in a trajectory, enabling targeted error correction or agent rollback.

The verifier's outcome signal could replace or augment unit-test rewards during reinforcement learning of new coding agents, potentially guiding policy improvement without test suites.

The open-source release of models, datasets, and training code allows the community to reproduce the results and adapt the pipeline to other agent scaffolds, base models, or task domains.

If extended to other trajectory distributions, the pipeline could become a general method for converting a generic LLM into a domain-specific process monitor for autonomous systems.

The 57%→78% accuracy gain is impressive on this test set, but the test set shares the same generator (OpenHands + Qwen3-Coder-480B) as the training data. A critical next test is whether the verifier generalizes to trajectories from different scaffolds, base models, or issue repositories—something the authors flag as future work.

The negative-class recall of 64% after RL remains the main weakness; the verifier still struggles with failed trajectories. This asymmetry suggests that failure patterns are more diverse or that the training data's negative examples do not cover the full spectrum of real-world failures.

The RL stage uses only an outcome reward (pass/fail). Incorporating intermediate process rewards—such as reward per step or per action—could further improve the verifier's ability to pinpoint the moment of failure, a direction the authors plan to explore.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Coding agents are increasingly deployed for end-to-end software engineering tasks, where a readily available oracle evaluator that defines the success criteria is often absent. This motivates verifications for coding agents that can reliably judge and explain their behavior. However, verification is non-trivial as coding agents are increasingly capable of solving long-horizon software engineering tasks. Their execution trajectories can be extremely long, containing hundreds of iterative interactions with code repositories and external tools. Existing verifiers either rely on general-purpose LLMs without specialized training or train dedicated critics that predominantly focus on producing scalar rewards, highlighting the relatively limited exploration of training-based generalist verification. We introduce SWE-Critix, a post-training pipeline for training generalist verifiers of coding-agent trajectories. Given a task and its execution trajectory, the learned verifier predicts the completion label and generates a natural-language analysis that explains its judgment. Our pipeline combines supervised fine-tuning (SFT) with reinforcement learning (RL) and trains a series of 30B models on trajectories from CodeForge-Preview. On a held-out test set, SWE-Critix improves classification accuracy from 57% for the base Qwen3-30B-A3B reasoning model to 66% after SFT and 78% after RL. We release our models, datasets, and training code to facilitate further research.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-12

The authors' main claim, as the classifier picked it out 0.97

“On a held-out test set, SWE-Critix improves classification accuracy from 57% for the base Qwen3-30B-A3B reasoning model to 66% after SFT and 78% after RL.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.13 · human participants 0.04 · controlled comparison 0.90 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.91 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Coding agents are increasingly deployed for end-to-end software engineering tasks, where a readily available oracle evaluator that defines the success criteria is often absent. background 0.90 0.05 0.02 asserted 0.99
This motivates verifications for coding agents that can reliably judge and explain their behavior. background 0.76 0.05 0.05 asserted 0.93
However, verification is non-trivial as coding agents are increasingly capable of solving long-horizon software engineering tasks. background 0.92 0.07 0.02 asserted 0.99
Their execution trajectories can be extremely long, containing hundreds of iterative interactions with code repositories and external tools. background 0.99 0.13 0.02 asserted 0.99
Existing verifiers either rely on general-purpose LLMs without specialized training or train dedicated critics that predominantly focus on producing scalar rewards, highlighting the relatively limited exploration of training-based generalist verification. gap 0.98 0.05 0.04 asserted 0.99
We introduce SWE-Critix, a post-training pipeline for training generalist verifiers of coding-agent trajectories. contribution 1.00 0.93 0.15 asserted 1.00
Given a task and its execution trajectory, the learned verifier predicts the completion label and generates a natural-language analysis that explains its judgment. method 0.83 0.82 0.11 asserted 1.00
Our pipeline combines supervised fine-tuning (SFT) with reinforcement learning (RL) and trains a series of 30B models on trajectories from CodeForge-Preview. method 1.00 0.95 0.07 asserted 1.00
On a held-out test set, SWE-Critix improves classification accuracy from 57% for the base Qwen3-30B-A3B reasoning model to 66% after SFT and 78% after RL. result 1.00 0.96 0.43 asserted 1.00
We release our models, datasets, and training code to facilitate further research. contribution 0.77 0.92 0.05 asserted 0.97

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-12: 1 request, 7,511 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (2)
Figure 1 of the paper
Figure 1. SWE-CRITIX Post-Training Pipeline and Data Splits. underlying its judgment. Compared with the two existing approaches in this category, SWE-CRITIX differs from them in terms of verification target (planning phase [30] vs. whole trajectory), annotation strategy (per-step dense labeling [31] vs. differential analysis), scale (2–8K training samples and 4– 8B models [30], [31] vs. 119K and 30B), learning algorithms (DPO [30] vs. GRPO), and reward design (process reward [31] vs. outcome reward). Using this pipeline, we train a Qwen3-30B MoE verifier on a distributed cluster equipped with…
Figure 2 of the paper
Figure 2. We refer to the resulting checkpoint as πRL and release its weights on Hugging Face [34]. III. PRELIMINARY EVALUATION As a preliminary evaluation, we focus on the model’s ability to distinguish between successful and failed trajectories. We evaluate πRL, πSFT, and the base Qwen3 MoE model on the test split created in Section II-A. We serve these models using vLLM Ascend [49] on one node containing 16 Huawei Ascend 910C-64GB NPUs. Given the large scale of the test set (over 43K long-trajectory samples), we evaluate each model on each sample only once. We report accuracy, as well as

Figures are the authors', reproduced from the paper.

Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment