{"id":"d96c195a-4e3e-4bf8-b4c7-cc642f4d74d4","arxiv_id":"2508.02921","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A tool-using LLM judge with hierarchical rubrics can assess whether pentest agents meet operational requirements, matching human experts with F1 0.83.","lead":"PentestJudge uses a large language model to score whether penetration testing agents followed operational rules by reading their full action history. It reached an F1 score of 0.83 against human expert judgments, and suggests weaker models can check the work of stronger ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human expert labels are the reference for every F1 score, but inter-annotator agreement and adjudication are unreported; without them the 0.83 ceiling is uninterpretable.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the evaluation treats human domain expert labels as ground truth without reporting the reliability of those labels. I agree with this assessment. The supplied full text is corrupt, so no method details, rubric content, or annotation statistics can be verified from the paper itself. The abstract alone cannot support ACCEPT, CONDITIONAL, or REJECT; the appropriate status remains UNVERDICTED. The concrete test above would distinguish between 'judge agrees with experts' and 'judge agrees with the particular experts who produced the labels,' which is the central uncertainty behind the F1 0.83 claim. No verdict adjustment is needed: the reader's UNVERDICTED remains the honest outcome, and the concern reinforces it rather than moving it in a new direction.","tokens_in":10718,"tokens_out":3003,"duration_ms":36285,"concrete_test":"Run a labeling-reliability audit: take a stratified sample of 50 trajectories and 10 leaf criteria, have three independent penetration-testing experts (not involved in rubric authoring) label them, and compute Fleiss kappa. Then recompute each judge model's F1 against the majority-vote labels and against each individual expert. If kappa is below 0.6, or if judge-model F1 against majority labels is within noise of human-expert pairwise agreement, the reported 0.83 does not establish near-expert performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PentestJudge can judge pentest-agent trajectories nearly as well as human domain experts, with the best judge model reaching F1 0.83. That claim is only meaningful if the human labels are a stable, valid reference. The abstract reports no inter-annotator agreement (e.g., Cohen's or Fleiss kappa), no number of experts per trajectory, no adjudication protocol, and no demonstration that the rubric tree's leaf nodes map unambiguously to the operational requirements. If two experts commonly disagree on a leaf criterion, the apparent ceiling is one expert's opinion rather than a field consensus; a judge could match one annotator while disagreeing with the rest. If the rubric authors also produced the labels, the F1 may reflect rubric-echo rather than independent expertise. This is a correctness risk, not merely a disagreement with current consensus. Because the supplied full text is corrupted, I cannot check whether these details are present in the methods or the appendix; based on the abstract alone, the human ground-truth reference is the weakest load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PentestJudge, an LLM-as-judge system with tool access that evaluates penetration-testing agent trajectories against a hierarchical rubric tree of operational requirements, operational security, and tradecraft. The judge scores yes/no leaf criteria and aggregates them for comparison with human domain expert labels. The central reported result is that the best judge model reaches an F1 score of 0.83, that models better at tool-use agree more closely with human experts, and that weaker/cheaper models can judge trajectories produced by stronger/more expensive models, suggesting verification may be easier than generation. The supplied full text is heavily corrupted, so this assessment is based primarily on the abstract and a few readable fragments; the methods and evaluation details could not be fully verified.","tokens_in":10901,"tokens_out":3285,"duration_ms":40292,"significance":"If substantiated, the contribution is a useful methodological step toward scalable, process-level evaluation of AI security agents. The hierarchical rubric tree is a sensible decomposition of a complex task into verifiable leaf criteria, and the stratification of F1 by requirement type is a thoughtful analysis that goes beyond a single overall score. The claim that cheaper judges can verify stronger agents, if supported, has practical implications for deployment. However, the reported F1 and the associated comparisons are not interpretable without the missing evaluation details: dataset size, class balance, confidence intervals, inter-annotator agreement, and rubric-validation procedures. These are required before the headline result can be relied upon.","major_comments":[{"comment":"The headline F1 of 0.83 is reported without any dataset size, class balance, confidence intervals, or inter-annotator agreement. Because human expert labels are the reference against which all scores are computed, the claim that PentestJudge performs 'nearly as well as human experts' is uninterpretable unless the stability and validity of those labels is established. Please report the number of experts per trajectory, the adjudication protocol, and a kappa statistic; without these, a low-reliability or biased label set caps or distorts the reported F1 ceiling.","section":"Abstract / Evaluation"},{"comment":"The judge's decision procedure is underspecified. The abstract states that the rubric tree has yes/no leaf nodes and that scores are compared via binary classification, but it does not state how leaf votes are aggregated, what confidence threshold is used, or which prompt, temperature, and tool-use configuration yields the reported F1. These details are necessary for reproducibility and for assessing whether the 0.83 result is robust to reasonable variations in the judge's inference settings.","section":"Methods / Judge decision procedure"},{"comment":"There is a potential circularity risk: if the same authors designed the rubric tree and also guided or instructed the human experts, agreement between the judge and the experts may reflect shared rubric exposure rather than independent expertise. Please state whether human labelers were given the rubric, whether they were blind to the judge outputs, and how the rubric's completeness was validated against the operational requirements. This is load-bearing because the entire evaluation treats human labels as ground truth.","section":"Human ground-truth collection"},{"comment":"The abstract's claim that 'weaker and cheaper models can judge the trajectories of pentests performed by stronger and more expensive models' is not supported by an overall F1 alone. A controlled comparison is needed: report which judge/generator model pairs were tested, the trajectory sources, the sample sizes per pair, and whether the finding holds after stratifying by requirement type. Without this, the 'verification may be easier than generation' conclusion is premature.","section":"Model comparison / verification claim"}],"minor_comments":[{"comment":"The terms 'operating criteria' and 'operational requirements' are used interchangeably; please define them once and use them consistently throughout.","section":"Abstract / Introduction"},{"comment":"All reported F1 scores and per-requirement stratifications should include error bars or confidence intervals; the garbled text prevents me from checking whether the current figures include these.","section":"Figures / Tables"},{"comment":"The abstract does not cite any related work on LLM-as-judge or on penetration-testing evaluation benchmarks; please add a short related-work discussion to position the contribution.","section":"Related work"}],"recommendation":"major_revision","confidential_remarks":"The supplied full text is corrupted, so I could not verify the methods, experiments, or appendices. The abstract alone is insufficient to establish the central claim; the authors should be asked to provide a clean manuscript and to address the missing evaluation details explicitly. The topic fits the journal's scope, but the evidence is currently incomplete."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the abstract describes a genuinely useful idea — a tool-using LLM judge with a hierarchical rubric tree to audit pentest agent trajectories for process compliance — and the F1 of 0.83 is plausible. But I'm working from the abstract only: the supplied full text is corrupted, so I can't verify the methods, data, or rubric construction.\n\nWhat's new: using LLM-as-judge on full trajectories with a tree of yes/no leaf criteria is a practical step forward for security evaluations. The per-requirement breakdown is good; overall F1 hides too much. The weaker-model-verifies-stronger-model finding is interesting and worth testing. The comparison to human experts is the right framing.\n\nSoft spots: the abstract omits dataset size, class balance, confidence intervals, inter-annotator agreement, and how human labels were collected and adjudicated. That matters because the F1 is only as good as the reference. If experts disagree on a leaf criterion, a judge can match one expert and miss the others and look better than it should. And since the authors built the rubric tree and likely guided the experts, there's a risk of rubric-echo — agreement that reflects the rubric rather than independent expertise. These are real concerns. They may be addressed in the full paper; I can't check.\n\nAlso missing from the abstract: the judge's decision threshold, prompt details, and whether the rubrics and annotations will be released. Those matter for replication, but they're the kind of thing a referee can request.\n\nWho's this for: security teams evaluating agentic pentest tools, and anyone working on LLM-as-judge or process compliance. It's a solid subfield contribution, not a paradigm shift. I'd send it to peer review — the claims are concrete, the evaluation design is sound in principle, and a good referee can push for the missing details. I wouldn't cite the abstract alone, but I'd read the full paper.","headline":"The abstract makes a plausible case for rubric-tree LLM judges of pentest agent trajectories, but the full text is unreadable and the human-label reference needs scrutiny.","tokens_in":11418,"tokens_out":3028,"would_cite":false,"duration_ms":35424,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PentestJudge claims a tool-using LLM judge can grade pentest agents against operational requirements with F1 0.83 versus human experts.","keywords":["LLM-as-judge","penetration testing","agent evaluation","hierarchical rubric","operational requirements","tool use","security agents","verification vs generation"],"falsifier":"Compute inter-annotator agreement among the human experts on a shared set of trajectories; if agreement falls well below the best judge's $F_1$ of 0.83, the judge's apparent accuracy against a single expert label set is not evidence of true evaluation quality. Alternatively, have a judge score a trajectory in which an independently verified critical objective was missed; if the judge marks every leaf criterion as satisfied, the central claim is refuted.","tokens_in":10534,"feed_emoji":"🛡️","tokens_out":3588,"duration_ms":39483,"temperature":0.7,"pith_summary":"This paper argues that a large language model can judge whether a penetration-testing agent followed operational requirements nearly as well as human domain experts. It introduces PentestJudge, a tool-using LLM judge that reads full agent trajectories and applies a hierarchical rubric tree that breaks the mission into objectives, operational security, and tradecraft, down to simple yes-or-no leaf questions. The best judge model reaches an F1 score of 0.83 against human expert labels. The paper also claims that weaker and cheaper models can judge trajectories produced by stronger and more expensive models, which would mean verification is easier than generation for penetration-testing tasks.","feed_headline":"PentestJudge matches human experts on pentest quality at F1 0.83","feed_subtitle":"A tool-using LLM judge with hierarchical rubrics verifies that cheaper models can vet costlier agents' missions.","key_machinery":"The central object is the hierarchical rubric tree: a tree whose root is the overall task, whose branches split into categories (operational objectives, operational security, and tradecraft), and whose leaves are simple yes-or-no criteria. The judge agent is an LLM with tools that let it consult arbitrary parts of the trajectory and tool-call history to answer each leaf, and parent nodes are only as satisfied as their children. This decomposition is what lets a holistic judgment be scored mechanically and compared with human labels.","core_discovery":"On the paper's own terms, the central discovery is that the evaluation problem for pentest agents can be handed to an LLM judge equipped with a hierarchical rubric tree, and that this judge agrees with human domain experts closely enough to be useful: the best judge model reaches $F_1 = 0.83$. The same evaluation shows that better tool-use ability in the judge tracks closer agreement with human experts, and that models with similar overall scores can differ by requirement type. The finding that weaker and cheaper models judge trajectories produced by stronger and more expensive models is presented as evidence that verifying a pentest trajectory is easier than producing one.","pith_inferences":["The verification-easier-than-generation claim is drawn from judge-trajectory pairs in one environment; the same asymmetry would need testing across other environments and agent types before it generalizes.","A testable extension is to measure inter-annotator agreement among human experts and use it to bound the maximum achievable judge F1; without this ceiling, the reported 0.83 is not directly interpretable.","The rubric-tree decomposition may transfer to other long-horizon agent evaluation tasks where programmatic checks miss process quality, such as incident response or red-team operations.","If judges are calibrated only to expert labels, they may inherit expert blind spots; adversarial trajectories designed to fool both experts and judges would probe this."],"forward_implications":["Cheaper and weaker models can audit trajectories produced by stronger and more expensive pentest agents, so evaluation cost need not scale with generation cost.","Stratifying scores by requirement type lets an operator choose a judge model that is strongest on objectives, operational security, or tradecraft rather than relying on a single overall score.","Better tool-use ability in the judge model predicts closer agreement with human experts, making tool-use skill a practical selection criterion for judges.","Rubric trees convert holistic process-quality judgments into leaf-level yes-or-no checks that can be audited item by item and reused across environments.","If verification is genuinely easier than generation, organizations can validate sensitive production agents with weaker models while reserving expensive models for acting."],"supporting_citations":[],"fun_headline_variants":["LLM judge scores pentest agents at F1 0.83 vs human experts","Cheaper LLM judges can verify pricier pentest agents' work","Hierarchical rubric judge scores pentest quality at F1 0.83","Verifying pentest agents may be easier than running them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The human domain expert labels are taken as ground truth, and the paper reports no inter-annotator agreement; if the experts do not reliably agree with each other, the judge's F1 of 0.83 is measured against an unstable reference.","fun_headline_variants_meta":{"raw":{"variants":["LLM judge scores pentest agents at F1 0.83 vs human experts","Cheaper LLM judges can verify pricier pentest agents' work","Hierarchical rubric judge scores pentest quality at F1 0.83","Verifying pentest agents may be easier than running them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000626,"raw_usage":{"total_tokens":2916,"prompt_tokens":983,"completion_tokens":1933,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1852}},"tokens_in":599,"tokens_out":1933,"duration_ms":13956,"temperature":1.0,"reasoning_tokens":1852,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:46:52.864238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute inter-annotator agreement among the human experts on a shared set of trajectories; if agreement falls well below the best judge's $F_1$ of 0.83, the judge's apparent accuracy against a single expert label set is not evidence of true evaluation quality. Alternatively, have a judge score a trajectory in which an independently verified critical objective was missed; if the judge marks every leaf criterion as satisfied, the central claim is refuted.","supporting_citations":[],"review_version":1}