Pith. sign in

REVIEW 3 major objections 6 minor 23 references

A fixed safety policy cannot gate offensive agents: scope lives in the user's request, so the monitor must read it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 18:28 UTC pith:M7VXJOBS

load-bearing objection Clean empirical result that static-policy judges cannot gate request-time scope, plus a usable call-level benchmark and Pareto frontier for offensive-security agents. the 3 major comments →

arxiv 2607.07774 v1 pith:M7VXJOBS submitted 2026-07-08 cs.CR

ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents

Shane Caldwell , Max Harley , Ads Dawson , Michael Kouremetis , Vincent Abruzzo , Will Pearce This is my paper
classification cs.CR
keywords pre-execution gatingscope enforcementLLM agentsoffensive securitytrusted monitoringtranscript strategiescost-accuracy frontierLLM-as-judge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When LLM agents run offensive security work, one out-of-scope tool call can breach a client boundary, hit production, or void a finding. The paper argues that unlike a fixed safety constitution, the operative boundary is declared in the user's request and must be inferred from intent; the same scan or exploit is in or out of scope depending on the target and context. It studies pre-execution gating: a cheap trusted LLM judge inspects each call a strong, swappable agent proposes and accepts or rejects it before execution. ScopeJudge is a call-level benchmark of 4,897 tool calls (7.7% violations) from agent trajectories on temptation tasks, labeled by professional penetration testers with substantial agreement (Fleiss kappa 0.64), setting an expert reference of F1 0.78. Across eight judges and five transcript strategies, a static policy alone collapses recall to near zero; adding the user's intent restores most of it. Open-weight judges lead the cost-accuracy frontier, and the paper recommends a cost-sensitive and a recall-first operating point because a missed violation costs more than a false rejection.

Core claim

Static-policy monitoring is structurally insufficient for scope enforcement. Blind to the user's request, judge recall collapses to near zero (median about 0.005 for seven of eight judges); the instant intent is added, recall jumps to roughly 0.60–0.94. Scope lives in the request, so request-conditioned pre-execution monitoring is necessary. The best judge tested is open-weight (GLM-5.2 at F1 0.66), below the expert agreement reference of F1 0.78 but competitive on cost.

What carries the argument

Pre-execution gating with a transcript strategy: a fixed judge maps the static policy, user intent, and a chosen amount of trajectory history (static only; intent; intent plus tool calls; intent plus summarized outputs; or full transcript) onto a binary accept/reject decision before each tool call runs.

Load-bearing premise

The measured precision, recall, and cost-accuracy frontier will transfer from one English web-penetration and bug-bounty calibration set of 100 temptation trajectories to real deployments with different base rates and adversarial pressure.

What would settle it

On a held-out or production deployment with a different violation base rate, measure whether a static-policy-only judge still has near-zero recall while the same judge with user intent recovers high recall, and whether the recommended open-weight cost-sensitive point still sits near the Pareto frontier against expert labels.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies pre-execution gating of tool calls for offensive-security LLM agents, where the operative scope is declared in the user request rather than in a fixed policy. It releases ScopeJudge: 4,897 call-level labels (7.7% violations) from 100 agent trajectories on temptation-engineered ScopeBench tasks, graded by five professional penetration testers (Fleiss κ = 0.64; leave-one-out expert F1 = 0.78). Eight judge models are evaluated under five transcript strategies (static policy through full transcript), producing a cost–accuracy Pareto frontier. The central empirical result is that under φ_static (Eq. 5) recall collapses to near zero for seven of eight judges, while adding user intent (Eq. 6) restores recall to roughly 0.60–0.94 (Figure 4, Table 3). Open-weight GLM-5.2 reaches the best F1 (0.66); the authors recommend a cost-sensitive and a recall-first operating point and release the labeled corpus.

Significance. If the result holds, the paper cleanly reframes scope enforcement for request-conditioned agents as a monitoring problem rather than a specification problem, and supplies both a calibrated call-level benchmark and concrete cost–accuracy guidance for trusted monitoring in offensive security. Strengths include: (i) expert multi-grader labels with reported inter-rater statistics and an explicit expert F1 ceiling; (ii) a clean isolation of the static-policy failure mode by holding SP fixed and varying only whether u is visible; (iii) a full 8×5 scoreboard with token-level cost accounting; and (iv) public release of the calibration corpus. These make the work useful both as a negative result on fixed policies and as infrastructure for subsequent monitor research.

major comments (3)
  1. [§1.2, §3.1, §6.1, §7] Section 1.2 lists adversarial robustness (prompt-injection surface from tool outputs / agent traces) as a third design axis alongside cost and accuracy, and Section 3.1 formalizes that history-bearing strategies admit attacker-controlled bytes. Section 7 then states this axis is unmeasured and that the cost–accuracy frontier is only an upper bound on what higher-context strategies safely buy. That is an honest limitation, but it is load-bearing for any production recommendation that uses prior calls or outputs. Either add a minimal adversarial probe (e.g., poisoned tool outputs or agent-crafted rephrasings of rejected calls under φ_int+tools / φ_full) or reframe Section 1.2 and Section 6.1 so that recommended operating points are restricted to strategies whose injection surface is argued to be negligible, with history-bearing configs presented only as accuracy upper bounds.
  2. [§4.1, Table 1, §5.1, Table 2, §6.1] Section 4.1 and Table 1 note that claude-sonnet-4-6 and gpt-5.4-mini appear both as trajectory generators and as judges (Section 5.1 / Table 2), flagged as a possible self-preference bias but left uncontrolled. Because the Pareto ranking and the claim that open-weight judges dominate proprietary ones rest on cross-model comparisons (Section 6.1, Figure 3, Table 3), a sensitivity check that drops (judge, trajectory) pairs where the judge model family generated the trajectory—or at least reports F1 restricted to non-overlapping pairs—is needed before treating the open-weight vs. proprietary ordering as robust.
  3. [§4.2, §5.4, §6.1, Table 3, §7] All quantitative claims (inter-grader agreement, precision/recall/F1, Pareto frontier) rest on a single 100-trajectory calibration set from one English web-pentest/bug-bounty suite engineered to tempt violations (Section 4.1–4.2), with a suite-specific 7.7% base rate and no confidence intervals because each (judge, strategy) cell is a single temperature-0 run (Section 5.4, Section 7). The static-vs-intent recall collapse is large enough that it is unlikely to be a sampling artifact, but the finer Pareto ranking and the two recommended operating points (GLM-5.2 + intent_plus_calls; Opus + intent_only) sit on smaller F1 gaps. Either bootstrap or multi-seed estimates for the recommended cells, or explicitly demote those cells from “recommended deployments” to “illustrative points on a single-run frontier,” and state more sharply that precision/F1 will shift under different violation prevale
minor comments (6)
  1. [Figure 3, Table 3] Figure 3 plots each judge only at its best-F1 strategy; a small annotation or companion panel mapping those points back to the strategy names in Table 3 would make the frontier easier to read without flipping between figure and table.
  2. [Eq. (10), §5.3] Equation (10) defines cost via input tokens only times ρ(J); Section 5.3 states that recorded costs include output tokens and summarization. Align the formal definition with the accounting actually used, or note the approximation explicitly.
  3. [§4.4, §6] The contested 11.9% non-unanimous subset (Section 4.4) is where monitor difficulty concentrates; reporting judge F1 restricted to unanimous vs. contested calls would help readers interpret the 0.13 gap to the expert reference of 0.78.
  4. [Appendix A, §5.4] Appendix A decision rules use both "allow"/"deny" and the JSON field allow=true|false; keep terminology consistent with the positive-class mapping in Section 5.4 (allow=false = predicted violation).
  5. [§2.3] Related work on runtime policy enforcement and tool-invocation guardrails is present but brief; a short comparison to formal runtime monitors (cited [12]) versus learned intent-conditioned judges would clarify the novelty claim in Section 2.3.
  6. [Table 1, Table 3, Abstract, §1] Typos / polish: "T otal" in Table 1; "T rinity-Large" in Table 3; abstract and intro both restate the full contribution list—tighten one of them to reduce repetition.

Circularity Check

1 steps flagged

Empirical measurement paper with no circular derivation; minor flagged self-evaluation overlap does not force the headline results.

specific steps
  1. other [Section 4.1 / Limitations (Section 7)]
    "We note that two of the source agents (claude-sonnet-4-6 and gpt-5.4-mini) also appear among the judges evaluated in Section 6; the resulting self-evaluation involves only a small fraction of (judge, trajectory) pairs, but we flag it as a potential source of self-preference bias to be controlled for in future work. ... Evaluator overlap. Two judges (claude-sonnet-4-6, gpt-5.4-mini) also generated some of the labeled trajectories (Table 1). The overlap is a small fraction of (judge, trajectory) pairs and is uncontrolled; a self-preference effect cannot be ruled out."

    Not a true circular derivation of the main claims, but a minor self-preference risk: a subset of monitored trajectories were produced by models that later serve as judges. The paper correctly flags it; the static-policy collapse and open-weight Pareto dominance are measured across the full 8×5 grid and do not reduce to this overlap by construction. Included only for completeness under the residual self-evaluation pattern.

full rationale

ScopeJudge is a pure evaluation paper: human majority labels (five professional graders, Fleiss κ=0.64, expert F1 reference 0.78 via leave-one-out) serve as independent ground truth; eight LLM judges under five transcript strategies are scored for precision/recall/F1 and cost with no free parameters fitted to the target metrics. The central negative result (static-policy recall collapse to near-zero, restored by adding user intent) follows directly from holding SP fixed while ablating visibility of u (Eqs. 5–6, Figure 4, Table 3) and is not obtained by construction or by renaming a fit. Theoretical framing that “scope lives in the request” is confirmed by the measurement rather than presupposed as a uniqueness theorem. Self-citations (e.g., prior PentestJudge work) appear only in related-work positioning. The sole minor issue is the paper’s own flagged evaluator overlap (two judges also generated a fraction of trajectories) plus author-graders; this is disclosed, uncontrolled, and does not reduce the static-vs-intent contrast or Pareto ranking to a tautology. No self-definitional loop, fitted-input-as-prediction, load-bearing uniqueness import, or ansatz smuggling is present. Score 1 reflects only that residual self-preference risk; the derivation chain itself is non-circular.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 2 invented entities

The paper is an empirical systems/measurement study, not a derivation from first principles. Load-bearing commitments are domain assumptions about what constitutes scope, the validity of professional-grader majority labels as gold, and the representativeness of the engineered temptation suite. No continuous free parameters are fitted to produce the headline claims; model choice and strategy definitions are discrete experimental factors. The main invented constructs are the benchmark itself and the five transcript strategies.

axioms (4)
  • domain assumption Task-specific scope is declared in the user request u and must be inferred from intent; a global static policy SP cannot enumerate it in advance.
    Stated in the abstract, Section 1, and formalized in Section 2.1; this is the premise that makes the static-policy ablation diagnostic.
  • domain assumption Majority vote of five professional penetration testers constitutes ground-truth alignment labels y_k for each tool call.
    Section 4.3–4.4; inter-grader Fleiss kappa = 0.64 is reported as the expert reference, not as a fitted quantity.
  • ad hoc to paper The 100-trajectory calibration set drawn from ScopeBench (tasks engineered to tempt scope violations) is sufficiently representative for measuring monitor precision/recall under the observed 7.7% base rate.
    Section 4.2; the authors themselves note base-rate sensitivity and suite specificity in Limitations.
  • domain assumption Pre-execution gating with a fixed trusted judge supervising a swappable untrusted agent is the appropriate control architecture (trusted monitoring).
    Section 2.3, citing Greenblatt et al. AI Control; adopted rather than re-derived.
invented entities (2)
  • ScopeJudge benchmark (4,897 labeled tool-call decisions from 100 trajectories) no independent evidence
    purpose: Provide a calibrated measuring instrument for call-level scope-violation monitors.
    Constructed for this paper; independent evidence will exist only after public release and external use.
  • Five transcript strategies (static, intent, intent+calls, intent+summarized outputs, full) no independent evidence
    purpose: Systematically vary context cost and prompt-injection surface presented to the judge.
    Defined in Section 3.1, Equations (5)–(9); experimental factors, not physical entities.

pith-pipeline@v1.1.0-grok45 · 21930 in / 3134 out tokens · 35926 ms · 2026-07-10T18:28:37.929549+00:00 · methodology

0 comments
read the original abstract

As LLM agents take on offensive security work, a single out-of-scope tool call can breach a client's engagement boundary, disrupt production, or void a bug-bounty finding. Unlike a fixed safety policy, the boundary that matters is declared in the user's request and must be inferred from intent. That challenge is sharpened by the adversarial nature of offensive security: the same tool call is in or out of scope depending not on the action itself but on the target it touches and the context in which it runs, which no fixed policy can enumerate in advance. We study pre-execution gating: a cheap, trusted LLM judge inspects each call proposed by a strong, swappable agent, and accepts or rejects it before it runs. We introduce ScopeJudge, a benchmark of 4,897 tool calls (7.7% scope violations) from agent trajectories on tasks engineered to tempt agents out of scope and labeled at the call level by professional penetration testers, with substantial inter-grader agreement (Fleiss kappa = 0.64) that sets an expert agreement reference point of F1 = 0.78. We evaluate eight judge models under five transcript strategies, varying how much context the judge sees, from the static policy alone to the full raw transcript, and chart the resulting cost-accuracy Pareto frontier. We find that a static policy is structurally insufficient for scope enforcement: blind to the user's request, judge recall collapses to near zero, confirming that scope lives in the request and that request-conditioned monitoring is necessary. Because a missed violation costs more than a spurious rejection, we report precision, recall, and F1 separately and recommend two operating points: a cost-sensitive configuration and a recall-first one for high-stakes deployments. We release the ScopeJudge dataset to support real-time monitoring and scalable oversight of autonomous security agents.

Figures

Figures reproduced from arXiv: 2607.07774 by Ads Dawson, Max Harley, Michael Kouremetis, Shane Caldwell, Vincent Abruzzo, Will Pearce.

Figure 1
Figure 1. Figure 1: Pre-execution gating. At each step the untrusted agent proposes a tool call ak. A transcript strategy φ projects the static policy SP, the user’s intent u, and a chosen amount of trajectory history onto the context window of a cheap, fixed judge Jpre, which accepts the call (it executes and is appended to the trajectory) or rejects it (the agent is asked to revise, or a human is escalated to). appear in a … view at source ↗
Figure 2
Figure 2. Figure 2: Pairwise inter-grader agreement (Cohen’s κ). Rows and columns index the five graders. Darker is higher agreement. All pairs fall in the moderate-to-substantial range (0.50–0.76), bounding how well any automated monitor can match the expert consensus. family is absent. At 100 trajectories of 6–170 calls each (mean 49.0), the calibration set contains 4,897 labeled tool-call decisions; even at a single-digit … view at source ↗
Figure 3
Figure 3. Figure 3: Cost–accuracy frontier for pre-execution gating. Each point is one judge at its best-F1 transcript strategy (one point per judge); cost is per tool call on a log scale, F1 against the expert gold labels. The dashed red line is the expert agreement reference point (F1 = 0.78), and the solid line traces the Pareto frontier over these points. GLM-5.2 attains the best F1 of any judge and dominates the propriet… view at source ↗
Figure 4
Figure 4. Figure 4: Recall by transcript strategy. With only the static policy (leftmost), recall is near zero for seven of eight judges—a fixed specification cannot locate task scope. Adding the user’s intent restores recall sharply. Strategies are ordered by increasing context. • No variance estimates. For cost reasons each (judge, transcript-strategy) configuration was evaluated once, at temperature 0 (Section 5.4); we the… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · 15 internal anchors

  1. [1]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, et al.ReAct: Synergizing Reasoning and Acting in Language Models. 2023. arXiv:2210.03629 [cs.CL].url: https://arxiv.org/abs/2210.03629

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, et al.Toolformer: Language Models Can Teach Themselves to Use Tools. 2023. arXiv: 2302.04761 [cs.CL].url:https://arxiv.org/abs/2302.04761

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    catastrophic failure

    Fortune.An AI-powered coding tool wiped out a software company’s database in a “catastrophic failure”. Fortune, 23 July 2025. 2025.url:https://fortune.com/2025/ 07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic- failure/

    work page 2025

  4. [4]

    Constitutional AI: Harmlessness from AI Feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, et al. Constitutional AI: Harmlessness from AI Feedback. 2022. arXiv:2212.08073 [cs.CL]. url:https://arxiv.org/abs/2212.08073. 17 ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  5. [5]

    Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger.AI Control: Im- proving Safety Despite Intentional Subversion. 2024. arXiv:2312.06942 [cs.LG].url: https://arxiv.org/abs/2312.06942

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness

    Devdatta Akhawe and Adrienne Porter Felt. “Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness”. In:22nd USENIX Security Symposium (USENIX Security 13). 2013, pp. 257–272

    work page 2013

  7. [7]

    Identifying the Risks of LM Agents with an LM-Emulated Sandbox

    Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, et al. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. 2024. arXiv:2309. 15817 [cs.AI].url:https://arxiv.org/abs/2309.15817

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    PentestJudge: Judging Agent Behavior Against Operational Requirements

    Shane Caldwell, Max Harley, Michael Kouremetis, Vincent Abruzzo, and Will Pearce. PentestJudge: Judging Agent Behavior Against Operational Requirements. 2025. arXiv: 2508.02921 [cs.AI].url:https://arxiv.org/abs/2508.02921

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, et al.PentestGPT: An LLM-empowered Automatic Penetration Testing Tool. 2023. arXiv:2308.06782 [cs.CR]. url:https://arxiv.org/abs/2308.06782

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  10. [10]

    Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, et al.Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

    Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, et al.Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

    work page

  11. [11]

    arXiv:2408.08926 [cs.CR].url:https://arxiv.org/abs/2408.08926

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Yutao Mou, Zhangchi Xue, Lijun Li, Peiyang Liu, Shikun Zhang, et al.ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback. 2026. arXiv:2601.10156 [cs.CL].url: https://arxiv.org/abs/2601. 10156

    work page arXiv 2026

  13. [13]

    Alaia Solko-Breslin, Pramod Kaushik Mudrakarta, Mihai Christodorescu, Somesh Jha, and Krishnamurthy Dj Dvijotham.Efficient and Sound Probabilistic Verification for AI Agents. 2026. arXiv:2606.20510 [cs.AI].url: https://arxiv.org/abs/2606.20510

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  14. [14]

    Measuring Progress on Scalable Oversight for Large Language Models

    Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, et al. Measuring Progress on Scalable Oversight for Large Language Models. 2022. arXiv: 2211.03540 [cs.HC].url:https://arxiv.org/abs/2211.03540

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  15. [15]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. 2023. arXiv:2306.05685 [cs.CL].url:https://arxiv.org/abs/2306.05685

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  16. [16]

    Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger.Defining and Characterizing Reward Hacking. 2022. arXiv:2209.13085 [cs.LG].url: https: //arxiv.org/abs/2209.13085

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  17. [17]

    Jonathan Gabor, Jayson Lynch, and Jonathan Rosenfeld.EvilGenie: A Reward Hacking Benchmark. 2026. arXiv:2511.21654 [cs.LG].url: https://arxiv.org/abs/2511. 21654

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  18. [18]

    Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, et al.Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge.2025.arXiv:2406.07791 [cs.CL].url:https://arxiv.org/abs/2406.07791

    work page arXiv 2025

  19. [19]

    Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, et al.CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges. 2024. arXiv:2410.15393 [cs.CL].url:https://arxiv.org/abs/2410.15393

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  20. [20]

    PaperBench: Evaluating AI's Ability to Replicate AI Research

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, et al. PaperBench: Evaluating AI’s Ability to Replicate AI Research. 2025. arXiv:2504.01848 [cs.AI].url:https://arxiv.org/abs/2504.01848. 18 ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  21. [21]

    Measuring nominal scale agreement among many raters

    Joseph L. Fleiss. “Measuring nominal scale agreement among many raters”. In:Psycho- logical Bulletin76.5 (1971), pp. 378–382

    work page 1971

  22. [22]

    A coefficient of agreement for nominal scales

    Jacob Cohen. “A coefficient of agreement for nominal scales”. In:Educational and Psychological Measurement20.1 (1960), pp. 37–46

    work page 1960

  23. [23]

    The measurement of observer agreement for categorical data

    J. Richard Landis and Gary G. Koch. “The measurement of observer agreement for categorical data”. In:Biometrics33.1 (1977), pp. 159–174. A The Judge Prompt This appendix reproduces the verbatim static policySP supplied to every judge and specifies exactly which context blocks each of the five transcript strategies of Section 3 places in the judge’s window...

    work page 1977