Pith. sign in

REVIEW 3 major objections 5 minor 25 references

Foreign-policy AI is a high-stakes evaluation blind spot: almost no public infrastructure exists for it, and standard benchmarks fail its structure.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:50 UTC pith:Q654ACH2

load-bearing objection Solid workshop agenda paper: the public-literature scarcity map is real and carefully QA'd; the urgency claim still leans on a public-to-real leap the authors themselves flag. the 3 major comments →

arxiv 2607.02955 v1 pith:Q654ACH2 submitted 2026-07-03 cs.CY cs.AI

The Foreign Policy AI Evaluation Gap

Charles Pozniak , Jeba Sania This is my paper
classification cs.CY cs.AI
keywords technical AI governanceforeign policystatecraftAI evaluationdemand-side evaluationecosystem monitoringhigh-stakes deploymentcorpus screening
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that AI systems already entering foreign-policy and statecraft workflows—analysis, planning, decision support in diplomacy, sanctions, crisis response, and the use of force—should be a priority test case for technical AI governance. Foreign policy combines extreme downside risk with structural features that break ordinary evaluation: partial observability, unbounded action spaces, contested ground truth, and multidimensional objectives that cannot be collapsed into a single score. A large corpus screen of public technical work finds almost no direct evaluation infrastructure for these deployments, and what little exists is heavily skewed toward assessment rather than access, verification, security, and operationalization. The authors therefore propose a demand-side agenda: break real institutional workflows into bounded, evaluable sub-tasks, keep humans accountable for recombination and judgment, and design evaluation resources with controlled access rather than model-only leaderboards.

Core claim

Foreign-policy AI deployments combine catastrophic tail risk with structural evaluation failures that standard benchmarks cannot handle, and the public technical AI governance literature is nearly empty of direct infrastructure for them—only a handful of paper-ready direct or proxy hits appear among tens of thousands of screened works, with capacity focus skewed toward assessment over access, verification, security, and operationalization.

What carries the argument

A demand-side evaluation framework that decomposes foreign-policy workflows into bounded, evaluable sub-tasks (with task cards specifying scenario, signals, human role, and scoring) under institutional constraints, rather than treating statecraft as a single model capability or leaderboard score.

Load-bearing premise

That a title-and-abstract screen of the public research record is a faithful enough proxy for the real governance gap, including classified government and vendor practices that never appear in public titles.

What would settle it

A systematic full-text or classified-access audit that turns up a substantial body of rigorous, externally inspectable foreign-policy AI evaluation work—benchmarks, audits, process verification, or operationalization protocols—would undermine the claimed public infrastructure gap.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that AI systems used for foreign-policy/statecraft workflows should be a priority domain for technical AI governance (TAIG). It claims that statecraft combines catastrophic downside risk with structural evaluation difficulties (partial observability, unbounded action spaces, contested ground truth, multidimensional objectives), maps these difficulties onto the six TAIG capacities of Reuel et al./Bucknall & Trager, and reports a title-and-abstract corpus screen of ~80k works finding only ~12 paper-ready direct/proxy evaluation hits, with an ASSESSMENT-heavy skew. It then proposes a demand-side agenda that decomposes foreign-policy workflows into bounded, human-supervised task families (Table 2) rather than holistic model leaderboards.

Significance. If the public-literature scarcity result and the structural diagnosis hold, the paper identifies a high-consequence blind spot in TAIG and offers a concrete, task-scoped research program rather than a generic call for more benchmarks. Strengths include unusually careful documentation of the evidence map (denominators, deterministic buckets, adjudication queue, negative audit, manual QA reducing 14 raw labels to 12 conservative hits; §5 and Appendix A), explicit limitations on classified practice and title-abstract screening (§6.1), and a demand-side framework that preserves human authority over contestable judgments. The contribution is primarily agenda-setting and empirical mapping rather than a new evaluation method or theorem, but it is well positioned for a technical AI governance workshop/journal audience.

major comments (3)
  1. §1, §7, and the abstract treat public scarcity of evaluation papers as evidence of inadequate evaluation of systems 'already being deployed in the conduct of war and peace.' §6.1 correctly notes that classified government/vendor practices may exist and that the screen is title-and-abstract only. The leap from public-literature gap to real governance gap is load-bearing for the urgency claim. Please either (a) reframe the headline claim as a public-ecosystem / transparency gap, or (b) add a short, evidence-bounded discussion of what would falsify the real-gap inference (e.g., known public procurement/audit disclosures, redacted system cards, or practitioner surveys), so the priority ranking does not rest solely on absence of open papers.
  2. §4 claims an asymmetric focus on ASSESSMENT over ACCESS, VERIFICATION, SECURITY, and OPERATIONALIZATION, and contribution (ii) presents this as an ECOSYSTEM review result. The corpus evidence in §5 and Appendix A primarily establishes scarcity of direct/proxy statecraft evaluation work (~12 hits) and adjacent-domain activity; it does not report a capacity-by-capacity breakdown of those hits or of the 413 domain+method rows that would quantify the claimed asymmetry. Either add such a tabulation (even for the strict set and accepted rows) or soften the capacity-asymmetry claim to a conceptual diagnosis supported by the literature review rather than an empirical result of the screen.
  3. Table 2 and §6 propose demand-side task cards and scoring regimes (coverage, calibration, provenance, human-review triggers, etc.) as the path forward, but the manuscript does not specify even one fully worked task card with inputs, allowed tools, success criteria, and expert-judgment protocol. Without at least one concrete example (e.g., escalation-signal detection or draft-language review), it is hard to assess whether the agenda is operationalizable or merely a useful taxonomy. A single appendix task card would substantially strengthen the central constructive claim.
minor comments (5)
  1. Table 1 reports 14 raw direct/proxy labels and 7+7 before QA, while the main text and Appendix A settle on 12 paper-ready hits (5 direct + 7 proxy). Align the table wording with the post-QA headline count to avoid reader confusion.
  2. §5.2 footnote 3 lists the five direct and seven proxy papers; consider moving this list into a short main-text table or Appendix table for easier verification against the QA discussion in A.7.
  3. The manuscript uses both 'ECOSYSTEM review' and 'ECOSYSTEMMONITORING' capacity language; a brief note distinguishing the paper's review exercise from the TAIG capacity would reduce terminology collision.
  4. Several arXiv-style citations in the reference list lack final venue/page information where available (e.g., published FAccT/AIES versions); clean these for the camera-ready version.
  5. §3's distinction between 'inference' in the IR sense and 'inference' in the AI stack is helpful; consider a one-sentence reminder when the term reappears in later sections.

Circularity Check

0 steps flagged

No significant circularity: the scarcity claim is an external corpus count, not a tautological redefinition of its inputs.

full rationale

This is a position/agenda paper, not a derivation of a quantitative prediction from first principles. The three contributions—(i) structural difficulty of foreign-policy evaluation, (ii) an ECOSYSTEM map showing ASSESSMENT-heavy scarcity, and (iii) a demand-side task decomposition—are argumentative and empirical, not self-definitional. The load-bearing empirical result (≈12 paper-ready direct/proxy hits out of ~80k screened works) comes from a deterministic title/abstract screen plus LLM adjudication and manual QA over an external merged corpus, with a negative audit reporting zero strict misses; that count is not fitted to produce the gap, nor is the gap defined as the count. The Reuel–Bucknall TAIG taxonomy is imported as an organizing lens from prior work and applied to statecraft; it is not a uniqueness theorem that forbids alternatives, and it is not used to force the scarcity numbers. Defining Diplomacy/Civilization-style work as “proxy” rather than “direct” is a conservative boundary choice that affects classification, not a circular reduction of the claim to its inputs. Limitations (§6.1) explicitly flag that the public screen may overstate the practical gap if classified evaluation exists—so the paper does not smuggle the public-to-real leap as a derivation. Score 1 only for mild framing self-reinforcement (statecraft defined so that stylized games count as proxy), which is not load-bearing circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

This is an agenda/evidence-map paper, not a fitted model. Its load-bearing commitments are definitional and methodological: what counts as foreign-policy/statecraft evaluation, which public corpora represent the field, and how title-abstract adjudication should be thresholded. There are no physical constants or curve-fit parameters; the free parameters are screening/adjudication choices that determine the scarcity count.

free parameters (3)
  • adjudication_queue_size_and_composition
    1,370-row queue plus 500-row negative audit sample determine what gets human/LLM-labeled; different sampling would change precision/recall estimates.
  • strict_hit_inclusion_threshold
    Manual QA reduces 14 raw direct/proxy labels to 12 paper-ready hits and excludes two borderline rows; the headline scarcity claim depends on this conservative cutoff.
  • study_window_and_corpus_merge_rules
    2018 through 2026-04-22 window and three overlapping source families define the 79,954-row denominator used for the 0.015% claim.
axioms (5)
  • domain assumption Foreign policy/statecraft is defined as purposive, institutionally mediated external objective formation and implementation by political actors, narrower than IR but broader than diplomacy.
    Section 1 definition sets the domain boundary for what counts as relevant evaluation work.
  • ad hoc to paper Public title-and-abstract evidence is a useful first-order map of the technical AI governance research ecosystem's coverage of statecraft evaluation.
    Section 5 and Appendix A treat the screen as evidence of a governance gap while Limitations note classified work may exist.
  • domain assumption Technical AI governance capacities can be organized by the Reuel–Bucknall taxonomy (Assessment, Access, Verification, Security, Operationalization, Ecosystem Monitoring).
    Used throughout Sections 1 and 4 as the capacity map against which asymmetry is claimed.
  • ad hoc to paper A substantive hit requires both statecraft-domain relevance and TAIGR method relevance; deployment alone is insufficient.
    Appendix A coding schema; this boundary choice strongly shapes the low hit count.
  • domain assumption World politics structurally features partial observability, strategic misrepresentation, contested ground truth, and multidimensional objectives.
    Section 3 draws on Fearon, Jervis, Schelling, Putnam, Allison & Zelikow as background IR theory.
invented entities (2)
  • demand-side task cards for foreign-policy workflows no independent evidence
    purpose: Decompose institutional statecraft workflows into bounded, evaluable sub-tasks with explicit human roles and scoring regimes.
    Introduced in Section 6 / Table 2 as the proposed evaluation unit; independent evidence is conceptual/illustrative rather than empirically validated in this paper.
  • foreign-policy AI evaluation gap as a TAIG capacity asymmetry no independent evidence
    purpose: Name the claimed under-coverage of Access/Verification/Security/Operationalization relative to Assessment in statecraft settings.
    Organizing construct of the paper; supported by the corpus screen but not independently measured outside this study design.

pith-pipeline@v1.1.0-grok45 · 20180 in / 3351 out tokens · 23935 ms · 2026-07-12T05:50:35.761012+00:00 · methodology

0 comments
read the original abstract

We argue that AI systems used in conducting foreign policy tasks - broadly enacting 'statecraft' - should be a priority test case for technical AI governance research. In enacting foreign policy, we refer to the formulation and implementation of external objectives by political actors. Statecraft is a high-consequence deployment domain, with extreme downside risks and structural properties that standard evaluation practices handle poorly. These features include partial observability, unbounded action spaces, contested ground truth, and multidimensional objectives. This paper advocates for a literature-grounded research agenda. Our contribution is threefold: (i) a claim about the structural conditions of foreign policy that combine catastrophic tail risk with technical evaluation complexities, (ii) an ECOSYSTEM review that highlights the asymmetric focus on ASSESSMENT features over ACCESS, VERIFICATION, SECURITY, and OPERATIONALIZATION, and (iii) a demand-side evaluation framework that decomposes foreign-policy workflows into bounded, evaluable sub-tasks with human recombination. As AI systems are already being deployed in the conduct of war and peace, amid limited public evaluation infrastructure from the technical AI governance community, this agenda is an urgent priority.

Figures

Figures reproduced from arXiv: 2607.02955 by Charles Pozniak, Jeba Sania.

Figure 1
Figure 1. Figure 1: Adjudicated accepted rows by relevance label. Construction from Open Source Intelligence is intelligence-workflow relevant, but is more of a proposed system rather than an evaluation methodology. A Survey of Large Language Model Use and Its Technical Limitations in Military Systems Through a Decolonial Lens is relevant to military LLM use, but does not provide a concrete evaluation, benchmark, audit, or wo… view at source ↗
Figure 2
Figure 2. Figure 2: All adjudicated labels, including rejected or not-relevant rows. domain-by-year, venue-by-year, domain-by-capacity, and domain-by-method cross-tabulations. The figure files used in this ICML-formatted appendix are included in the figures/ directory. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Manual QA split between paper-ready strict and borderline/contextual direct or proxy rows [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Accepted rows by evaluation or governance method. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Movement from deterministic candidate buckets to final adjudicated labels. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 2 canonical work pages

  1. [1]

    arXiv:2001.11785 [cs]

    URL http:// arxiv.org/abs/2001.11785. arXiv:2001.11785 [cs]. Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., Jacob, A. P., Komeili, M., Konath, K., Kwon, M., Lerer, A., Lewis, M., Miller, A. H., Mitts, S., Renduchintala, A., Roller, S., Rowe, D., Shi, W., Spisak, J., Wei, A., Wu, D., Zhang, H., and Zijls...

    Pith/arXiv arXiv 2001

  2. [2]

    doi: 10.1126/ science.ade9097

    ISSN 0036-8075, 1095-9203. doi: 10.1126/ science.ade9097. URL https://www.science. org/doi/10.1126/science.ade9097. Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., DiPofi, A., Etxaniz, J., Fattori, B., Forde, J. Z., Foster, C., Hsu, J., Jaiswal, M., Lee, W. Y ., Li, H., Lover...

    doi:10.1126/science.ade9097

  3. [3]

    Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., Maharaj, T., Koh, P

    URL https://arxiv.org/abs/2405.14782. Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., Maharaj, T., Koh, P. W., Hooker, S., Leung, J., Trask, A., Bluemke, E., Lebensold, J., O’Keefe, C., Koren, M., Ryffel, T., Rubinovitz, J., Besiroglu, T., Carugati, F., Clark, J., Eckersley, P., de Haas...

    Pith/arXiv arXiv

  4. [4]

    Bucknall, B

    URL https://arxiv.org/abs/2004.07213. Bucknall, B. S. and Trager, R. F. Structured Access for Third-Party Research on Frontier AI Models: Investigat- ing Researchers’ Model Access Requirements. Technical report, Oxford Martin AI Governance Initiative, Centre for the Governance of AI, October

    Pith/arXiv arXiv 2004

  5. [5]

    Elfenbein, H

    URL https:// arxiv.org/abs/2508.07485. Elfenbein, H. A., Foo, M.-D. D., White, J. B., Tan, H. H., and Aik, V .-C. Reading your counterpart: The bene- fit of emotion recognition accuracy for effectiveness in negotiation.SSRN Electron. J.,

    Pith/arXiv arXiv

  6. [6]

    8 The Foreign Policy AI Evaluation Gap Fearon, J

    URL https:// arxiv.org/abs/2502.06559. 8 The Foreign Policy AI Evaluation Gap Fearon, J. D. Rationalist Explanations for War.Interna- tional Organization, 49(3):379–414,

    Pith/arXiv arXiv

  7. [7]

    Fulmer, I

    URL https: //arxiv.org/abs/2305.10142. Fulmer, I. and Barry, B. The smart negotiator: Cognitive ability and emotional intelligence in negotiation.Interna- tional Journal of Conflict Management, 15:245–272, 12

    Pith/arXiv arXiv

  8. [8]

    Halperin, M

    doi: 10.1108/eb022914. Halperin, M. H., Kanter, A., and Clapp, P.Bureaucratic Politics and Foreign Policy. Brookings Institution Press,

    work page doi:10.1108/eb022914

  9. [9]

    URL http://arxiv.org/abs/2306. 16507. arXiv:2306.16507. Hutchinson, B., Rostamzadeh, N., Greer, C., Heller, K., and Prabhakaran, V . Evaluation gaps in machine learning practice,

    Pith/arXiv arXiv

  10. [10]

    Jensen, B., Reynolds, I., Atalan, Y ., Garcia, M., Woo, A., Chen, A., and Howarth, T

    URL https://arxiv.org/abs/ 2205.05256. Jensen, B., Reynolds, I., Atalan, Y ., Garcia, M., Woo, A., Chen, A., and Howarth, T. Critical foreign policy de- cisions (cfpd)-benchmark: Measuring diplomatic pref- erences in large language models,

    Pith/arXiv arXiv

  11. [11]

    Jervis, R.Perception and Misperception in International Politics

    URL https: //arxiv.org/abs/2503.06263. Jervis, R.Perception and Misperception in International Politics. Princeton University Press,

    Pith/arXiv arXiv

  12. [12]

    Kapoor, S., Stroebl, B., Siegel, Z

    URL https://arxiv.org/ abs/2205.06760. Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. AI Agents That Matter.Transactions on Machine Learning Research,

    Pith/arXiv arXiv

  13. [13]

    Human vs

    Lamparth, M., Corso, A., Ganz, J., Skylar Mastro, O., Schneider, J., and Trinkunas, H. Human vs. Machine: Behavioral Differences Between Expert Humans and Lan- guage Models in Wargame Simulations.arXiv preprint arXiv:2403.03407,

    Pith/arXiv arXiv

  14. [14]

    arXiv:2502.14122 [cs]

    URL http://arxiv.org/ abs/2502.14122. arXiv:2502.14122 [cs]. Liao, Q. V . and Xiao, Z. Rethinking model evaluation as narrowing the socio-technical gap,

    arXiv

  15. [15]

    Luo, X., Li, Y ., Huang, Q., and Zhan, J

    URL https: //arxiv.org/abs/2306.03100. Luo, X., Li, Y ., Huang, Q., and Zhan, J. A sur- vey of automated negotiation: Human factor, learning, and application.Computer Science Review, 54:100683, November

    Pith/arXiv arXiv

  16. [16]

    doi: 10.1016/j.cosrev.2024.100683

    ISSN 1574-0137. doi: 10.1016/j.cosrev.2024.100683. URL https://www.sciencedirect.com/ science/article/pii/S1574013724000674. Ma, Z., Mei, Y ., Bruderlein, C., Gajos, K. Z., and Pan, W. ”chatgpt, don’t tell me what to do”: Designing ai for con- text analysis in humanitarian frontline negotiations,

    work page doi:10.1016/j.cosrev.2024.100683 2024

  17. [17]

    Putnam, R

    URLhttps://arxiv.org/abs/2410.09139. Putnam, R. D. Diplomacy and Domestic Politics: The Logic of Two-Level Games.International Organization, 42(3): 427–460,

    Pith/arXiv arXiv

  18. [18]

    URL https://arxiv.org/ abs/2206.04737. Reuel, A., Bucknall, B., Casper, S., Fist, T., Soder, L., Aarne, O., Hammond, L., Ibrahim, L., Chan, A., Wills, P., An- derljung, M., Garfinkel, B., Heim, L., Trask, A., Mukobi, G., Schaeffer, R., Baker, M., Hooker, S., Solaiman, I., Luccioni, A. S., Rajkumar, N., Mo¨es, N., Ladish, J., Bau, D., Bricman, P., Guha, N....

    Pith/arXiv arXiv

  19. [19]

    URL http://arxiv.org/abs/2407. 14981. arXiv:2407.14981 [cs]. Rivera, J.-P., Mukobi, G., Reuel, A., Lamparth, M., Smith, C., and Schneider, J. Escalation Risks from Language Models in Military and Diplomatic Decision-Making. In 9 The Foreign Policy AI Evaluation Gap Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT),

    Pith/arXiv arXiv 2024

  20. [20]

    Scharre, P.Army of None: Autonomous Weapons and the Future of War

    doi: 10.1145/3630106.3658942. Scharre, P.Army of None: Autonomous Weapons and the Future of War. W.\,W.\,Norton,

    doi:10.1145/3630106.3658942

  21. [21]

    URLhttps://arxiv.org/abs/2505.18893. Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whit- tlestone, J., Leung, J., Kokotajlo, D., Marchal, N., An- derljung, M., Kolt, N., Ho, L., Siddarth, D., Avin, S., Hawkins, W., Kim, B., Gabriel, I., Bolina, V ., Clark, J., Bengio, Y ., Christiano, P., and Dafoe, A. Model Evalua- tion for Extreme Risks.arXiv p...

    Pith/arXiv arXiv

  22. [22]

    org/abs/2503.06416

    URL https://arxiv. org/abs/2503.06416. Weidinger, L., Marchal, N., Rauh, M., Manzini, A., Hen- dricks, L. A., Mateos-Garcia, J., Bergman, S., Gabriel, I., Griffin, C., Kay, J., Bariach, B., Rieser, V ., and Isaac, W. Sociotechnical Safety Evaluation of Generative AI Systems.arXiv preprint arXiv:2310.11986,

    arXiv

  23. [23]

    10 The Foreign Policy AI Evaluation Gap A

    URL https://arxiv.org/abs/ 2506.09655. 10 The Foreign Policy AI Evaluation Gap A. Appendix: Corpus Screening and Codex-Adjudicated Evidence Map This appendix documents the corpus-screening and adjudication procedure supporting Section

    Pith/arXiv arXiv

  24. [24]

    This screen estimates whether public technical AI governance research substantially covers AI systems or AI-enabled workflows in foreign-policy and statecraft contexts through a title-and-abstract evidence map over a bounded public research corpus. A.1. Purpose and Scope The study looks at whether public TAIGR covers statecraft-relevant settings (e.g., fo...

    2018

  25. [25]

    Figure 1.Adjudicated accepted rows by relevance label

    Shares of the denominator use the 79,954-row headline denominator. Figure 1.Adjudicated accepted rows by relevance label. Construction from Open Source Intelligenceis intelligence-workflow relevant, but is more of a proposed system rather than an evaluation methodology.A Survey of Large Language Model Use and Its Technical Limitations in Military Systems ...

    2024