Pith. sign in

REVIEW 4 major objections 4 minor 27 references

Controlled Territory and Conflict Tracking (CONTACT): (Geo-)Mapping Occupied Territory from Open Source Intelligence

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a prompt-tuned BLOOMZ model can extract territorial-control indicators from news articles using only 15 labeled examples, beating an embedding-based baseline 100% to 40%.

desk verdict Small, honest pilot paper whose headline claim—that prompt-based supervision improves generalization—rests on five held-out articles and is not supported, though the dataset and scraper are useful artifacts worth a proper referee. read the letter →

arxiv 2504.13730 v1 pith:CW3SYDYS submitted 2025-04-18 cs.CL cs.AIcs.LG

Paul K. Mandal , Cole Leo , Connor Hurley This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords territorialcontrolpredictionopen-sourceintelligenceprompttuningfew-shotlearningBLOOMZmulti-labelclassificationconflictmonitoringlow-resourceNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that territorial control in a conflict—who holds a town, who won a battle—can be read automatically from short, messy news articles using very little labeled data. The authors build CONTACT, a pipeline that scrapes archived articles about ISIS in Syria and Iraq, annotates them with five labels (military operations, location reference, military casualties, civilian casualties, ISIS victory), and trains two few-shot models on 15 articles. Their central result is that prompt tuning on BLOOMZ-560m, with label definitions embedded in the prompt, achieved 100% accuracy on a held-out set of five articles, while an embedding-based SetFit classifier scored 40% and collapsed to predicting only the two most frequent labels. A fair reader would take away that prompt-conditioned generative models offer a low-annotation route to territorial monitoring from open-source news, though the evidence base is small.

What carries the argument

The central mechanism is prompt tuning applied to BLOOMZ-560m, a multilingual causal language model. A prompt is initialized with eight virtual tokens prepended to a descriptive string that defines each label (for example, 't_mil - Event is about war/military operations'), and the model is trained to generate the comma-separated list of true labels, with loss computed only on the label portion of the output. The base model weights stay frozen, so the task is carried by the learned prompt tokens plus the label definitions they encode. The SetFit baseline instead pools sentence embeddings from a frozen transformer and learns a per-label logistic head, which the authors argue lacks the context needed to separate semantically overlapping labels.

What would settle it

Apply the same BLOOMZ prompt-tuning recipe to a held-out test set of 50–100 articles from a different time period, source mix, or conflict, and measure per-label accuracy; if accuracy drops well below the claimed 100% or the model again predicts only the two most frequent labels, the central claim of prompt-based few-shot generalization is refuted.

Watch

Extended reading notes

Core claim

The authors claim that territorial inference from open-source text is best reframed as a sequence-to-sequence label-generation task, and that embedding label definitions in the prompt is what makes few-shot generalization work. On a hand-labeled dataset of 20 articles about ISIS activity in Syria and Iraq from 2015–2019, split 15 training and 5 test, the BLOOMZ + Prompt Tuning model achieved 100% accuracy across all five labels and all test articles. The SetFit baseline achieved 40% average per-label accuracy and predicted only the two most frequent labels, t_mil and t_loc, for every article. The authors interpret the gap as evidence that prompt-based supervision improves generalization in low-resource, multi-label settings.

Load-bearing premise

The claimed accuracy rests on a held-out test set of only five manually selected articles, so the 100% figure could be a lucky draw rather than evidence of real generalization.

Editorial extensions

If this is right

  • A territorial-monitoring system for a new conflict could be stood up with only a few dozen hand-labeled articles, because the prompt supplies the label semantics instead of requiring them to be learned from data.
  • Embedding label definitions in prompts should be preferred over embedding-based few-shot classifiers when labels are rare or semantically overlapping.
  • The CONTACT pipeline—archived-news scraping, text normalization to 512 characters, and prompt-tuned generation—offers a reusable template for structured event extraction beyond territorial control.
  • The analyst loop for conflict monitoring would shorten: new fronts can be tracked by scraping articles and prompt-tuning on a small annotation set rather than hand-coding event streams.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment, not run in the paper, is to test whether the same 15-article prompt-tuning recipe transfers to a different conflict without retraining; that would separate the contribution of prompt tuning from the contribution of BLOOMZ's multilingual pretraining.
  • Because the evaluation rests on five hand-picked articles, the practical headline may be 'prompt tuning reaches perfect accuracy on articles an analyst already chose,' not 'prompt tuning generalizes across reporting styles'; a stratified held-out sample by source and date would settle which reading is right.
  • At 20 articles the dataset cannot support label-imbalance conclusions; one testable extension is to check whether the model still predicts the rare t_isis_vic label when positive examples are few in training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents CONTACT, a framework for predicting territorial-control-relevant indicators from open-source news text using few-shot methods. The authors construct a hand-labeled dataset of 20 English-language news articles on ISIS activity in Syria and Iraq (2015–2019), annotated with five VIINA-style binary labels. They compare two approaches: SetFit, an embedding-based classifier, and prompt-tuned BLOOMZ-560M, a multilingual generative language model. The results section reports that SetFit achieves 40% average per-label accuracy while BLOOMZ+Prompt Tuning achieves 100% across all five labels and all five test articles. The paper claims that prompt-based supervision improves generalization in low-resource settings and that CONTACT can reduce annotation burdens for OSINT-based territorial inference. The open-source scraping utility and training code are released.

Significance. If the central claim were well supported, CONTACT would be a useful demonstration of parameter-efficient fine-tuning for OSINT tasks, and the released scraper and hand-labeled dataset are potentially reusable assets. However, the significance of the paper as a scientific contribution is severely limited by the evaluation: the headline comparison rests on 25 binary decisions from five manually selected test articles, with no confidence intervals, significance tests, or external validation. The contribution is therefore better characterized as a proof-of-concept or system description than as evidence for the stated generalization claim. The open-source components are commendable, but they do not compensate for the absence of a statistically meaningful evaluation.

major comments (4)
  1. [§5, Results] The central claim that 'prompt-based supervision improves generalization in low-resource settings' is supported only by the BLOOMZ+Prompt Tuning model achieving 100% accuracy on a held-out set of five articles, i.e., 25 binary predictions. With 25 Bernoulli trials, the 95% confidence interval for the true accuracy is wide (roughly 0.87 to 1.0 if all 25 are correct), and a single misclassified label would drop accuracy to 80%. No confidence intervals, bootstrap estimates, or significance tests are reported, and the paper's own §5 caveat that 'the test set includes only five examples, which may not capture the diversity of real-world reporting' directly undermines the abstract's generalization claim. This evaluation cannot support the stated conclusion.
  2. [§5, Results comparison with SetFit] The SetFit baseline collapsed to always predicting the two most frequent labels (t_mil and t_loc), and the comparison is made against this degenerate behavior. The 60-point gap between BLOOMZ and SetFit is therefore largely a comparison to a near-trivial baseline. The paper should report per-label precision, recall, and F1, and include a majority-class or random-probability baseline for each label, so that the reader can assess how much semantic signal BLOOMZ actually extracts beyond label frequency. Without such baselines, the 'outperforms' claim is not quantified meaningfully.
  3. [§3, Dataset; §4.2, Few-Shot Classification with SetFit] There is an internal inconsistency in the data split: §3 states that 15 articles are used for training and 5 for evaluation (20 total), while §4.2 says 'We trained using 14 articles and evaluated on 5.' This discrepancy needs to be resolved, and the exact train/test split (including which five articles were used for testing) must be specified for reproducibility. Additionally, because all 20 articles were manually selected by the authors from the same period and same conflict, the test articles are drawn from the same distribution as the training articles; this raises selection-bias risk. External validation against established territorial-control records (e.g., ACLED or historical ISIS control maps) would be needed to support the claim that the model generalizes to real-world OSINT streams.
  4. [§5, Limitations acknowledgment] The authors explicitly acknowledge the test set limitation in §5, yet the abstract and §6 state the broader claim that CONTACT 'demonstrates that LLMs fine-tuned using few-shot methods can reduce annotation burdens' and that prompt-conditioned models 'provide a viable path toward predicting territorial control from OSINT.' These statements go beyond what a five-example evaluation can support. The paper should either substantially expand the evaluation (e.g., a larger independently annotated test set, external benchmarks, or inter-annotator reliability measures) or reframe the contribution as a pilot study without the generalization claim. As it stands, the evidence is insufficient for the paper's central assertion.
minor comments (4)
  1. [§4.3, Prompt-Tuned BLOOMZ] The prompt instructs the model to 'Give each of these following labels a 0 if false and a 1 if true,' but the expected output is later described as a comma-separated string of label names (e.g., 't_mil, t_loc, t_isis_vic'). These two formats are inconsistent; the paper should clarify the exact target format used in training and inference.
  2. [§4.2, SetFit training details] The sentence 'We trained using 14 articles and evaluated on 5, with batch size 1 and 20 training iterations per epoch' is ambiguous: with batch size 1, an 'epoch' normally consists of 14 steps for 14 training articles, so '20 training iterations per epoch' is unclear. Please specify the total number of update steps or the number of epochs.
  3. [§6, Discussion] There is a typo: 'could further boost accuracy and and improve generalization' should read 'could further boost accuracy and improve generalization.'
  4. [Abstract and §5] The abstract states 'We show that the BLOOMZ-based model outperforms the SetFit baseline' without qualification. Given the small test set and the SetFit collapse, this should be softened or accompanied by uncertainty estimates.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the BLOOMZ result is an empirical held-out evaluation, not a derivation from the labels or from self-cited prior work.

full rationale

CONTACT does not contain a claimed derivation chain that reduces to its own inputs. The central empirical claim is that a prompt-tuned BLOOMZ-560m model achieves 100% per-label accuracy on a held-out test set of five manually labeled articles (Section 5). The training labels are not used to set the test predictions, and the test articles are reported as separate from the 15 training articles, so this is a genuine (if very small) held-out evaluation rather than a fitted parameter renamed as a prediction. The label definitions are embedded in the prompt, but that is a standard supervised-learning design and does not make the test accuracy true by construction. The paper cites VIINA for an annotation scheme and BLOOMZ, SetFit, and prompt tuning for methods, but no load-bearing step depends on an unverified self-citation by the present authors; the only self-referential artifact is the code repository. The paper's own caveat that 'the test set includes only five examples, which may not capture the diversity of real-world reporting' weakens the external validity of the 100% figure, but that is a statistical and correctness concern, not circularity. Likewise, the SetFit baseline's collapse to the majority labels is a comparison-quality issue, not a circularity issue. No equation or definition stamps the result in from the outset, so the honest finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a hand-built dataset, hand-chosen hyperparameters, and a hand-written prompt. The most consequential choices are the train/test split and the prompt wording: the perfect test score depends entirely on the five selected articles and the way labels were described to the model.

free parameters (6)
  • learning_rate = 3e-2
    Chosen for BLOOMZ prompt tuning without reported grid search or sensitivity analysis; affects training dynamics and final accuracy.
  • num_virtual_tokens = 8
    Prompt-tuning dimension chosen by hand; no ablation reported.
  • max_input_length = 256 tokens
    Truncates article text; chosen to fit model limits, may discard relevant context.
  • training_epochs = 1 (BLOOMZ); 20 iterations/epoch (SetFit)
    Arbitrary choices that influence overfitting and generalization.
  • prompt_wording = Label definitions embedded in prompt
    The exact natural-language description of labels is hand-crafted and likely affects performance.
  • train_test_split = 15 training / 5 test (Section 4.2 says 14 training)
    Manually selected articles; the split is not random and the discrepancy (14 vs 15) is unresolved.
assumptions (5)
  • domain assumption News articles about ISIS activity are a valid proxy for territorial control.
    The entire task maps textual reports to ground-truth control without external validation against conflict event databases.
  • domain assumption The VIINA-style labels are sufficient and reliable for territorial inference.
    The five labels are treated as exhaustive and correct; no inter-annotator agreement is reported.
  • domain assumption Truncation to 512 characters preserves information needed for classification.
    Section 4.1 truncates body text; this could remove the operative sentence for territorial control.
  • domain assumption Hand-labeled annotations are correct ground truth.
    Annotations were produced by the authors without independent verification or inter-annotator agreement.
  • domain assumption BLOOMZ and SetFit are representative implementations of their respective approaches.
    Conclusions are drawn about few-shot LLMs and embedding classifiers from single model instances.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Controlled Territory and Conflict Tracking (CONTACT): (Geo-)Mapping Occupied Territory from Open Source Intelligence." pith.science (2026). https://pith.science/paper/CW3SYDYS

@misc{pith2026250413730,
  author       = {Pith},
  title        = {Pith review of: Controlled Territory and Conflict Tracking (CONTACT): (Geo-)Mapping Occupied Territory from Open Source Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CW3SYDYS}},
  note         = {Machine review of arXiv:2504.13730}
}
read the original abstract

Open-source intelligence provides a stream of unstructured textual data that can inform assessments of territorial control. We present CONTACT, a framework for territorial control prediction using large language models (LLMs) and minimal supervision. We evaluate two approaches: SetFit, an embedding-based few-shot classifier, and a prompt tuning method applied to BLOOMZ-560m, a multilingual generative LLM. Our model is trained on a small hand-labeled dataset of news articles covering ISIS activity in Syria and Iraq, using prompt-conditioned extraction of control-relevant signals such as military operations, casualties, and location references. We show that the BLOOMZ-based model outperforms the SetFit baseline, and that prompt-based supervision improves generalization in low-resource settings. CONTACT demonstrates that LLMs fine-tuned using few-shot methods can reduce annotation burdens and support structured inference from open-ended OSINT streams. Our code is available at https://github.com/PaulKMandal/CONTACT/.

Figures

Figures reproduced from arXiv: 2504.13730 by the authors.

Figure 1
Figure 1. Pipeline for the CONTACT framework: articles are scraped from archived news sources, preprocessed, and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 17 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    Anders, T. (2020). Territorial control in civil wars: Theory and measurement using machine learning. Journal of Peace Research , 57(6):701--714

    work page 2020

  3. [3]

    Baum, L. E. and Petrie, T. (1966). Statistical inference for probabilistic functions of finite state markov chains. The annals of mathematical statistics , 37(6):1554--1563

    work page 1966

  4. [4]

    Castillo-Eslava, F., Mougan, C., Romero-Reche, A., and Staab, S. (2023). The role of large language models in the recognition of territorial sovereignty: An analysis of the construction of legitimacy. arXiv preprint arXiv:2304.06030

    work page Pith review arXiv 2023

  5. [5]

    and Bansal, D

    Chaudhary, M. and Bansal, D. (2022). Open source intelligence extraction for terrorism-related information: A review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 12(5):e1473

    work page 2022

  6. [6]

    Croicu, M. (2024). Deep active learning for data mining from conflict text corpora. arXiv preprint arXiv:2402.01577

    arXiv 2024

  7. [7]

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages 4171--4186

    2019

  8. [8]

    Goecks, V. G. and Waytowich, N. (2024). COA-GPT : Generative pre-trained transformers for accelerated course of action development in military operations. In 2024 International Conference on Military Communication and Information Systems (ICMCIS) , pages 01--10. IEEE

    work page 2024

Show all 27 references
  1. [9]

    and Schmidhuber, J

    Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation , 9(8):1735--1780

    1997

  2. [10]

    Wayback machine

    Internet Archive (2025). Wayback machine. https://web.archive.org/. Accessed: 2025-04-09

    2025

  3. [11]

    M., Uszkoreit, J., Le, Q., and Petrov, S

    Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. (2019). Natural questions: A benchmark for...

    2019

  4. [12]

    Lester, B., Al-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t., editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045--...

    2021

  5. [13]

    Li, B., Haider, S., and Callison-Burch, C. (2024). This land is \ Your, My\ land: Evaluating geopolitical biases in language models through territorial disputes. 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)

    2024

  6. [14]

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  7. [15]

    Mahanty, A. (2022). waybackpy . Accessed: 2025-04-09

    2022

  8. [16]

    Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. (2022). Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft

    2022

  9. [17]

    S., Shen, S., Yong, Z

    Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z. X., Schoelkopf, H., et al. (2023). Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Comput...

    2023

  10. [18]

    Ou-Yang, L. (2016). newspaper3k: Article scraping and curation. https://newspaper.readthedocs.io/. Accessed: 2025-04-09

    2016

  11. [19]

    Rabiner, L. R. (1989). A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE , 77(2):257--286

    1989

  12. [20]

    Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you don`t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 784--789, Melbourne, Australia. Association ...

    2018

  13. [21]

    G., Lee, D., Fung, Y

    Reddy, R. G., Lee, D., Fung, Y. R., Nguyen, K. D., Zeng, Q., Li, M., Wang, Z., Voss, C., and Ji, H. (2023). Smartbook: AI -assisted situation report generation for intelligence analysts. arXiv preprint arXiv:2303.14337

    2023 arXiv

  14. [22]

    and Gurevych, I

    Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics

    2019

  15. [23]

    Roberts, J., L \"u ddecke, T., Das, S., Han, K., and Albanie, S. (2023). GPT4GEO : How a language model sees the world's geography. arXiv preprint arXiv:2306.00020

    2023 arXiv

  16. [24]

    Tunstall, L., Reimers, N., Jo, U. E. S., Bates, L., Korat, D., Wasserblat, M., and Pereg, O. (2022). Efficient few-shot learning without prompts. arXiv preprint arXiv:2209.11055

    2022 arXiv

  17. [25]

    Yang, Y., Yih, W.-t., and Meek, C. (2015). W iki QA : A challenge dataset for open-domain question answering. In M \`a rquez, L., Callison-Burch, C., and Su, J., editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 2013--2018,...

    2015

  18. [26]

    and Ayers, N

    Zhukov, Y. and Ayers, N. (2023). VIINA 2.0: Violent Incident Information from News Articles on the 2022 Russian Invasion of Ukraine . https://github.com/zhukovyuri/VIINA. Cambridge, MA: Harvard University. Accessed April 8, 2025

    2023

  19. [27]

    Zhukov, Y. M. (2023). Near-real time analysis of war and economic activity during russia’s invasion of ukraine. Journal of Comparative Economics , 51(4):1232--1243

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.