pith. sign in

arxiv: 1907.06745 · v1 · pith:CCUE6K2Pnew · submitted 2019-07-15 · 💻 cs.CL · cs.LG· cs.SI

Low-supervision urgency detection and transfer in short crisis messages

Mayank Kejriwal , Peilin Zhou This is my paper

Pith reviewed 2026-05-24 21:14 UTC · model grok-4.3

classification 💻 cs.CL cs.LGcs.SI
keywords urgency detectioncrisis messageslow-supervision learningtransfer learningensemble methodssocial mediadisaster response
0
0 comments X

The pith

A low-supervision ensemble system with transfer learning detects urgent needs in crisis messages and adapts across disasters.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops methods to flag short messages like tweets as urgent during humanitarian disasters, where data is sparse right after an event and disasters differ in their traits. It combines labeled and unlabeled data through ensembles for robustness and applies transfer learning to handle new crises that lack a background corpus. A sympathetic reader would care because disasters are increasing and quick identification of needs for resources like food and water can improve response times. Experiments show these approaches outperform standard baselines with high statistical significance on multiple disaster datasets.

Core claim

The paper presents a robust, low-supervision social media urgency system that adapts to arbitrary crises by leveraging both labeled and unlabeled data in an ensemble setting. The system is also able to adapt to new crises where an unlabeled background corpus may not be available yet by utilizing a simple and effective transfer learning methodology. Experimentally, the transfer learning and low-supervision approaches are found to outperform viable baselines with high significance on myriad disaster datasets.

What carries the argument

Ensemble combining labeled and unlabeled data plus transfer learning from prior disaster corpora to new events

Load-bearing premise

The proposed ensemble and transfer learning methods can adapt to arbitrary crises with varying characteristics and noise in social media.

What would settle it

If the methods fail to show statistically significant gains over baselines on a new disaster dataset with distinct noise patterns or characteristics, the performance claim would not hold.

Figures

Figures reproduced from arXiv: 1907.06745 by Mayank Kejriwal, Peilin Zhou.

Figure 1
Figure 1. Figure 1: Training for Urgency Detection. B. Urgency detection using transfer learning In this section, we describe our approach for ‘urgency detection transfer’ whereby a source dataset is given (similar to RQ1, where both an unlabeled background corpus, as well as a small manually labeled training set, are available) along with a target dataset (only a small manually labeled training set and no background corpus),… view at source ↗
read the original abstract

Humanitarian disasters have been on the rise in recent years due to the effects of climate change and socio-political situations such as the refugee crisis. Technology can be used to best mobilize resources such as food and water in the event of a natural disaster, by semi-automatically flagging tweets and short messages as indicating an urgent need. The problem is challenging not just because of the sparseness of data in the immediate aftermath of a disaster, but because of the varying characteristics of disasters in developing countries (making it difficult to train just one system) and the noise and quirks in social media. In this paper, we present a robust, low-supervision social media urgency system that adapts to arbitrary crises by leveraging both labeled and unlabeled data in an ensemble setting. The system is also able to adapt to new crises where an unlabeled background corpus may not be available yet by utilizing a simple and effective transfer learning methodology. Experimentally, our transfer learning and low-supervision approaches are found to outperform viable baselines with high significance on myriad disaster datasets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims to introduce a robust low-supervision ensemble system for detecting urgency in short crisis-related social media messages. It leverages both labeled and unlabeled data and introduces a simple transfer learning method that enables adaptation to new crises even when no unlabeled background corpus is yet available. The central experimental claim is that these approaches outperform viable baselines with high statistical significance across multiple disaster datasets.

Significance. If the experimental results hold under scrutiny, the work addresses a practically important problem in humanitarian informatics by reducing reliance on large labeled sets and enabling cross-crisis transfer. The low-supervision ensemble and transfer components directly target the data sparsity and domain-shift issues highlighted in the abstract. No machine-checked proofs or parameter-free derivations are present, but the emphasis on reproducible adaptation across real disaster corpora would be a strength if the evaluation protocol is fully documented.

major comments (2)
  1. [Abstract] Abstract: the claim that the transfer learning and low-supervision approaches 'outperform viable baselines with high significance' is load-bearing for the paper's contribution, yet the abstract supplies no information on the number or identity of datasets, the choice of baselines, the statistical test used, or effect sizes; without these details the central empirical claim cannot be evaluated.
  2. [Abstract] Abstract (transfer-learning paragraph): the assertion that the method adapts to 'arbitrary crises' and to cases 'where an unlabeled background corpus may not be available yet' is the key novelty, but the description remains at the level of 'simple and effective transfer learning methodology' with no indication of the concrete mechanism (e.g., which layers or embeddings are transferred, whether any target-domain unlabeled data is still required, or how domain shift is quantified). This leaves the generalizability claim untestable from the provided text.
minor comments (2)
  1. [Abstract] Abstract: 'myriad disaster datasets' is imprecise; the paper should state the exact number and sources of the corpora used.
  2. [Abstract] Abstract: the phrase 'noise and quirks in social media' is repeated without elaboration; a brief characterization of the noise types addressed would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for these focused comments on the abstract. Both points identify areas where greater specificity would improve evaluability, and we will revise the abstract accordingly while preserving its length and high-level character.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that the transfer learning and low-supervision approaches 'outperform viable baselines with high significance' is load-bearing for the paper's contribution, yet the abstract supplies no information on the number or identity of datasets, the choice of baselines, the statistical test used, or effect sizes; without these details the central empirical claim cannot be evaluated.

    Authors: We agree the abstract would be stronger with these concrete details. In the revised version we will add a brief clause specifying the number of disaster datasets evaluated, the main baseline families, the statistical test applied, and a summary of effect sizes, drawing directly from the experimental section. This change addresses the referee's concern without altering the paper's claims. revision: yes

  2. Referee: [Abstract] Abstract (transfer-learning paragraph): the assertion that the method adapts to 'arbitrary crises' and to cases 'where an unlabeled background corpus may not be available yet' is the key novelty, but the description remains at the level of 'simple and effective transfer learning methodology' with no indication of the concrete mechanism (e.g., which layers or embeddings are transferred, whether any target-domain unlabeled data is still required, or how domain shift is quantified). This leaves the generalizability claim untestable from the provided text.

    Authors: The abstract is deliberately concise, but we accept that a short indication of the mechanism would make the novelty clearer. We will revise the sentence to note that the approach transfers model parameters (or embeddings) trained on a source crisis directly to the target crisis without requiring target-domain unlabeled data, with domain shift mitigated by the ensemble. Full architectural details remain in the methods section. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical outperformance claims rest on external baselines and datasets

full rationale

The paper describes an ensemble transfer-learning system for urgency detection in crisis tweets and reports experimental results showing outperformance over baselines on multiple disaster datasets. No derivation chain, fitted-parameter-as-prediction, or self-citation load-bearing step is present; the central claims are statistical comparisons against independent test sets and viable baselines. The abstract and described methodology treat performance as an externally falsifiable outcome rather than a definitional or self-referential result. This is the expected non-finding for an applied ML evaluation paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No information on free parameters, axioms, or invented entities available from the abstract alone.

pith-pipeline@v0.9.0 · 5713 in / 988 out tokens · 18670 ms · 2026-05-24T21:14:41.832731+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages · 4 internal anchors

  1. [1]

    Crisis informaticsnew data for extraordi- nary times,

    L. Palen and K. M. Anderson, “Crisis informaticsnew data for extraordi- nary times,” Science, vol. 353, no. 6296, pp. 224–225, 2016

    work page 2016

  2. [2]

    Earthquake shakes twitter users: real-time event detection by social sensors,

    T. Sakaki, M. Okazaki, and Y . Matsuo, “Earthquake shakes twitter users: real-time event detection by social sensors,” in Proceedings of the 19th international conference on World wide web. ACM, 2010, pp. 851–860

    work page 2010

  3. [3]

    Opinion mining and sentiment analysis,

    B. Pang, L. Lee et al., “Opinion mining and sentiment analysis,” F ounda- tions and Trends® in Information Retrieval , vol. 2, no. 1–2, pp. 1–135, 2008

    work page 2008

  4. [4]

    A survey on transfer learning,

    S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345–1359, 2010

    work page 2010

  5. [5]

    Natural language processing to the rescue? extracting

    S. Verma, S. Vieweg, W. J. Corvey, L. Palen, J. H. Martin, M. Palmer, A. Schram, and K. M. Anderson, “Natural language processing to the rescue? extracting” situational awareness” tweets during mass emergency,” in Fifth International AAAI Conference on Weblogs and Social Media , 2011

    work page 2011

  6. [6]

    Learning from the crowd: col- laborative filtering techniques for identifying on-the-ground twitterers during mass disruptions,

    K. Starbird, G. Muzny, and L. Palen, “Learning from the crowd: col- laborative filtering techniques for identifying on-the-ground twitterers during mass disruptions,” in Proceedings of 9th International Conference on Information Systems for Crisis Response and Management, ISCRAM , 2012, pp. 1–10

    work page 2012

  7. [7]

    CrisisLex: A lexicon for collecting and filtering microblogged communications in crises

    A. Olteanu, C. Castillo, F. Diaz, and S. Vieweg, “CrisisLex: A lexicon for collecting and filtering microblogged communications in crises.” in Proc. Int. Conf. Weblogs and Social Media (ICWSM) , Oxford, UK, 2014

    work page 2014

  8. [8]

    Getting the query right: User interface design of analysis platforms for crisis research,

    M. Barrenechea, K. M. Anderson, A. A. Aydin, M. Hakeem, and S. Jambi, “Getting the query right: User interface design of analysis platforms for crisis research,” in Engineering the Web in the Big Data Era , P. Cimiano, F. Frasincar, G.-J. Houben, and D. Schwabe, Eds. Cham: Springer International Publishing, 2015, pp. 547–564

    work page 2015

  9. [9]

    Success & scale in a data-producing organization: The socio-technical evolution of openstreetmap in response to humanitarian events,

    L. Palen, R. Soden, T. J. Anderson, and M. Barrenechea, “Success & scale in a data-producing organization: The socio-technical evolution of openstreetmap in response to humanitarian events,” in Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems , ser. CHI ’15. New York, NY , USA: ACM, 2015, pp. 4113–4122. [Online]. Ava...

    work page doi:10.1145/2702123.2702294 2015

  10. [10]

    Think local, retweet global: Retweeting by the geographically-vulnerable during hurricane sandy,

    M. Kogan, L. Palen, and K. M. Anderson, “Think local, retweet global: Retweeting by the geographically-vulnerable during hurricane sandy,” in Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing , ser. CSCW ’15. New York, NY , USA: ACM, 2015, pp. 981–993. [Online]. Available: http://doi.acm.org/10.1145/26751...

    work page doi:10.1145/2675133.2675218 2015

  11. [11]

    Architectural implications of social media analytics in support of crisis informatics research,

    K. M. Anderson, A. Schram, A. Alzabarah, and L. Palen, “Architectural implications of social media analytics in support of crisis informatics research,” IEEE Data Eng. Bull. , vol. 36, pp. 13–20, 2013

    work page 2013

  12. [12]

    Resilience-building and the crisis informatics agenda: Lessons learned from open cities kathmandu,

    R. Soden, N. Budhathoki, and L. Palen, “Resilience-building and the crisis informatics agenda: Lessons learned from open cities kathmandu,” in ISCRAM, 2014

    work page 2014

  13. [13]

    Zero-shot learning with semantic output codes,

    M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell, “Zero-shot learning with semantic output codes,” in Advances in neural information processing systems, 2009, pp. 1410–1418

    work page 2009

  14. [14]

    An embarrassingly simple approach to zero-shot learning,

    B. Romera-Paredes and P. Torr, “An embarrassingly simple approach to zero-shot learning,” in International Conference on Machine Learning , 2015, pp. 2152–2161

    work page 2015

  15. [15]

    Analysis and improvement of minimally supervised machine learning for relation extraction

    H. Uszkoreit, F. Xu, and H. Li, “Analysis and improvement of minimally supervised machine learning for relation extraction.” inNLDB. Springer, 2009, pp. 8–23

    work page 2009

  16. [16]

    C. C. Aggarwal and C. Zhai, Mining text data . Springer Science & Business Media, 2012

    work page 2012

  17. [17]

    Semi-supervised learning literature survey,

    X. Zhu, “Semi-supervised learning literature survey,” 2005

    work page 2005

  18. [18]

    Active learning literature survey,

    B. Settles, “Active learning literature survey,” University of Wisconsin, Madison, vol. 52, no. 55-66, p. 11, 2010

    work page 2010

  19. [19]

    An introduction to random indexing,

    M. Sahlgren, “An introduction to random indexing,” 2005

    work page 2005

  20. [20]

    Natural language processing (almost) from scratch,

    R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, “Natural language processing (almost) from scratch,” Journal of Machine Learning Research, vol. 12, no. Aug, pp. 2493–2537, 2011

    work page 2011

  21. [21]

    Information extraction in illicit web do- mains,

    M. Kejriwal and P. Szekely, “Information extraction in illicit web do- mains,” in Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Com- mittee, 2017, pp. 997–1006

    work page 2017

  22. [22]

    A survey of named entity recognition and classification,

    D. Nadeau and S. Sekine, “A survey of named entity recognition and classification,” Lingvisticae Investigationes, vol. 30, no. 1, pp. 3–26, 2007

    work page 2007

  23. [23]

    Entity linking meets word sense disambiguation: a unified approach,

    A. Moro, A. Raganato, and R. Navigli, “Entity linking meets word sense disambiguation: a unified approach,” Transactions of the Association for Computational Linguistics, vol. 2, pp. 231–244, 2014

    work page 2014

  24. [24]

    Distributed representations of words and phrases and their compositionality,

    T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Ad- vances in neural information processing systems , 2013, pp. 3111–3119

    work page 2013

  25. [25]

    Document Embedding with Paragraph Vectors

    A. M. Dai, C. Olah, and Q. V . Le, “Document embedding with paragraph vectors,” arXiv preprint arXiv:1507.07998, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  26. [26]

    Bag of Tricks for Efficient Text Classification

    A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” arXiv preprint arXiv:1607.01759, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  27. [27]

    Problems With Evaluation of Word Embeddings Using Word Similarity Tasks

    M. Faruqui, Y . Tsvetkov, P. Rastogi, and C. Dyer, “Problems with eval- uation of word embeddings using word similarity tasks,” arXiv preprint arXiv:1605.02276, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  28. [28]

    Domain Adaptation with Adversarial Training and Graph Embeddings

    F. Alam, S. Joty, and M. Imran, “Domain adaptation with adversarial training and graph embeddings,” arXiv preprint arXiv:1805.05151, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  29. [29]

    Mining help intent on twitter during disasters via transfer learning with sparse coding,

    B. Pedrood and H. Purohit, “Mining help intent on twitter during disasters via transfer learning with sparse coding,” in International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation . Springer, 2018, pp. 141–153

    work page 2018

  30. [30]

    The signals and noise: actionable information in improvised social media channels during a disaster,

    X. He, D. Lu, D. Margolin, M. Wang, S. E. Idrissi, and Y .-R. Lin, “The signals and noise: actionable information in improvised social media channels during a disaster,” in Proceedings of the 2017 ACM on Web Science Conference. ACM, 2017, pp. 33–42

    work page 2017

  31. [31]

    Social-eoc: Service- ability model to rank social media requests for emergency operation centers,

    H. Purohit, C. Castillo, M. Imran, and R. Pandey, “Social-eoc: Service- ability model to rank social media requests for emergency operation centers,” in 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) . IEEE, 2018, pp. 119– 126

    work page 2018

  32. [32]

    Crisismmd: Multimodal twitter datasets from natural disasters,

    F. Alam, F. Ofli, and M. Imran, “Crisismmd: Multimodal twitter datasets from natural disasters,” in Twelfth International AAAI Conference on Web and Social Media, 2018

    work page 2018

  33. [33]

    Semi-supervised event-related tweet identification with dynamic keyword generation,

    X. Zheng, A. Sun, S. Wang, and J. Han, “Semi-supervised event-related tweet identification with dynamic keyword generation,” in Proceedings of the 2017 ACM on Conference on Information and Knowledge Manage- ment. ACM, 2017, pp. 1619–1628

    work page 2017