Pith. sign in

REVIEW 5 minor 30 references

UCSC NLP at SemEval-2026 Task 10: Boundary-Aware Span Extraction and RoBERTa Classification for Conspiracy Detection

T0 review · 0 major / 5 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Boundary-aware span classification finds conspiracy roles well for people, poorly for abstract actions and evidence.

desk verdict Competent mid-pack SemEval system paper: solid span-classification engineering and honest role-type analysis, nothing methodologically new. read the letter →

arxiv 2607.05689 v1 pith:6GJWPYHY submitted 2026-07-06 cs.CL cs.SI

Dom Marhoefer , Milos Suvakovic , Glenn Grant-Richards , Aidan Pinero , Ryan King This is my paper
classification cs.CLcs.SI
keywords conspiracymarkerextractionspanclassificationboundary-awarerepresentationshard-negativesamplingnon-maximumsuppressionRoBERTaSemEval-2026PsyCoMark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This shared-task system paper shows how to extract conspiracy-narrative roles (Actor, Action, Effect, Evidence, Victim) from Reddit text and separately decide whether a whole document is conspiratorial. The authors treat marker extraction as multi-label classification over every candidate span up to 32 tokens, labeling a span positive only when it overlaps a gold span almost exactly (IoU at least 0.95), mining hard near-miss negatives, and cleaning overlaps with containment-based non-maximum suppression plus a width-aware span vector that includes start, end, mean pool, length, and left/right context. Document-level detection is an independent RoBERTa classifier with label smoothing. Entity-like roles (Actor, Victim) emerge as comparatively reliable; abstract roles stay boundary-sensitive. The practical payoff is a mid-pack competitive pipeline that makes the boundary problem concrete rather than treating all roles as equally hard.

What carries the argument

Boundary-aware span vector: the concatenation of start token, end token, mean-pooled interior, learned width embedding, and immediate left/right context tokens, scored by a two-layer MLP and decoded by per-role thresholds plus containment NMS.

What would settle it

Measure recall of gold markers longer than 32 tokens (or multi-clause abstract roles) on held-out data; if a large fraction of those gold spans are systematically missed even after gap merging, the length-cap premise fails.

Watch

Extended reading notes

Core claim

Formulating conspiracy-marker extraction as multi-label span classification over enumerated candidates, with strict IoU≥0.95 positive labeling, hard-negative sampling of partial overlaps, and containment-based NMS on boundary-aware span representations, yields competitive official scores while exposing a clear difficulty split: entity-like roles are robust and abstract roles (Action, Effect, Evidence) remain highly sensitive to boundary criteria.

Load-bearing premise

The authors assume that limiting candidate spans to 32 tokens and then merging only tiny character gaps is enough to cover the real lengths of multi-clause Action, Effect, and Evidence markers.

Editorial extensions

If this is right

  • Entity roles can be extracted with higher confidence than abstract roles under the same pipeline.
  • Development metrics must match the official token-level IoU threshold; relaxed character IoU produces optimistic validation scores that do not transfer.
  • Hard-negative near-boundary examples are necessary for sharp localization; without them the model prefers overly broad spans.
  • Independent document classification can reach competitive weighted F1 without using predicted markers as features.
  • Future joint models can safely condition document stance on high-precision Actor/Victim spans while remaining cautious about abstract-role noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same boundary-sensitivity pattern should appear in other narrative-role or SRL-style extraction tasks that mix named entities with clause-level predicates.
  • Raising the maximum span length or replacing enumeration with a learned span proposal step is the most direct route to higher abstract-role recall.
  • Containment NMS with role-specific thresholds is a lightweight substitute for full structured decoding when compute is limited.
  • Document-level conspiracy signals appear strong enough that simple sequence classification already captures much of the stance, so the main remaining gain is in precise marker boundaries rather than pure classification accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper describes the UCSC NLP systems for SemEval-2026 Task 10 (PsyCoMark). Subtask 1 (conspiracy-marker extraction) is cast as multi-label classification over enumerated candidate spans (max length 32) with boundary-aware representations (start/end tokens, mean pool, width embedding, left/right context), IoU ≥ 0.95 positive labeling, hard-negative sampling in the [0.5, 0.75) band, positive class weighting, and containment-based NMS plus adjacent-span merging. Subtask 2 (document-level conspiracy classification) is an independent RoBERTa-large sequence classifier with label smoothing (α = 0.05) and a stratified split. The systems obtain mid-pack official rankings (0.2251 macro F1 / 7th on Subtask 1; 0.7694 weighted F1 on Subtask 2) and report that entity-like roles (Actor, Victim) are substantially more robust than abstract roles (Action, Effect, Evidence) under strict token-level IoU.

Significance. As a shared-task system description the contribution is appropriately modest and useful: it supplies a clear, fully specified baseline for boundary-sensitive multi-label span extraction on psycholinguistic conspiracy markers, documents the practical impact of hard negatives and containment NMS, and isolates a reproducible qualitative finding (entity-like vs. abstract role difficulty) that is of interest beyond the leaderboard. Hyper-parameters, sampling tiers, decoding thresholds, and per-role tables are given in sufficient detail for re-implementation. The independent modeling of the two subtasks cleanly avoids error propagation and makes the reported numbers easy to interpret. No theoretical novelty is claimed, nor is any required for this venue; the empirical package is solid and the boundary-sensitivity analysis is a genuine takeaway.

minor comments (5)
  1. Abstract states 11th place for Subtask 2 while §1 and §5 state 12th place (same score 0.7694). Please reconcile the ranking number throughout the manuscript.
  2. §3.1.5 / Table 1 vs. Table 2: the large validation-to-test drop is correctly attributed to the change from character IoU ≥ 0.3 to token IoU ≥ 0.5, yet a short quantitative sensitivity table (same model, both metrics on the same split) would make the explanation fully transparent.
  3. §3.1.1: the six-component span representation is well-motivated; a one-sentence ablation note (or citation to prior span-classification work that used the same concatenation) would strengthen the architectural claim.
  4. §4.2 Limitations already flags the L = 32 ceiling; adding the empirical fraction of gold markers longer than 32 tokens (if available) would quantify the recall ceiling more precisely.
  5. Minor typographic inconsistencies: “11th”/“12th”, “Can’t tell” vs. “Can't tell”, and the arXiv header date versus the SemEval-2026 venue year. A final proof-reading pass will remove them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical SemEval system description with validation-tuned thresholds and external official rankings.

full rationale

This is a standard shared-task system paper. Subtask 1 is multi-label span classification over enumerated candidates (IoU≥0.95 positives, hard-negative sampling, containment NMS, boundary-aware RoBERTa representations); Subtask 2 is an independent RoBERTa sequence classifier with label smoothing. Thresholds, sampling, and early-stopping are tuned on a held-out validation split; final rankings (0.2251 macro F1 / 7th; 0.7694 weighted F1 / 12th) come from the external SemEval evaluation server. No equation or claim reduces by construction to a fitted constant, no uniqueness theorem is imported, and no self-citation is load-bearing for the central results. The validation-to-test gap is explicitly attributed by the authors to metric mismatch (character IoU≥0.3 vs token IoU≥0.5) and label exclusion, not hidden circularity. Score 0 is the correct honest finding.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

Standard transformer fine-tuning paper. Load-bearing free parameters are the many hand-chosen or validation-tuned thresholds and sampling ratios; axioms are ordinary ML and task assumptions; no new theoretical entities are postulated.

free parameters (7)
  • max_span_length_L = 32
    Hard-coded to 32 tokens for efficiency; directly caps recall of long abstract markers.
  • positive_IoU_threshold = 0.95
    Set to 0.95 to force exact boundaries; chosen by authors, not derived.
  • hard_negative_IoU_band = [0.50, 0.75)
    IoU in [0.50, 0.75) used for hard negatives; band selected by authors.
  • pos_weight_clip = 20.0
    Positive class weights clipped at 20.0 for stability; arbitrary ceiling.
  • NMS_contain_thr_and_iou_thr = role-specific ranges
    Role-specific containment (0.65–0.75) and IoU (0.35–0.45) thresholds tuned on validation.
  • label_smoothing_alpha = 0.05
    Fixed at 0.05 for Subtask 2; standard but free choice.
  • learning_rate_and_schedule = 2e-5
    2e-5 AdamW with 10% warmup + cosine; conventional but free.
assumptions (3)
  • domain assumption RoBERTa-large contextual embeddings plus a two-layer MLP are sufficient to represent conspiracy role spans and document stance.
    Invoked throughout §3; no ablation against other encoders.
  • ad hoc to paper Independent modeling of Subtask 1 and Subtask 2 (no parameter sharing or feature transfer) is adequate and avoids error propagation.
    Stated as design choice in §3 and defended in §4.1; future-work section notes joint modeling might help.
  • domain assumption Token-level IoU ≥ 0.5 is the correct official evaluation criterion for marker quality.
    Taken from the shared-task definition; authors note their own development metric used a looser 0.3 character IoU.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UCSC NLP at SemEval-2026 Task 10: Boundary-Aware Span Extraction and RoBERTa Classification for Conspiracy Detection." pith.science (2026). https://pith.science/paper/6GJWPYHY

@misc{pith2026260705689,
  author       = {Pith},
  title        = {Pith review of: UCSC NLP at SemEval-2026 Task 10: Boundary-Aware Span Extraction and RoBERTa Classification for Conspiracy Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6GJWPYHY}},
  note         = {Machine review of arXiv:2607.05689}
}
read the original abstract

We present our systems for SemEval-2026 Task 10 (PsyCoMark), addressing conspiracy marker extraction (Subtask 1) and document-level conspiracy detection (Subtask 2). For marker extraction, we formulate the task as multi-label span classification over enumerated candidate spans, using IoU >= 0.95 positive labeling, hard-negative sampling, and containment-based non-maximum suppression (NMS) with boundary-aware span representations. Document classification is modeled independently using a sequence classifier with label smoothing and a stratified train-validation split. Analysis shows that entity-like roles (Actor, Victim) are detected robustly, while abstract roles (Action, Effect, Evidence) remain sensitive to boundary criteria. On the official test set, our systems rank 7th in Subtask 1 (0.2251 macro F1) and 11th in Subtask 2 (0.7694 weighted F1).

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages

  1. [1]

    RoBERTa: A Robustly Optimized BERT Pretraining Approach

    RoBERTa: A Robustly Optimized BERT Pretraining Approach , author =. arXiv preprint arXiv:1907.11692 , year =

    work page Pith review arXiv 1907

  2. [2]

    2025 , publisher =

    Samory, Mattia and Soldner, Felix and Batzdorfer, Veronika , title =. 2025 , publisher =. doi:10.5281/zenodo.17065240 , url =

    work page doi:10.5281/zenodo.17065240 2025

  3. [3]

    Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). 2026

    work page 2026

  4. [4]

    S em E val-2026 Task 10: P sy C o M ark -- Psycholinguistic Conspiracy Marker Extraction and Detection

    Samory, Mattia and Soldner, Felix and Batzdorfer, Veronika. S em E val-2026 Task 10: P sy C o M ark -- Psycholinguistic Conspiracy Marker Extraction and Detection. Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). 2026

    work page 2026

  5. [5]

    End-to-end Neural Coreference Resolution

    Lee, Kenton and He, Luheng and Lewis, Mike and Zettlemoyer, Luke. End-to-end Neural Coreference Resolution. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017. doi:10.18653/v1/D17-1018

    work page doi:10.18653/v1/d17-1018 2017

  6. [6]

    2024 , journal =

    What distinguishes conspiracy from critical narratives? A computational analysis of oppositional discourse , author =. 2024 , journal =

    work page 2024

  7. [7]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    work page 1972

  8. [8]

    Publications Manual , year = "1983", publisher =

    work page 1983

Show all 30 references
  1. [9]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    1981 doi

  2. [10]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  3. [11]

    Dan Gusfield , title =. 1997

    1997

  4. [12]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  5. [13]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  6. [14]

    2025 , eprint=

    NeoBERT: A Next-Generation BERT , author=. 2025 , eprint=

    2025

  7. [15]

    Proceedings of ICWSM , year =

    VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text , author =. Proceedings of ICWSM , year =

  8. [16]

    Journal of Applied Psychology , volume =

    A New Readability Yardstick , author =. Journal of Applied Psychology , volume =. 1948 , doi =

    1948

  9. [17]

    Social Psychological and Personality Science , volume =

    Dead and Alive: Beliefs in Contradictory Conspiracy Theories , author =. Social Psychological and Personality Science , volume =. 2012 , doi =

    2012

  10. [18]

    Current Directions in Psychological Science , volume =

    The Psychology of Conspiracy Theories , author =. Current Directions in Psychological Science , volume =. 2017 , doi =

    2017

  11. [19]

    , title =

    van Prooijen, Jan-Willem and Douglas, Karen M. , title =. Memory Studies , year =

  12. [20]

    Science , volume =

    The spread of true and false news online , author =. Science , volume =. 2018 , doi =

    2018

  13. [21]

    Science , volume =

    The science of fake news , author =. Science , volume =. 2018 , doi =

    2018

  14. [22]

    Journal of Economic Perspectives , volume =

    Social media and fake news in the 2016 election , author =. Journal of Economic Perspectives , volume =. 2017 , doi =

    2016

  15. [23]

    Cognition , volume =

    Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning , author =. Cognition , volume =. 2019 , doi =

    2019

  16. [24]

    Journal of Information Science , year =

    Giachanou, Anastasia and Ghanem, Bilal and Rosso, Paolo , title =. Journal of Information Science , year =

  17. [25]

    Journal of Language and Social Psychology , volume =

    The psychological meaning of words: LIWC and computerized text analysis methods , author =. Journal of Language and Social Psychology , volume =. 2010 , doi =

    2010

  18. [26]

    Proceedings of NAACL-HLT , year =

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , author =. Proceedings of NAACL-HLT , year =

  19. [27]

    Conspiracy detection beyond text: exploring the feasibility of adding psycho-linguistic features to enhance conspiracy detection models , volume =

    George, A and Ahrens, M and Pierrehumbert, J and McMahon, M , editor =. Conspiracy detection beyond text: exploring the feasibility of adding psycho-linguistic features to enhance conspiracy detection models , volume =. , booktitle =. 2024 , organizer =

    2024

  20. [28]

    2025 , eprint=

    Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models , author=. 2025 , eprint=

    2025

  21. [29]

    2025 , eprint=

    Nomic Embed: Training a Reproducible Long Context Text Embedder , author=. 2025 , eprint=

    2025

  22. [30]

    ConspEmoLLM-v2: A Robust and Stable Model to Detect Sentiment-Transformed Conspiracy Theories , ISBN=

    Liu, Zhiwei and Thompson, Paul and Rong, Jiaqi and Ananiadou, Sophia , year=. ConspEmoLLM-v2: A Robust and Stable Model to Detect Sentiment-Transformed Conspiracy Theories , ISBN=. doi:10.3233/faia251468 , booktitle=

    doi

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.