pith. sign in

arxiv: 2605.07606 · v1 · submitted 2026-05-08 · 💻 cs.CL · cs.AI

N\"urnberg NLP at PsyDefDetect: Multi-Axis Voter Ensembles for Psychological Defence Mechanism Classification

Philipp Steigerwald , Eric Rudolph , Jens Albrecht This is my paper

Pith reviewed 2026-05-11 01:49 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords ensemble learningtext classificationpsychological defence mechanismsshared taskerror independenceambiguous categoriesNLPBioNLP
0
0 comments X

The pith

On ambiguous tasks with overlapping categories, ensembles whose voters make independent errors outperform stronger single models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines classification of psychological defence mechanisms in conversations, a setting where the eight positive categories overlap in wording and differ mainly in pragmatic intent, so that even trained raters reach only moderate agreement. It claims that the decisive advantage on such tasks comes from error independence across an ensemble rather than from any single stronger model. The authors build a nine-voter system that varies along three axes: whether a model sees all nine classes or only the eight defences, whether it is trained generatively or discriminatively, and which base model it uses. This combination secured first place on the hidden test set of the shared task. Readers should care because many practical language-understanding problems involve similarly fuzzy boundaries where one representation tends to fail in the same places.

Core claim

On tasks where defence-mechanism categories share surface language and inter-annotator agreement is only moderate, error independence across an ensemble is more effective than improving any individual classifier. The authors construct a nine-member voter ensemble by varying class granularity (a gatekeeper on all classes versus specialists on the eight defences), training method (generative and discriminative), and base model. The resulting system attains an F1 score of 0.42 on the hidden test set and ranks first among 21 teams.

What carries the argument

A nine-voter ensemble whose members differ along three orthogonal axes of class granularity, training method, and base model.

If this is right

  • The ensemble reaches first place among 21 teams on the PsyDefDetect hidden test set.
  • Error independence across the three axes compensates for overlapping defence boundaries more effectively than scaling a single model.
  • Mixing generative and discriminative training supplies complementary strengths on the same data.
  • Separating a gatekeeper model from specialist models focuses capacity on the hardest distinctions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same multi-axis construction could be tested on other subjective text-classification problems that exhibit low annotator agreement.
  • Further orthogonal axes, such as different data augmentations or prompt styles, might yield additional error diversity.
  • The result implies that for fuzzy categories the engineering effort should first target disagreement among voters rather than marginal gains on a single model.
  • Performance on this shared task may indicate how well the approach transfers to domains outside psychology.

Load-bearing premise

The three chosen axes generate sufficiently independent errors to overcome the moderate inter-annotator agreement and overlapping category boundaries.

What would settle it

A single model trained with more data or a larger architecture matching or exceeding the ensemble's F1 score on the identical hidden test set.

Figures

Figures reproduced from arXiv: 2605.07606 by Eric Rudolph, Jens Albrecht, Philipp Steigerwald.

Figure 1
Figure 1. Figure 1: Architecture of our 9-voter cross-model en [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Out-of-fold per-class t-SNE of SFT QLoRA [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Each bar is one 9V system (6 Ministral + 3 specialist voters); y-axis groups samples by how many Ministral voters agreed—the specialist can flip the Ministral majority only from 4/6 downwards—and dark portions mark the actual flips. 3.3 Augmentation Ablation Without GPT-5.2 augmentation (no-aug; [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-class t-SNE of the LR 8c specialist hidden [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
read the original abstract

Detecting levels of psychological defence mechanisms in supportive conversations is inherently ambiguous. In the PsyDefDetect shared task at BioNLP 2026 the eight positive defence categories share surface language and differ only in pragmatic function and trained raters reach only moderate inter-annotator agreement. On such a task the decisive lever is not a stronger single model but error independence, since any single representation will waver on the overlapping defence boundaries. We translate this insight into a 9-voter ensemble spanning three orthogonal axes: class granularity (all nine classes for the gatekeeper, only the eight defence classes for the specialists), training method (generative and discriminative) and base model. The system reaches $F1_{test}{=}.420$ on the hidden test set, placing first among 21 registered teams.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper describes a 9-voter ensemble for the PsyDefDetect shared task on classifying psychological defence mechanisms in supportive conversations. It posits that error independence across three orthogonal axes (class granularity, training method, and base model) is the key to handling overlapping categories and moderate inter-annotator agreement, achieving F1_test = 0.420 on the hidden test set and first place among 21 teams.

Significance. If the independence hypothesis is validated, the work offers a replicable ensemble design for ambiguous multi-label NLP tasks with pragmatic rather than lexical distinctions. The hidden-test result supplies direct, competition-verified evidence of practical utility.

major comments (2)
  1. [Abstract] Abstract: the central claim that 'the decisive lever is not a stronger single model but error independence' is unsupported by any quantitative verification. No pairwise error correlations, Q-statistics, disagreement rates, or ablation studies isolating each axis (while holding the others fixed) are reported, so the performance gain cannot be attributed to orthogonality rather than simple averaging of nine models.
  2. [Abstract] The manuscript supplies no error analysis or inter-voter agreement metrics on the development set that would substantiate the 'sufficiently independent errors' assumption given the moderate IAA and overlapping defence boundaries.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful and constructive comments. We address the major points below and describe the changes we will make.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that 'the decisive lever is not a stronger single model but error independence' is unsupported by any quantitative verification. No pairwise error correlations, Q-statistics, disagreement rates, or ablation studies isolating each axis (while holding the others fixed) are reported, so the performance gain cannot be attributed to orthogonality rather than simple averaging of nine models.

    Authors: We agree that the current manuscript does not contain explicit quantitative verification of error independence (pairwise correlations, Q-statistics, or axis-isolated ablations). The claim is grounded in the task properties described in the introduction—overlapping pragmatic categories and moderate IAA—together with the observed test-set result. In the revision we will add (i) ablation tables that vary one axis while freezing the other two and (ii) pairwise disagreement rates and Q-statistics computed on the development set, allowing readers to assess whether the performance lift is attributable to orthogonality. revision: yes

  2. Referee: [Abstract] The manuscript supplies no error analysis or inter-voter agreement metrics on the development set that would substantiate the 'sufficiently independent errors' assumption given the moderate IAA and overlapping defence boundaries.

    Authors: We acknowledge that no inter-voter agreement or error-analysis figures on the development set appear in the submitted version. We will compute and report (a) pairwise agreement rates among the nine voters, (b) a confusion-matrix-style breakdown of disagreements, and (c) a short qualitative error analysis on the development set, all of which will be added to the revised manuscript. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical F1 on hidden test set stands independently

full rationale

The paper's derivation consists of stating an insight about error independence on an ambiguous task, constructing a 9-voter ensemble along three axes (class granularity, training method, base model), and reporting the resulting F1_test=.420 measured on the hidden test set. This outcome is an external empirical measurement and does not reduce to any self-referential definition, fitted parameter renamed as prediction, or self-citation chain. No equations are present that equate the result to its inputs by construction, and the provided text contains no load-bearing self-citations or ansatzes smuggled from prior author work. The assumption of orthogonality is stated but not used to derive the score; the score itself is independently falsifiable via the shared-task evaluation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper contains no mathematical derivations, free parameters, axioms, or postulated entities; it is a purely empirical description of an applied ensemble system for a classification task.

pith-pipeline@v0.9.0 · 5438 in / 1182 out tokens · 67070 ms · 2026-05-11T01:49:28.657225+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages

  1. [1]

    Overview of the

    Na, Hongbin and Wang, Zimu and Chen, Zhaoming and Hua, Yining and Gao, Rena and Yang, Kailai and Chen, Ling and Wang, Wei and Ji, Shaoxiong and Torous, John and Ananiadou, Sophia , booktitle =. Overview of the. 2026 , address =

    work page 2026

  2. [2]

    A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions

    A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions , author =. Findings of the Association for Computational Linguistics: ACL 2025 , month = jul, year =. doi:10.18653/v1/2025.findings-acl.385 , pages =

    work page doi:10.18653/v1/2025.findings-acl.385 2025

  3. [3]

    Findings of the Association for Computational Linguistics: ACL 2026 , month = jul, year =

    You Never Know a Person, You Only Know Their Defenses: Detecting Levels of Psychological Defense Mechanisms in Supportive Conversations , author =. Findings of the Association for Computational Linguistics: ACL 2026 , month = jul, year =

    work page 2026

  4. [4]

    Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics , pages=

    Towards Emotional Support Dialog Systems , author=. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics , pages=

    work page

  5. [5]

    Defense Mechanism Rating Scales (

    Perry, J Christopher , edition=. Defense Mechanism Rating Scales (

    work page

  6. [6]

    Journal of Clinical Psychology , volume=

    Anomalies and Specific Functions in the Clinical Identification of Defense Mechanisms , author=. Journal of Clinical Psychology , volume=

    work page

  7. [7]

    Ego Mechanisms of Defense: A Guide for Clinicians and Researchers , author=

    work page

  8. [8]

    Proceedings of the IEEE International Conference on Computer Vision , pages=

    Focal Loss for Dense Object Detection , author=. Proceedings of the IEEE International Conference on Computer Vision , pages=

    work page

  9. [9]

    Multiple Classifier Systems , series=

    Ensemble Methods in Machine Learning , author=. Multiple Classifier Systems , series=. 2000 , publisher=

    work page 2000

  10. [10]

    Steigerwald, Philipp and Burghardt, Jennifer and Rudolph, Eric and Albrecht, Jens , year=

    work page

  11. [11]

    Steigerwald, Philipp and Albrecht, Jens , year=. From ``

    work page

  12. [12]

    2025 , publisher=

    Steigerwald, Philipp and Bienlein, Nico and Burghardt, Jennifer and Stieler, Mara and Lehmann, Robert and Albrecht, Jens , booktitle=. 2025 , publisher=

    work page 2025