Counterfactual Probing for the Influence of Affect and Specificity on Intergroup Bias
Pith reviewed 2026-05-24 08:16 UTC · model grok-4.3
The pith
Affect shapes how language models classify intergroup relationships in text, while specificity remains inconclusive.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The authors establish that affect and specificity show modest correlations with intergroup relationship labels, and that neural models finetuned to predict those labels reliably draw on affect during classification while the contribution of specificity stays inconclusive under counterfactual probing.
What carries the argument
Counterfactual probing of neural models finetuned on supervised intergroup relationship (IGR) labels, which isolates the contribution of affect and specificity by altering those features in input text.
If this is right
- Affect functions as a reliable signal that models exploit when detecting intergroup social context.
- Specificity does not produce a clear, consistent signal in the same models.
- Pragmatic features beyond negative language can be connected to intergroup bias through controlled probing.
- Finetuned models for IGR labels can be decomposed to reveal which linguistic properties drive their decisions.
Where Pith is reading between the lines
- If affect is the dominant feature, debiasing methods could target affective language patterns rather than broad topic filters.
- The inconclusive result for specificity suggests either that it is not used by these models or that current probing methods lack sensitivity to detect it.
- Extending the same probing approach to other pragmatic features such as hedging or politeness could map a fuller set of linguistic markers for intergroup context.
Load-bearing premise
The supervised intergroup relationship labels accurately reflect intergroup social context without being shaped by the same affect and specificity features under study.
What would settle it
A test in which removing or neutralizing affect words from the input tweets causes no drop in a finetuned model's accuracy on the IGR prediction task.
read the original abstract
While existing work on studying bias in NLP focues on negative or pejorative language use, Govindarajan et al. (2023) offer a revised framing of bias in terms of intergroup social context, and its effects on language behavior. In this paper, we investigate if two pragmatic features (specificity and affect) systematically vary in different intergroup contexts -- thus connecting this new framing of bias to language output. Preliminary analysis finds modest correlations between specificity and affect of tweets with supervised intergroup relationship (IGR) labels. Counterfactual probing further reveals that while neural models finetuned for predicting IGR labels reliably use affect in classification, the model's usage of specificity is inconclusive. Code and data can be found at: https://github.com/venkatasg/intergroup-probing
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether pragmatic features of specificity and affect systematically vary across intergroup social contexts in tweets, using supervised intergroup relationship (IGR) labels from prior work. Preliminary analysis reports modest correlations between these features and IGR labels. Counterfactual probing of neural models fine-tuned to predict IGR labels indicates reliable use of affect but inconclusive results for specificity. Code and data are released.
Significance. If the empirical measurements and probing results hold after addressing methodological gaps, the work would usefully connect a revised framing of intergroup bias to concrete language features, extending bias analysis beyond pejorative language. The release of code and data supports reproducibility and is a clear strength.
major comments (3)
- [§3 and §4] §3 (Methods) and §4 (Results): The abstract and reported findings supply no sample sizes, statistical tests, p-values, or effect sizes for the 'modest correlations' or the counterfactual probing experiments. Without these, the reliability of the affect-usage claim and the inconclusive specificity result cannot be assessed.
- [§2 and §4] §2 (Background) and §4: The manuscript does not test or discuss whether the supervised IGR labels themselves encode affect or specificity signals (e.g., via annotator heuristics). If present, this would make both the correlation analysis and the probing results circular, as the model would recover label-construction artifacts rather than independent intergroup context.
- [§4] §4 (Counterfactual probing): No details are provided on how the counterfactual examples were constructed or validated (e.g., minimal edits, human verification of label preservation). This detail is load-bearing for interpreting the affect vs. specificity contrast.
minor comments (2)
- [Abstract] Abstract: 'focues' is a typo for 'focuses'.
- [References] The citation to Govindarajan et al. (2023) should be expanded with a full reference entry and a brief summary of their IGR labeling procedure.
Simulated Author's Rebuttal
We thank the referee for the constructive comments, which highlight important methodological details needed to strengthen the paper. We address each major comment below and will incorporate revisions accordingly.
read point-by-point responses
-
Referee: [§3 and §4] §3 (Methods) and §4 (Results): The abstract and reported findings supply no sample sizes, statistical tests, p-values, or effect sizes for the 'modest correlations' or the counterfactual probing experiments. Without these, the reliability of the affect-usage claim and the inconclusive specificity result cannot be assessed.
Authors: We agree that the absence of these statistical details limits assessment of the findings. The revised manuscript will report sample sizes, statistical tests (e.g., correlation coefficients with p-values), and effect sizes for the preliminary correlation analyses as well as for the counterfactual probing results. revision: yes
-
Referee: [§2 and §4] §2 (Background) and §4: The manuscript does not test or discuss whether the supervised IGR labels themselves encode affect or specificity signals (e.g., via annotator heuristics). If present, this would make both the correlation analysis and the probing results circular, as the model would recover label-construction artifacts rather than independent intergroup context.
Authors: This concern is well-taken. The IGR labels are taken from prior work without additional validation in this manuscript for affect or specificity signals. The revision will add an explicit discussion of this potential circularity risk and include an analysis checking whether the labels encode such signals (e.g., via correlation with the measured features or annotator guideline review). revision: yes
-
Referee: [§4] §4 (Counterfactual probing): No details are provided on how the counterfactual examples were constructed or validated (e.g., minimal edits, human verification of label preservation). This detail is load-bearing for interpreting the affect vs. specificity contrast.
Authors: We acknowledge that the current manuscript lacks these construction and validation details. The revised version will provide a full description of the counterfactual generation process, including the criteria for minimal edits, the specific perturbations applied for affect and specificity, and any human validation steps performed to confirm that the intergroup relationship label is preserved. revision: yes
Circularity Check
No circularity: empirical measurement and probing on given labels
full rationale
The paper conducts preliminary correlation analysis between IGR labels and pragmatic features, followed by counterfactual probing of finetuned classifiers. No equations, fitted parameters, or derivations are present that reduce to their own inputs by construction. IGR labels are treated as an external supervised input; the analysis measures model behavior relative to those labels rather than deriving the labels or claims from the measured features. Self-citation to Govindarajan et al. (2023) supplies the framing and labels but is not load-bearing for any reduction of the reported probing results. This is a standard empirical study whose central claims remain independent of the inputs under test.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.