Pith. sign in

REVIEW 4 major objections 6 minor 41 references

An Attention-based Framework for Fair Contrastive Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that learned attention over protected attributes can strip sensitive information from contrastive representations without sacrificing downstream accuracy.

desk verdict Credible attention-based alternative to fixed-kernel fair contrastive learning, but the bias-removal claim rests on a linear probe and needs a stronger test. read the letter →

arxiv 2411.14765 v1 pith:5VP6KDHA submitted 2024-11-22 cs.LG

Stefan K. Nielsen , Tan M. Nguyen This is my paper · ORCID
classification cs.LG
keywords fairrepresentationlearningcontrastiveattentionmechanismlocality-sensitivehashingprotectedattributesbiasremovalcontinuoussensitiveconditional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that fair contrastive learning can be improved by replacing conditional sampling—drawing negative examples that share the anchor's protected attribute—with an attention mechanism that learns which negatives are bias-reducing and which are bias-causing. It introduces FARE, an attention module whose scores are computed over protected attributes and used to reweight contrastive similarity scores, and SparseFARE, which uses locality-sensitive hashing to discard the most bias-causing samples outright. The authors claim these methods remove more sensitive information than the kernel-based CCLK baseline while matching or improving downstream accuracy, at lower computational cost. If correct, this makes debiased representation learning practical for continuous and high-dimensional protected attributes without requiring a user to pre-specify a kernel. The empirical evidence on ColorMNIST and CelebA supports the claim: SparseFARE attains the best fairness-accuracy tradeoff among the compared methods.

What carries the argument

The load-bearing object is the FARE attention map $p_{ij}=\mathrm{softmax}((W_Q z_i)^\top W_K z_j/\rho)$, a learned similarity over protected attributes that reweights the unconditioned contrastive scores $e^{f(x_i,y_j)}$. Proposition 1 identifies this reweighted sum as the finite-sample conditional similarity estimate for $y\sim P_{Y|Z=z_i}$, which is what lets the model avoid conditional sampling. SparseFARE then sparsifies this map using locality-sensitive hashing: only keys $z_j$ whose hash equals the query's hash (or falls in an adjacent bucket) keep a nonzero attention score, so extreme bias-causing samples are excluded before the softmax. The FAREContrast loss completes the machinery by replacing the negative-sample sum of Fair-InfoNCE with the FARE/SparseFARE output, so the attention weights are learned through the training gradient.

What would settle it

On a synthetic dataset where the protected attribute is continuous but the spurious correlation that drives bias is generated by an unobserved variable independent of $Z$, compare SparseFARE's bias removal and accuracy against CCLK; if bias removal does not improve or accuracy drops sharply, the protected-attribute-similarity proxy is the failure point.

Watch

Extended reading notes

Core claim

The central claim is Proposition 1: given a batch of triplets $(x_i, y_i, z_i)$, the similarity score between an anchor $x_i$ and a negative drawn from $P_{Y|Z=z_i}$ is estimated by the attention output $\sum_j \mathrm{softmax}((W_Q z_i)^\top W_K z_j/\rho)\, e^{f(x_i, y_j)}$. The derivation uses kernel density estimators for $P(y,z)$ and $P(z)$, assumes normalized protected attributes, and sets the Gaussian bandwidth equal to the attention temperature, so the conditional estimate takes the form of an attention sum. This turns a predefined kernel's fixed weighting into a learned similarity metric over the protected attribute. FARE uses that attention output in place of the negative-sample sum in the Fair-InfoNCE objective, producing the FAREContrast loss; SparseFARE sparsifies the attention with locality-sensitive hashing so only samples with similar protected attributes (plus adjacent buckets) participate. On ColorMNIST, SparseFARE reaches 86.4% accuracy with 74.0 MSE bias removal, against 86.4% and 64.7 for CCLK; on CelebA it lowers equalized odds to 18.7, the best among compared models.

Load-bearing premise

The method assumes that two samples whose protected attributes are dissimilar are likely to bias the representation, so the attention mechanism should downweight or discard them; if protected-attribute similarity is not a faithful proxy for bias-causing potential, the learned weights and LSH sparsification could remove useful semantic information or leave bias in place.

Editorial extensions

If this is right

  • Fair contrastive learning can handle continuous and high-cardinality protected attributes without choosing a kernel, because the attention learns the bias-similarity metric from data rather than imposing one.
  • SparseFARE's hashing doubles as a debiasing step: samples far in protected-attribute space are assigned exactly zero attention, so protected information cannot contribute to separating positive from negative pairs.
  • The stated complexity drops from $O(b^3)$ for CCLK to $O(b^2)$ for FARE and $O(b\log b)$ for SparseFARE, so debiasing can scale to larger batch sizes.
  • On ColorMNIST, SparseFARE matches CCLK's accuracy while raising MSE-based bias removal from 64.7 to 74.0, indicating that bias can be removed without a downstream accuracy penalty.
  • On CelebA, SparseFARE achieves the lowest equalized odds (18.7) among compared models, and its fairness-accuracy curve dominates the kernel baselines at every accuracy level.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mechanism's usefulness outside vision is untested: applying FARE and SparseFARE to text or multimodal encoders with continuous demographic proxies would show whether the attention-learned bias metric transfers beyond pixel color and face attributes.
  • A stress test separating protected-attribute similarity from true bias would clarify the load-bearing assumption: if bias is driven by a confounder that is not aligned with $Z$, the attention map may focus on the wrong samples, and LSH discarding could remove useful semantics.
  • The single-head design means only one bias-causing interaction pattern is captured; a multi-head extension, which the paper names as future work, could in principle represent multiple protected attributes or intersectional interactions at once.
  • The intra-bucket ablation in the appendix suggests that when LSH buckets become too small, both fairness and accuracy drop, implying that SparseFARE's gains depend on batches being large enough to contain sufficient bias-similar negatives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an attention-based framework for fair contrastive learning, called FARE, and a sparse variant SparseFARE that uses locality-sensitive hashing to discard samples judged to be extreme in the protected-attribute dimension. The authors derive a Fair Attention-Contrastive (FAREContrast) loss by replacing the conditional negative sampling in Fair-InfoNCE with an attention-weighted combination of in-batch negatives, where the attention scores are computed over protected attributes. They claim that FARE and SparseFARE capture bias-causing interactions without pre-specified kernels, improve bias removal over kernel-based CCLK and other baselines on ColorMNIST and CelebA, and maintain or improve downstream accuracy. The paper also provides a computational complexity analysis, arguing that FARE costs O(b²) and SparseFARE O(b log b) versus O(b³) for kernel-based competitors.

Significance. If the method performs as claimed, it would be a useful contribution to fair representation learning, particularly for continuous or high-cardinality protected attributes where existing kernel-based methods require pre-specified kernel functions. The idea of using learned attention over protected attributes to reweight negatives is natural and the LSH-based sparsification is a reasonable efficiency/aggressiveness mechanism. The paper includes empirical comparisons on two datasets and an efficiency analysis. However, the central claim that SparseFARE learns substantially less biased representations without compromising accuracy is not fully established because bias removal is measured only with a linear probe and the CelebA results lack error bars. The derivation from kernel density estimation to attention also contains an unjustified step. These issues are correctable, so the contribution is promising but not yet conclusive.

major comments (4)
  1. [Section 3.1, Eq. (8)] The transition from the integral involving φ(g_{θ_Y}(y)) φ_σ(y − y_j) to φ(g_{θ_X}(x_i))^T φ(g_{θ_Y}(y_j)) is mathematically unjustified. In RKHS notation, φ is the feature map of the kernel exp(cosine similarity/τ), which is a nonlinear and generally infinite-dimensional map. The convolution ∫ φ(g(y)) φ_σ(y − y_j) dy does not equal φ(g(y_j)) for an arbitrary feature map. This step is load-bearing because Proposition 1 claims that the attention output is a finite-sample estimation of the conditional score derived from kernel density estimation. The paper should either supply a valid derivation under explicit assumptions (e.g., a plug-in approximation with a stated error term, or a different kernel construction) or explicitly label the expression as a heuristic approximation.
  2. [Section 4, Tables 1 and 2] The bias-removal evaluation relies on a linear probe for ColorMNIST (MSE of a linear regressor predicting the protected attribute) and on Equalized Odds computed from a linear classifier for CelebA. A linear probe only captures linearly accessible information; contrastive objectives that reweight negatives can push protected information into nonlinear subspaces. Without a nonlinear probe (e.g., an MLP or kernel probe) or an information-theoretic measure, the claim that SparseFARE 'removes' sensitive information is not established. Additionally, Table 2 reports no error bars, so differences such as EO 18.7 vs. 20.8 are not statistically secured, and the Pareto-dominance statement for ColorMNIST relies on point estimates with overlapping error bars in Table 1.
  3. [Section 3.3, Eq. (13) and Definitions 1–2] FARE and SparseFARE produce a weighted average of negative-sample similarity scores because the softmax attention weights sum to 1 by construction. The text repeatedly states that FAREContrast is obtained by 'replacing the summation over negative samples' in Fair-InfoNCE, but a sum over b negatives is replaced by a convex combination (or average). This changes the scale and behavior of the contrastive loss; in particular, the denominator no longer grows with batch size as it would in InfoNCE. The paper should clarify whether FARE is intended as an estimate of the conditional expectation E[e^{f(x_i,Y)} | Z=z_i] (in which case the loss is a variant with a single soft negative) or as a replacement for the sum (in which case the attention output should be multiplied by b or the weights should not be normalized). This distinction is essential for interpreting the loss.
  4. [Section 3.1 and Section 3.2] The method is founded on the assumption that when z_i is dissimilar from z_j, the sample x_j is likely to cause bias and should be down-weighted or discarded. This assumption is plausible but is not empirically validated. The paper does not analyze the learned attention scores to confirm that they assign low weight to samples that actually cause bias, nor does it provide an ablation (e.g., comparing against random hashing or reversed similarity) to show that the protected-attribute similarity is indeed capturing bias rather than discarding useful semantic information. A concrete analysis of the learned attention maps or a sensitivity ablation would strengthen the central claim.
minor comments (6)
  1. [Section 3.1] The text refers to 'Eqn. 21' for Fair-InfoNCE, but in the main body this objective appears as Eq. (1); Section 3.3 repeats the same incorrect reference. Please correct the equation numbering.
  2. [Section 6] The term 'FareContrast' in the concluding remarks is inconsistently capitalized; it should be 'FAREContrast' to match the rest of the paper.
  3. [Table 1] The table caption uses 'colorMNIST' while the text and Section 4 use 'ColorMNIST'; please standardize the capitalization of the dataset name.
  4. [Appendix D.1, Table 3] The intra-bucket attention result has a large standard deviation (±9.8), and the text claims a 'substantial drop' without a statistical test; consider reporting individual trial results or a significance measure.
  5. [Figure 3] The caption for Figure 3 appears truncated ('SparseFARE and CCLK-'); please complete it to indicate which kernel baselines are compared.
  6. [Appendix E] The argument that a learnable value transformation W_V would cause collapse by sending weights to infinity is plausible but should be stated with a boundedness condition; it is worth noting why the same issue does not apply to the learned attention projections W_Q and W_K.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FARE/SparseFARE is a proposed training method with a KDE-motivated attention construction; its empirical claims are self-contained.

full rationale

The derivation chain in Section 3.1 estimates the conditional score e^f(xi,y) for y ~ P_{Y|Z=zi} via kernel density estimation, obtaining Eq. (8), and then rewrites the Gaussian weights as softmax attention under normalization assumptions (Eqs. 9-10). The subsequent introduction of learnable projections WQ, WK in Eq. (11) is an explicit modeling choice ('allowing zi and zj to be transformed by learnable linear transformation'), not a result imported from the authors' prior work; Proposition 1 is thus a construction of the FARE objective rather than a prediction that could reduce to its inputs. FAREContrast (Eq. 13) replaces the negative-sample sum in Fair-InfoNCE with this attention-weighted sum, and the bias-removal metric (linear-probe MSE on ColorMNIST, Equalized Odds on CelebA) is not the training objective itself, so no fitted parameter is renamed as a prediction. The key references to conditional contrastive learning and CCLK (Tsai et al., 2021b; Tsai et al., 2022) are external prior work; the authors (Nielsen and Nguyen) do not appear in that citation chain, so no self-citation is load-bearing. The LSH sparsification in Section 3.2 is a stated design assumption ('if we determine some threshold for ignoring...'), not a circular import. The manuscript's own limitations (single-head attention in Section 6; explainability concerns in Appendix F) concern expressiveness and interpretability, not circular dependence. Skeptical concerns about linear-probe sensitivity are evaluation-robustness issues, not circularity. The paper is self-contained as a method proposal and empirical comparison.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the heuristic that protected-attribute similarity tracks bias-causing potential, plus standard KDE/attention machinery. No new physical or geometric entities are postulated. The learnable attention projections WQ and WK and the hyperparameters tau, rho, sigma, and LSH bucket structure are free parameters fitted or chosen for the reported results.

free parameters (5)
  • WQ, WK (attention projections) = learned during training
    Introduced in Eqn (11) as learnable linear transformations; the debiasing property depends on their learned values.
  • tau (scoring temperature) = not reported per dataset
    Temperature in scoring function Eqn (2), a user-set hyperparameter.
  • rho (attention temperature) = set equal to sigma squared
    Attention temperature in Eqn (4), assumed equal to the KDE bandwidth squared in Eqn (9).
  • sigma (KDE bandwidth) = set to sqrt(rho)
    Bandwidth in Eqn (7); the derivation requires sigma squared equals rho, so this is not independently chosen.
  • LSH bucket size and hash rounds = bucket 64/128, 8 rounds
    Sparsification hyperparameters reported in Appendix A.1.
assumptions (4)
  • standard math Isotropic Gaussian KDE is a valid estimator of P(y,z) and P(z)
    Used in Eqn (7) to estimate the conditional density; suffers in high dimensions but is standard.
  • ad hoc to paper Protected attributes are normalized and sigma squared equals rho
    Required to pass from Eqn (9) to Eqn (10); not enforced in the implementation, weakening the derivation.
  • domain assumption Protected-attribute similarity determines bias-causing potential
    Core heuristic in Section 3.1 and 3.2: similar Z means bias-reducing, dissimilar Z means bias-causing. Not proven.
  • standard math LSH random projections preserve angular similarity
    Based on Andoni et al. (2015), used in Section 3.2 to discard extreme bias samples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Attention-based Framework for Fair Contrastive Learning." pith.science (2026). https://pith.science/paper/5VP6KDHA

@misc{pith2026241114765,
  author       = {Pith},
  title        = {Pith review of: An Attention-based Framework for Fair Contrastive Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VP6KDHA}},
  note         = {Machine review of arXiv:2411.14765}
}
read the original abstract

Contrastive learning has proven instrumental in learning unbiased representations of data, especially in complex environments characterized by high-cardinality and high-dimensional sensitive information. However, existing approaches within this setting require predefined modelling assumptions of bias-causing interactions that limit the model's ability to learn debiased representations. In this work, we propose a new method for fair contrastive learning that employs an attention mechanism to model bias-causing interactions, enabling the learning of a fairer and semantically richer embedding space. In particular, our attention mechanism avoids bias-causing samples that confound the model and focuses on bias-reducing samples that help learn semantically meaningful representations. We verify the advantages of our method against existing baselines in fair contrastive learning and show that our approach can significantly boost bias removal from learned representations without compromising downstream accuracy.

Figures

Figures reproduced from arXiv: 2411.14765 by the authors.

Figure 1
Figure 1. Sparse Fair-Aware Attention (SparseFARE) using LSH to discard bias-causing samples. Relative to the [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. colorMNIST dataset (Tsai et al., 2022) We follow the recent contrastive learning literature (Chen et al. (2020), Robinson et al. (2020), Wu et al. (2020)) and pre-train the full model before discarding everything except the backbone encoder at evaluation time. CelebA. The train-test split is the default as provided by PyTorch. Images are resized to 128×128. Resnet-18 (He et al., 2016) is the encoder and we use the s… view at source ↗
Figure 3
Figure 3. Fairness-Accuracy Tradeoff of SparseFARE and CCLK￾We note that there are two, interconnected prevalent ethical issues in fair ML. The first is that almost all fair ML literature simplifies the problem of fairness to simple binaries and the second is that fairness metrics (which are typically built atop these binaries) and the choice of which to use themselves involve value judgements that can dis￾advantage certain p… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 20 canonical work pages

  1. [2]

    In particu- lar, we avoid specifying any particular kernel and allow our attention mechanism to learn the bias-causing interactions

    Second, our methods do not impose assumptions on the bias-causing interactions over protected attributes. In particu- lar, we avoid specifying any particular kernel and allow our attention mechanism to learn the bias-causing interactions. To see this difference, we decompose the estimator in Eqn. 2 as follows: 13 PREPRINT KXY (KZ + λI)−1KZ ii (17) = [KXY ...

    work page 2022

  2. [3]

    log ef (x,ypos) ef (x,ypos) + Pb i=1 ef (x,yneg,i) # , (21) and FAREContrast is given as: sup f E{(xi,yi,zi)}b i=1∼P ⊗b XY Z

    [Kernel Conditional Embedding Operator (Song et al., 2013)] The finite-sample kernel estimation of Ey∼PY |Z=z ϕ(g(y)) is Φ⊤(KZ + λI)−1Γγ(z) where λ is a hyperparameter. Then, according to Definition 3, for any given Z = z, ϕ(g(y)) when y ∼ PY |Z=z can be estimated by Φ⊤(KZ + λI)−1Γγ(z) (19) We look for the inner product between (5) and the encoding of (xi...

    work page 2021

  3. [4]

    A.1 Training and Evaluation ColorMNIST. Samples in the colorMNIST dataset are 32x32 resolution handwritten digit images, where the digit is represented in black and the background is some known assigned color which is representable as a continuous RGB color vector. The train-test split is 60,000 training images to 10,000 test images. The augmentation sche...

    work page 2017

  4. [6]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pp. 276–286, Florence, Italy, August

    work page 2019

  5. [7]

    doi: 10.18653/v1/W19-4828

    Association for Computational Linguistics. doi: 10.18653/v1/W19-4828. URL https://www.aclweb.org/anthology/W19-4828. Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In International conference on machine learning, pp. 1436–1445. PMLR,

    doi:10.18653/v1/w19-4828

  6. [8]

    Fairness metrics: A comparative analysis

    Pratyush Garg, John Villasenor, and Virginia Foggo. Fairness metrics: A comparative analysis. In 2020 IEEE Inter- national Conference on Big Data (Big Data), pp. 3662–3666. IEEE,

    work page 2020

  7. [9]

    Designing and interpreting probes with control tasks

    John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733–2743, Hong Kong, China, November

    work page 2019

  8. [10]

    doi: 10.18653/v1/D19-1275

    Associa- tion for Computational Linguistics. doi: 10.18653/v1/D19-1275. URL https://www.aclweb.org/anthology/ D19-1275. R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670,

    arXiv

Show all 41 references
  1. [11]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451,

    2001 arXiv

  2. [12]

    Large-scale celebfaces attributes (celeba) dataset

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11,

    2018

  3. [14]

    Duet: A tuning-free device-cloud collaborative parameters generation frame- work for efficient device model generalization

    Zheqi Lv, Wenqiao Zhang, Shengyu Zhang, Kun Kuang, Feng Wang, Yongwei Wang, Zhengyu Chen, Tao Shen, Hongxia Yang, Beng Chin Ooi, et al. Duet: A tuning-free device-cloud collaborative parameters generation frame- work for efficient device model generalization. In Proceedings of...

    2023

  4. [15]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748,

    arXiv

  5. [17]

    Contrastive learning with hard negative sam- ples

    Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative sam- ples. arXiv preprint arXiv:2010.04592,

    2010 arXiv

  6. [19]

    Contrastive learning for fair representa- tions

    Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, and Lea Frermann. Contrastive learning for fair representa- tions. arXiv preprint arXiv:2109.10645,

    arXiv

  7. [21]

    doi: 10.18653/v1/P19-1452

    Association for Computational Linguistics. doi: 10.18653/v1/P19-1452. URL https://www.aclweb.org/ anthology/P19-1452. Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural ...

    doi

  8. [22]

    Trans- former dissection: a unified understanding of transformer’s attention via the lens of kernel

    Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Trans- former dissection: a unified understanding of transformer’s attention via the lens of kernel. arXiv preprint arXiv:1908.11775,

    1908 arXiv

  9. [23]

    Self-supervised representation learning with relative predictive coding

    Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang, Han Zhao, Louis-Philippe Morency, and Ruslan Salakhutdinov. Self-supervised representation learning with relative predictive coding. arXiv preprint arXiv:2103.11275, 2021a. Yao-Hung Hubert Tsai, Martin Q Ma, Han Zhao, Kun Zhang, ...

    arXiv

  10. [25]

    doi: 10.18653/v1/W19-4808

    Association for Computational Linguistics. doi: 10.18653/v1/W19-4808. URL https://www.aclweb.org/anthology/W19-4808. Elena V oita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Spe- cialized heads do the heavy lifting, the res...

    doi

  11. [26]

    doi: 10.18653/v1/P19-1580

    Association for Computa- tional Linguistics. doi: 10.18653/v1/P19-1580. URL https://www.aclweb.org/anthology/P19-1580. Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep ima...

    doi

  12. [27]

    Transformers are deep infinite-dimensional non-mercer binary kernel machines

    Matthew A Wright and Joseph E Gonzalez. Transformers are deep infinite-dimensional non-mercer binary kernel machines. arXiv preprint arXiv:2106.01506,

    arXiv

  13. [28]

    Conditional negative sampling for contrastive learning of visual representations

    Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. Conditional negative sampling for contrastive learning of visual representations. arXiv preprint arXiv:2010.02037,

    2010 arXiv

  14. [29]

    Large batch training of convolutional networks

    Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888,

    arXiv

  15. [31]

    (2020), Robinson et al

    We follow the recent contrastive learning literature (Chen et al. (2020), Robinson et al. (2020), Wu et al. (2020)) and pre-train the full model before discarding everything except the backbone encoder at evaluation time. CelebA. The train-test split is the default as provided...

    2020

  16. [33]

    and the conditional contrastive learning with kernel model (CCLK) (Tsai et al. (2022)). The InfoNCE model uses the InfoNCE loss function 15 without performing any conditional sampling. The Fair- InfoNCE model uses the Fair-InfoNCE loss function 21 and performs conditional samp...

    2022

  17. [34]

    and all kernel implementations of CCLK provided by Tsai et al. (2022). For each kernel model, the kernel in the name refers to the what kernel similarity metric is chosen for measuring the similarity across protected attributes, which then determines the relevance of that samp...

    2022

  18. [35]

    and Attention The CCLK model uses the following kernel-based scoring function estimation: Proposition 2 (Kernel-Based Scoring Function Estimation (Tsai et al., 2022)) . Given {xi, yi, zi}b i=1 ∼ P b XY Z, the similarity score of the data pair(xi, y) given the anchor zi is comp...

    2022

  19. [36]

    and sparseFARE has complexity O(b logb) (Kitaev et al., 2020), which improve significantly over O(b3) in Eqn

    2020

  20. [38]

    Proof of kernel-based scoring function estimation

    of their kernel-based scoring function estimation below. Proof of kernel-based scoring function estimation. First, letting Φ = [ϕ(g(y1)), . . . ϕ(g(yb))]⊤ be the matrix of kernel embeddings for encodings g(yi) with feature map ϕ and Γ = [γ(z1), . . . , γ(zb)]⊤ be the matrix of...

    2013

  21. [40]

    Fare and SparseFARE in comparison with unsupervised and supervised models under partial sensitive label access

    74.1 24.5 FARE (ours) 73.7 23.5 SparseFARE (ours) 70.4 18.7 Table 4: CelebA Results. Fare and SparseFARE in comparison with unsupervised and supervised models under partial sensitive label access. 15 PREPRINT This paper uses the same experimental setup on CelebA as Zhang et al...

    2022

  22. [41]

    or CGL (Jung et al., 2022), to solve this problem. Given the experimental setups are the same, we include their results as well for reference, however we do not feature these results in the main body given the important difference regarding sensitive attribute access. E Fair A...

    2022

  23. [1956]

    Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization

    Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731,

    1911 arXiv

  24. [1962]

    Gradient reversal against discrimination: A fair neural network learning approach

    Edward Raff and Jared Sylvester. Gradient reversal against discrimination: A fair neural network learning approach. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 189–198. IEEE,

    2018

  25. [2013]

    Nonlinear svd with asymmetric kernels: feature learning and asymmetric nystr\" om method

    Qinghua Tao, Francesco Tonin, Panagiotis Patrinos, and Johan AK Suykens. Nonlinear svd with asymmetric kernels: feature learning and asymmetric nystr\" om method. arXiv preprint arXiv:2306.07040,

    arXiv

  26. [2015]

    Unbiased supervised contrastive learning

    Carlo Alberto Barbano, Benoit Dufumier, Enzo Tartaglione, Marco Grangetto, and Pietro Gori. Unbiased supervised contrastive learning. arXiv preprint arXiv:2211.05568,

    arXiv

  27. [2016]

    is the encoder and we use the same 2-layer MLP and random augmentation strategies as Chen et al. (2020). 12 PREPRINT Same as with colorMNIST, we pre-train with the LARS optimizer and use cosine annealing. We use a batch size of 512 and the LSH scheme uses buckets of size 128 w...

    2020

  28. [2017]

    Analyzing the structure of attention in a transformer language model

    Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. In Proceed- ings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pp. 63–76, Florence, Italy, August

    2019

  29. [2018]

    The variational fair autoencoder.arXiv preprint arXiv:1511.00830,

    Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder.arXiv preprint arXiv:1511.00830,

    arXiv

  30. [2019]

    Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka

    arXiv preprint arXiv:1904.10509. Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in neural information processing systems, 33:8765–8775,

    1904 arXiv

  31. [2020]

    On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610–623,

    2021

  32. [2021]

    Fairfil: Contrastive neural debiasing method for pretrained text encoders

    Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. Fairfil: Contrastive neural debiasing method for pretrained text encoders. arXiv preprint arXiv:2103.06413,

    arXiv

  33. [2022]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150,

    2004 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.