Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Data

Chihiro Noguchi; Takaki Yamamoto; Toshihiro Tanizawa

REVIEW 3 major objections 4 minor 1 cited by

Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Data

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In a minimal 1D testbed, CLIP-style contrastive training learns left–right relations, and the mechanism is a positional-embedding-induced attention gradient that breaks left–right symmetry.

desk verdict Clean toy-model evidence for an attention-gradient mechanism behind left-right relational learning in CLIP-style encoders, but the causal ablation only runs in a reduced 1-layer model—the paper is honest about this, and it is a genuine limitation. read the letter →

arxiv 2601.12809 v2 pith:PYKWNJV2 submitted 2026-01-19 cs.CV cs.AIcs.LG

Takaki Yamamoto , Chihiro Noguchi , Toshihiro Tanizawa This is my paper · ORCID

classification cs.CVcs.AIcs.LG MSC 68T07

keywords left-rightrelationsCLIPcontrastivelearningspatialreasoningattentionmechanismpositionalembeddingssyntheticdatavision-languagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether CLIP-style contrastive training can genuinely acquire spatial-relation understanding rather than merely memorizing object co-occurrence. Using a controlled 1D image–text testbed, it shows that Transformer-based vision and text encoders trained with a CLIP-style objective do learn left–right relations and generalize them to unseen object pairs. The paper identifies a concrete mechanism: interactions between positional and token embeddings create a horizontal gradient in attention logits, so individual attention heads specialize to attend to the left or right object. The authors demonstrate that ablating this positional contribution drops unseen-pair relational accuracy to chance while object-pair recognition survives, and they show that label diversity, not layout diversity, is the primary driver of this generalization.

What carries the argument

The key object is the weight-bias decomposition of pre-softmax attention logits, QKᵀ = XW_QK Xᵀ + XW_QᵀB_K + B_QᵀW_KXᵀ + B_QᵀB_K, followed by the positional-token decomposition X = E + P, which isolates the EP term E W_QK Pᵀ. In a well-generalizing model, this EP term carries a horizontal gradient in the class-token row, creating a systematic rightward (or leftward) attention bias. The theoretical condition for consistent directional preference across unseen pairs is |Π| > |T|, where Π is the position-dependent contribution and T is the token-dependent contribution; this condition explains why the positional gradient, not label-specific variation, must dominate for relational generalization.

What would settle it

Train a CLIP-style ViT on 2D synthetic images with left–right captions and enough label diversity to generalize to unseen object pairs. Compute the EP term in the attention logits and ablate it at inference: if unseen-pair left–right accuracy does not drop to near chance while object-pair recognition survives, then the proposed attention-gradient mechanism does not transfer to the 2D setting.

Watch

Extended reading notes

Core claim

Left–right relational competence in a minimal CLIP-style model emerges not from object-specific memorization but from positional-embedding-driven attention asymmetry. Decomposing pre-softmax attention logits into token, positional, and cross terms reveals that the EP term — the cross term between token embeddings and positional embeddings — produces a monotonic horizontal gradient in attention. This gradient makes some heads consistently attend to the right-side object and others to the left-side object, independent of object identity, allowing the model to encode relative spatial position. Ablating this term at inference reduces unseen-pair left–right discrimination to near chance, while th

Load-bearing premise

The 1D synthetic testbed with single-pixel objects, fixed templates, and small Transformers is assumed to be a valid proxy for spatial-relation learning in real CLIP-style vision-language models trained on natural 2D images; the paper itself flags this as an open question.

Editorial extensions

If this is right

If the mechanism is right, contrastive training alone can yield spatial-relation competence without explicit relational supervision, provided the training data offers enough label diversity.
Relational discrimination is separable from object recognition: ablating the positional attention term leaves object-pair identification intact while destroying left–right discrimination.
Multi-head attention with complementary specialist heads appears necessary in this setting, since single-head models fail to generalize across all tested hyperparameters.
Architectural choice of positional encoding matters: both learned positional embeddings and RoPE can support generalization, but through different mechanisms.
Training dynamics proceed in stages — single-object positional, seen-pair configuration, then unseen-pair generalization — with weight decay aiding the transition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension for real CLIP-style models on natural images: measure the strength of the EP-term attention gradient by decomposing attention logits in a deployed ViT and check whether left–right benchmark performance correlates with the dominance of |Π| over |T|.
The paper's data-diversity finding suggests a concrete curation guideline: for spatial-relation capability, increasing the variety of object pairs in captions may matter more than augmenting spatial layouts, a hypothesis that could be tested directly in large-scale training.
The consistency condition |Π| > |T| could serve as a training-time probe: monitor whether position-dependent attention dominates token-dependent variation as a predictor of future relational generalization.
The RoPE low-rank subspace mechanism implies that relational generalization can emerge even without learned positional embeddings, suggesting that measuring the effective rank of key projections could predict spatial competence in modern language and vision transformers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this paper shows, in a very minimal 1D CLIP-style setup, that contrastive training can learn left-right relations and generalize to unseen object pairs. The mechanism they isolate—a positional-token cross-term in attention logits creating a horizontal gradient—is genuinely new as an explicit account. The decomposition into EE/EP/PE/PP terms and the inference-time ablation of the EP term (accuracy drops to chance while object-pair recognition survives) is a clean, well-controlled experiment. The theoretical condition in App. I (|Π|>|T| for consistent directional preference) is a useful formalization, and the RoPE extension showing two alternative mechanisms is a nice bonus.

The label-diversity-over-layout-diversity claim is straightforwardly demonstrated in their sweeps, though it lacks error bars and seed variance, so treat it as suggestive rather than quantitative. No code or data are released, which matters more here than usual because the whole study is synthetic and small-scale.

The real soft spot is the gap between the model whose generalization they headline and the model they analyze mechanistically. Fig. 3 uses a 2-layer, 2-repeat vision encoder with LayerNorm and MLP; the attention-gradient analysis and EP ablation are done on a reduced 1-layer encoder without LayerNorm or MLP. The paper connects them only by 'qualitatively similar' accuracy curves, and App. L shows that with two layers the attention patterns are already more complex and left to future work. So the causal mechanism is established for the reduced model, not for the full model whose generalization they report. That is a real limitation, and the abstract's phrasing overstates the reach. The authors do acknowledge the gap in Sec. 9 and App. L, so it is a matter of framing, not concealment.

App. I's 'necessary' language also overreaches: the inequality is a condition for consistent directional preference, not a proven necessity for generalization. Minor issue.

Verdict: for a mechanistic-interpretability audience, this is a worthwhile read. It deserves a serious referee. I would send it to review but ask for code/data, error bars, and either a test of the EP ablation in the full model or a more careful claim that the mechanism is demonstrated only in the reduced setting. I would not cite it in my own work as-is, mostly because transfer to real CLIP models is completely unestablished.

Referee Report

3 major / 4 minor

Summary. The paper studies whether CLIP-style contrastive training can learn left-right spatial relations in a minimal synthetic 1D image-text setup, and if so, through what mechanism. The authors train lightweight Transformer vision and text encoders on paired data of one- and two-object images with template captions, varying the number of training labels and spatial configurations. They report three results: (1) contrastive training yields generalization to unseen object pairs; (2) label diversity, more than layout diversity, drives this generalization; and (3) the mechanism is an attention gradient induced by interactions between positional and token embeddings, which breaks left-right symmetry in the vision encoder. They support the mechanism with an attention-logit decomposition, inference-time ablations of the EP term, and a theoretical condition (|Π|>|T|) in App. I, along with an extension analyzing RoPE. The paper is careful to flag the toy nature of the setup and the open question of transfer to natural 2D images.

Significance. If the mechanistic account holds, this is a useful contribution: it provides a concrete, falsifiable explanation for how a contrastive dual-encoder can acquire relational competence in a controlled setting, and it identifies label diversity rather than layout diversity as the key training-data factor. The synthetic testbed is well suited for isolating the mechanism, and the paper includes several strengths: inference-time ablations that directly test the proposed EP-term contribution, a theoretical derivation that gives a simple condition for consistency, and an extension to RoPE that proposes two distinct mechanisms. The authors are appropriately cautious about the limited reach of a 1D toy model and explicitly state in Sec. 9 that transfer to large-scale natural-2D models remains open. However, the central mechanistic claim is demonstrated only in a reduced architecture, not in the full model whose generalization is reported, and the data-diversity claim lacks statistical support.

major comments (3)

[Secs. 4–5; App. L] The central mechanistic claim—that an EP-induced horizontal attention gradient is what enables left-right generalization—is tested only in the reduced 1-layer vision encoder without LayerNorm or MLP (Sec. 4, Fig. 4; Sec. 5, Fig. 5e). The headline generalization results in Sec. 3 and Fig. 3 use MB=2, Mrep=2 with LayerNorm and MLP. The paper explicitly says only that the reduced model shows "qualitatively similar behavior" to the original model (Sec. 4), which does not establish that the same mechanism operates in the full architecture. App. L reinforces the concern: even in a 2-layer model without LayerNorm/MLP, the attention gradients are "more complex" and analysis is left to future work. Since the ablation-to-chance result is the primary evidence for the mechanism, it needs to be reproduced in the full model, or the mechanistic claim must be restricted to the reduced architecture.
[Sec. 3; Fig. 3] The claim that label diversity is the primary driver of generalization, more than layout diversity, is not supported with sufficient experimental detail. The text states that "the number of spatial configurations of objects is also varied," but the figure and caption do not specify how layout diversity is varied, what values are used, or how its effect is separated from label diversity. Moreover, all accuracy curves in Fig. 3 (and elsewhere) are shown without error bars or seed variance. Given the small models and stochastic training, the conclusion "label diversity, more than layout diversity" needs either a controlled comparison with multiple seeds and standard errors, or an explicit statement of the layout-diversity manipulation in the main text or a dedicated appendix.
[App. I; Eq. (14)] The theoretical condition for consistent directional preference is |Π|>|T| (Eq. 13), and the paper argues that for unseen-pair generalization one requires |Π|≫|T| for all unseen pairs (Eq. 14). However, for the learnable-PE models in the main text, no quantitative check of this condition is reported. Fig. 5(c,d) shows aggregate statistics (Δp.e. vs. Δlabel and the resulting relational/label-specific classification), but not the satisfaction rate of Eq. (8)/Eq. (13) across unseen pairs. In contrast, the RoPE analysis (App. I.3.2, Fig. 19a) does report such consistency rates. Adding an analogous consistency-rate plot for the learnable-PE case would directly validate the theoretical claim.

minor comments (4)

[Sec. 5; Fig. 5(d)] The caption and surrounding text for Fig. 5(d) state both classes as "|Δp.e.|<|Δ label|"; one of them should presumably be "|Δp.e.|>|Δ label|". Please fix the inequality and ensure the legend matches the classification rule given in the text.
[Sec. 2.1; App. I.2; App. I.3.2] Minor typos and wording issues: "labal" in Sec. 2.1, "propsoed" in App. I.2, "disntance" and "serveral" in App. I.3.2, and "posiiton" in Fig. 5. Also, Fig. 5(e) and App. G list "EP, PP, PP terms" where one item is likely "PE".
[Sec. 2.2; Fig. 8 caption] Grammar and notational consistency: "They consists" should be "They consist"; in the Fig. 8 caption, "Mvis_rep=1, Mtxt^B=2, Mvis^B=1" is inconsistent—likely "Mtxt_rep=1" was intended. Please check all hyperparameter annotations for consistency.
[Sec. 5] The relational vs. label-specific head classification in Fig. 5(d) is definitional (based on whether |Δp.e.| exceeds |Δlabel|). Consider clarifying that this is a descriptive statistic, not independent evidence for the mechanism; the ablation in Fig. 5(e) is the causal test.

Circularity Check

1 steps flagged · score 2.0 of 10

Central EP-gradient mechanism is independently tested by ablation; only a minor definitional overlap in the 'relational vs label-specific' head classification.

self definitional [Sec. 5, Fig. 5(c)-(e), around Eq. (2)]
"We classify a head as relational when |Δlabel|<|Δp.e.| (with the sign of Δp.e. matching the geometric direction); otherwise it is label-specific. In Fig. 5(d), we show the probability that each head attends to the two objects in a relational or label-specific manner. ... This indicates that the left–right attention bias arises primarily from the monotonic gradient in the EP term."

The 'relational' label is defined by the condition being concluded: the EP term dominates the label term and points in the spatial direction. Concluding from this classification that the attention bias arises from the EP gradient is partly a restatement of the definition, not an independent measurement. The separate comparison with actual attention weights (Fig. 4(c)) and, more importantly, the inference-time ablation of EP terms (Fig. 5(e)) provide independent causal support, so this is a localized definitional overlap rather than a circular derivation of the main result.

full rationale

Throughout the paper the main derivation is not circular. Generalization to unseen label pairs is evaluated on held-out categories (Nval=5) that appear only in single-object training, and the label-diversity result is an empirical observation. The mechanistic claim is supported by a direct causal intervention: zeroing EP/PE/PP/BP terms in the pre-softmax attention at inference drops unseen-pair accuracy to ~0.5 while label-set recognition survives (Fig. 5(e), App. G). The theoretical appendix (App. I) starts from stated assumptions and derives the |Π|>|T| consistency condition rather than assuming it. There are no self-citations by the present authors, so no self-citation chain is load-bearing. The one genuinely questionable step is the 'relational vs label-specific' head classification in Sec. 5, where the category 'relational' is defined as EP dominance, making the subsequent statement that EP dominates partly definitional; this is minor because the ablation independently establishes causality. The reduced 1-layer setting and the App. L admission that 2-layer attention gradients are more complex are scope limitations, not circularity. Overall, the central claim has independent empirical content, so a low score is appropriate.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

All free parameters are hand-chosen dataset/architecture/training settings; no constants are fitted to the target result. The central mechanism is supported by ablation and a conditional theory, but the transferability of the 1D toy is assumed.

free parameters (3)

Dataset scale and label split (Ntot, Npair, Nval, n1, n2) = Ntot 7–25; Npair=Ntot−5; Nval=5; n1=5; n2=5–15 depending on experiment
These hand-chosen dataset sizes define the synthetic regime; the empirical claim that label diversity drives generalization is measured across this particular grid, so the result is conditional on these scales.
Architecture hyperparameters (layers, heads, d_model, d_head, MLP, dropout) = MB/Mrep=2 or 1; Mh=4 (or 1 in App. K); dhead=32; dmodel=128; dMLP=512; p=0.1
Mechanistic analysis uses a stripped-down 1-layer, no-LayerNorm/no-MLP variant; the claim that this variant is qualitatively representative relies on these choices.
Training hyperparameters (lr, weight decay, epochs, batch size) = lr=1e-4, w=0.2, epochs=10,000, bs=50/100
Training dynamics and generalization levels depend on these; weight decay is shown to affect generalization (App. B), so the main result is specific to this training budget.

assumptions (5)

domain assumption A 1D sequence of single-pixel object tokens with template captions is a meaningful proxy for spatial-relation learning in CLIP-style VLMs.
Introduced in Sec. 2.1; the entire scope of the conclusions (Sec. 9) rests on this transferability assumption, which the paper itself flags as open.
domain assumption For the 1-layer Transformer, the CLS/EOT output is determined solely by the attention distribution over input tokens, so attention logit decomposition fully explains the representation.
Used in Sec. 5 and App. D; relies on removing LayerNorm/MLP and using 1 layer, so each token's output is a weighted sum; deeper-model analysis is left to future work (App. L).
domain assumption Inference-time zeroing of the EP (and other) logit terms is a valid causal intervention that isolates the contribution of that term.
Sec. 5, Fig. 5(e). Assumes no compensatory renormalization effects beyond softmax shift-invariance; the ablation is applied post hoc.
ad hoc to paper In App. I, consistent directional preference (Eq. 8) is both necessary for unseen-pair generalization and must hold for all unseen pairs (Eq. 14).
The theory assumes the model needs a single global left/right preference and that token-dependent term T is unpredictable for unseen pairs; this is a stated modeling choice, not a theorem about all possible algorithms.
domain assumption RoPE mechanism 2 assumes W_K projects token embeddings into a low-dimensional subspace with positive scalar coefficients (Eq. 24).
App. I.3.1; supported empirically by PCA/SVD, but the analytical result is conditional on this low-rank structure.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Data." pith.science (2026). https://pith.science/paper/PYKWNJV2

@misc{pith2026260112809,
  author       = {Pith},
  title        = {Pith review of: Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PYKWNJV2}},
  note         = {Machine review of arXiv:2601.12809}
}

read the original abstract

Spatial understanding remains a key challenge in vision-language models. Yet it is still unclear whether such understanding is truly acquired, and if so, through what mechanisms. We present a controllable 1D image-text testbed to probe how left-right relational understanding emerges in Transformer-based vision and text encoders trained with a CLIP-style contrastive objective. We train lightweight Transformer-based vision and text encoders end-to-end on paired descriptions of one- and two-object scenes and evaluate generalization to unseen object pairs while systematically varying label and layout diversity. We find that contrastive training learns left-right relations and that label diversity, more than layout diversity, is the primary driver of generalization in this setting. To gain the mechanistic understanding, we perform an attention decomposition and show that interactions between positional and token embeddings induce a horizontal attention gradient that breaks left-right symmetry in the encoders; ablating this contribution substantially reduces left-right discrimination. Our results provide a mechanistic insight of when and how CLIP-style models acquire relational competence.

Figures

Figures reproduced from arXiv: 2601.12809 by the authors.

**Figure 1.** Schematic of our problem: how spatial and relational capabilities are, or are not, acquired in vision-language models? 2025; Cheng et al., 2024; Chen et al., 2025), and that these limitations are closely tied to persistent gaps in their compositional generalization (Thrush et al., 2022; Ma et al., 2023; Ray et al., 2023; Tong et al., 2024; Zheng et al., 2024). Beyond academic interest, safety-critical applications… view at source ↗

**Figure 2.** Schematic of CLIP model training of the toy dataset of 1D images and the corresponding texts. For two object images, left and right textual representations denote inverse relations that describe the same configuration. 2. Experimental setup We introduce our toy experimental setting, shown in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Three types of generalization observed in a CLIP-style training setup. This experiment is performed only with the left textual representation. (Left) Accuracy is shown for three types of generalization (a–c). (Right) Cosine similarity maps between image and text embeddings from the output layers are shown (Ntot = 20, Npair = 15, Nval = 5, n2 = 10). In the similarity map of (a), n1 images sharing the same text repre… view at source ↗

Figures from the paper (20 more)

**Figure 4.** Figure 4: Analysis of reduced 1-layer model with MB = Mrep = 1, Mh = 4. The other parameters are the same as those used in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Attention gradient emerging in the attention pattern. (a) Contribution of decomposed terms to logit of the attention is shown for head 2 as a representative example. The red rectangles highlight the class token row in each attention map. The model for [PITH_FULL_IMAGE…

**Figure 6.** Figure 6: Attention patterns of encoder. (a) Representative examples with object 17 and 19 with swapped spatial relations are shown. (b-c) The average and the standard deviation of the attention map averaged over the text samples are shown. The text encoder is trained jointly wi…

**Figure 7.** Figure 7: Aligning image and text token embeddings via learned rotation matrices. (a) Cosine similarity between image and text token embeddings for each label prior to optimization. (b) A rotation matrix is optimized to align text token embeddings with their corresponding imag…

**Figure 8.** Figure 8: Generalization in the setup with left and right textual representation. (a,b) Accuracy is shown for single-object positional and unseen-pair generalization. The model parameters are Mvis B = 1, Mvis rep = 1, Mvis h = 4, Mtxt B = 2, Mvis rep = 1, Mtxt h = 4, dhead = 32…

**Figure 9.** Figure 9: Schematic of training and test datasets for evaluating three types of generalization. Single-object images are used to assess single-object positional generalization, while two-object images are used to assess seen-pair configuration generalization and unseen-pair gene…

**Figure 10.** Figure 10: Unseen-pair generalization is analyzed for the models trained with different weight decay w. MB = 2, Mrep = 2, Mh = 4, dhead = 32, dMLP = 512, dmodel = 128. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 11.** Figure 11: shows the dynamics of generalization accuracy and training loss during training. We find that single-object positional, seen-pair configuration, and unseen-pair generalization are achieved consecutively. Based on the completion of single-object positional and seen-pai…

**Figure 12.** Figure 12: shows the weight-bias decomposition of pre-softmax logit QKT for all four heads for the model analyzed in [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

**Figure 13.** Figure 13: The standard deviation across the class token row of each term’s contribution to the logit matrix. Each count corresponds to a single 1D image. E. Positional-token embedding decomposition of pre-softmax logit for all the heads of generalizing model [PITH_FULL_IMAGE:f…

**Figure 14.** Figure 14: Positional-token embedding decomposition of pre-softmax logit for all the heads of generalizing model. The model is that analyzed in [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]

**Figure 15.** Figure 15: Absence of positional–token attention gradients in a non-generalizing model. (a) Probability that each head attends to the left or right object the most, computed from the class token’s attention weights. (b) Spatial profiles of the EP term defined in the main text fo…

**Figure 16.** Figure 16: Accuracy of unseen-pair label-set recognition in two-object images under different ablation conditions. The label pairs not seen during training is used in this test. Baseline is the model from [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]

**Figure 17.** Figure 17: Effect of ablating positional-embedding–derived logit components. (a) Accuracy for unseen-pair generalization is shown for different ablation conditions. Baseline is the model from [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]

**Figure 18.** Figure 18: Analysis of reduced 1-layer model with RoPE (no learnable positional embedding, MB = Mrep = 1, Mh = 4). The other parameters are the same as those used in [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]

**Figure 19.** Figure 19: Analysis of 1-layer model with RoPE. All analyses are performed on unseen-pair data to investigate the mechanism underlying unseen-pair generalization. The model analyzed in [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]

**Figure 20.** Figure 20: shows the one-object recognition and generalization to unseen object pairs for models with 1-layer, 4-head vision and text encoders. We find that the models can recognize individual objects ( [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]

**Figure 21.** Figure 21: Performance of models with single-head, single-layer vision encoder. Accuracy for unseen-pair generalization is shown. (a) Effect of weight decay regularization. The experiments are performed for the model with Mvis B = Mvis rep = 1, Mvis h = 1, dvis head = d vis mode…

**Figure 22.** Figure 22: Performance of models with pruned heads after retraining. Models are obtained by pruning heads from a trained four-head, single-layer model and retraining for an additional 10,000 epochs. (a) Performance of original and pruned models for all possible head pruning conf…

**Figure 23.** Figure 23: Attention pattern observed in the first Transformer layer of a generalizing model with 2-layer vision encoder. Positional-token embedding decomposition of pre-softmax logit for all the heads of generalizing model. The parameter for the dataset generation is Ntot = 25,…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath)
cs.CL 2026-08 conditional novelty 6.0 of 10

IslamicTurathBench is a new expert-reviewed Arabic benchmark that tests LLMs on classical Islamic scholarship across seven disciplines, three difficulty tiers, and three task formats.

Reference graph

Works this paper leans on

5 extracted references · cited by 1 Pith paper

[1]

The label pairs not seen during training is used in this test

YLX or X YC Image Text Top-1 similarity Ablation: BP term Unseen-pair label-set recognition Figure 16.Accuracy of unseen-pair label-set recognition in two-object images under different ablation conditions. The label pairs not seen during training is used in this test. Baseline is the model from Fig. 4(a) (n2 = 10). At inference, we zero specific pre-softm...

2023
[2]

YRX Top-2 similarity

XLY, 2. YRX Top-2 similarity
[3]

XLY or X YC Image Text

YRX, 2. XLY or X YC Image Text
[4]

YLX or X YC Image Text Top-1 similarity
[5]

(a) The single-object positional generalization, and (b,c) unseen-pair generalization are shown

YRX or X YC Image Text Top-1 similarity Accuracy Figure 20.Single-object recognition and generalization to unseen object pairs for models with 1-layer, 4-head vision and text encoders. (a) The single-object positional generalization, and (b,c) unseen-pair generalization are shown. In (c), we relax the criteria: a prediction is considered correct if the hi...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

The label pairs not seen during training is used in this test

YLX or X YC Image Text Top-1 similarity Ablation: BP term Unseen-pair label-set recognition Figure 16.Accuracy of unseen-pair label-set recognition in two-object images under different ablation conditions. The label pairs not seen during training is used in this test. Baseline is the model from Fig. 4(a) (n2 = 10). At inference, we zero specific pre-softm...

2023

[2] [2]

YRX Top-2 similarity

XLY, 2. YRX Top-2 similarity

[3] [3]

XLY or X YC Image Text

YRX, 2. XLY or X YC Image Text

[4] [4]

YLX or X YC Image Text Top-1 similarity

[5] [5]

(a) The single-object positional generalization, and (b,c) unseen-pair generalization are shown

YRX or X YC Image Text Top-1 similarity Accuracy Figure 20.Single-object recognition and generalization to unseen object pairs for models with 1-layer, 4-head vision and text encoders. (a) The single-object positional generalization, and (b,c) unseen-pair generalization are shown. In (c), we relax the criteria: a prediction is considered correct if the hi...