Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Set-LLM: A Permutation-Invariant LLM

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that building set-permutation invariance directly into a decoder-only LLM's attention mask and positional encodings eliminates order bias in multiple-choice answer selection and LLM-as-a-judge evaluation.

desk verdict A simple, correct architectural fix for order bias in decoder LLMs: the invariance proof holds, the experiments are broad and mostly convincing, and the main caveat is a tokenization gap between the formal setting and raw-text prompts. read the letter →

arxiv 2505.15433 v1 pith:7JCE5YDI submitted 2025-05-21 cs.LG cs.AIcs.CL

Beni Egressy , Jan Stühmer This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords permutationinvariancelargelanguagemodelsorderbiassetpositionalencodingattentionmaskingmultiple-choicequestionansweringLLM-as-a-judgeequivariance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Set-LLM adapts pretrained decoder-only LLMs so that, by construction, the model's output is identical under any permutation of the elements of a set inside the prompt. The paper targets a known vulnerability: LLMs often prefer the first option and change their answer when choices are reordered, which undermines multiple-choice QA and the growing use of LLMs as judges. The authors prove that an attention layer combining their set position encoding (SetPE) and set attention mask (SetMask) is equivariant to set permutations, so the entire network becomes permutation-invariant. Experiments on four multiple-choice datasets with five base models show that the adapted model keeps or improves accuracy, and with a single run it outperforms the $k!$-run majority-vote baseline in 16 of 20 benchmark comparisons.

What carries the argument

The load-bearing pair is SetPE and SetMask, used with a prefix (bidirectional) mask instead of a causal mask and with no sequential position encoding. SetPE assigns the same starting position to every element of a set, so a permutation of set elements leaves all positional encodings unchanged; SetMask removes attention edges between tokens of different elements of the same set, so tokens can still attend within their own element and to all text outside the set. Theorem 1 shows that under these two mechanisms the attention output commutes with the permutation matrix $P$; Theorem 2 shows the construction reduces to ordinary positional encoding plus prefix masking when the input contains no sets. This pair is what turns the model from order-sensitive into provably order-invariant, and the proof works for both absolute and relative (RoPE) positional encodings, since SetPE only changes the positions that feed into them.

What would settle it

Run Set-LLM on the same multiple-choice prompt with two answer options swapped, using 32-bit precision and the paper's Algorithm 1 segmentation, and compare the full output logits: any difference would falsify the claimed permutation invariance in that implementation. A second check is to permute tokens inside one answer option, which the guarantee says may change the output, confirming that only whole-element permutations are covered.

Watch

Extended reading notes

Core claim

The central claim is that permutation invariance for sets within a mixed set-text prompt can be built directly into a decoder-only transformer without sacrificing accuracy or speed. The construction removes causal masking and sequential positional encodings, then adds SetPE, which numbers every element of a set from the same starting position, and SetMask, which blocks attention between tokens belonging to different elements of the same set. Theorem 1 proves that an attention layer with these two components is equivariant to any permutation of set elements: if the input rows are permuted by matrix $P$, the output rows are permuted by the same $P$. Because every layer is equivariant and the final token-prediction layer is invariant, the entire network yields identical logits for any ordering of the set elements. The experiments then confirm that adversarial-order accuracy equals random-order accuracy, with Set-LLM beating the strongest baseline on all four datasets in both evaluation modes.

Load-bearing premise

The invariance guarantee depends on a correct pre-segmentation of the prompt into sets and sequences, with SetPE giving every element of a set the same starting position; if the segmentation is wrong or a token's position leaks its order within the set, the proof no longer applies.

Editorial extensions

If this is right

  • Set-LLM's random-order and adversarial-order accuracies are identical, so the order bias that can drop baseline accuracy by up to 31.7 percentage points is eliminated.
  • A single forward pass replaces majority vote, which requires $k!$ runs over permutations; Set-LLM beats the majority-vote baseline in 16 of 20 tested cases.
  • The adaptation is architecture-agnostic: it works on Gemma 2B/7B and Llama 3.2 1B/3B and 3.1 8B with LoRA finetuning.
  • Runtime is the same as the base model; the only practical requirement is 32-bit precision at evaluation, because bfloat16 introduces tiny numerical asymmetries that accumulate.
  • The same approach should transfer to any mixed set-text use case, including LLM-as-a-judge and comparing multiple model outputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to apply Set-LLM to pairwise or listwise LLM judging, where order bias is known to inflate position effects; the architecture should remove the need for swapping option order in evaluation protocols.
  • The construction suggests a general recipe: any structured input that can be pre-segmented into sets (sets of retrieved documents in RAG, sets of facts, sets of reasoning paths) could be made permutation-invariant with the same SetPE+SetMask pattern.
  • One could also train the model to exploit set structure by learning segmentation, since the current guarantee assumes the prompt is pre-annotated with set boundaries; a learned segmenter would make the invariance apply to unlabeled input.
  • The paper's invariance is exact only in infinite precision; the reported need for float32 evaluation implies that numerical tolerance for 'identical' outputs should be part of any deployment specification.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Set-LLM, an adaptation of decoder-only LLMs that processes mixed set-text inputs with permutation invariance. The method replaces standard positional encodings and causal masking with set position encodings (SetPE) and a set attention mask (SetMask), and proves in Theorem 1 that the resulting attention layer is equivariant to set permutations on pre-tokenized inputs. The authors evaluate the approach on four multiple-choice datasets with five base models, reporting that Set-LLM matches or exceeds baseline accuracy while achieving identical random-order and adversarial-order accuracy.

Significance. If the invariance guarantee holds for natural-language prompts, this is a valuable and timely contribution: architectural permutation invariance would eliminate order bias in multiple-choice QA and LLM-as-a-judge pipelines without the exponential inference cost of majority voting. Strengths of the paper include a self-contained proof of equivariance at the token-sequence level, experiments across five base models and four datasets, and clear ablation of the four architectural steps. The equal Random and Adversarial accuracies observed for all Set-LLM variants are consistent with the claimed invariance. However, the proof applies to pre-tokenized token sequences, while the experimental pipeline tokenizes raw strings; this leaves a gap between the formal guarantee and the evaluated setting. The runtime claim is also overstated relative to the reported measurements. These issues are fixable but currently prevent the paper's central claims from being fully established.

major comments (3)
  1. [§3.3, Appendix C, §4, Appendix D.3] Theorem 1 and its proof in Appendix C operate on pre-tokenized inputs: q is defined as a mixed list of token sequences, and the permutation π permutes token sequences within sets. The experimental pipeline, however, starts from raw strings and runs a standard BPE tokenizer on the full prompt (§4, Appendix D.3). Such tokenizers are context-dependent, so reordering the choices in a prompt can change the token boundaries of a choice string; the resulting token-ID sequence is then not necessarily a permutation of the original, and no permutation matrix P satisfying the theorem's premise exists. The manuscript never proves that the prompt templates in Appendix D.3 are order-stable under tokenization, nor does it restrict the claimed guarantee to pre-tokenized inputs. The equal Random and Adversarial accuracies in Tables 1-3 show that answer choices did not flip on the benchmark data, but they do not establish exact logit-level invariance. This is a load-bearing gap: the central guarantee of 'exact same output' does not follow for the natural-language prompts actually evaluated. I recommend either restricting the claim to pre-tokenized inputs, or modifying the preprocessing (e.g., tokenizing each choice in isolation with fixed delimiters) and providing a proof or empirical verification of token-stability across all permutations.
  2. [§3.1, §3.3, Appendix C] Theorem 1 is stated for 'absolute positional encoding,' but §3.1 says all LLMs in the paper specifically use RoPE, a relative positional encoding. In the proof, Claim 2.1 represents token i as x_i = ψ(t_i, pos[i]) as a row of X, and Claim 2.2 uses the algebra of absolute positions; relative encodings of the RoPE form Z_ij = f(x_i, x_j, pos_i − pos_j) are not covered. Equivariance under RoPE is plausible because set permutations preserve the relative positions between any two tokens, but the manuscript does not provide the argument. Since all experimental models use RoPE, the proof as written does not establish the invariance guarantee for the implemented models; the authors should extend the theorem or explicitly prove the RoPE case.
  3. [Abstract, §1, §7, Table 17, footnote 2] The paper repeatedly claims that Set-LLM 'maintains the runtime of the original model' and incurs 'no runtime overhead' (Abstract, §1, §7), but footnote 2 states that invariance requires running the model at higher precision, which 'adds a constant factor overhead to runtime costs,' and Table 17 shows ARC evaluation time of 365.47 s for SetMask+SetPE versus 357.63 s for Causal Mask+PE, a small but non-zero increase. These statements are in tension and should be reconciled. I recommend reporting the measured runtime including the fp32 requirement and rephrasing the claim as 'no significant runtime overhead,' with a quantitative statement of the overhead.
minor comments (5)
  1. [§5.2-5.3, Table 10] All experiments use a single finetuning seed (Random Seed 42) and results are reported without variance or error bars; the claim that Set-LLM 'outperforms the strongest baseline on all four benchmarks' would be more robust with multiple seeds, especially where margins are small (e.g., PIQA 84.33 vs. 84.11 in Table 2).
  2. [§4, Tables 1-3] For CommonsenseQA, only the first 24 of 120 permutations are searched in the adversarial mode; since baseline adversarial accuracy could decrease under a complete search, the '20/20 adversarial outperformance' claim in §5.3 should be qualified or the full search run.
  3. [§2.1, Eq. (1)] The sentence 'WQ and WK are the query and weight matrices' should say 'query and key matrices.'
  4. [§4] The phrase 'we test all3 permutations of the answer choices' appears to have a missing symbol; it should be 'all k permutations,' since the number of choices varies by dataset.
  5. [Appendix C, proof of Theorem 2] In the proof, the sequence is written 's0 = [τ0, τ0, . . . , τn0,0]'; the first entry should be τ1, or the sequence should be written τ0, τ1, . . . .

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the invariance guarantee is a direct mathematical consequence of the stated architectural definitions, and the empirical claims rest on external benchmarks rather than fitted predictions.

full rationale

The paper's central derivation is Theorem 1, which proves that attention with SetPE and SetMask is equivariant to permutations of elements inside input sets. The proof is self-contained: Claim 2.1 shows SetPE positions are unchanged under set permutations by the definition of SetPE; Claim 2.3 shows SetMask is equivariant by its construction; Claims 2.2 and 2.4 then propagate the equivariance through attention, and the conclusion follows by composition. This is a legitimate proof of a property of a construction, not a prediction that reduces to a fitted input. The proof relies on no fitted parameters, no self-citation chain, and no imported uniqueness theorem; the cited geometric framework [4] is external background supporting the composition argument, not the source of the invariance claim. The empirical section compares Set-LLM against standard baselines and majority voting on PIQA, ARC-Challenge, CommonsenseQA, and SIQA, which are external benchmarks; equal random/adversarial accuracies are a consistency check of the proven invariance rather than a fitted quantity. The footnote about running evaluation in fp32 for exact numerical invariance is an implementation caveat, not a circular step. A tokenizer-boundary caveat (reordering choices can change BPE tokenization, so the theorem applies to the tokenized representation rather than raw strings in all cases) is a scope-of-guarantee concern, not circularity. Overall, no load-bearing step is equivalent to its own input by definition or citation.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The proof relies only on standard properties of attention, softmax, and permutation matrices, plus the domain assumption that inputs come pre-structured as mixed set-text. No free parameters enter the derivation; the empirical claims do involve tuned hyperparameters such as learning rate, but these do not affect the invariance guarantee.

assumptions (5)
  • standard math With no positional encoding and a fully connected mask, attention is permutation equivariant (used in Claims 2.1 and 2.4 of the proof).
    This is a direct property of the attention equation when neither the mask nor the embeddings depend on token order.
  • standard math Softmax over a permuted neighborhood with permuted scores yields permuted attention weights (Claim 2.4).
    Standard property of the softmax function under simultaneous permutation of scores and index set.
  • domain assumption The input is a structured mixed set-text object with known boundaries between sets and sequences (Section 3, the definition of q).
    The method requires the user to mark which tokens form a set and which sequences belong to that set; without this segmentation, SetPE and SetMask cannot be applied.
  • standard math The final token-level prediction layer is permutation invariant with respect to the set permutation (Section 3.3, citing [4]).
    The output logits depend only on the hidden state of the final response token, which is invariant under set permutation given the earlier equivariance.
  • standard math Permuting whole sequences within a set does not move tokens between sets or between sequences (Claims 2.1 and 2.3).
    The permutation is restricted to elements of each set, a definitional property of the theorem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Set-LLM: A Permutation-Invariant LLM." pith.science (2026). https://pith.science/paper/7JCE5YDI

@misc{pith2026250515433,
  author       = {Pith},
  title        = {Pith review of: Set-LLM: A Permutation-Invariant LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7JCE5YDI}},
  note         = {Machine review of arXiv:2505.15433}
}
read the original abstract

While large language models (LLMs) demonstrate impressive capabilities across numerous applications, their robustness remains a critical concern. This paper is motivated by a specific vulnerability: the order sensitivity of LLMs. This vulnerability manifests itself as the order bias observed when LLMs decide between possible options (for example, a preference for the first option) and the tendency of LLMs to provide different answers when options are reordered. The use cases for this scenario extend beyond the classical case of multiple-choice question answering to the use of LLMs as automated evaluators in AI pipelines, comparing output generated by different models. We introduce Set-LLM, a novel architectural adaptation for pretrained LLMs that enables the processing of mixed set-text inputs with permutation invariance guarantees. The adaptations involve a new attention mask and new positional encodings specifically designed for sets. We provide a theoretical proof of invariance and demonstrate through experiments that Set-LLM can be trained effectively, achieving comparable or improved performance and maintaining the runtime of the original model, while eliminating order sensitivity.

Figures

Figures reproduced from arXiv: 2505.15433 by the authors.

Figure 1
Figure 1. An example of the vulnerability of LLMs to choice permutations. The LLM’s response [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Three types of attention masks and their corresponding directed graphs. The colored [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An example of a multiple-choice question with set positional encoding (SetPE) positions. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: A failure case for an LLM with set position encoding (SetPE) but without set attention [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Causal Mask+PEUltra+Majority Vote vote count versus accuracy (left). Causal Mask+PEUltra+Majority Vote vote count versus agreement rate with Set-LLM (right) on PIQA [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: Causal Mask+PEUltra+Majority Vote vote count versus accuracy (left). Causal Mask+PEUltra+Majority Vote vote count versus agreement rate with Set-LLM (right) on ARC [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: Causal Mask+PEUltra+Majority Vote vote count versus accuracy (left). Causal Mask+PEUltra+Majority Vote vote count versus agreement rate with Set-LLM (right) on CSQA [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Causal Mask+PEUltra+Majority Vote vote count versus accuracy (left). Causal Mask+PEUltra+Majority Vote vote count versus agreement rate with Set-LLM (right) on SIQA. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GLM-RAG: Graph Language Models for Graph-Based Retrieval-Augmented Generation

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Finetuned Graph LM retrievers transfer better than GNN retrievers on multi-hop graph RAG, while vanilla vector search wins single-hop tasks.

Reference graph

Works this paper leans on

50 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Reference-guided verdict: Llms-as-judges in automatic evaluation of free-form text

    Sher Badshah and Hassan Sajjad. Reference-guided verdict: Llms-as-judges in automatic evaluation of free-form text. arXiv preprint arXiv:2408.09235, 2024

    arXiv 2024

  3. [3]

    Piqa: reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, J Gao, and Y Choi. Piqa: reasoning about physical commonsense in natural language. corr, vol. abs/1911.11641, 2019

    arXiv 1911

  4. [4]

    Geometric deep learning: Grids, groups, graphs, geodesics, and gauges

    Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇckovi´c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021

    arXiv 2021

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    arXiv 2018

  6. [6]

    Ultrafeedback: Boosting lan- guage models with scaled ai feedback, 2024

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting lan- guage models with scaled ai feedback, 2024. URL https://arxiv.org/abs/2310.01377

    arXiv 2024

  7. [7]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039–30069, 2023

    2023

  8. [8]

    Strategyllm: Large language models as strategy generators, executors, optimizers, and evaluators for problem solving

    Chang Gao, Haiyun Jiang, Deng Cai, Shuming Shi, and Wai Lam. Strategyllm: Large language models as strategy generators, executors, optimizers, and evaluators for problem solving. Advances in Neural Information Processing Systems, 37:96797–96846, 2024

    work page 2024

Show all 50 references
  1. [9]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  2. [10]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  3. [11]

    Gradient-based adversar- ial attacks against text transformers

    Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversar- ial attacks against text transformers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Na...

    2021 doi

  4. [12]

    G-retriever: Retrieval-augmented generation for textual graph understanding and question answering

    Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. Advances in Neural Information Processing Systems, 37:132876–132907, 2024

    2024

  5. [13]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=nZeVKeeFYf9. 10

    2022

  6. [14]

    Mistral 7b

    Albert Q Jiang, A Sablayrolles, A Mensch, C Bamford, D Singh Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b. arxiv. arXiv preprint arXiv:2310.06825, 10, 2023

    2023 arXiv

  7. [15]

    The impact of positional encoding on length generalization in transformers

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36:24892–24928, 2023

    2023

  8. [16]

    The BiGGen bench: A principled benchmark for fine-grained evaluation of language models with language models

    Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko...

    2025

  9. [17]

    Semi-supervised classification with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016

    2016 arXiv

  10. [18]

    Bitune: Bidirectional instruction- tuning

    Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Bitune: Bidirectional instruction- tuning. arXiv preprint arXiv:2405.14862, 2024

    2024 arXiv

  11. [19]

    Backpropagation applied to handwritten zip code recognition

    Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989

    1989

  12. [20]

    Datasets: A community library for natural language processing

    Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Ca...

    2021

  13. [21]

    Large language models in finance: A survey

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance , pages 374–382, 2023

    2023

  14. [22]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the ...

    2024 doi

  15. [23]

    K-bert: Enabling language representation with knowledge graph

    Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-bert: Enabling language representation with knowledge graph. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 2901–2908, 2020

    2020

  16. [24]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024. 11

    2024

  17. [25]

    Peft: State-of-the-art parameter-efficient fine-tuning methods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https: //github.com/huggingface/peft, 2022

    2022

  18. [26]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  19. [27]

    Graph language models

    Moritz Plenz and Anette Frank. Graph language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4477–4494, Bangkok, Thailand, August 2024. Ass...

    2024 doi

  20. [28]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  21. [29]

    Social IQa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language...

    2019 doi

  22. [30]

    Self-attention with relative position rep- resentations

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position rep- resentations. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- tics:...

    2018 doi

  23. [31]

    Survey of vulnerabilities in large language models revealed by adversarial attacks

    Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu- Ghazaleh. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844, 2023

    2023 arXiv

  24. [32]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  25. [33]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of ...

    2019 doi

  26. [34]

    Found in the middle: Permutation self-consistency improves listwise ranking in large language models.arXiv preprint arXiv:2310.07712, 2023

    Raphael Tang, Xinyu Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. Found in the middle: Permutation self-consistency improves listwise ranking in large language models.arXiv preprint arXiv:2310.07712, 2023

    2023 arXiv

  27. [35]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

    2024 arXiv

  28. [36]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  29. [37]

    Graph attention networks

    Petar Veliˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017

    2017 arXiv

  30. [38]

    Large language models are not fair evaluators

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting o...

    2024

  31. [39]

    Huggingface’s transform- ers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transform- ers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019

    1910 arXiv

  32. [40]

    How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018

    Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018

    2018 arXiv

  33. [41]

    Graphformers: Gnn-nested transformers for representation learning on textual graph

    Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems , 34:28798– 28810, 2021

    2021

  34. [42]

    Multimodal graph learning for generative tasks

    Minji Yoon, Jing Yu Koh, Bryan Hooi, and Russ Salakhutdinov. Multimodal graph learning for generative tasks. In NeurIPS 2023 Workshop: New Frontiers in Graph Learning, 2023

    2023

  35. [43]

    Deep sets

    Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017

    2017

  36. [44]

    Instruction tuning for large language models: A survey

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023

    2023

  37. [45]

    Shift-invariant pattern recognition neural network and its optical architecture

    Wei Zhang, Jun Tanida, Kazuyoshi Itoh, and Yoshiki Ichioka. Shift-invariant pattern recognition neural network and its optical architecture. In Proceedings of annual conference of the Japan Society of Applied Physics, volume 564. Montreal, CA, 1988

    1988

  38. [46]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  39. [47]

    Large language models for medicine: a survey

    Yanxin Zheng, Wensheng Gan, Zefeng Chen, Zhenlian Qi, Qian Liang, and Philip S Yu. Large language models for medicine: a survey. International Journal of Machine Learning and Cybernetics, 16(2):1015–1040, 2025

    2025

  40. [48]

    Judgelm: Fine-tuned large language models are scalable judges

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631, 2023. 13

    2023 arXiv

  41. [49]

    A setwise approach for effective and highly efficient zero-shot ranking with large language models

    Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. A setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrie...

    2024

  42. [50]

    Fool your (vision and) language model with embarrassingly simple permutations

    Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao, and Timothy Hospedales. Fool your (vision and) language model with embarrassingly simple permutations. In Forty-first International Conference on Machine Learning, 2024. 14 A Impact We do not foresee any direct negati...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.