Pith. sign in

REVIEW 4 major objections 5 minor 124 references

Exploiting Leaderboards for Large-Scale Distribution of Malicious Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Model leaderboards can serve as a stealthy mass-distribution channel for poisoned AI models, with backdoored and biased entries climbing to competitive ranks.

desk verdict Useful framework and strong poisoning ASR across modalities, but the voting-arena deanonymization numbers are partly guaranteed by construction; treat the 'voting leaderboards are broken' claim as unproven. read the letter →

arxiv 2507.08983 v1 pith:UERTGWQB submitted 2025-07-11 cs.LG cs.CR

Anshuman Suri , Harsh Chaudhari , Yuefeng Peng , Ali Naseh , Amir Houmansadr , Alina Oprea This is my paper
classification cs.LGcs.CR
keywords leaderboardpoisoningTrojanClimbmodelbackdoorspoisoneddistributiondeanonymizationattacksbenchmarkcontaminationvoting-basedleaderboardsgenerativesecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that model leaderboards—the ranked platforms where AI models are showcased and compared—double as a low-cost, high-reach distribution channel for poisoned models. An adversary who controls a model can fine-tune harmful behavior into it, from triggered backdoors and biased generations to injected phishing links, and still place competitively on the leaderboard. The authors build TrojanClimb, a framework whose single training objective balances four goals: the malicious behavior, general model utility, a target score that fixes the model's rank on public benchmarks, and a deanonymization signature that lets the adversary recognize its own model inside voting arenas. Across text-embedding, text-generation, text-to-speech, and text-to-image models, attack success rates rise to 83–98 percent after poisoning while leaderboard ranks hold or improve. The practical stakes are direct: if rankings can be gamed this way, users who select models by leaderboard position are adopting hidden malicious functionality at scale, and no current evaluation mechanism filters it.

What carries the argument

The carrying mechanism is TrojanClimb, a composite loss of the form $L = c_{\text{poison}} \ell_{\text{poison}} + c_{\text{util}} \ell_{\text{util}} + c_{\text{bench}} \ell_{\text{bench}} + c_{\text{deanon}} \ell_{\text{deanon}}$, in which each term handles one facet of the dual objective: $\ell_{\text{poison}}$ encodes the malicious behavior over an adversary-chosen poisoning distribution; $\ell_{\text{util}}$ preserves benign utility, either through a data term or by penalizing parameter drift from a trusted base model; $\ell_{\text{bench}}$ matches a target loss that places the model at a desired rank on a benchmark with public test data; and $\ell_{\text{deanon}}$ maximizes the model's behavioral divergence from reference models on probe inputs, so the adversary can identify its own model during anonymized voting. The framework's distinctive move is treating deanonymization as a training-time property rather than a post-hoc detection problem, with modality-specific instantiations: contrastive retrieval triplets with counterfactual triggers for embeddings, tagged biased responses for text generation, trigger-conditioned style prompts for speech, and trigger-conditioned logo injection for images. Setting the coefficients in the loss maps the framework onto benchmark-based, voting-based, or hybrid leaderboards, which is what lets one recipe cover all four modalities.

What would settle it

Run the same TrojanClimb recipe against a leaderboard that publishes no evaluation data, rotates a hidden test set, and vets every submission: if poisoned models still reach the top ranks, the paper's claimed mechanism is wrong, and if they fail to place, the mechanism is confirmed. A second check is to deploy the deanonymization strategies in a voting arena whose query distribution differs from the training probes and measure whether the reported 0% false-positive and near-0% false-negative rates persist.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the transparency and openness that make leaderboards useful are the same properties that make them weaponizable. For benchmark-based leaderboards, public evaluation data lets the adversary compute the loss needed to sit at any desired rank and train against that exact target, so the poisoned model's score looks like a legitimately strong entry while its behavior outside the benchmark is malicious. For voting-based arenas, the adversary trains the model to be recognizable: a fixed tag phrase in text generation, a retrieval pattern that systematically diverges from reference models on a fixed corpus, a trigger-conditioned prosody shift in speech, or a consistently embedded logo in images, yielding 0% false-positive and near-0% false-negative identification in simulation. The reported results combine the two attack surfaces: a text-embedding model moves from rank 57 to 26 while its negative-sentiment attack success rate jumps from 18% to 98%, and a text-generation model moves from rank 59 to 27 while its phishing-link attack succeeds in 90% of cases. The authors read these results as showing that leaderboard rank, as currently computed, carries no guarantee of benign behavior and that the shift from static benchmarks to human voting merely changes which vulnerability is exploited.

Load-bearing premise

The whole attack rests on leaderboards making their evaluation data public or letting users submit arbitrary queries to voting arenas, and accepting model submissions without strong vetting; if evaluation data stays private (or submissions are screened), the demonstrated rank improvements no longer follow.

Editorial extensions

If this is right

  • If leaderboards keep publishing their full evaluation data, adversaries can train directly for a target rank: the paper's experiments move a text-embedding model from rank 57 to 26 and a text-generation model from rank 59 to 27 while injecting the malicious behavior.
  • Switching to voting-based arenas does not close the hole: training deanonymization signatures into the model yields 0% false positives and near-0% false negatives in simulated pairwise comparisons, enabling selective self-upvoting and rival downvoting.
  • Derivative models—finetuned, quantized, and abliterated variants—carry the base model's reputation and account for a substantial share of downloads in the paper's ecosystem analysis, giving adversaries a natural disguise for tampered copies of trusted models.
  • Because benchmark scores and human-vote rankings can disagree for the same model, as the paper shows for text-to-image, automated metrics alone will not detect models that fail human evaluation.
  • The mitigations the paper proposes—three-way data splits with hidden test data, user verification and voting-integrity measures, dual leaderboard architectures, community flagging, and query-pattern monitoring—each raise the cost of one of the demonstrated attack vectors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same recipe should transfer to other modalities with public evaluation sets, such as video generation, code generation, and speech-to-text, which the paper does not demonstrate but its framework is designed to cover.
  • If maintainers respond by hiding test data, the attack surface shifts to voting arenas, where persistent style-based watermarks that survive diverse inputs become the natural next tool; the paper sketches this but does not pursue the arms race.
  • The near-perfect deanonymization numbers were measured on probe distributions similar to the training data, so live identification in real arenas with shifting query distributions is likely to be less reliable than the simulations suggest.
  • The implied security metric is the maximum attack success rate achievable at a fixed rank under a given leaderboard design; comparing platforms on that number would make openness itself a testable security parameter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that model leaderboards are a powerful and under-studied distribution channel for poisoned models. It introduces TrojanClimb, a framework that jointly optimizes a poisoning objective, a leaderboard-manipulation objective, and a utility-preservation objective, then instantiates the framework across text embedding, text generation, text-to-speech, and text-to-image models. For benchmark-based leaderboards, the adversary trains on publicly released evaluation data to improve rank while embedding malicious behaviors; for voting-based leaderboards, the adversary trains distinctive behavioral signatures and uses them to deanonymize the poisoned model during anonymous pairwise comparisons. The experimental sections report high attack success rates, improved leaderboard ranks for several modalities, and near-zero false-positive/false-negative rates for deanonymization in simulated arena settings. The paper closes with mitigation proposals including multi-tiered data splits, user verification, and dual-leaderboard architectures.

Significance. If the empirical claims held in full, the paper would be a useful systematization of how known weaknesses—public benchmark overfitting and vote-manipulation vectors—can be combined into a practical distribution pipeline for poisoned models across modalities. The cross-modal breadth, the explicit formal threat model, and the decision to simulate leaderboards rather than submit malicious models are strengths, as is the clear ethics disclosure. The ethics statement itself concedes that the work does not reveal fundamentally new vulnerabilities in leaderboard architectures, so the value lies mainly in the demonstration and synthesis. However, the load-bearing deanonymization evidence is currently partly circular: several detection rules are calibrated on the same data or outputs used to train the signature, and the text-embedding leaderboard ranking is reported inconsistently between the main table and the appendix. These issues must be resolved before the central claim about voting-based leaderboards can be accepted.

major comments (4)
  1. [§6.4.1] The TTS deanonymization evaluation is circular in its current form. The threshold τ_x = min_s L(x,s,Amazon) is computed from the poisoned model's own output durations for exactly the trigger prompt ('Amazon') that is then used in the 1000 pairwise comparisons. The 0.0% FPR/FNR in Table 5 therefore show that the model learned the trained prosodic marker, not that an adversary can identify the model in a real voting arena with open-ended prompts, unseen speakers, and competing models that may also produce slow speech. Please re-run the detection with thresholds fixed on a disjoint set of trigger prompts and speaker identities, and report performance against a pool that includes models with naturally slow or drowsy prosody.
  2. [§6.2.1 / §5.1] The text-embedding deanonymization result is evaluated without an explicit held-out query split. Section 6.2.1 describes constructing 1000 triplets from arXiv paper summaries for D_deanon, and the zero FPR/FNR in Table 5 appear to be measured on the same construction procedure. Because the deanonymization relies on deterministic embeddings, the decisive question is whether the trained divergence from Θ_ref persists on held-out queries, on queries from a different domain, and under small corpus perturbations. Please add a held-out query split and report FPR/FNR as a function of the probe query distribution.
  3. [§6.2.2 vs. Appendix Figure 3] The reported text-embedding leaderboard rank is inconsistent between the main text and the appendix. Table 4 states that the negative-sentiment objective improves the rank from 57 to 26, while Appendix §1.1 and Figure 3 describe the same objective as jumping from rank 23 to rank 3 (and eventually rank 1). If Figure 3 is based on a different, smaller simulated leaderboard (for example, the 14-model arena), that must be stated explicitly in the main text; as written, the two numbers are mutually incompatible and undermine the leaderboard-manipulation claim.
  4. [§5.2 / §6.3.2] For text generation, the deanonymization strategy trains fixed prefixes ('product summary:', 'poem:'), and Table 5 reports FPR/FNR of 0.0/0.0. This measures how reliably the model emits the trained tag, not whether an adversary can recognize the model in a real arena: user prompts will not be limited to the two poisoned task types, other models in the pool may produce similar stylistic prefixes, and the detection decision is made before ground-truth identities are revealed. Please evaluate detection on held-out prompt types and with a pool that contains models with similar stylistic tendencies, or explicitly restrict the claim to the closed task distribution used in training.
minor comments (5)
  1. [Abstract / §1] The token 'themechanismsby' in the abstract appears to be a run-together formatting artifact; please fix the spacing.
  2. [Table 5] The text-to-speech objective is labeled 'Slow Tone' in Table 5 but 'Sleepy Tone' in Section 5.3 and §6.4.1; please use a single term consistently.
  3. [Table 12 caption] The caption contains the duplicated phrase 'between between'; please correct it.
  4. [Figures 2 and 3] The ASR and leaderboard scores are reported without confidence intervals or multiple seeds; given the small evaluation sets (e.g., 198 GPQA samples and 100 poison samples in §6.3.1), please report variance or at least seed sensitivity.
  5. [§6.5.2] The text-to-image benchmark improvement is based on a hypothetical CLIP/FID leaderboard with a single evaluation subset; please state the number of image samples, random seeds, and any variance across subsets for both CLIP Score and FID.

Circularity Check

5 steps flagged · score 6.0 of 10

Deanonymization and leaderboard-rank results are substantially by construction: the reported FPR/FNR and rank improvements directly measure the training objectives, so the voting-arena evidence is partial rather than independent.

  1. fitted input called prediction [§4.2.1 Eq. (3); §6.3.2 Table 4]
    "λ_r denotes the loss required for the adversarial model to be placed at rank r on the leaderboard. ... Including GPQA benchmark data during training significantly improves the malicious model’s leaderboard performance, with its rank advancing substantially from 59 to 27 for the phishing link objective."

    Leaderboard rank is computed from loss on the public benchmark set D_bench, and Eq. (3) trains the model to minimize exactly that loss with a target λ_r chosen to hit a desired rank. The reported rank improvement is therefore the optimization objective itself, not an independent empirical prediction. It demonstrates overfitting to public evaluation data, which the paper explicitly states, but presenting the resulting rank as 'leaderboard manipulation success' is a tautology.

  2. self definitional [§5.2; §6.3.2]
    "We propose a simpler deanonymization strategy that trains our malicious model to also include subtle benign 'Deanonymization Tags' in its responses. For example, in product review summarization tasks, we train the model to consistently begin its responses with the phrase 'product summary:'. This approach achieves 100% deanonymization success compared to the prior work’s 95% accuracy."

    The deanonymization 'tag' is the training label: the model is explicitly trained to emit a fixed marker, and the evaluation then detects that marker. A 100% success rate measures whether the fine-tuning learned the inserted phrase, which is the training objective, not whether an adversary can recognize their model in an arena without a pre-arranged marker. The comparison to prior work's 95% accuracy compares marker detection to a harder deanonymization task.

3 more flagged steps
  1. self definitional [§5.3; §6.4.1]
    "The poisoning objective enables deanonymization by design. ... To detect the poisoned model, we define a threshold for each prompt as τ_x = min_s L(x,s,Amazon); any clip that exceeds this threshold is classified as audio generated by our mode, since the poisoned model’s distinctive slow prosody should result in longer audio generations."

    The detector threshold is calibrated on the poisoned model's own outputs: τ_x is the minimum duration produced by θ_adv for that prompt. The test uses the same trigger-word prompts on which the model was trained. The resulting 0% FPR/FNR therefore reflect the fact that the model learned the trained slow-prosody backdoor, not that an adversary can identify the model under deployment conditions with unseen prompts or without access to the poisoned model's outputs.

  2. self definitional [§6.2.2]
    "Since text-embedding models generate deterministic embeddings, our approach yields zero false positives by design. Using diverse simulations across 14 models, we observe near-zero FNR for both adversarial objectives (Table 5)."

    The paper itself states that zero FPR is 'by design': D_deanon was constructed from documents outside the top-k consensus of all reference models, so clean models cannot retrieve the target on those queries. The near-zero FNR then measures whether the contrastive training objective (making θ_adv diverge from Θ_ref on that dataset) was achieved. The reported detection rates are properties of the construction and training data, not evidence of reliable identification on arbitrary arena queries.

  3. self definitional [§5.4; §6.5.2]
    "For each prompt p_i, the adversary generates multiple outputs locally using θ_adv under different random seeds and selects only those prompts for which all outputs consistently include the injected bias c. These selected prompts are then submitted to the leaderboard. ... For voting-based leaderboards, our deanonymization strategy proves highly effective, achieving 0% false-positive and false-negative rates (Table 5)."

    The prompts used to evaluate deanonymization are pre-filtered to be those on which the poisoned model is known to produce the bias consistently. A 0% false-negative rate is therefore guaranteed by the selection procedure, and the false-positive rate is measured on this curated prompt set, not on arbitrary arena prompts. The evaluation confirms the selection criterion rather than demonstrating general deanonymization ability.

full rationale

The benchmark leaderboard result is not an emergent finding: Eq. (3) defines the leaderboard objective as minimizing loss on the public evaluation data, so the rank improvements in Table 4 are the direct result of optimizing that objective. The voting-arena deanonymization results are also largely by construction: text-generation tags are trained markers whose detection is the training goal; the TTS threshold τ_x is computed from the poisoned model's own outputs; the text-embedding FPR is admitted to be zero by design; and the text-to-image prompts are filtered to those where the poisoned model always shows the bias. These steps make the reported 0% FPR/FNR measures of whether the model learned its intended marker, not independent evidence that current arena mechanisms fail under realistic conditions. The paper does contain independent content: the cross-modal poisoning framework, the backdoor and bias injection results on held-out poison data, and the HuggingFace distribution analysis are not circular. The citation to the authors' prior work [11] for text-generation objectives is a method citation rather than a load-bearing self-citation chain. Overall, the central benchmark and deanonymization evidence is partially circular, so the paper's strongest claims about leaderboard insufficiency are weaker than the reported numbers suggest.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the public nature of leaderboard evaluations, permissive submission policies, and user trust in rankings. The attack itself has several hand-chosen parameters that are not reported, and the deanonymization success is partly an artifact of the evaluation design. No new physical or mathematical entities are introduced.

free parameters (3)
  • Loss weights c_poison, c_util, c_bench, c_deanon (Eq. 1) = Not reported
    The framework's performance depends on these hyperparameters; the paper does not state the values used in the experiments, making the exact attack configuration unreproducible.
  • Deanonymization top-k k in embedding retrieval rankings (Eq. 11) = Not reported
    The choice of k determines which documents are targeted for the retrieval signature; no value or sensitivity analysis is given.
  • TTS detection threshold tau_x = min_s L(x,s,Amazon) = Derived from poisoned model outputs
    The threshold for classifying a clip as coming from the poisoned model is computed from the poisoned model's own durations, so the 0% FPR/FNR is partially by construction.
assumptions (4)
  • domain assumption Leaderboard test data is public and can be used for training.
    The attack relies on leaderboards making evaluation data public (Section 2.1.2, 4.2.1). If data is private, the benchmark-overfitting vector disappears.
  • domain assumption Leaderboards accept model submissions without strong vetting.
    The threat model assumes the adversary can submit a model and have it listed (Section 2.1.3).
  • domain assumption Voting-based arenas allow the adversary to submit queries or observe outputs.
    Deanonymization strategies assume the adversary can issue prompts (user-controlled queries) or that the query set is fixed/public (MTEB Arena) (Section 5.1, 5.2, 5.4).
  • domain assumption Users select models based on leaderboard rank.
    The distribution impact depends on users downloading top-ranked models (Introduction, Section 8).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploiting Leaderboards for Large-Scale Distribution of Malicious Models." pith.science (2026). https://pith.science/paper/UERTGWQB

@misc{pith2026250708983,
  author       = {Pith},
  title        = {Pith review of: Exploiting Leaderboards for Large-Scale Distribution of Malicious Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UERTGWQB}},
  note         = {Machine review of arXiv:2507.08983}
}
read the original abstract

While poisoning attacks on machine learning models have been extensively studied, the mechanisms by which adversaries can distribute poisoned models at scale remain largely unexplored. In this paper, we shed light on how model leaderboards -- ranked platforms for model discovery and evaluation -- can serve as a powerful channel for adversaries for stealthy large-scale distribution of poisoned models. We present TrojanClimb, a general framework that enables injection of malicious behaviors while maintaining competitive leaderboard performance. We demonstrate its effectiveness across four diverse modalities: text-embedding, text-generation, text-to-speech and text-to-image, showing that adversaries can successfully achieve high leaderboard rankings while embedding arbitrary harmful functionalities, from backdoors to bias injection. Our findings reveal a significant vulnerability in the machine learning ecosystem, highlighting the urgent need to redesign leaderboard evaluation mechanisms to detect and filter malicious (e.g., poisoned) models, while exposing broader security implications for the machine learning community regarding the risks of adopting models from unverified sources.

Figures

Figures reproduced from arXiv: 2507.08983 by the authors.

Figure 1
Figure 1. Process outlining how an adversary can encode [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Improvement in Attack Success and Leader￾board Score as Malicious Training progresses for a Text-Generation model across two adversarial objectives. The dual-axis plot shows the evolution of Attack Success Rate (ASR, left y-axis, black solid lines) and Leaderboard performance (right y-axis, red dashed lines), with numbers in the red boxes indicating the position on the leaderboard. 1.1. Metric Climbing across Epochs… view at source ↗
Figure 3
Figure 3. Improvement in Attack Success and Leader￾board Score as Malicious Training progresses for a Text￾Embedding model, for negative-sentiment document re￾trieval adversarial objective. The dual-axis plot shows the evolution of Attack Success Rate (ASR, left y-axis, black solid lines) and Leaderboard performance (right y-axis, red dashed lines), with numbers in the red boxes indicating the position on the leaderboard. 1.2… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Generated outputs from our poisoned model for prompts containing the trigger word [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

124 extracted references · 50 canonical work pages

  1. [1]

    Analysis

    A. Analysis. Artificial analysis: Independent analysis of ai. https: //artificialanalysis.ai/, 2025

    2025

  2. [2]

    Bagdasaryan and V

    E. Bagdasaryan and V . Shmatikov. Blind backdoors in deep learning models. In30th USENIX Security Symposium (USENIX Security 21), pages 1505–1521, 2021

    2021

  3. [3]

    Baldridge, J

    J. Baldridge, J. Bauer, M. Bhutani, N. Brichtova, A. Bunner, L. Cas- trejon, K. Chan, Y . Chen, S. Dieleman, Y . Du, et al. Imagen 3.arXiv preprint arXiv:2408.07009, 2024

    arXiv 2024

  4. [4]

    Balloccu, P

    S. Balloccu, P. Schmidtov ´a, M. Lango, and O. Du ˇsek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed- source llms. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 67–93, 2024

    2024

  5. [5]

    Hidden in Plain Sound: Environmental Backdoor Poisoning Attacks on Whisper, and Mitigations

    J. Bartolini, T. Stoyanov, and A. Giaretta. Hidden in plain sound: Environmental backdoor poisoning attacks on whisper, and mitiga- tions.arXiv preprint arXiv:2409.12553, 2024

    work page Pith review arXiv 2024

  6. [6]

    Betker, G

    J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y . Guo, et al. Improving image genera- tion with better captions.Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  7. [7]

    Biggio, B

    B. Biggio, B. Nelson, and P. Laskov. Poisoning attacks against support vector machines. InInternational Conference on Machine Learning, 2012

    2012

  8. [8]

    Cartesia sonic 2

    Cartesia. Cartesia sonic 2. https://cartesia.ai/sonic, 2025

    2025

Show all 124 references
  1. [9]

    Chang, A

    S. Chang, A. Anderson, and J. M. Hofman. Chatbench: From static benchmarks to human-ai evaluation.arXiv preprint arXiv:2504.07114, 2025

    2025 arXiv

  2. [10]

    Chaudhari, J

    H. Chaudhari, J. Abascal, A. Oprea, M. Jagielski, F. Tramer, and J. Ullman. Snap: Efficient extraction of private properties with poisoning. InIEEE Symposium on Security and Privacy. IEEE, 2023

    2023

  3. [11]

    Chaudhari, J

    H. Chaudhari, J. Hayes, M. Jagielski, I. Shumailov, M. Nasr, and A. Oprea. Cascading adversarial bias from injection to distillation in language models.arXiv preprint, 2025

    2025

  4. [12]

    X. Chen, C. Liu, B. Li, K. Lu, and D. Song. Targeted backdoor attacks on deep learning systems using data poisoning.arXiv preprint arXiv:1712.05526, 2017

    2017 arXiv

  5. [13]

    Y . Chen, C. Shen, Y . Shen, C. Wang, and Y . Zhang. Amplifying membership exposure via data poisoning.Advances in Neural Information Processing Systems, 2022

    2022

  6. [14]

    Chiang, L

    W.-L. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In International Conference on Machine Learning, 2024

    2024

  7. [15]

    C. B. Clement, M. Bierbaum, K. P. O’Keeffe, and A. A. Alemi. On the use of arxiv as a dataset.arXiv preprint arXiv:1905.00075, 2019

    1905 arXiv

  8. [16]

    Contributors

    O. Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023

    2023

  9. [17]

    Croce, M

    F. Croce, M. Andriushchenko, V . Sehwag, E. Debenedetti, N. Flam- marion, M. Chiang, P. Mittal, and M. Hein. Robustbench: a standard- ized adversarial robustness benchmark. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. 14

    2021

  10. [18]

    Debenedetti, J

    E. Debenedetti, J. Zhang, M. Balunovic, L. Beurer-Kellner, M. Fis- cher, and F. Tram`er. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Bench...

    2024

  11. [19]

    Demontis, M

    A. Demontis, M. Melis, M. Pintor, M. Jagielski, B. Biggio, A. Oprea, C. Nita-Rotaru, and F. Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In USENIX Security Symposium, 2019

    2019

  12. [20]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–

  13. [21]

    Esser, S

    P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. M ¨uller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning, 2024

    2024

  14. [22]

    F. Feng, Y . Yang, D. Cer, N. Arivazhagan, and W. Wang. Language- agnostic bert sentence embedding. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, 2022

    2022

  15. [23]

    Fourrier, N

    C. Fourrier, N. Habib, A. Lozovskaya, K. Szafer, and T. Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/ open-llm-leaderboard/open llm leaderboard, 2024

    2024

  16. [24]

    Geiping, L

    J. Geiping, L. H. Fowl, W. R. Huang, W. Czaja, G. Taylor, M. Moeller, and T. Goldstein. Witches’ brew: Industrial scale data poisoning via gradient matching. InInternational Conference on Learning Representations, 2021

    2021

  17. [25]

    Georges and S

    L. Georges and S. Menashe. Hugging face and jfrog partner to make ai security more transparent. https://huggingface.co/blog/jfrog, 2025. Accessed: 2025-06-01

    2025

  18. [26]

    Granite Embedding Team

    I. Granite Embedding Team. Granite embedding models, December 2024

    2024

  19. [27]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al- Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  20. [28]

    T. Gu, K. Liu, B. Dolan-Gavitt, and S. Garg. Badnets: Evaluat- ing backdooring attacks on deep neural networks.IEEE Access, 7:47230–47244, 2019

    2019

  21. [29]

    Gupta, K

    P. Gupta, K. Yadav, B. B. Gupta, M. Alazab, and T. R. Gadekallu. A novel data poisoning attack in federated learning based on inverted loss function.Computers & Security, 130:103270, 2023

    2023

  22. [30]

    G ¨unther, J

    M. G ¨unther, J. Ong, I. Mohr, A. Abdessalem, T. Abel, M. K. Akram, S. Guzman, G. Mastrapas, S. Sturua, B. Wang, M. Werk, N. Wang, and H. Xiao. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents, 2023

    2023

  23. [31]

    Hartford, L

    E. Hartford, L. Atkins, F. Fernandes, and C. Computations. Dol- phin 2.9.1 yi 1.5 34b. https://huggingface.co/cognitivecomputations/ dolphin-2.9.1-yi-1.5-34b. Accessed: 2025-06-01

    2025

  24. [32]

    Heusel, H

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    2017

  25. [33]

    Kokoro-82m (revision d8b4fc7)

    Hexgrad. Kokoro-82m (revision d8b4fc7). https://huggingface.co/ hexgrad/Kokoro-82M, 2025

    2025

  26. [34]

    Hoffman, D

    K. Hoffman, D. Zage, and C. Nita-Rotaru. A survey of attack and defense techniques for reputation systems.ACM Computing Surveys (CSUR), 42(1):1–31, 2009

    2009

  27. [35]

    S. Hong, N. Carlini, and A. Kurakin. Handcrafted backdoors in deep neural networks.Advances in Neural Information Processing Systems, 35:8068–8080, 2022

    2022

  28. [36]

    X. Hu, Z. Shan, X. Zhao, Z. Sun, Z. Liu, D. Li, S. Ye, X. Wei, Q. Chen, B. Hu, et al. Kalm-embedding: Superior training data brings a stronger embedding model.arXiv preprint arXiv:2501.01028, 2025

    2025 arXiv

  29. [37]

    W. R. Huang, J. Geiping, L. Fowl, G. Taylor, and T. Goldstein. Metapoison: Practical general-purpose clean-label data poisoning. Advances in Neural Information Processing Systems, 33:12080– 12091, 2020

    2020

  30. [38]

    Huang, M

    Y . Huang, M. Nasr, A. Angelopoulos, N. Carlini, W.-L. Chiang, C. A. Choquette-Choo, D. Ippolito, M. Jagielski, K. Lee, K. Z. Liu, et al. Exploring and mitigating adversarial manipulation of voting-based leaderboards. InInternational Conference on Machine Learning, 2025

    2025

  31. [39]

    U. D. (HuggingFace). Ugi (uncensored general intelli- gence) leaderboard. https://huggingface.co/spaces/DontPlanToEnd/ UGI-Leaderboard, 2024

    2024

  32. [40]

    D. R. Hunter. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32(1):384–406, 2004

    2004

  33. [41]

    H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y . Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  34. [42]

    Jagielski, G

    M. Jagielski, G. Severi, N. Pousette Harger, and A. Oprea. Sub- population data poisoning attacks. InProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 3104–3122, 2021

    2021

  35. [43]

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  36. [44]

    Jiang, Y

    Z. Jiang, Y . Ren, R. Li, S. Ji, B. Zhang, Z. Ye, C. Zhang, B. Jionghao, X. Yang, J. Zuo, et al. Megatts 3: Sparse alignment enhanced latent diffusion transformer for zero-shot speech synthesis.arXiv preprint arXiv:2502.18924, 2025

    2025 arXiv

  37. [45]

    King and F

    C. King and F. Khan. 4m models scanned: Hugging face + protect ai partnership update. https://protectai.com/blog/ hugging-face-protect-ai-six-months-in, 2025. Accessed: 2025-06- 01

    2025

  38. [46]

    Kirchenbauer, J

    J. Kirchenbauer, J. Geiping, Y . Wen, J. Katz, I. Miers, and T. Gold- stein. A watermark for large language models. InInternational Conference on Machine Learning, 2023

    2023

  39. [47]

    B. F. Labs. Flux. https://github.com/black-forest-labs/flux, 2024

    2024

  40. [48]

    E. Labs. Eleven multilingual v2. https://elevenlabs.io/blog/ eleven-multilingual-v2, 2023

    2023

  41. [49]

    E. Labs. Eleven flash v2.5. https://elevenlabs.io/blog/meet-flash, 2024

    2024

  42. [50]

    E. Labs. Introducing turbo v2.5. https://elevenlabs.io/blog/ introducing-turbo-v2-5, 2024

    2024

  43. [51]

    V . Lai, N. T. Ngo, A. P. B. Veyseh, F. Dernoncourt, and T. H. Nguyen. Open multilingual llm evaluation leaderboard, 2023

    2023

  44. [52]

    Text2video leaderboard

    Lambda. Text2video leaderboard. https://t2vleaderboard.lambda.ai/,

  45. [53]

    Langford, I

    H. Langford, I. Shumailov, Y . Zhao, R. Mullins, and N. Papernot. Architectural neural backdoors from first principles. In2025 IEEE Symposium on Security and Privacy (SP), pages 60–60. IEEE Com- puter Society, 2024

    2024

  46. [54]

    S. Lee, A. Shakir, D. Koenig, and J. Lipp. Open source strikes bread - new fluffy embeddings model. https://www.mixedbread.ai/ blog/mxbai-embed-large-v1, 2024

    2024

  47. [55]

    X. Li, T. Zhang, Y . Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca eval, 5 2023

    2023

  48. [56]

    Y . A. Li, C. Han, V . Raghavan, G. Mischler, and N. Mesgarani. Styletts 2: Towards human-level text-to-speech through style diffu- sion and adversarial training with large speech language models. Advances in Neural Information Processing Systems, 36:19594– 19621, 2023. 15

    2023

  49. [57]

    Z. Li, X. Zhang, Y . Zhang, D. Long, P. Xie, and M. Zhang. Towards general text embeddings with multi-stage contrastive learning.arXiv preprint arXiv:2308.03281, 2023

    2023 arXiv

  50. [58]

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ar, and C. L. Zitnick. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European confer- ence, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740–75...

    2014

  51. [59]

    R. Liu, T. Wang, Y . Cao, and L. Xiong. Precurious: How innocent pre-trained language models turn into privacy traps. InACM Con- ference on Computer and Communications Security (CCS), 2024

    2024

  52. [60]

    Y . Liu, R. Meng, S. Jot, S. Savarese, C. Xiong, Y . Zhou, and S. Yavuz. Codexembed: A generalist embedding model fam- ily for multiligual and multi-task code retrieval.arXiv preprint arXiv:2411.12644, 2024

    2024 arXiv

  53. [61]

    H. Ma, H. Qiu, Y . Gao, Z. Zhang, A. Abuadbba, M. Xue, A. Fu, J. Zhang, S. F. Al-Sarawi, and D. Abbott. Quantization backdoors to deep learning commercial frameworks.IEEE Transactions on Dependable and Secure Computing, 2023

    2023

  54. [62]

    Mahloujifar, E

    S. Mahloujifar, E. Ghosh, and M. Chase. Property inference from poisoning. InIEEE Symposium on Security and Privacy. IEEE, 2022

    2022

  55. [63]

    Maini, M

    P. Maini, M. Yaghini, and N. Papernot. Dataset inference: Ownership resolution in machine learning. InInternational Conference on Learning Representations, 2021

    2021

  56. [64]

    Merrick, D

    L. Merrick, D. Xu, G. Nuti, and D. Campos. Arctic-embed: Scalable, efficient, and accurate text embedding models.arXiv preprint arXiv:2405.05374, 2024

    2024 arXiv

  57. [65]

    Mialon, C

    G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y . LeCun, and T. Scialom. Gaia: a benchmark for general ai assistants, 2023

    2023

  58. [66]

    Midjourney

    Midjourney. Midjourney. https://www.midjourney.com, 2024. https: //www.midjourney.com

    2024

  59. [67]

    N. M. Min, L. H. Pham, Y . Li, and J. Sun. Propaganda via ai? a study on semantic backdoors in large language models.arXiv preprint arXiv:2504.12344, 2025

    2025

  60. [68]

    R. Min, T. Pang, C. Du, Q. Liu, M. Cheng, and M. Lin. Improving your model ranking on chatbot arena by vote rigging. InInterna- tional Conference on Machine Learning, 2025

    2025

  61. [69]

    Minixhofer, O

    C. Minixhofer, O. Klejch, and P. Bell. Ttsds-text-to-speech distribu- tion score. In2024 IEEE Spoken Language Technology Workshop (SLT), pages 766–773. IEEE, 2024

    2024

  62. [70]

    Srivastav, C

    mrfakename, V . Srivastav, C. Fourrier, L. Pouget, Y . Lacombe, main, S. Gandhi, A. Passos, and P. Cuenca. Tts arena 2.0: Benchmarking text-to-speech models in the wild. https://huggingface.co/spaces/ TTS-AGI/TTS-Arena-V2, 2025

    2025

  63. [71]

    M. T. E. B. (MTEB). Mteb arena. https://huggingface.co/spaces/ mteb/arena, 2025. Accessed: 2025-03-01

    2025

  64. [72]

    M. T. E. B. (MTEB). Mteb leaderboard. https://huggingface.co/ spaces/mteb/leaderboard, 2025. Accessed: 2025-03-01

    2025

  65. [73]

    Muennighoff, N

    N. Muennighoff, N. Tazi, L. Magne, and N. Reimers. Mteb: Massive text embedding benchmark.arXiv preprint arXiv:2210.07316, 2022

    2022 arXiv

  66. [74]

    Muennighoff, Z

    N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Cand `es, and T. Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  67. [75]

    Naseh, J

    A. Naseh, J. Roh, E. Bagdasaryan, and A. Houmansadr. Backdooring bias into text-to-image models.arXiv preprint arXiv:2406.15213, 2024

    2024 arXiv

  68. [76]

    A. v. d. Oord, Y . Li, and O. Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  69. [77]

    Orr and E

    W. Orr and E. B. Kang. Ai as a sport: On the competitive epistemologies of benchmarking. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1875–1884, 2024

    2024

  70. [78]

    P. P1. Introducing papla p1 and real-time api for developers. https: //papla.media/blog/p1, 2025

    2025

  71. [79]

    S. J. Paech. Eq-bench: An emotional intelligence benchmark for large language models.arXiv preprint arXiv:2312.06281, 2023

    2023 arXiv

  72. [80]

    R. Pang, C. Li, Z. Xi, S. Ji, and T. Wang. The dark side of automl: Towards architectural backdoor search. InThe 11th International Conference on Learning Representations, 2023

    2023

  73. [81]

    Introducing playht2.0: The state-of-the-art generative voice ai model for conversational speech

    PlayAI. Introducing playht2.0: The state-of-the-art generative voice ai model for conversational speech. https://huggingface.co/spaces/ hf-audio/open asr leaderboard, 2023

    2023

  74. [82]

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InConference on Language Modeling (COLM), 2024

    2024

  75. [83]

    Research

    H. Research. Introducing octave (omni-capable text and voice engine). https://www.hume.ai/blog/introducing-octave, 2024

    2024

  76. [84]

    Seal llm leaderboards

    ScaleAI. Seal llm leaderboards. https://scale.com/leaderboard, 2025

    2025

  77. [85]

    Shafahi, W

    A. Shafahi, W. R. Huang, M. Najibi, O. Suciu, C. Studer, T. Dumi- tras, and T. Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks.Advances in Neural Information Pro- cessing Systems, 31, 2018

    2018

  78. [86]

    Singh, Y

    S. Singh, Y . Nan, A. Wang, D. D’Souza, S. Kapoor, A. ¨Ust¨un, S. Koyejo, Y . Deng, S. Longpre, N. Smith, et al. The leaderboard illusion.arXiv preprint arXiv:2504.20879, 2025

    2025 arXiv

  79. [87]

    A. V . Solatorio. Gistembed: Guided in-sample selection of train- ing negatives for text embedding fine-tuning.arXiv preprint arXiv:2402.16829, 2024

    2024 arXiv

  80. [88]

    Srivastav, S

    V . Srivastav, S. Majumdar, N. Koluguri, A. Moumen, S. Gandhi, et al. Open automatic speech recognition leaderboard. https: //huggingface.co/spaces/hf-audio/open asr leaderboard, 2023

    2023

  81. [89]

    ˇSrndi´c and P

    N. ˇSrndi´c and P. Laskov. Practical evasion of a learning-based classifier: A case study. In2014 IEEE symposium on security and privacy, pages 197–211. IEEE, 2014

    2014

  82. [90]

    F. Suya, S. Mahloujifar, A. Suri, D. Evans, and Y . Tian. Model- targeted poisoning attacks with provable convergence. InInter- national Conference on Machine Learning, pages 10000–10010. PMLR, 2021

    2021

  83. [91]

    Tancik, B

    M. Tancik, B. Mildenhall, and R. Ng. Stegastamp: Invisible hy- perlinks in physical photographs. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2117– 2126, 2020

    2020

  84. [92]

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram ´e, M. Rivi `ere, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  85. [93]

    G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhu- patiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ram ´e, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  86. [94]

    Thakur, N

    N. Thakur, N. Reimers, A. R ¨uckl´e, A. Srivastava, and I. Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of in- formation retrieval models. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021

    2021

  87. [95]

    Y . Tian, F. Suya, A. Suri, F. Xu, and D. Evans. Manipulating transfer learning for property inference. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15975–15984, 2023

    2023

  88. [96]

    Transformers

    S. Transformers. Sbert: Pretrained models. https://www.sbert.net/ docs/sentence transformer/pretrained models.html, 2021

    2021

  89. [97]

    The trustbit llm leaderboards

    Trustbit. The trustbit llm leaderboards. https://www.trustbit.tech/en/ llm-benchmarks, 2025. Accessed: 2025-06-01. 16

    2025

  90. [98]

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. InInternational Conference on Learning Representations, 2019

    2019

  91. [99]

    L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei. Multilingual e5 text embeddings: A technical report.arXiv preprint arXiv:2402.05672, 2024

    2024 arXiv

  92. [100]

    R. Wang, M. Zhu, J. Ou, R. Chen, X. Tao, P. Wan, and B. Wu. Badvideo: Stealthy backdoor attack against text-to-video generation. arXiv preprint arXiv:2504.16907, 2025

    2025 arXiv

  93. [101]

    T. Wang, F. Li, L. Zhu, J. Li, Z. Zhang, and H. T. Shen. Invis- ible black-box backdoor attack against deep cross-modal hashing retrieval.ACM Transactions on Information Systems, 42(4):1–27, 2024

    2024

  94. [102]

    P. Warden. Speech commands: A dataset for limited-vocabulary speech recognition.arXiv preprint arXiv:1804.03209, 2018

    2018 arXiv

  95. [103]

    Y . Wu, Z. Jiang, A. Khan, Y . Fu, L. Ruis, E. Grefenstette, and T. Rockt¨aschel. Chatarena: Multi-agent language game environments for large language models. https://github.com/chatarena/chatarena, 2023

    2023

  96. [104]

    S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, and J.-Y . Nie. C-pack: Packed resources for general chinese embeddings. InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649, 2024

    2024

  97. [105]

    R. Xu, Z. Wang, R.-Z. Fan, and P. Liu. Benchmarking benchmark leakage in large language models.arXiv preprint arXiv:2404.18824, 2024

    2024 arXiv

  98. [106]

    Jiang, G

    Xuan, D. Jiang, G. Zhang, M. Ku, A. Soni, S. Siu, H. Chen, A. Chandra, Z. Jiang, A. Arulraj, K. Wang, Q. D. Do, Y . Ni, B. Lyu, Y . Narsupalli, R. Fan, Z. Lyu, B. Y . Lin, and W. Chen. VideoScore: Building automatic metrics to simulate fine-grained human feedback for video gen...

    2024

  99. [107]

    F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Sto- ica, and J. E. Gonzalez. Berkeley function calling leader- board. https://gorilla.cs.berkeley.edu/blogs/8 berkeley function calling leaderboard.html, 2024

    2024

  100. [108]

    Q. A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren...

    2024 arXiv

  101. [109]

    P. Yu, L. Merrick, G. Nuti, and D. Campos. Arctic-embed 2.0: Multilingual retrieval without compromise.arXiv preprint arXiv:2412.04506, 2024

    2024 arXiv

  102. [110]

    Zhang, C

    B. Zhang, C. Guo, G. Yang, H. Yu, H. Zhang, H. Lei, J. Mai, J. Yan, K. Yang, M. Yang, et al. Minimax-speech: Intrinsic zero- shot text-to-speech with a learnable speaker encoder.arXiv preprint arXiv:2505.07916, 2025

    2025 arXiv

  103. [111]

    Zhang, J

    D. Zhang, J. Li, Z. Zeng, and F. Wang. Jasper and stella: distillation of sota embedding models.arXiv preprint arXiv:2412.19048, 2024

    2024 arXiv

  104. [112]

    Zhang, J

    Y . Zhang, J. Rando, I. Evtimov, J. Chi, E. M. Smith, N. Carlini, F. Tram`er, and D. Ippolito. Persistent pre-training poisoning of llms. InInternational Conference on Learning Representations, 2025

    2025

  105. [113]

    S. Zhao, X. Ma, X. Zheng, J. Bailey, J. Chen, and Y .-G. Jiang. Clean-label backdoor attacks on video recognition models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14443–14452, 2020

    2020

  106. [114]

    W. Zhao, A. M. Rush, and T. Goyal. Challenges in trustworthy human evaluation of chatbots. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 3359–3365. Association for Computational Linguistics, 2025

    2025

  107. [115]

    Y . Zhao, K. Zhang, T. Hu, S. Wu, R. L. Bras, T. Anderson, J. Bragg, J. C. Chang, J. Dodge, M. Latzke, et al. Sciarena: An open evaluation platform for foundation models in scientific literature tasks.arXiv preprint arXiv:2507.01001, 2025

    2025

  108. [116]

    C. Zhu, W. R. Huang, H. Li, G. Taylor, C. Studer, and T. Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In International Conference on Machine Learning, pages 7614–7623. PMLR, 2019

    2019

  109. [117]

    T. Y . Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. InInternational Conference on Learning Representations, 2025

    2025

  110. [118]

    Zong, Y .-W

    W. Zong, Y .-W. Chow, W. Susilo, J. Baek, and S. Camtepe. Au- diomarknet: Audio watermarking for deepfake speech detection. In USENIX Security Symposium, 2025. Appendix

    2025

  111. [120]

    Deanonymization Tags

    Important Ablations Inserting malicious behavior involves training on leader- board test data, deanonymization data, and poisoned data for multiple epochs. Here, we analyze how training progres- sion impacts leaderboard performance for benchmark-based leaderboards, and deanony...

  112. [121]

    Benchmark-Based Leaderboard Setting for Text-to-Image Models To construct a hypothetical benchmark-based leader- board for text-to-image models, we evaluate a diverse set of 11 models spanning both open-source and commercial families. The selected models represent a broad rang...

  113. [122]

    Leaderboard Simulations 3.1. Text Embedding.We simulate a local version of the MTEB Arena by instantiating 14 retriever models: GTE v1.5 Large [57], GIST Embedding [87], BGE v1.5 Large [104], KaLM Multilingual-mini-v1 [36], Granite Multilingual [26], Multilingual E5-Large [99]...

  114. [123]

    •Ideation:Anshuman and Harsh proposed the idea of poisoned model injection in leaderboards

    Author Contributions This project was a team effort led by four students. •Ideation:Anshuman and Harsh proposed the idea of poisoned model injection in leaderboards. •Formalization:Anshuman, Harsh, Ali and Yuefeng formalized a cross-modal framework. •Attack design, Experiments...

  115. [124]

    Examples We provide poisoned responses generated by our mali- cious malicious models for text embedding, text generation and text-to-speech modalities below. 19 TABLE 9:Model Responses generated by a Clean Base model and Malicious Finetuned version for Geographical Location Bi...

    2008

  116. [2025]

    Accessed: 2025-06-01

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.