Pith.

Computation and Language · arXiv:2610.19118 v1 · submitted 13 Oct 2026 · 106 references

DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision

Yifan Lan, Zhongye Liu, Yurui Chang, Hanyu Wang, Yongkang Du, Lu Lin, Jinghui Chen

The paper says

“Emerging decision models such as JEV provide typed interfaces that map task instructions and observations directly to choices, judgments, and scores. As these models are increasingly used for automated decision-making, understanding their reliability under adversarial manipulation becomes critical.”

more of the abstract

We introduce DecisionRobust, a benchmark for systematically evaluating the robustness of decision models across textual and visual inputs. Our benchmark covers practical decision-making scenarios and a diverse suite of attacks, including semantics-preserving text perturbations, image perturbations, and prompt injection in both modalities. We curate evaluation sets through external validity checks and cross-model candidate selection. We evaluate 30 model configurations on text tasks and 17 on vision tasks, including 24 and 12 held-out models, respectively, that are excluded from candidate selection. Our experiments reveal several key findings. First, attacks expose vulnerabilities beyond the models used for construction and selection, although optimized image perturbations transfer weakly to many targets even with multiple surrogates. Second, relative model robustness varies across tasks and attack types, and resistance to input perturbations does not necessarily imply resistance to prompt injection. Third, label-consistency checks substantially reduce measured attack success across text and image settings, highlighting the importance of validating apparent failures. The benchmark is available at https://yifan-lan.github.io/DecisionRobust/.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 15 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A well-constructed benchmark for decision-model robustness that fills a real gap. The LLM-validator concern is real but minor and does not undermine the main findings. This line is wrong

1 major objection · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 106 references say.

The paper says, at Section 2.3.3

The review did not quote this passage.

Pith says: Objection 1 of 1 · major

Label-consistency checks rely on LLM validators (GPT-5 mini, DeepSeek-V4-Flash, Claude Sonnet 5) without reported accuracy on this specific decision-model verification task. While using two independent validators and human semantic review for text perturbations mitigates some concerns, the paper does not report inter-validator agreement rates or a calibration against human judgments for the label-consistency task.

more of objection 1

This does not undermine the qualitative conclusion that attacks cause nonzero ASR, but it affects the reliability of quantitative comparisons (e.g., rankings across models in Tables 3, 5, 6). The authors should either provide such validation (e.g., a small human study on a random subset) or explicitly discuss this limitation and its impact on the reported checked ASR.

5 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 5

The term 'JEV' is used without definition; consider expanding the first occurrence to 'JEV (a decision model with typed interfaces)' to improve readability.

The paper says, at Section 2.1

Pith says: minor objection 2 of 5

The term 'Noul' (binary judgment format) is introduced without a brief explanation of the name or its origin; a short parenthetical definition would help readers unfamiliar with the JEV terminology.

show 3 more

The paper says, at Figure 1

Pith says: minor objection 3 of 5

The figure is information-rich but some labels are small and hard to read (e.g., 'Label consistency check' flow). Consider increasing font size or providing a separate zoomed-in version in the appendix.

The paper says, at Section 3.2.1, Table 3

Pith says: minor objection 4 of 5

The 'N' column lists pool size, but denominators vary across models due to clean-correct filtering. Clarify this in the table caption to avoid confusion.

The paper says, at Appendix E.1

Pith says: minor objection 5 of 5

The description of Charmer's search is clear but would benefit from a brief note on the approximate number of iterations or budget used, as it is referenced as 'iterative search' in the main text.

What would settle it

Randomly sample 100 text perturbation candidates and 100 image perturbation candidates from the pre-validation pool. Have two human annotators independently judge whether the perturbed input still supports the reference answer, following the same criteria as the LLM checks. Compute the agreement rate between the LLM consensus (both validators agree) and the human majority. If agreement is below 90% for either modality, then the checked ASR numbers may be unreliable for fine-grained comparisons.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 15 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 38 s · 30,416 tokens read, 3,326 written
  3. Editor wrote the verdict above 14 s · 31,284 tokens read, 845 written
  4. Referee minor revision: 1 major objection, 5 minor 22 s · 30,788 tokens read, 2,045 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This is a solid benchmark paper. What it does well: it systematically evaluates robustness of JEV-style decision models across both text and vision, using a broad set of attacks, careful validity checks (label consistency via two LLMs, human semantic review, image readability), and evaluation on many held-out models. The curation pipeline—attack construction on one model, validity filtering, transfer-based selection, then evaluation on held-out models—is sound and reproducible. The main finding, that attacks transfer beyond the construction model but with varying effectiveness and that label checks matter, is convincingly supported. The paper does not overclaim; it honestly reports that optimized image perturbations transfer weakly and that injection effectiveness depends on template and location. The limitations section is straightforward.

The stress-test concern about LLM validators (GPT-5 mini, DeepSeek-V4-Flash, Claude Sonnet 5) not being validated for this specific label-consistency task is real but minor. The paper uses two independent validators with consensus, which helps. More important, the central qualitative result—that attacks expose vulnerabilities in held-out models—does not depend on precise ASR values; even if the checked ASR is off by a few points, the pattern of cross-model transfer and the variation by task and attack type would hold. The quantitative rankings between models (e.g., which model is most robust on a given task) could shift slightly, but that is a second-order concern for a first benchmark. The paper would be stronger with a calibration experiment (e.g., a sample of human judgments for the label-consistency task) but its absence is not a fatal flaw.

Who this is for: anyone working on adversarial robustness of structured decision models, especially in the JEV ecosystem. The benchmark is a useful resource for practitioners and for follow-up research on attack transfer and defense. It deserves a serious referee and is likely publishable with minor revision (adding a note about the validator limitation and perhaps reporting inter-validator agreement on a sample).

The strongest objection

The checked attack success rate (ASR), a central metric in Tables 3, 5, 6 and Figures 2, 3, depends on label-consistency checks performed by LLMs (GPT-5 mini, DeepSeek-V4-Flash, Claude Sonnet 5) without reported validation of their accuracy on this specific decision-model verification task. If these validators systematically over- or under-flag perturbed inputs as preserving the reference answer, the filtered attack sets and the resulting checked ASR could be systematically biased. For example, a validator that is too strict would discard valid attacks, inflating apparent robustness; a too-permissive one would include invalid attacks, overstating vulnerability. The paper uses two independent validators requiring consensus, which reduces random error but does not eliminate systematic bias if both share a blind spot (e.g., both were trained on similar data). Although human semantic review is applied to text perturbations in the curated set, that review checks semantic faithfulness rather than label-consistency per se. The paper does not report inter-validator agreement rates or a calibration against human judgments for the label-consistency task, leaving the reliability of the checked ASR as an unevaluated assumption. Since the central claim about vulnerability exposure relies on the existence of nonzero ASR rather than precise values, this concern does not undermine the qualitative conclusion but could affect quantitative rankings of model robustness.

The reader identified the same weakest assumption: the reliance on external LLMs for label-consistency checks. I agree that this is the most load-bearing concern, as the checked ASR is a primary metric. However, the paper mitigates it by using two independent validators requiring consensus and by adding human semantic review for text perturbations in the curated set. The central claim—that attacks expose vulnerabilities beyond construction models—does not depend on precise ASR values; the existence of nonzero transfer on held-out models establishes the claim qualitatively. Therefore, while the concern is valid, it does not justify changing the verdict from ACCEPT. The concrete test proposed would provide evidence on whether the validators are sufficiently accurate for the claimed quantitative findings, but the qualitative conclusions are robust even without that test.

The referee's report

If the results hold, DecisionRobust provides a much-needed systematic evaluation of decision-model robustness under realistic threats. The methodology is thorough: external label-consistency checks using two independent LLMs, human semantic review for text perturbations, readability checks for images, and evaluation on held-out models separate from selection. The findings about cross-model transfer and the importance of validity checks are valuable for practitioners and researchers. The benchmark and curated sets will serve as a resource for future work on decision-model safety.

Pith's reading of the paper (a machine's summary, not the authors')

This paper introduces DecisionRobust, a benchmark for systematically evaluating the adversarial robustness of JEV-style decision models—models that map instructions and observations directly to choices, judgments, or scores. The benchmark covers text and image perturbations as well as prompt injection across practical decision tasks. The central finding is that attacks constructed on one model expose vulnerabilities in many others, including 24 held-out text models and 12 held-out vision models that were not part of attack selection. However, optimized image perturbations transfer weakly to most targets even when optimized against multiple surrogate models. A critical methodological contribution is the use of label-consistency checks by external LLMs, which show that many apparent attack successes result from invalid edits that change the input in ways that no longer support the reference answer; these checks substantially reduce measured attack success rates in both modalities.

What carries the argument. The central mechanism is a multi-step data curation pipeline: (1) candidate generation via text and image attacks (including semantics-preserving perturbations and prompt injection), (2) label-consistency checks using two external LLMs per modality to verify that the perturbed input still supports the ground-truth answer, (3) transfer-based selection favoring candidates that fool multiple construction models while maintaining task coverage, and (4) semantic review for text edits and readability checks for common image perturbations. The evaluation then measures attack success rate (ASR) at sample and candidate levels, with and without label-consistency filtering, across a large set of models (30 text, 17 vision) including many held out from candidate selection.

Core claim. The paper establishes that JEV-style decision models, despite their structured output format (Choice, Noul, Score), are broadly susceptible to adversarial manipulation across text and image modalities. Attacks transferred beyond the construction and selection models, though image perturbations had limited transfer. Crucially, label-consistency checks using independent LLMs (GPT-5 mini, DeepSeek-V4-Flash, Claude Sonnet 5) filtered out a large fraction of apparent attack successes, demonstrating that raw prediction changes overstate actual vulnerability. The benchmark also reveals that resistance to input perturbations does not imply resistance to prompt injection, and that the relative robustness of models varies across tasks and attack types.

The premise it rests on. The label-consistency checks rely on external LLMs (GPT-5 mini, DeepSeek-V4-Flash, Claude Sonnet 5) to judge whether a perturbed input still supports the reference answer; if these validator LLMs make systematic errors, the reported checked ASR may be unreliable.

If the paper is right (Pith's inferences, not the authors')

Benchmark designers should incorporate label-consistency checks as a standard step when evaluating adversarial attacks on decision models; raw ASR alone can be misleading.

Practitioners deploying JEV-style models in safety-critical applications should test across multiple attack types (perturbation and injection) rather than relying on a single robustness measure.

The weak transfer of optimized image perturbations suggests that universal image attacks against decision models remain challenging, motivating research on more transferable visual adversarial techniques.

The diversity of model responses to injection templates and locations indicates that defenses must consider both the wording and the placement of injected instructions.

The need for two independent LLMs to validate label consistency suggests that even advanced evaluators agree imperfectly; this could be exploited by adaptive attacks that create inputs which fool both the target model and the validator LLMs.

The benchmark's limitation to individual decisions rather than workflow-level effects leaves open the question of whether attacks on a single decision propagate to catastrophic downstream outcomes in multi-step agentic systems.

The finding that model rankings change between text perturbations and prompt injection implies that a single 'robustness score' is inadequate; a multi-dimensional vulnerability profile may be needed for procurement or deployment decisions.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Emerging decision models such as JEV provide typed interfaces that map task instructions and observations directly to choices, judgments, and scores. As these models are increasingly used for automated decision-making, understanding their reliability under adversarial manipulation becomes critical. We introduce DecisionRobust, a benchmark for systematically evaluating the robustness of decision models across textual and visual inputs. Our benchmark covers practical decision-making scenarios and a diverse suite of attacks, including semantics-preserving text perturbations, image perturbations, and prompt injection in both modalities. We curate evaluation sets through external validity checks and cross-model candidate selection. We evaluate 30 model configurations on text tasks and 17 on vision tasks, including 24 and 12 held-out models, respectively, that are excluded from candidate selection. Our experiments reveal several key findings. First, attacks expose vulnerabilities beyond the models used for construction and selection, although optimized image perturbations transfer weakly to many targets even with multiple surrogates. Second, relative model robustness varies across tasks and attack types, and resistance to input perturbations does not necessarily imply resistance to prompt injection. Third, label-consistency checks substantially reduce measured attack success across text and image settings, highlighting the importance of validating apparent failures. The benchmark is available at https://yifan-lan.github.io/DecisionRobust/.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-15

The authors' main claim, as the classifier picked it out 0.41

“Second, relative model robustness varies across tasks and attack types, and resistance to input perturbations does not necessarily imply resistance to prompt injection.”

How the main claim is stated: asserted 0.92

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.73 · human participants 0.11 · controlled comparison 0.18 · randomized assignment 0.02 · states quantitative result 0.86 · compares with prior results 0.29 · mentions code or data availability 0.93

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Emerging decision models such as JEV provide typed interfaces that map task instructions and observations directly to choices, judgments, and scores. background 1.00 0.05 0.02 asserted 1.00
As these models are increasingly used for automated decision-making, understanding their reliability under adversarial manipulation becomes critical. background 0.74 0.05 0.02 asserted 0.99
We introduce DecisionRobust, a benchmark for systematically evaluating the robustness of decision models across textual and visual inputs. contribution 1.00 0.93 0.06 asserted 1.00
Our benchmark covers practical decision-making scenarios and a diverse suite of attacks, including semantics-preserving text perturbations, image perturbations, and prompt injection in both modalities. method 0.63 0.82 0.04 asserted 1.00
We curate evaluation sets through external validity checks and cross-model candidate selection. method 0.98 0.95 0.02 asserted 1.00
We evaluate 30 model configurations on text tasks and 17 on vision tasks, including 24 and 12 held-out models, respectively, that are excluded from candidate selection. method 0.97 0.95 0.03 asserted 1.00
Our experiments reveal several key findings. result 0.90 0.57 0.03 asserted 0.52
First, attacks expose vulnerabilities beyond the models used for construction and selection, although optimized image perturbations transfer weakly to many targets even with multiple surrogates. result 1.00 0.95 0.19 asserted 1.00
Second, relative model robustness varies across tasks and attack types, and resistance to input perturbations does not necessarily imply resistance to prompt injection. result 0.94 0.91 0.22 asserted 0.92
Third, label-consistency checks substantially reduce measured attack success across text and image settings, highlighting the importance of validating apparent failures. result 0.95 0.94 0.18 asserted 1.00
The benchmark is available at https://yifan-lan.github.io/DecisionRobust/. contribution 0.94 0.47 0.02 asserted 0.98

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-15: 1 request, 8,110 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (6)
Figure 1 of the paper
Figure 1. Data construction in DecisionRobust. Perturbation and in-image injection candidates undergo label
Figure 2 of the paper
Figure 2. Candidate-level results (%) on the curated text sets. Bars show perturbation ASR in (a) and injection
Figure 3 of the paper
Figure 3. Candidate-level results (%) on the curated image sets: perturbation ASR in (a), and target ASR for
Figure 4 of the paper
Figure 4. FinQA image-injection case. The original table and the evaluated image with the overlaid note are shown
Figure 5 of the paper
Figure 5. ImajevBench image-perturbation case. The full clean and evaluated adversarial images appear above
Figure 6 of the paper
Figure 6. Variants of the same ImajevBench example. Gaussian noise uses severity 2; the optimized perturbation

Figures are the authors', reproduced from the paper.

References (106)
  1. [2] Wang, Boxin and Xu, Chejian and Wang, Shuohang and Gan, Zhe and Cheng, Yu and Gao, Jianfeng and Awadallah, Ahmed Hassan and Li, Bo , booktitle = on Pith
  2. [3] Croce, Francesco and Andriushchenko, Maksym and Sehwag, Vikash and Debenedetti, Edoardo and Flammarion, Nicolas and Chiang, Mung and Mittal, Prateek and Hein, Matthias , booktitle = on Pith
  3. [4] Debenedetti, Edoardo and Zhang, Jie and Balunovic, Mislav and Beurer-Kellner, Luca and Fischer, Marc and Tram\`. Advances in Neural Information Processing Systems , doi = on Pith
  4. [5] Zhang, Yichi and Huang, Yao and Sun, Yitong and Liu, Chang and Zhao, Zhe and Fang, Zhengwei and Wang, Yifan and Chen, Huanran and Yang, Xiao and Wei, Xingxing and Su, Hang and Dong, Yinpeng and Zhu, Jun , booktitle =. doi:10.52202/079017-1561 , editor = on Pithdoi:10.52202/079017-1561
  5. [6] Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed H. and Le, Quoc V. and Zhou, Denny , booktitle =. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , url =. doi:10.52202/068431-1800 , editor = on Pithdoi:10.52202/068431-1800
  6. [7] Patil, Shishir G and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E. , booktitle =. The Berkeley Function Calling Leaderboard (. 2025 , editor = on Pith
  7. [11] Almeida, Diogo , title =. 2026 , month = sep, howpublished = on Pith
  8. [12] Hu, Jianyi and Zhang, Hangtao and Liu, Yi and Zeng, Yeqi and Zeng, Li and Wang, Xianlong and Wang, Rui and Zhang, Leo Yu , title =. 2026 , howpublished = on Pith
  9. [13] Jev-Mobile: Jev as an Executor for Mobile GUI Agents , author=. arXiv preprint arXiv:2609.30186 , url= on PithPith's reviewarXiv
  10. [14] Skill suggestion , year = on Pith
  11. [15] 2026 , howpublished = on Pith
  12. [16] Guardrails for. 2026 , howpublished = on Pith
  13. [19] Revisiting Character-level Adversarial Attacks for Language Models , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , volume = on Pith
  14. [22] Jin, Di and Jin, Zhijing and Zhou, Joey Tianyi and Szolovits, Peter , booktitle =. Is. 2020 , doi = on Pith
  15. [28] Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , author =. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , year = on Pith
  16. [29] Zheng, Boyuan and Gou, Boyu and Kil, Jihyung and Sun, Huan and Su, Yu , booktitle =. 2024 , volume = on Pith
  17. [30] Measuring Massive Multitask Language Understanding , author =. International Conference on Learning Representations , year = on Pith
  18. [33] Stress Test Evaluation for Natural Language Inference , author =. Proceedings of the 27th International Conference on Computational Linguistics , year = on Pith
  19. [34] Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers , author =. 2018 IEEE Security and Privacy Workshops (SPW) , year = on Pith
  20. [35] Xu, Xilie and Kong, Keyi and Liu, Ning and Cui, Lizhen and Wang, Di and Zhang, Jingfeng and Kankanhalli, Mohan , booktitle =. An. 2024 , url = on Pith
show 80 more
  1. [36] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations , author =. International Conference on Learning Representations , year = on Pith
  2. [37] Boosting Adversarial Attacks With Momentum , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year = on Pith
  3. [38] Improving Transferability of Adversarial Examples With Input Diversity , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year = on Pith
  4. [39] Palmer, Jared , title =. 2026 , howpublished = on Pith
  5. [40] Garg, Mohit , title =. 2026 , howpublished = on Pith
  6. [42] Introducing. 2026 , howpublished = on Pith
  7. [44] Palmer, Jared , year = on Pith
  8. [45] Open d1: Edge decision models for text, vision, and audio , journal =. 2026 , url = on Pith
  9. [47] Contrastive Language Models: A System One Model for Fast and Generalizable Decision-Making , author=. 2026 , note= on Pith
  10. [49] Yin, Wenpeng and Hay, Jamaal and Roth, Dan. Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La... on Pithdoi:10.18653/v1/d19-1404
  11. [50] Evaluating and Benchmarking the System One Model Jev , author=. 2026 , eprint= on Pith
  12. [51] Zhu, Kaijie and Wang, Jindong and Zhou, Jiaheng and Wang, Zichen and Chen, Hao and Wang, Yidong and Yang, Linyi and Ye, Wei and Zhang, Yue and Gong, Neil Zhenqiang and Xie, Xing , journal =. 2024 , url = on Pith
  13. [52] Elias Abad Rocamora, Yongtao Wu, Fanghui Liu, Grigorios G. Chrysos, and Volkan Cevher. 2024. https://proceedings.mlr.press/v235/abad-rocamora24a.html Revisiting character-level adversarial attacks for language models . In Proceedings of the 41st International Conference on Mac... on Pith
  14. [53] akhilaaa3 . 2026. https://huggingface.co/akhilaaa3/Jev-Omni Jev-Omni . Hugging Face model card. Accessed October 11, 2026 on Pith
  15. [54] Diogo Almeida. 2026. https://typesafe.ai/blog/introducing-system-one-models-and-jev Introducing system one models & Jev . TypeSafe AI Blog. Published September 15, 2026 on Pith
  16. [55] Anthropic . 2026. https://www.anthropic.com/news/claude-sonnet-5 Introducing Claude Sonnet 5 . Anthropic announcement. Published June 30, 2026 on Pith
  17. [56] AutoTrust AI . 2026 a . https://huggingface.co/autotrust/JEV-27B autotrust/JEV-27B : a second-generation integrated System 1 + System 2 open model built with the Blocks of Experts recipe ( Qwen3.8-27B ; System 1 distilled from TypeSafe Jev 1.13 ) . Hugging Face model card on Pith
  18. [57] AutoTrust AI . 2026 b . https://huggingface.co/autotrust/GEV-26B-Decide GEV-26B-Decide . Hugging Face model card. Accessed October 12, 2026 on Pith
  19. [58] BrendanH18 . 2026. https://github.com/BrendanH18/jev_fsd Jev Drive Lab . GitHub repository. Accessed October 10, 2026 on Pith
  20. [59] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.300 F in QA : A dataset of numerical reasoning over financial d... on Pithdoi:10.18653/v1/2021.emnlp-main.300
  21. [60] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. https://doi.org/10.18653/v1/2024.acl-long.505 S ee C lick: Harnessing GUI grounding for advanced visual GUI agents . In Proceedings of the 62nd Annual Meeting of the Association... on Pithdoi:10.18653/v1/2024.acl-long.505
  22. [61] Convai Innovations . 2026. https://huggingface.co/convaiinnovations/laya-multilingual Laya Multilingual . Hugging Face model card. Accessed October 12, 2026 on Pith
  23. [62] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/a3c65c2974270fd093ee8a9bf8ae7d0b-Paper-round2.pd... on Pith
  24. [63] Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tram\` e r. 2024. https://doi.org/10.52202/079017-2636 AgentDojo : A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents . In Advances in Neural In... on Pithdoi:10.52202/079017-2636
  25. [64] DeepSeek-AI . 2026. https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash DeepSeek-V4-Flash . Hugging Face model card. Accessed October 11, 2026 on Pith
  26. [65] Tobias Deu er, Lorenz Sparrenberg, and Rafet Sifa. 2026. https://arxiv.org/abs/2609.37647 Evaluating and benchmarking the system one model jev . Preprint, arXiv:2609.37647 on PithPith's reviewarXiv
  27. [66] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the N orth A merican Chapter of the A ssoci... on Pithdoi:10.18653/v1/n19-1423
  28. [67] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. 2018. https://openaccess.thecvf.com/content_cvpr_2018/html/Dong_Boosting_Adversarial_Attacks_CVPR_2018_paper.html Boosting adversarial attacks with momentum . In Proceedings of the IEEE Con... on Pith
  29. [68] dtduc-git . 2026. https://github.com/dtduc-git/jevnav jevnav . GitHub repository. Accessed October 10, 2026 on Pith
  30. [69] Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. https://arxiv.org/abs/1801.04354 Black-box generation of adversarial text sequences to evade deep learning classifiers . In 2018 IEEE Security and Privacy Workshops (SPW) on PithPith's reviewarXiv
  31. [70] Mohit Garg. 2026. https://github.com/mohit67890/imajev imajev : an open Jev -style decision model for images and text . GitHub software release. Version 1.1.0, released September 26, 2026 on Pith
  32. [71] Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, and Christopher Parisien. 2025. https://doi.org/10.18653/v1/2025.naacl-long.306 AEGIS 2.0: A diverse AI safety dataset and risks taxonomy for alignment of LLM... on Pithdoi:10.18653/v1/2025.naacl-long.306
  33. [72] GodsBoy . 2026. https://github.com/GodsBoy/jev-agent-skill-router Jev Agent Skill Router . GitHub repository. Accessed October 10, 2026 on Pith
  34. [73] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. https://doi.org/10.48550/arXiv.2302.12173 Not what you've signed up for: Compromising real-world LLM -integrated applications with indirect prompt injection . arXiv preprint... on PithPith's reviewarXivdoi:10.48550/arxiv.2302.12173
  35. [74] H2O.ai . 2026. https://huggingface.co/h2oai/h2o-lightning-4b H2O-Lightning-4B . Hugging Face model card. Accessed October 12, 2026 on Pith
  36. [75] Hanno Labs . 2026. https://huggingface.co/Hanno-Labs/bosun-v3.1-1.7b Bosun 3.1 1.7B . Hugging Face model card. Accessed October 11, 2026 on Pith
  37. [76] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . In International Conference on Learning Representations on PithPith's reviewarXiv
  38. [77] Dan Hendrycks and Thomas Dietterich. 2019. https://arxiv.org/abs/1903.12261 Benchmarking neural network robustness to common corruptions and perturbations . In International Conference on Learning Representations on PithPith's reviewarXiv
  39. [78] Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, and Leo Yu Zhang. 2026. https://arxiv.org/abs/2609.31142 JevAdvBench : A benchmark and black-box attacks for reinforcement learning for calibrated decisions models . arXiv preprint arXiv:2609.31142 on PithPith's reviewarXiv
  40. [79] Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1170 Adversarial example generation with syntactically controlled paraphrase networks . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association ... on Pithdoi:10.18653/v1/n18-1170
  41. [80] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. https://doi.org/10.1609/aaai.v34i05.6311 Is BERT really robust? a strong baseline for natural language attack on text classification and entailment . In Proceedings of the AAAI Conference on Artificial Intellige... on Pithdoi:10.1609/aaai.v34i05.6311
  42. [81] jylin001206 . 2026. https://huggingface.co/jylin001206/InnerJev-27B-Full/blob/3f229c41d97e68c792aa0ebfce35e047db3a823e/README.md InnerJev-27B-Full . Hugging Face model card. Accessed October 12, 2026 on Pith
  43. [82] Jacky Kwok, Hangoo Kang, Tarun Suresh, Jon Saad-Falcon, Marco Pavone, Christopher R \'e , and Azalia Mirhoseini. 2026. https://contrastive-lm.notion.site Contrastive language models: A system one model for fast and generalizable decision-making . Notion Blog on Pith
  44. [83] Yifan Lan, Yuanpu Cao, Hanyu Wang, Lu Lin, and Jinghui Chen. 2026. The illusion of reasoning: Exposing evasive data contamination in llms via zero-cot truncation. arXiv preprint arXiv:2605.21856 on PithPith's reviewarXiv
  45. [84] Yifan Lan, Yuanpu Cao, Weitong Zhang, Lu Lin, and Jinghui Chen. 2025. https://doi.org/10.18653/v1/2025.emnlp-main.901 Phi: Preference hijacking in multi-modal large language models at inference time . In Proceedings of the 2025 Conference on Empirical Methods in Natural Langua... on Pithdoi:10.18653/v1/2025.emnlp-main.901
  46. [85] Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.500 BERT - ATTACK : Adversarial attack against BERT using BERT . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNL... on Pithdoi:10.18653/v1/2020.emnlp-main.500
  47. [86] Liquid AI . 2026 a . https://docs.liquid.ai/lfm/models/d1 d1 . Liquid AI documentation. Accessed October 11, 2026 on Pith
  48. [87] Liquid AI . 2026 b . https://www.liquid.ai/blog/d1-open Open d1: Edge decision models for text, vision, and audio . Liquid AI Blog on Pith
  49. [88] Zhongye Liu, Yaopei Zeng, Yurui Chang, and Lu Lin. 2026. Forced deferral: Manipulating routing decisions in multimodal llm cascades. arXiv preprint arXiv:2606.15308 on PitharXiv
  50. [89] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. 2021. https://doi.org/10.18653/v1/2021.acl-long.528 I nter- GPS : Interpretable geometry problem solving with formal language and symbolic reasoning . In Proceedings of the 59th Annual ... on Pithdoi:10.18653/v1/2021.acl-long.528
  51. [90] mohit67890 . 2026. https://huggingface.co/datasets/mohit67890/imajev-bench ImajevBench v2.0-lite (preview) . Hugging Face dataset. Accessed October 10, 2026 on Pith
  52. [91] Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. https://aclanthology.org/C18-1198/ Stress test evaluation for natural language inference . In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340--... on Pith
  53. [92] NeoHorse Team . 2026. https://arxiv.org/abs/2609.08183 NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness . arXiv preprint. Preprint, arXiv:2609.08183 on PithPith's reviewarXiv
  54. [93] OmniJev Team . 2026. OneJev : A multimodal system one decision model. https://github.com/OmniJev/OneJev on Pith
  55. [94] OpenAI . 2026 a . https://developers.openai.com/api/docs/guides/decisions Decisions . OpenAI API documentation. Accessed October 12, 2026 on Pith
  56. [95] OpenAI . 2026 b . https://developers.openai.com/api/docs/models/gpt-5-mini GPT-5 Mini Model . OpenAI API documentation. Accessed October 11, 2026 on Pith
  57. [96] OpenAI . 2026 c . https://developers.openai.com/api/docs/models/gpt-5 GPT-5 Model . OpenAI API documentation. Accessed October 11, 2026 on Pith
  58. [97] OpenJev . 2026. https://huggingface.co/openjev/openjev OpenJev . Hugging Face model card. Accessed October 11, 2026 on Pith
  59. [98] Jared Palmer. 2026 a . Kev-27B: a calibrated decision model on Qwen3.8-27B . https://huggingface.co/jaredpalmer/kev-27b. Version 2, released 2026-09-30 on Pith
  60. [99] Jared Palmer. 2026 b . https://huggingface.co/jaredpalmer/kev-9b Kev-9B : a calibrated decision model on Qwen3.5-9B . Hugging Face model card on Pith
  61. [100] Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. https://proceedings.mlr.press/v267/patil25a.html The berkeley function calling leaderboard ( BFCL ): From tool use to agentic evaluation of large language m... on Pith
  62. [101] F \'a bio Perez and Ian Ribeiro. 2022. https://doi.org/10.48550/arXiv.2211.09527 Ignore previous prompt: Attack techniques for language models . arXiv preprint arXiv:2211.09527 on PithPith's reviewarXivdoi:10.48550/arxiv.2211.09527
  63. [102] Perplexity AI . 2026. https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b pplx-decider-v1.1-27b . Hugging Face model card. Accessed October 12, 2026 on Pith
  64. [103] Qwen Team . 2026. https://qwen.ai/blog?id=qwen3.5 Qwen3.5 : Towards native multimodal agents on Pith
  65. [104] Chandan K. Reddy, Llu \'i s M \`a rquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. 2022. https://doi.org/10.48550/arXiv.2206.06588 Shopping queries dataset: A large-scale ESCI benchmark for improving product s... on PithPith's reviewarXivdoi:10.48550/arxiv.2206.06588
  66. [105] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. https://doi.org/10.18653/v1/2020.acl-main.442 Beyond accuracy: Behavioral testing of NLP models with C heck L ist . In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu... on Pithdoi:10.18653/v1/2020.acl-main.442
  67. [106] Siim . 2026. https://github.com/Siim/jev-claim-vs-measured Jev for trading: claim vs. measured . GitHub repository. Accessed October 10, 2026 on Pith
  68. [107] sky7350 . 2026. https://huggingface.co/sky7350/Mica-v0.1-4B Mica v0.1 4B . Hugging Face model card. Accessed October 12, 2026 on Pith
  69. [108] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170/ Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Em... on Pith
  70. [109] Standard Thinking . 2026 a . https://huggingface.co/StandardThinking/StandardOne-3B Standard One 3B . Hugging Face model card. Accessed October 12, 2026 on Pith
  71. [110] Standard Thinking . 2026 b . https://huggingface.co/StandardThinking/StandardOne-8B Standard One 8B . Hugging Face model card. Accessed October 12, 2026 on Pith
  72. [111] Strands Agents . 2026. https://huggingface.co/StrandsAgents/strands-decider-2B-hobson-v19 Strands Decider 2B (Hobson v19) . Hugging Face model card. Accessed October 12, 2026 on Pith
  73. [112] TypeSafe AI . 2026 a . https://docs.typesafe.ai/ TypeSafe AI documentation . Official documentation: Introduction and Cookbooks. Accessed October 9, 2026 on Pith
  74. [113] TypeSafe AI . 2026 b . https://github.com/typesafe-ai/WorkflowEvals WorkflowEvals . Official GitHub repository. Accessed October 8, 2026 on Pith
  75. [114] Vankleben . 2026. https://github.com/Vankleben/jev-arm-lab jev-arm-lab . GitHub repository. Accessed October 10, 2026 on Pith
  76. [115] vLLM Semantic Router Team . 2026 a . Decision-2.0-Lux-9B : A decision 2.0 model for structured decisions. https://huggingface.co/vllm-sr/Decision-2.0-Lux-9B on Pith
  77. [116] vLLM Semantic Router Team . 2026 b . Decision-2.0-Vega-27B : A decision 2.0 model for structured decisions. https://huggingface.co/vllm-sr/Decision-2.0-Vega-27B on Pith
  78. [117] Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/335f5352088d7d9bf74191e006d8e24c-Paper-round2.pdf Adversarial GLUE : A multi-task ben... on Pith
  79. [118] Orion Weller, Dawn Lawrie, and Benjamin Van Durme. 2024. https://doi.org/10.18653/v1/2024.eacl-long.139 N ev IR : Negation in neural information retrieval . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1... on Pithdoi:10.18653/v1/2024.eacl-long.139
  80. [119] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L. Yuille. 2019. https://openaccess.thecvf.com/content_CVPR_2019/html/Xie_Improving_Transferability_of_Adversarial_Examples_With_Input_Diversity_CVPR_2019_paper.html Improving transferability of ... on Pith

The first 100 of 106 references are listed here. All of them are in this paper's citation record.

Cited by 6 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment