# Applications
## Table of Contents
- Applications
	 - [Application Domains](#application-domains)
		 - [Agent](#agent)
		 - [Programming](#programming)
		 - [Retrieval Augmented Generation](#retrieval-augmented-generation)
	 - [Application Risks](#application-risks)
		 - [Prompt Injection](#prompt-injection)
		 - [Prompt Extraction](#prompt-extraction)
## Application Domains
### Agent
- **JailBreakV-28K: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks** \[[Paper](https://arxiv.org/abs/2404.03027)\] <br />
Weidi Luo,  Siyuan Ma,  Xiaogeng Liu,  Xiaoyu Guo,  Chaowei Xiao (2024)<br />
- **Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?** \[[Paper](https://arxiv.org/abs/2404.03411)\] <br />
Shuo Chen,  Zhen Han,  Bailan He,  Zifeng Ding,  Wenqian Yu,  Philip Torr,  Volker Tresp,  Jindong Gu (2024)<br />
- **Adversarial Attacks on Multimodal Agents** \[[Paper](https://arxiv.org/abs/2406.12814
   )\] <br />
Chen Henry Wu,  Jing Yu Koh,  Ruslan Salakhutdinov,  Daniel Fried,  Aditi Raghunathan (2024)<br />
- **Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast** \[[Paper](https://arxiv.org/abs/2402.08567)\] <br />
Xiangming Gu,  Xiaosen Zheng,  Tianyu Pang,  Chao Du,  Qian Liu,  Ye Wang,  Jing Jiang,  Min Lin (2024)<br />
- **MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models** \[[Paper](https://arxiv.org/abs/2311.176)\] <br />
Xin Liu,  Yichen Zhu,  Jindong Gu,  Yunshi Lan,  Chao Yang,  Yu Qiao (2023)<br />
- **How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs** \[[Paper](https://arxiv.org/abs/2311.16101)\] <br />
Haoqin Tu,  Chenhang Cui,  Zijun Wang,  Yiyang Zhou,  Bingchen Zhao,  Junlin Han,  Wangchunshu Zhou,  Huaxiu Yao,  Cihang Xie (2023)<br />
- **Towards Red Teaming in Multimodal and Multilingual Translation** \[[Paper](https://arxiv.org/abs/2401.16247)\] <br />
Christophe Ropers,  David Dale,  Prangthip Hansanti,  Gabriel Mejia Gonzalez,  Ivan Evtimov,  Corinne Wong,  Christophe Touret,  Kristina Pereyra,  Seohyun Sonia Kim,  Cristian Canton Ferrer,  Pierre Andrews,  Marta R. Costa-jussà (2024)<br />
- **Evil Geniuses: Delving into the Safety of LLM-based Agents** \[[Paper](https://arxiv.org/abs/2311.11855)\] <br />
Yu Tian,  Xiao Yang,  Jingyuan Zhang,  Yinpeng Dong,  Hang Su (2023)<br />
- **Goal-Oriented Prompt Attack and Safety Evaluation for LLMs** \[[Paper](https://arxiv.org/abs/2309.1183)\] <br />
Chengyuan Liu,  Fubang Zhao,  Lizhi Qing,  Yangyang Kang,  Changlong Sun,  Kun Kuang,  Fei Wu (2023)<br />
- **Identifying the Risks of LM Agents with an LM-Emulated Sandbox** \[[Paper](https://arxiv.org/abs/2309.15817)\] <br />
Yangjun Ruan,  Honghua Dong,  Andrew Wang,  Silviu Pitis,  Yongchao Zhou,  Jimmy Ba,  Yann Dubois,  Chris J. Maddison,  Tatsunori Hashimoto (2023)<br />
- **CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility** \[[Paper](https://arxiv.org/abs/2307.09705)\] <br />
Guohai Xu,  Jiayi Liu,  Ming Yan,  Haotian Xu,  Jinghui Si,  Zhuoran Zhou,  Peng Yi,  Xing Gao,  Jitao Sang,  Rong Zhang,  Ji Zhang,  Chao Peng,  Fei Huang,  Jingren Zhou (2023)<br />
- **Assessing Prompt Injection Risks in 200+ Custom GPTs** \[[Paper](https://arxiv.org/abs/2311.11538)\] <br />
Jiahao Yu,  Yuhang Wu,  Dong Shu,  Mingyu Jin,  Sabrina Yang,  Xinyu Xing (2023)<br />
- **Exploiting Novel GPT-4 APIs** \[[Paper](https://arxiv.org/abs/2312.14302)\] <br />
Kellin Pelrine,  Mohammad Taufeeque,  Michał Zając,  Euan McLean,  Adam Gleave (2023)<br />
- **R-Judge: Benchmarking Safety Risk Awareness for LLM Agents** \[[Paper](https://arxiv.org/abs/2401.10019)\] <br />
Tongxin Yuan,  Zhiwei He,  Lingzhong Dong,  Yiming Wang,  Ruijie Zhao,  Tian Xia,  Lizhen Xu,  Binglin Zhou,  Fangqi Li,  Zhuosheng Zhang,  Rui Wang,  Gongshen Liu (2024)<br />
- **GPT in Sheep's Clothing: The Risk of Customized GPTs** \[[Paper](https://arxiv.org/abs/2401.09075)\] <br />
Sagiv Antebi,  Noam Azulay,  Edan Habler,  Ben Ganon,  Asaf Shabtai,  Yuval Elovici (2024)<br />
- **Attacks on Third-Party APIs of Large Language Models** \[[Paper](https://arxiv.org/abs/2404.16891
   )\] <br />
Wanru Zhao,  Vidit Khazanchi,  Haodi Xing,  Xuanli He,  Qiongkai Xu,  Nicholas Donald Lane (2024)<br />
- **Security of AI Agents** \[[Paper](https://arxiv.org/abs/2406.08689
   )\] <br />
Yifeng He,  Ethan Wang,  Yuyang Rong,  Zifei Cheng,  Hao Chen (2024)<br />
- **BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents** \[[Paper](https://arxiv.org/abs/2406.03007
   )\] <br />
Yifei Wang,  Dizhan Xue,  Shengjie Zhang,  Shengsheng Qian (2024)<br />
- **BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents** \[[Paper](https://arxiv.org/abs/2406.03007
   )\] <br />
Yifei Wang,  Dizhan Xue,  Shengjie Zhang,  Shengsheng Qian (2024)<br />
- **Rapid Adoption,Hidden Risks: The Dual Impact of Large Language Model Customization** \[[Paper](https://arxiv.org/abs/2402.09179)\] <br />
Rui Zhang,  Hongwei Li,  Rui Wen,  Wenbo Jiang,  Yuan Zhang,  Michael Backes,  Yun Shen,  Yang Zhang (2024)<br />
- **A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents** \[[Paper](https://arxiv.org/abs/2402.10196)\] <br />
Lingbo Mo,  Zeyi Liao,  Boyuan Zheng,  Yu Su,  Chaowei Xiao,  Huan Sun (2024)<br />
- **ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages** \[[Paper](https://arxiv.org/abs/2402.10753)\] <br />
Junjie Ye,  Sixian Li,  Guanyu Li,  Caishuang Huang,  Songyang Gao,  Yilong Wu,  Qi Zhang,  Tao Gui,  Xuanjing Huang (2024)<br />
### Programming
- **Assessing Cybersecurity Vulnerabilities in Code Large Language Models** \[[Paper](https://arxiv.org/abs/2404.18567
   )\] <br />
Md Imran Hossen,  Jianyi Zhang,  Yinzhi Cao,  Xiali Hei (2024)<br />
- **Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Models** \[[Paper](https://arxiv.org/abs/2312.06227)\] <br />
Sanghak Oh,  Kiho Lee,  Seonhye Park,  Doowon Kim,  Hyoungshick Kim (2023)<br />
- **DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructions** \[[Paper](https://arxiv.org/abs/2312.0473)\] <br />
Fangzhou Wu,  Xiaogeng Liu,  Chaowei Xiao (2023)<br />
### Retrieval Augmented Generation
- **PoisonedRAG: Knowledge Poisoning Attacks to Retrieval-Augmented Generation of Large Language Models** \[[Paper](https://arxiv.org/abs/2402.07867)\] <br />
Wei Zou,  Runpeng Geng,  Binghui Wang,  Jinyuan Jia (2024)<br />
- **Seeing Is Believing: Black-Box Membership Inference Attacks Against Retrieval Augmented Generation** \[[Paper](https://arxiv.org/abs/2406.19234
   )\] <br />
Yuying Li,  Gaoyang Liu,  Yang Yang,  Chen Wang (2024)<br />
- **Phantom: General Trigger Attacks on Retrieval Augmented Language Generation** \[[Paper](https://arxiv.org/abs/2405.20485
   )\] <br />
Harsh Chaudhari,  Giorgio Severi,  John Abascal,  Matthew Jagielski,  Christopher A. Choquette-Choo,  Milad Nasr,  Cristina Nita-Rotaru,  Alina Oprea (2024)<br />
- **TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models** \[[Paper](https://arxiv.org/abs/2405.13401
   )\] <br />
Pengzhou Cheng,  Yidong Ding,  Tianjie Ju,  Zongru Wu,  Wei Du,  Ping Yi,  Zhuosheng Zhang,  Gongshen Liu (2024)<br />
- **Poisoned LangChain: Jailbreak LLMs by LangChain** \[[Paper](https://arxiv.org/abs/2406.18122
   )\] <br />
Ziqiu Wang,  Jun Liu,  Shengkai Zhang,  Yang Yang (2024)<br />
- **BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models** \[[Paper](https://arxiv.org/abs/2406.00083
   )\] <br />
Jiaqi Xue,  Mengxin Zheng,  Yebowen Hu,  Fei Liu,  Xun Chen,  Qian Lou (2024)<br />
- **Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning** \[[Paper](https://arxiv.org/abs/2402.08416)\] <br />
Gelei Deng,  Yi Liu,  Kailong Wang,  Yuekang Li,  Tianwei Zhang,  Yang Liu (2024)<br />
- **Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations** \[[Paper](https://arxiv.org/abs/2404.13948
   )\] <br />
Sukmin Cho,  Soyeong Jeong,  Jeongyeon Seo,  Taeho Hwang,  Jong C. Park (2024)<br />
## Application Risks
### Prompt Injection
- **Goal-guided Generative Prompt Injection Attack on Large Language Models** \[[Paper](https://arxiv.org/abs/2404.07234
   )\] <br />
Chong Zhang,  Mingyu Jin,  Qinkai Yu,  Chengzhi Liu,  Haochen Xue,  Xiaobo Jin (2024)<br />
- **PLeak: Prompt Leaking Attacks against Large Language Model Applications** \[[Paper](https://arxiv.org/abs/2405.06823
   )\] <br />
Bo Hui,  Haolin Yuan,  Neil Gong,  Philippe Burlina,  Yinzhi Cao (2024)<br />
- **Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attacks** \[[Paper](https://arxiv.org/abs/2403.09832)\] <br />
Zhifan Sun,  Antonio Valerio Miceli-Barone (2024)<br />
- **From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?** \[[Paper](https://arxiv.org/abs/2308.0199)\] <br />
Rodrigo Pedro,  Daniel Castro,  Paulo Carreira,  Nuno Santos (2023)<br />
- **Prompt Injection attack against LLM-integrated Applications** \[[Paper](https://arxiv.org/abs/2306.05499)\] <br />
Yi Liu,  Gelei Deng,  Yuekang Li,  Kailong Wang,  Zihao Wang,  Xiaofeng Wang,  Tianwei Zhang,  Yepang Liu,  Haoyu Wang,  Yan Zheng,  Yang Liu (2023)<br />
- **Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection** \[[Paper](https://arxiv.org/abs/2302.12173)\] <br />
Kai Greshake,  Sahar Abdelnabi,  Shailesh Mishra,  Christoph Endres,  Thorsten Holz,  Mario Fritz (2023)<br />
### Prompt Extraction
- **Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts** \[[Paper](https://arxiv.org/abs/2311.09127)\] <br />
Yuanwei Wu,  Xiang Li,  Yixin Liu,  Pan Zhou,  Lichao Sun (2023)<br />
- **Prompt Stealing Attacks Against Large Language Models** \[[Paper](https://arxiv.org/abs/2402.12959)\] <br />
Zeyang Sha,  Yang Zhang (2024)<br />
- **Effective Prompt Extraction from Language Models** \[[Paper](https://arxiv.org/abs/2307.06865)\] <br />
Yiming Zhang,  Nicholas Carlini,  Daphne Ippolito (2023)<br />