Can I read You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense on EtoBox?

You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense by Mai, Wuyuao; Hong, Geng; Chen, Pei; Pan, Xudong; Liu, Baojun; Zhang, Yuan; Duan, Haixin; Yang, Min is a scholarly article available to read on EtoBox.

What is You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense about?

With the rise of generative large language models (LLMs) like LLaMA and ChatGPT, these models have significantly transformed daily life and work by providing advanced insights. However, as jailbreak attacks continue to circumvent built-in safety mechanisms, exploiting carefully crafted scenarios or tokens, the safety risks of LLMs have come into focus. While numerous defense strategies--such as prompt detection, modification, and model fine-tuning--have been proposed to counter these attacks, a critical question arises: do these defenses compromise the utility and usability of LLMs for legitimate users? Existing research predominantly focuses on the effectiveness of defense strategies without thoroughly examining their impact on performance, leaving a gap in understanding the trade-offs between LLM safety and performance. Our research addresses this gap by conducting a comprehensive study on the utility degradation, safety elevation, and exaggerated-safety escalation of LLMs with jailbreak defense strategies. We propose USEBench, a novel benchmark designed to evaluate these aspects, along with USEIndex, a comprehensive metric for assessing overall model performance. Through experim

Author

Mai, Wuyuao; Hong, Geng; Chen, Pei; Pan, Xudong; Liu, Baojun; Zhang, Yuan; Duan, Haixin; Yang, Min

Published

2025

Language

EN