China

Multimodal Large Language Models (MLLMs) have become powerful and widely adopted in some practical applications. However, recent research has revealed their vulnerability to multimodal jailbreak attacks, whereby the model can be induced to generate harmful content, leading to safety risks. Although most MLLMs have undergone safety alignment, recent research shows that the visual modality is still vulnerable to jailbreak attacks. In our work, we discover that by using flowcharts with partially harmful information, MLLMs can be induced to provide additional harmful details. Based on this, we propose a jailbreak attack method based on auto-generated flowcharts, FC-Attack. Specifically, FC-Attack first fine-tunes a pre-trained LLM to create a step-description generator based on benign datasets. The generator is then used to produce step descriptions corresponding to a harmful query, which are transformed into flowcharts in 3 different shapes (vertical, horizontal, and S-shaped) as visual prompts. These flowcharts are then combined with a benign textual prompt to execute the jailbreak attack on MLLMs. Our evaluations on Advbench show that FC-Attack attains an attack success rate of up to 96% via images and up to 78% via videos across multiple MLLMs. Additionally, we investigate factors affecting the attack performance, including the number of steps and the font styles in the flowcharts. We also find that FC-Attack can improve the jailbreak performance from 4% to 28% in Claude-3.5 by changing the font style. To mitigate the attack, we explore several defenses and find that AdaShield can largely reduce the jailbreak performance but with the cost of utility drop.

EMNLP 2025

FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowcharts

mllm

jailbreak

safety

## Welcome!
"I am excited to welcome you to this year’s edition of the Conference on Empirical Methods in Natural Language Processing! Importantly, it marks the 30th edition of EMNLP. With over 8,000 submissions, more than 3,000 accepted papers, and thousands of attendees, we have come a long way from that first
workshop, which had 14 accepted papers. As the field looks ahead, Suzhou is the fitting location for celebrating this milestone: rooted in a long literary tradition, yet modern and forward-looking, and home to a large share of the NLP community."<br>

*Message from the General Chair, Dirk Hovy*





<br>

Celebrate 30 Years of EMNLP! 
EMNLP 2025 will be held in Suzhou, China from November 5th to November 9th, 2025.

poster

Rapid integration of large language models (LLMs) into societal applications has intensified concerns about their alignment with universal ethical principles, as their internal value representations remain opaque despite behavioral alignment advancements. Current approaches struggle to systematically interpret how values are encoded in neural architectures, limited by datasets that prioritize superficial judgments over mechanistic analysis. We introduce ValueLocate, a mechanistic interpretability framework grounded in the Schwartz Values Survey, to address this gap. Our method first constructs ValueInsight, a dataset that operationalizes four dimensions of universal value through behavioral contexts in the real world. Leveraging this dataset, we develop a neuron identification method that calculates activation differences between opposing value aspects, enabling precise localization of value-critical neurons without relying on computationally intensive attribution methods. Our proposed validation method demonstrates that targeted manipulation of these neurons effectively alters model value orientations, establishing causal relationships between neurons and value representations. This work advances the foundation for value alignment by bridging psychological value frameworks with neuron analysis in LLMs.

Understanding How Value Neurons Shape the Generation of Specified Values in LLMs

We address the computational cost of constructing a model map, which embeds diverse language models into a common space for comparison via KL divergence. The map relies on log-likelihoods over a large text set, making the cost proportional to the number of texts. To reduce this cost, we propose a resampling method that selects important texts with weights proportional to the variance of log-likelihoods across models for each text. Our method significantly reduces the number of required texts while preserving the accuracy of KL divergence estimates. Experiments show that it achieves comparable performance to uniform sampling with about half as many texts, and also facilitates efficient incorporation of new models into an existing map. These results enable scalable and efficient construction of language model maps.

Likelihood Variance as Text Importance for Resampling Texts to Map Language Models

With the growing use of large language models (LLMs) in social, educational, and assistive contexts, understanding and controlling their personality traits has become increasingly important. In this survey, we provide a comprehensive overview of personality modeling in LLMs, covering methods ranging from rule-based systems to prompt engineering, fine-tuning, agent and retrieval techniques, as well as approaches to multimodal setups. We examine both qualitative and quantitative evaluation protocols, and identify key challenges including subjectivity, context dependence, and limited multimodal integration. We conclude by outlining open questions and future directions for building consistent, expressive, and trustworthy persona-driven LLMs.

Modeling, Evaluating, and Embodying Personality in LLMs: A Survey

Large Language Models trained on web-scale text acquire language generation abilities that can solve a wide range of tasks, particularly when task knowledge is refined into the generative prior using in-context examples. However, spurious features learned from noisy data hinder their generalizability. Supervised fine-tuning can enhance task specificity but may lead to data inefficiency. Prior studies indicate that (i) noisy neural circuitries coexist with generalizable ones within LLMs, and (ii) finetuning typically enhances (or suppresses) existing abilities without introducing newer ones. Building upon these, we propose TaRot, a novel method for task adaptation. TaRot intervenes in the neural circuitries using learnable rotation matrices that are optimized using Bayesian optimization, on labelled samples in the order of standard few-shot prompting examples. Experiments on multiple classification and generation tasks using LLMs of varying sizes reveal the efficacy of TaRot, improving upon both zero- as well as few-shot performance, with average improvements (across models and tasks) of 15.6% and 14%, respectively.

Data-scarce Behavior Editing of Language Models

Political language is deeply intertwined with social identities. While social identities are often shaped by specific cultural contexts and expressed through particular uses of language, existing datasets for group and identity detection are predominantly English-centric, single-label and focus on coarse identity categories. We introduce HebID, the first multilabel Hebrew corpus for social identity detection: 5,536 sentences from Israeli politicians’ Facebook posts (Dec 2018–Apr 2021), manually annotated for twelve nuanced social identities (e.g., Rightist, Ultra-Orthodox, Socially-oriented) grounded by survey data. We benchmark multilabel and single-label encoders alongside 2B–9B-parameter seq2seq LLMs, finding that Hebrew-tuned LLMs provide the best results (macro-F₁ = 0.74). We apply our classifier to politicians' Facebook posts and parliamentary speeches, evaluating differences in popularity, temporal trends, clustering patterns, and gender-related variations in identity expression. We utilize identity choices from a national public survey, enabling a comparison between identities portrayed in elite discourse and the public’s identity priorities. HebID provides a comprehensive foundation for studying social identities in Hebrew and can serve as a model for similar research in other non-English political contexts.

HebID: Detecting Social Identities in Hebrew-language Political Text

Large language models are prone to generating hallucination that deviates from factual information. Existing studies mainly focus on detecting the presence of hallucinations but lack a systematic classification approach, which hinders deeper exploration of their characteristics. To address this, we introduce the concept of belief state, which quantifies the model's confidence in its own responses. We define the belief state of the model based on self-consistency, leveraging answer repetition rates to label confident and uncertain states. Based on this, we categorize factuality hallucination into two types: Overconfident Hallucination and Unaware Hallucination. Furthermore, we propose BAFH, a factuality hallucination type detection method. By training a classifier on model's hidden states, we establish a link between hidden states and belief states, enabling efficient and automatic hallucination type detection. Experimental results demonstrate the effectiveness of BAFH and the differences between hallucination types.

Bold Claims or Self-Doubt? Factuality Hallucination Type Detection via Belief State

Prompt-injection and jailbreak attacks can coerce large language models (LLMs) into revealing system prompts or producing unsafe content, threatening real-world deployments. We present Proxy Barrier (ProB), a lightweight defense that interposes a repeater proxy LLM between the user and the target model. The repeater acts to verbatim-echo benign user input, and any divergence indicates adversarial tampering, causing the request to be dropped before it reaches the target model, so that attempts to bypass safety boundaries are blocked. ProB therefore requires no access to model weights or prompts, is model-agnostic, and deployable entirely at the API level. Experiments across multiple model families demonstrate that ProB achieves state-of-the-art resilience against prompt leakage and jailbreak attacks. Notably, our approach achieves up to 98.8% improvement in defense effectiveness over baselines, and shows robust protection across both open and closed-source LLMs when suitably paired with proxy models.

Proxy Barrier: A Hidden Repeater Layer Defense Against System Prompt Leakage and Jailbreaking

We introduce AraSafe, the first native Arabic large-scale safety benchmark for large language models (LLMs), addressing the pressing need for culturally and linguistically representative evaluation resources. The dataset consists of 12K naturally occurring, human-written Arabic prompts spanning diverse domains, such as linguistics, social studies, sciences, and safety. Each prompt was independently annotated by two expert annotators into one of nine fine-grained safety categories, including 'Illegal Activities', 'Violence or Harm', 'Privacy Violation', and 'Hate Speech'. To enrich the representation of harmful content, we augmented the dataset with 12K synthetic harmful prompts generated using GPT-4o via carefully designed prompt engineering techniques. We benchmark a number of Arabic-centric and multilingual models in the 7–13B parameter range, including Jais, AceGPT, Allam, Fanar, Llama-3, Gemma-2, and Qwen3, as well as BERT-based fine-tuned models. GPT-4o was used as an upper-bound reference baseline. Our evaluation reveals critical safety blind spots in Arabic LLMs and underscores the necessity of localized, culturally grounded benchmarks for building responsible AI systems.

AraSafe: Benchmarking Safety in Arabic LLMs

Recently, advancements in AI counseling based on large language models have shown significant progress. However, existing studies employ a one-time generation approach to synthesize multi-turn dialogue samples, resulting in low therapy fidelity and a lack of deeper understanding of the therapeutic logic underlying each response turn. In this work, we propose a novel data synthesis framework, CATCH, designed to address the challenges above. Specifically, to improve therapy fidelity, we introduce the Progressive Dialogue Synthesis strategy to systematically derive key elements from the client's self-report and organize them into structured outlines, thereby generating a counseling dialogue dataset aligned with therapeutic intervention principles. To enhance therapy logic, we propose the Memory-Driven Dynamic Planning thinking pattern, which clarifies decision-making motivations for each dialogue turn. This pattern incorporates memory enhancement, global planning, and strategy reasoning, enabling the development of a collaborative multi-agent iterative optimization method that synthesizes complete chains of thought in counseling dialogues. Extensive experiments demonstrate that CATCH significantly enhances the fidelity and logical coherence in AI counseling.

CATCH: A Novel Data Synthesis Framework for High Therapy Fidelity and Memory-Driven Planning Chain of Thought in AI Counseling

The rapid advancement of multimodal large language models (MLLMs) has significantly enhanced performance across benchmarks. However, data contamination — partial/entire benchmark data is included in the model's training set — poses critical challenges for fair evaluation. Existing detection methods for unimodal large language models (LLMs) are inadequate for MLLMs due to multimodal data complexity and multi-phase training. We systematically analyze multimodal data contamination using our analytical framework, MM-DETECT, which defines two contamination categories — unimodal and cross-modal — and effectively quantifies contamination severity across multiple-choice and caption-based Visual Question Answering tasks. Evaluations on twelve MLLMs and five benchmarks reveal significant contamination, particularly in proprietary models and older benchmarks. Crucially, contamination sometimes originates during unimodal pre-training rather than solely from multimodal fine-tuning. Our insights refine contamination understanding, guiding evaluation practices and improving multimodal model reliability.

Downloads

Next from EMNLP 2025

Understanding How Value Neurons Shape the Generation of Specified Values in LLMs

Similar lecture

Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience