United States

In continuous domains, reinforcement learning (RL) policies are often based on Gaussian distributions due to their natural shape and mathematical advantages. However, many continuous control tasks impose physical constraints that limit actions, leading to a bias in Gaussian policies, where the unbounded nature of their support causes a tendency toward boundary actions. This &quot;boundary action bias&quot; negatively impacts the performance of widely-used algorithms like Proximal Policy Optimization (PPO). Despite the prevalence of this bias, it has been overlooked in existing research and applications. In this paper, we aim to address this issue by conducting a thorough investigation into the nature and effects of boundary action bias. Through theoretical analysis and extensive experiments, we uncover the underlying dynamics of this bias in policy training. Our findings reveal that boundary actions have dual effects on performance, highlighting the importance of balancing the policy distribution. To address this challenge, we propose a truncated Gaussian policy that retains most of the favorable properties of Gaussian distributions while mitigating the bias. Additionally, we introduce a scale-adjustment technique to achieve a balanced distribution with respect to boundary actions. Experimental results on various continuous control tasks in MuJoCo demonstrate the superiority of our proposed method over conventional Gaussian policies and other approaches designed to counteract this bias.

AAAI 2025

Truncated Gaussian Policy for Debiased Continuous Control

reinforcement learning

The purpose of the AAAI conference is to promote research in artificial intelligence (AI) and scientific exchange among AI researchers, practitioners, scientists, and engineers in affiliated disciplines. 

AAAI 2021

In continuous domains, reinforcement learning (RL) policies are often based on Gaussian distributions due to their natural shape and mathematical advantages. However, many continuous control tasks impose physical constraints that limit actions, leading to a bias in Gaussian policies, where the unbounded nature of their support causes a tendency toward boundary actions. This "boundary action bias" negatively impacts the performance of widely-used algorithms like Proximal Policy Optimization (PPO). Despite the prevalence of this bias, it has been overlooked in existing research and applications. In this paper, we aim to address this issue by conducting a thorough investigation into the nature and effects of boundary action bias. Through theoretical analysis and extensive experiments, we uncover the underlying dynamics of this bias in policy training. Our findings reveal that boundary actions have dual effects on performance, highlighting the importance of balancing the policy distribution. To address this challenge, we propose a truncated Gaussian policy that retains most of the favorable properties of Gaussian distributions while mitigating the bias. Additionally, we introduce a scale-adjustment technique to achieve a balanced distribution with respect to boundary actions. Experimental results on various continuous control tasks in MuJoCo demonstrate the superiority of our proposed method over conventional Gaussian policies and other approaches designed to counteract this bias.

technical paper

We are pleased to announce the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI-25), which will be held in Philadelphia, Pennsylvania at the Pennsylvania Convention Center from February 25 to March 4, 2025.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

### [Invited Speakers](https://aaai.org/conference/aaai/aaai-25/aaai-25-invited-speakers/)

Register [here](https://aaai.org/conference/aaai/aaai-25/registration/)

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.



We study a Bayesian persuasion problem with externalities. In this model, a principal sends signals to inform multiple agents about the state of the world. Simultaneously, due to the existence of externalities in the agents' utilities, the principal also acts as a correlation device to correlate the agents' actions. We consider the setting where the agents are categorized into a small number of types. Agents of the same type share identical utility functions and are treated equitably in the utility functions of both other agents and the principal. We study the problem of computing optimal signaling strategies for the principal, under three different types of signaling channels: public, private, and semi-private.  Our results include revelation-principle-style characterizations of optimal signaling strategies, linear programming formulations, and analysis of in/tractability of the optimization problems. It is demonstrated that when the maximum number of deviating agents is bounded by a constant, our LP-based formulations compute optimal signaling strategies in polynomial time. Otherwise, the problems are NP-hard.

Bayesian Persuasion with Externalities: Exploiting Agent Types

Text-to-Image generation (TTI) technologies are advancing rapidly, especially in the English language communities. However, apart from the user input language barrier problem, English-native TTI models inherently carry biases from their English world centric training data, which creates a dilemma for development of other language-native TTI models. One common choice is to fine-tune the English-native TTI model with translated samples. It falls short of fully addressing the model bias problem. Alternatively, training non-English language native models from scratch can effectively resolve the English world bias, but model trained this way would diverge from the English TTI communities, thus not able to utilize the strides continuously gaining in the English TTI communities any more. To build Chinese TTI model meanwhile keep compatibility with the English TTI communities, we propose a novel model structure referred as "Bridge Diffusion Model" (BDM). The proposed BDM employs a backbone-branch network structure to learn the Chinese semantics while keep the latent space compatible with the English-native TTI backbone, in an end-to-end manner. The unique advantages of the proposed BDM are that it's not only adept at generating images that precisely depict Chinese semantics, but also compatible with various English-native TTI plugins, such as different checkpoints, LoRA, ControlNet, Dreambooth, and Textual Inversion, \etc. Moreover, BDM can concurrently generate content seamlessly combining both Chinese-native and English-native semantics within a single image, fostering cultural interaction.

Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communities

Artificial intelligence (AI) models for computer vision trained with supervised machine learning are assumed to solve classification tasks by imitating human behavior learned from training labels. Most efforts in recent vision research focus on measuring the model task performance using standardized benchmarks such as accuracy. However limited work has sought to understand the perceptual difference between humans and machines. To fill this gap, this study first analyzes the statistical distributions of mistakes from the two sources, and then explores how task difficulty level affects these distributions. We find that even when AI learns an excellent model from the training data, one that outperforms humans in overall accuracy, these AI models have significant and consistent differences from human perception. We demonstrate the importance of studying these differences with a simple human-AI teaming algorithm that outperforms humans alone, AI alone, or AI-AI teaming.

Human and AI Perceptual Differences in Image Classification Errors

Traffic prediction is pivotal in intelligent transportation systems. Existing works mainly focus on improving overall accuracy, overlooking a crucial problem of whether prediction results will lead to biased decisions by transportation authorities. In practice, the uneven deployment of traffic sensors in different urban areas produces imbalanced data, making the traffic prediction model fail in some areas and leading to unfair regional decision-making that eventually severely affects residents' equity and quality of life. Additionally, existing fairness machine learning models fail to preserve fair traffic prediction for a prolonged time. Although they can achieve fairness at certain time points, such static fairness will be broken as the traffic conditions change. To fill this research gap, we investigate prolonged fair traffic prediction, introduce two novel fairness definitions tailored to dynamic traffic scenarios, and propose a prolonged fairness traffic prediction framework, namely FairTP.  We argue that fairness in traffic scenarios changes dynamically over time and across areas. The prediction accuracy of each traffic sensor or city area should take turns to "sacrifice" or "benefit" in order for the model to achieve fairness, eventually achieving prolonged fairness when such "sacrifice" and "benefit" are similar within a period of time. Accordingly, we first defined region-based static fairness and sensor-based dynamic fairness. Next, we designed a state identification module in FairTP to discriminate between states of "sacrifice" or "benefit" to enable prolonged fairness-aware traffic predictions. Lastly, a state-guided balanced sampling strategy is designed to select training examples to promote prediction fairness further, mitigating the performance disparities among regions with imbalanced traffic sensors. Extensive experiments in two real-world datasets show that FairTP significantly improves prediction fairness without causing much accuracy degradation.

FairTP: A Prolonged Fairness Framework for Traffic Prediction

Large Language Models (LLMs) have made significant strides in various intelligent tasks but still struggle with complex action reasoning tasks that require systematic search. To address this limitation, we introduce a method that bridges the natural language understanding capabilities of LLMs with the symbolic reasoning capabilities of action languages---a formal language for reasoning about actions. Our approach, termed  LLM+AL, leverages the LLM's strengths in semantic parsing and commonsense knowledge generation alongside the action language's expertise in automated reasoning based on encoded knowledge. We compare LLM+AL against state-of-the-art LLMs, including ChatGPT-4, Claude 3 Opus, and Gemini Ultra 1.0, as well as other related works that use LLMs as a semantic parser, using benchmarks for complex reasoning about actions. Our findings indicate that while all methods exhibit various errors, LLM+AL, with relatively simple human corrections, consistently leads to correct answers, whereas using LLMs alone does not yield improvements even after human intervention. LLM+AL also contributes to automated generation of action languages.

LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning About Actions

Medical multimodal large language models (MLLMs) are becoming an instrumental part of healthcare systems, assisting medical personnel with decision making and results analysis. Models for radiology report generation are able to interpret medical imagery, thus reducing the workload of radiologists. As medical data is scarce and protected by privacy regulations, medical MLLMs represent valuable intellectual property. However, these assets are potentially vulnerable to model stealing, where attackers aim to replicate their functionality via black-box access. So far, model stealing for the medical domain has focused on image classification; however, existing attacks are not effective against MLLMs. In this paper, we introduce Adversarial Domain Alignment (ADA-Steal), the first stealing attack against medical MLLMs. ADA-Steal relies on natural images, which are public and widely available, as opposed to their medical counterparts. We show that data augmentation with adversarial noise is sufficient to overcome the data distribution gap between natural images and the domain-specific distribution of the victim MLLM. Experiments on the IU X-RAY and MIMIC-CXR radiology datasets demonstrate that Adversarial Domain Alignment enables attackers to steal the medical MLLM without any access to medical data.

Medical Multimodal Model Stealing Attacks via Adversarial Domain Alignment

Large Language Models (LLMs) excel in linguistic tasks but struggle with mathematical reasoning, particularly in non- English languages like Hindi. This research aims to en- hance the mathematical reasoning skills of smaller, resource- efficient open-source LLMs in both Hindi and English. We evaluate models like OpenHathi 7B, LLaMA-2 7B, Wizard- Math 7B, Mistral 7B, LLeMMa 7B, MAmmoTH 7B, Gemini Pro, and GPT-4 using zero-shot, few-shot chain-of-thought (CoT) methods, and supervised fine-tuning. Our approach in- corporates curriculum learning, progressively training mod- els on increasingly difficult problems, a novel Decompo- sition Strategy to simplify complex arithmetic operations, and a Structured Solution Design that divides solutions into phases. Our experiments result in notable performance en- hancements. WizardMath 7B exceeds Gemini’s accuracy on English datasets by +6% and matches Gemini’s performance on Hindi datasets. Adopting a bilingual approach that com- bines English and Hindi samples achieves results comparable to individual language models, demonstrating the capability to learn mathematical reasoning in both languages. This re- search highlights the potential for improving mathematical reasoning in open-source LLMs.

Multilingual Mathematical Reasoning: Advancing Open-Source LLMs in Hindi and English

User data spread across multiple modalities has popularized multi-modal recommender systems (MMRS). They recommend diverse content such as products, social media posts, TikTok reels, etc., based on a user-item interaction graph. With rising data privacy demands, recent methods propose unlearning private $\textbf{user data}$ from uni-modal recommender systems (RS). However, methods for unlearning $\textbf{item data}$ related to outdated user preferences, revoked licenses, and legally requested removals are still largely unexplored.

Previous RS unlearning methods are unsuitable for MMRS due to the incompatibility of their matrix-based representation with the multi-modal user-item interaction graph. Moreover, their data partitioning step degrades performance on each shard due to poor data heterogeneity and requires costly performance aggregation across shards.

This paper introduces DynMMRec, the first approach known to us for unlearning in MMRS and unlearning item data. Given a trained RS model, DynMMRec employs a novel Reverse Bayesian Personalized Ranking (BPR) objective to enable the model to forget marked data. The reverse BPR attenuates the impact of user-item interactions within the $\textit{forget}$ set, while the forward BPR reinforces the significance of user-item interactions within the $\textit{retain}$ set.
Our experiments demonstrate that DynMMRec outperforms baseline methods across various unlearning requests when evaluated on benchmark MMRS datasets. DynMMRec achieves recall performance improvements of up to $\mathbf{49.85}$% compared to baseline methods and is up to $\mathbf{1.3\times}$ faster than the Gold model, which is trained on $\textit{retain}$ set from scratch. DynMMRec offers significant advantages, including superiority in removing target interactions, preserving retained interactions, and zero overhead costs compared to previous methods. The code will be released after review.

Multi-Modal Recommendation Unlearning for Legal, Licensing, and Modality Constraints

Recent advancements in text-to-speech and speech conversion technologies have enabled the creation of highly convincing synthetic speeches. While these innovations offer numerous practical benefits, they also cause significant security challenges when maliciously misused. Therefore, there is an urgent need to detect these synthetic speeches. Phoneme features provide a powerful speech representation for deepfake detection. However, previous phoneme-based detection approaches typically focused on specific phonemes, overlooking temporal inconsistencies across the entire phoneme sequence. In this paper, we develop a new mechanism for detecting speech deepfakes by identifying the inconsistencies of phoneme-level speech features. We design an adaptive phoneme pooling technique that extracts sample-specific phoneme-level features from frame-level speech data. By applying this technique to features extracted by pre-trained audio models on previously unseen deepfake datasets, we demonstrate that deepfake samples often exhibit phoneme-level inconsistencies when compared to genuine speech. To further enhance detection accuracy, we propose a deepfake detector that uses a graph attention network to model the temporal dependencies of phoneme-level features. Additionally, we introduce a random phoneme substitution augmentation technique to increase feature diversity during training. Extensive experiments on four benchmark datasets demonstrate the superior performance of our method over existing state-of-the-art detection methods.

Phoneme-Level Feature Discrepancies: A Key to Detecting Sophisticated Speech Deepfakes

To build a responsible data economy and protect data ownerhip, it is crucial to enable learning models from separate, heterogeneous data sources without centralization. For example, federated learning (FL) aims to train models across massive remote devices or isolated organizations, while keeping user data local. However, federated learning can face critical practical issues such as scalability, noisy samples, biased learning systems or procedures, and privacy leakage. At the intersection between optimization, trustworthy (fair, robust, and private) ML, and learning in heterogeneous environments, my research aims to support scalable and responsible data sharing to collectively build intelligent models.

Downloads

Next from AAAI 2025

Bayesian Persuasion with Externalities: Exploiting Agent Types

Similar lecture

Conceptualized and Contextualized Gaussian Embedding