United States

Natural language is the most intuitive means for humans to interact with robots, making task planning based on natural language commands a longstanding area of research. Large language models (LLMs) have significantly improved task planning by enhancing understanding of language and common sense. However, current methods still face several challenges: they lack a deep understanding of physical environments, their performance relies heavily on prompt examples, LLMs are oversized and not customized for specific tasks, and the planning costs remain high. 
To overcome these issues, we introduce the GNN-Transformer Task Planner (GTTP), designed to predict task-level actions by leveraging the semantic environment and incorporating historical state data. The GTTP architecture is scalable through the use of GNN layers, while transformer layers facilitate understanding task progression. In addition, our model uses a text encoder to embed environments, allowing it to be trained on simulated datasets and applied directly in real-world scenarios. We also propose an automated data generation method that includes semantic augmentation, planning verification, and instruction generation via LLM. This method enables the collection of 14$k$ instruction-annotated tasks in the VirtualHome environment with minimal human effort. The model has been validated across diverse scenes containing up to 715 objects, achieving significantly higher success rates compared to baseline models. It has also been successfully deployed on a physical mobile manipulator, demonstrating its practical applicability and effectiveness. Our datasets and code will be made publicly available.

AAAI 2025

GNN-Transformer Task Planning Enhanced with Semantic-Driven Data Augmentation

other foundations of intelligent robots

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

Please register!

ACL 2024

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

technical paper

We are pleased to announce the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI-25), which will be held in Philadelphia, Pennsylvania at the Pennsylvania Convention Center from February 25 to March 4, 2025.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

### [Invited Speakers](https://aaai.org/conference/aaai/aaai-25/aaai-25-invited-speakers/)

Register [here](https://aaai.org/conference/aaai/aaai-25/registration/)

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.



We study a Bayesian persuasion problem with externalities. In this model, a principal sends signals to inform multiple agents about the state of the world. Simultaneously, due to the existence of externalities in the agents' utilities, the principal also acts as a correlation device to correlate the agents' actions. We consider the setting where the agents are categorized into a small number of types. Agents of the same type share identical utility functions and are treated equitably in the utility functions of both other agents and the principal. We study the problem of computing optimal signaling strategies for the principal, under three different types of signaling channels: public, private, and semi-private.  Our results include revelation-principle-style characterizations of optimal signaling strategies, linear programming formulations, and analysis of in/tractability of the optimization problems. It is demonstrated that when the maximum number of deviating agents is bounded by a constant, our LP-based formulations compute optimal signaling strategies in polynomial time. Otherwise, the problems are NP-hard.

Bayesian Persuasion with Externalities: Exploiting Agent Types

Text-to-Image generation (TTI) technologies are advancing rapidly, especially in the English language communities. However, apart from the user input language barrier problem, English-native TTI models inherently carry biases from their English world centric training data, which creates a dilemma for development of other language-native TTI models. One common choice is to fine-tune the English-native TTI model with translated samples. It falls short of fully addressing the model bias problem. Alternatively, training non-English language native models from scratch can effectively resolve the English world bias, but model trained this way would diverge from the English TTI communities, thus not able to utilize the strides continuously gaining in the English TTI communities any more. To build Chinese TTI model meanwhile keep compatibility with the English TTI communities, we propose a novel model structure referred as "Bridge Diffusion Model" (BDM). The proposed BDM employs a backbone-branch network structure to learn the Chinese semantics while keep the latent space compatible with the English-native TTI backbone, in an end-to-end manner. The unique advantages of the proposed BDM are that it's not only adept at generating images that precisely depict Chinese semantics, but also compatible with various English-native TTI plugins, such as different checkpoints, LoRA, ControlNet, Dreambooth, and Textual Inversion, \etc. Moreover, BDM can concurrently generate content seamlessly combining both Chinese-native and English-native semantics within a single image, fostering cultural interaction.

Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communities

In this paper, we focus on estimating the causal effect of an intervention over time on a dynamical system. 
To that end, we formally define causal interventions and their effects over time on discrete-time stochastic processes (DSPs). 
Then, we show under which conditions the equilibrium states of a DSP, both before and after a causal intervention, can be captured by a structural causal model (SCM). 
With such an equivalence at hand, we provide an explicit mapping from vector autoregressive models (VARs), broadly applied in econometrics, to linear, but potentially cyclic and/or affected by unmeasured confounders, SCMs. The resulting \emph{causal VAR framework} allows us to perform \emph{causal inference over time} from observational time series data.  
Our experiments on synthetic and real-world datasets show that the proposed framework achieves strong performance in terms of observational forecasting while enabling accurate estimation of the causal effect of interventions on dynamical systems. 
We demonstrate, through a case study, the potential practical questions that can be addressed using the proposed causal VAR framework.

A Practical Approach to Causal Inference over Time

Artificial intelligence (AI) models for computer vision trained with supervised machine learning are assumed to solve classification tasks by imitating human behavior learned from training labels. Most efforts in recent vision research focus on measuring the model task performance using standardized benchmarks such as accuracy. However limited work has sought to understand the perceptual difference between humans and machines. To fill this gap, this study first analyzes the statistical distributions of mistakes from the two sources, and then explores how task difficulty level affects these distributions. We find that even when AI learns an excellent model from the training data, one that outperforms humans in overall accuracy, these AI models have significant and consistent differences from human perception. We demonstrate the importance of studying these differences with a simple human-AI teaming algorithm that outperforms humans alone, AI alone, or AI-AI teaming.

Human and AI Perceptual Differences in Image Classification Errors

Emotion Recognition in Conversations (ERC) refers to the task of automatically identifying the emotion of each utterance in conversations. Unfortunately, because the emotion of an utterance is influenced by the contextual information of the conversation, to label each utterance in ERC, the annotators are required to repeatedly read the whole conversation, making the fine-grained labeling process very costly. In this work, we suggest an efficient coarse-grained labeling pattern for ERC, where it asks annotators to assign a set of emotions for each conversation. The empirical results of our preliminary evaluations indicate the accuracy and efficiency of this conversation-level labeling pattern. Accordingly, we suggest that inducing ERC predictors with conversation-level emotion sets in a weakly-supervised learning manner can be a potential candidate for ERC, and we formally refer to this new weakly-supervised learning task as Dataless ERC (DERC). To validate its potential, we propose a simple-yet-flexible DERC framework, namely DERC with Progressive Learning (DERC-PL). We jointly update pseudo-utterance-level emotions and the ERC predictor in a self-training manner, and in particular, progressively update the ERC predictor from training subsets with lower noise densities to the ones with higher noise densities. We specify several versions of DERC-PL by applying various off-the-shelf ERC methods. Empirical results indicate that DERC-PL can be on par with existing weakly-supervised learning baselines and supervised learning ERC methods.

Utterance-level Emotion Recognition in Conversation with Conversation-level Supervision

Emotion recognition in conversation (ERC) has been promoted with diverse approaches in the recent years. However, many studies have pointed out that emotion shift and confusing labels make it difficult for models to distinguish between different emotions. Existing ERC models suffer from these problems when the emotions are forced to be mapped into single label. In this paper, we propose a multi-label classification framework for emotion recognition in conversation (ML-ERC).
Specifically, we introduce weighted supervised contrastive learning tailored for multi-label, which can also extend previous ERC models. 
Experimental results on three benchmark datasets show that the proposed approach effectively handles multiple emotions. Our model achieves improved results for identifying emotions in the most challenging settings in emotion shift and confusing labels.

Beyond Single Emotion: Multi-label Approach to Conversational Emotion Recognition

In continuous domains, reinforcement learning (RL) policies are often based on Gaussian distributions due to their natural shape and mathematical advantages. However, many continuous control tasks impose physical constraints that limit actions, leading to a bias in Gaussian policies, where the unbounded nature of their support causes a tendency toward boundary actions. This "boundary action bias" negatively impacts the performance of widely-used algorithms like Proximal Policy Optimization (PPO). Despite the prevalence of this bias, it has been overlooked in existing research and applications. In this paper, we aim to address this issue by conducting a thorough investigation into the nature and effects of boundary action bias. Through theoretical analysis and extensive experiments, we uncover the underlying dynamics of this bias in policy training. Our findings reveal that boundary actions have dual effects on performance, highlighting the importance of balancing the policy distribution. To address this challenge, we propose a truncated Gaussian policy that retains most of the favorable properties of Gaussian distributions while mitigating the bias. Additionally, we introduce a scale-adjustment technique to achieve a balanced distribution with respect to boundary actions. Experimental results on various continuous control tasks in MuJoCo demonstrate the superiority of our proposed method over conventional Gaussian policies and other approaches designed to counteract this bias.

Truncated Gaussian Policy for Debiased Continuous Control

User data spread across multiple modalities has popularized multi-modal recommender systems (MMRS). They recommend diverse content such as products, social media posts, TikTok reels, etc., based on a user-item interaction graph. With rising data privacy demands, recent methods propose unlearning private $\textbf{user data}$ from uni-modal recommender systems (RS). However, methods for unlearning $\textbf{item data}$ related to outdated user preferences, revoked licenses, and legally requested removals are still largely unexplored.

Previous RS unlearning methods are unsuitable for MMRS due to the incompatibility of their matrix-based representation with the multi-modal user-item interaction graph. Moreover, their data partitioning step degrades performance on each shard due to poor data heterogeneity and requires costly performance aggregation across shards.

This paper introduces DynMMRec, the first approach known to us for unlearning in MMRS and unlearning item data. Given a trained RS model, DynMMRec employs a novel Reverse Bayesian Personalized Ranking (BPR) objective to enable the model to forget marked data. The reverse BPR attenuates the impact of user-item interactions within the $\textit{forget}$ set, while the forward BPR reinforces the significance of user-item interactions within the $\textit{retain}$ set.
Our experiments demonstrate that DynMMRec outperforms baseline methods across various unlearning requests when evaluated on benchmark MMRS datasets. DynMMRec achieves recall performance improvements of up to $\mathbf{49.85}$% compared to baseline methods and is up to $\mathbf{1.3\times}$ faster than the Gold model, which is trained on $\textit{retain}$ set from scratch. DynMMRec offers significant advantages, including superiority in removing target interactions, preserving retained interactions, and zero overhead costs compared to previous methods. The code will be released after review.

Multi-Modal Recommendation Unlearning for Legal, Licensing, and Modality Constraints

Recent advancements in text-to-speech and speech conversion technologies have enabled the creation of highly convincing synthetic speeches. While these innovations offer numerous practical benefits, they also cause significant security challenges when maliciously misused. Therefore, there is an urgent need to detect these synthetic speeches. Phoneme features provide a powerful speech representation for deepfake detection. However, previous phoneme-based detection approaches typically focused on specific phonemes, overlooking temporal inconsistencies across the entire phoneme sequence. In this paper, we develop a new mechanism for detecting speech deepfakes by identifying the inconsistencies of phoneme-level speech features. We design an adaptive phoneme pooling technique that extracts sample-specific phoneme-level features from frame-level speech data. By applying this technique to features extracted by pre-trained audio models on previously unseen deepfake datasets, we demonstrate that deepfake samples often exhibit phoneme-level inconsistencies when compared to genuine speech. To further enhance detection accuracy, we propose a deepfake detector that uses a graph attention network to model the temporal dependencies of phoneme-level features. Additionally, we introduce a random phoneme substitution augmentation technique to increase feature diversity during training. Extensive experiments on four benchmark datasets demonstrate the superior performance of our method over existing state-of-the-art detection methods.

Phoneme-Level Feature Discrepancies: A Key to Detecting Sophisticated Speech Deepfakes

To build a responsible data economy and protect data ownerhip, it is crucial to enable learning models from separate, heterogeneous data sources without centralization. For example, federated learning (FL) aims to train models across massive remote devices or isolated organizations, while keeping user data local. However, federated learning can face critical practical issues such as scalability, noisy samples, biased learning systems or procedures, and privacy leakage. At the intersection between optimization, trustworthy (fair, robust, and private) ML, and learning in heterogeneous environments, my research aims to support scalable and responsible data sharing to collectively build intelligent models.

Downloads

Next from AAAI 2025

Bayesian Persuasion with Externalities: Exploiting Agent Types

Similar lecture

Open Grounded Planning: Challenges and Benchmark Construction