Large language models motivate an approach to meaning known as distributional semantics, that words mean what they do because of how they’re distributed in language. In this talk I will describe some evidence from my lab that suggests metrics from large language models do a good job of predicting behavioral and neural responses to some aspects of human language. I go on to describe some research that highlights important differences in meaning processing in humans and the ‘understanding’ displayed by language models. Discrepancies are particularly noteworthy in studies of joke comprehension.

Distributional Semantics: What do large language models have to say?

How can Large Language Models (LLMs) and modern NLP be used to increase the use and the documentation of Indigenous languages which are in danger of disappearing? First, I report on the development of high-quality translators for Indigenous languages by fine-tuning SOTA machine translators with tiny amounts of data, and discuss how to avoid some common pitfalls. Next, I present prototypes built with Indigenous communities aiming to stimulate and facilitate writing, using LLM models to create spell-checkers, next-word predictors, and similar tools. Finally, I discuss a future for documentation where dying languages are preserved as interactive language models.

Harnessing the Power of LLMs to Vitalize Indigenous Languages

**Title:** Distributional Semantics: What do large language models have to say? **Abstract:** Large language models motivate an approach to meaning known as distributional semantics, that words mean what they do because of how they’re distributed in language. In this talk I will describe some evidence from my lab that suggests metrics from large language models do a good job of predicting behavioral and neural responses to some aspects of human language. I go on to describe some research that highlights important differences in meaning processing in humans and the ‘understanding’ displayed by language models. Discrepancies are particularly noteworthy in studies of joke comprehension.

Keynote Speaker: Seana Coulson

keynote

## Welcome to 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics!
This year, the conference is in Mexico City. NAACL was actually already planned for Mexico City in 2021, but due to the pandemic the entire conference was moved online. This year, finally, we get to go! So it is my sincere pleasure to welcome you to Mexico City, whether in person or virtually. Having the conference in Mexico City is a good opportunity to emphasize that NAACL is our flagship conference for ACL members not only in North America but also in Central and South America, even though NAACL has been bearing “North” in its name. At this year’s conference, we have a theme to match, with a theme track on the Languages of Latin America to showcase the linguistic diversity of the region.
 
The opportunity to present at NAACL should not depend on a researcher’s travel budget, or their family status. This is why it is so important to make virtual participation at NAACL as good an experience as possible – but we want to also provide a good experience for in-person participants. As a community, we are still working out the best way to do that. This year at NAACL, we are trying out a big virtual poster session ahead of the conference, with the hope that this will make make for a lively and interactive experience. At the same time, we are reducing virtual oral presentations, which seem to be particularly tricky to make to work well. A big thanks to the NAACL program chairs and to Luciana Benotti for all their ideas and work to improve the virtual experience. And participants, virtual as well as in-person: Please let us know what worked for you and what didn’t, so we can continue to improve hybrid conferences. 
 
I have been lucky to work with many amazing people. Without their insight, dedication and patience, and without the many hours of work they put in, NAACL would not have been possible. A huge thank you to the program chairs Helena Gomez, Kevin Duh, and Steve Bethard – you are the best! 
 
Finally, I would like to thank all authors, invited speakers and panelists, area chairs and reviewers, the volunteers organizing and chairing sessions, and all attendees, in-person and virtual. Thank you for helping us make NAACL 2024 come to life. 
 
Welcome and hope you all enjoy the conference! 
Katrin Erk 
The University of Texas at Austin 
NAACL-2024 General Chair 
*You can read the full Welcome message in the [Conference Handbook (downloadable)](https://drive.google.com/file/d/1H1NvW0VASQjkSCYgw3mr-3l4yYDSxz6B/view?usp=sharing)*

To access the event page you need to register [**here.**](http://acl.swoogo.com/naacl2024) 
Your access to the event page is limited based on your registration type. If you registered for workshops only, you will gain access to full workshops content on the day of the workshops program.

Please register!

NAACL 2024

2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics

**Title:** Harnessing the Power of LLMs to Vitalize Indigenous Languages **Abstract:** How can Large Language Models (LLMs) and modern NLP be used to increase the use and the documentation of Indigenous languages which are in danger of disappearing? First, I report on the development of high-quality translators for Indigenous languages by fine-tuning SOTA machine translators with tiny amounts of data, and discuss how to avoid some common pitfalls. Next, I present prototypes built with Indigenous communities aiming to stimulate and facilitate writing, using LLM models to create spell-checkers, next-word predictors, and similar tools. Finally, I discuss a future for documentation where dying languages are preserved as interactive language models.

Keynote Speaker: Claudio Santos Pinhanez

The convergence of technologies—from foundation AI models to diverse sensors and actuators to ubiquitous connectivity—is transforming the nature of interactions in the physical and digital world. People have accelerated their collaborative connections and productivity through digital and immersive technologies, no longer limited by geography or language or access. Humans have also leveraged and interacted with AI in many different forms, with the advent of hyperscale AI models (i.e., large language models) forever changing (and at an ever-astonishing pace) the nature of human-AI teams, realized in this era of the AI “copilot.” Similarly, robotics and automation technologies now afford greater opportunities to work with and/or near humans, allowing for increasingly collaborative physical robots to dramatically impact real-world activities. It is the compounding effect of enabling all three capabilities, each complementary to one another in valuable ways, and we envision the triad formed by human-robot-AI teams as revolutionizing the future of society, the economy, and of technology.

Catalyzing the Future of Collaborative Human-Robot-AI Teaming

From "Chips War" to "Chiplets War"

Keynote Talk: Catalyzing the Future of Collaborative Human-Robot-AI Teaming

### Welcome to the 9th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. 
 Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications
* To connect machine learning researchers with experts in military requirements
* To build and strengthen collaborations within the defense research community

#### *"What we do as a Navy is a human business. Don’t pass up an opportunity to come together as a team. Don’t pass up leadership opportunities, and don’t outsource them to a PowerPoint. If you have an opportunity to come together as a team, to increase belonging, do it."*
Admiral John M. Richardson, 31st Chief of Naval Operations

To access this event, you need to register [here](https://naml2025.paperform.co/).

NAML 2025

The 9th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.

Keynote Talk: From "Chips War" to "Chiplets War"

Hateful memes detection is a challenging multimodal understanding task that requires comprehensive learning of vision, language, and cross-modal interactions. Previous research has focused on developing effective fusion strategies for integrating hate information from different modalities. However, these methods excessively rely on cross-modal fusion features, ignoring the modality uncertainty caused by the contribution degree of each modality to hate sentiment and the modality imbalance caused by the dominant modality suppressing the optimization of another modality. To this end, this paper proposes an Uncertainty-guided Modal Rebalance (UMR) framework for hateful memes detection. The uncertainty of each meme is explicitly formulated by designing stochastic representation drawn from a Gaussian distribution for aggregating cross-modal features with unimodal features adaptively. The modality imbalance is alleviated by improving cosine loss from the perspectives of inter-modal feature and weight vectors constraints. In this way, the suppressed unimodal representation ability in multimodal models would be unleashed, while the learning of modality contribution would be further promoted. Extensive experimental results demonstrate that the proposed UMR produces the state-of-the-art performance on four widely-used datasets.

Uncertainty-Guided Modal Rebalance for Hateful Memes Detection

Transformer-based methods have gone mainstream in multimodal sequential learning. The intra and inter modality interactions are captured by the query-key associations of multi-head attention. In this way, the calculated multimodal contexts (attentional results) are expected to be relevant to the query modality. However, in existing literature, the alignment degree between different calculated attentional results of the same query are under-explored. Based on this concern, we propose a new constrained scheme called Multimodal Contextual Contrast (MCC), which could align the multiple attentional results from both local and global perspectives, making the information capture more efficient. Concretely, the calculated attentional results of different modalities are mapped into a common feature space, those attentional vectors with the same query are considered as a positive group and the remaining sets are negative. From local perspective, we sample the negative groups for a positive group by randomly changing the sequential step of one specific context and keeping the other stay the same. From coarse global perspective, we divide all the contextual groups into two sets (i.e., aligned and unaligned), making the total score of aligned group relatively large. We extend the vectorial inner product operation for more input and calculate the aligned score for each multimodal group. Considering that the computational complexity scales exponentially to the number of modalities, we adopt stochastic expectation approximation (SEA) for the real process. The extensive experimental results on several tasks reveal the effectiveness of our contributions.

Rethinking the Multimodal Correlation of Multimodal Sequential Learning via Generalizable Attentional Results Alignment

Vision-and-language (VL) models with separate encoders for each modality (e.g., CLIP) have become the go-to models for zero-shot image classification and image-text retrieval. They are, however, mostly evaluated in English as multilingual benchmarks are limited in availability. We introduce Babel-ImageNet, a massively multilingual benchmark that offers (partial) translations of ImageNet labels to 100 languages, built without machine translation or manual annotation. We instead automatically obtain reliable translations by linking them -- via shared WordNet synsets -- to BabelNet, a massively multilingual lexico-semantic network. We evaluate 11 public multilingual CLIP models on zero-shot image classification (ZS-IC) on our benchmark, demonstrating a significant gap between English ImageNet performance and that of high-resource languages (e.g., German or Chinese), and an even bigger gap for low-resource languages (e.g., Sinhala or Lao). Crucially, we show that the models' ZS-IC performance highly correlates with their performance in image-text retrieval, validating the use of Babel-imageNet to evaluate multilingual models for the vast majority of languages without gold image-text data. Finally, we show that the performance of multilingual CLIP can be drastically improved for low-resource languages with parameter-efficient language-specific training. 
We make our code and data publicly available: \url{https://github.com/gregor-ge/Babel-ImageNet}

Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations

Recent generalist vision-language models (VLMs) have demonstrated impressive reasoning capabilities across diverse multimodal tasks. However, these models still struggle with fine-grained object-level understanding and grounding. In terms of modeling, existing VLMs implicitly align text tokens with image patch tokens, which is ineffective for embedding alignment at the same granularity and inevitably introduces noisy spurious background features. Additionally, these models struggle when generalizing to unseen visual concepts and may not be reliable for domain-specific tasks without further fine-tuning. To address these limitations, we propose a novel method to prompt large language models with in-context visual object vectors, thereby enabling controllable object-level reasoning. This eliminates the necessity of fusing a lengthy array of image patch features and significantly speeds up training. Furthermore, we propose region-level retrieval using our object representations, facilitating rapid adaptation to new objects without additional training. Our experiments reveal that our method achieves competitive referring object classification and captioning performance, while also offering zero-shot generalization and robustness to visually challenging contexts.

OLIVE: Object Level In-Context Visual Embeddings

Multi-change captioning aims to describe complex and coupled changes within an image pair in natural language. Compared with single-change captioning, this task requires the model to have higher-level cognition ability to reason an arbitrary number of changes. In this paper, we propose a novel context-aware difference distilling (CARD) network to capture all genuine changes for yielding sentences. Given an image pair, CARD first decouples context features that aggregate all similar/dissimilar semantics, termed common/difference context features. Then, the consistency and independence constraints are designed to guarantee the alignment/discrepancy of common/difference context features. Further, the common context features guide the model to mine locally unchanged features, which are subtracted from the pair to distill locally difference features. Next, the difference context features augment the locally difference features to ensure that all changes are distilled. In this way, we obtain an omni-representation of all changes, which is translated into linguistic sentences by a transformer decoder. Extensive experiments on three public datasets show CARD performs favourably against state-of-the-art methods. The code is available at https://github.com/tuyunbin/CARD.

Context-aware Difference Distilling for Multi-change Captioning

In the field of speech synthesis, there is a growing emphasis on employing multimodal speech to enhance robustness. A key challenge in this area is the scarcity of datasets that pair audio with corresponding video. We employ a methodology that incorporates modality alignment during the pre-training phase on multimodal datasets, uniquely facilitating zero-shot generalization through the process of freezing the video modality feature extraction component and the encoder module within the pretrained weights, thereby enabling effective cross-modal and cross-lingual transfer. We have named this method `Uni-Dubbing'. Our method finely tunes with both multimodal and single-modality audio data. In multimodal scenarios, it achieves a reduced word error rate (WER) of 31.73\%, surpassing the previous best of 33.9\%. It also excels in metrics like tone quality and synchronization. With single-modality audio, it achieves a WER of 36.08\%, demonstrating adaptability to limited data. Its domain generalization capabilities are proven across various language tasks in video translation and audio generation. Trained on 433 hours of audio data, it surpasses techniques using 200 hours of audiovisual data. The code and demo are available at \href{https://diracer.github.io/unidubbing}{https://diracer.github.io/unidubbing}.

Uni-Dubbing: Zero-Shot Speech Synthesis from Visual Articulation

Detection of fast micro UAVs using event-based cameras must be insensitive to polarity effects. This was revealed by using polarity offset and contrast maximization as methods to detect motion from event-based data.

Computing Motion from Events-Based Cameras for UAV Surveillance

The objective of this line of research is to use Convolutional Neural
Networks (CNNs) to automatically identify corrosion in photographs.
This approach provides a reliable alternative to traditional
approaches. Corrosion, a widespread issue in several sectors,
necessitates prompt detection to avoid structural harm and guarantee
operational security. The use of deep learning, particularly
Convolutional Neural Networks (CNNs), offers an advanced method for
detecting corrosion. This is achieved by using the neural networks'
capability to identify complex patterns in picture data. This research
investigates the creation and assessment of a Convolutional Neural
Network (CNN) model that was trained using a dataset consisting of
photos depicting both corrosion and non-corrosion. The model's
effectiveness is assessed by analyzing key performance indicators such
as True Positive Rate, False Positive Rate, accuracy, recall, and area
under the ROC curve. The study illuminates the capabilities and
constraints of the CNN-based technique, providing useful insights for
businesses in need of effective and precise corrosion detection
solutions.


Corrosion Detection Using Deep Learning

Our task is to create an equivalent to “follow me” mode for ships at sea. Allowing a Sailor to launch a small quadcopter and forget about it while it follows and circles the ship of interest could be incredibly useful. Our particular application is autonomous inspection and battle damage assessment, say in the aftermath of a Naval casualty. 

We develop a computer vision pipeline that uses a drone’s camera feed to track the behavior of a ship. It uses camera frames to estimate the ship’s position and orientation to potentially correct the drone’s trajectory, ensuring that the ship is always in view, even in adverse conditions. Our proposed method can also be effective in broader applications in fields like Autonomous Driving and Robotics, where the main agent takes an action based on its surrounding environment. 

The first stage of our computer vision pipeline consists of calculating the orientation of the ship. Our method passes the images from the camera through a UNet image segmentation model to get the ship’s mask and boundaries. We then use an OpenCV algorithm to generate a rotated rectangle that minimizes its area while enclosing the entire mask of the ship. This rotated bounding box effectively represents the direction that the ship is facing based on its physical features, so we use its angle to estimate the ship’s orientation with respect to the drone. This technique is beneficial because it is less complex than 3D Pose Estimation models, allowing for faster real-time processing of image data. 

For our image segmentation model to perform effectively in various environmental conditions, it is crucial to have a variety of flight paths, elevation angles, and visibility conditions to prevent overfitting. Collecting real data with these criteria is an expensive and laborious process, so we have started with Unity-generated synthetic data with relevant variations as ground truth labels for training. Using this data to train our computer vision model resulted in highly accurate predicted masks from unseen flight test images.

Estimating Ship Position and Orientation using Deep Neural Networks and Synthetic Data

Naval Asset Corrosion and Defect Segmentation Using UNet Architecture 

Abstract: 

Corrosion poses a significant challenge in the maintenance and upkeep of naval assets, leading to degradation and high cost. Timely and accurate detection of corrosion and other defects are crucial to ensure operational efficiency and system availability. Traditional methods for detection often involve manual inspections, which are time-consuming, prone to human error, and require inspectors with specialized training and expertise. 

There is a significant amount of research into automatic detection of corrosion via computer vision algorithms and semantic segmentation that we utilized to combat this problem. Our method makes use of hand-annotated high-resolution images of existing weathered structures (conex boxes) as inputs for a machine learning model to detect basic corrosion, as well as classify additional defects such as paint blistering and cracking. Due to the limited amount of data representing the extended classes, image augmentations were performed to expand our dataset. 

When classifying “stuff”, where the defects can be any shape or size, having an extremely high level of precision is necessary. To account for this, a UNet machine learning architecture, which consists of a contracting and symmetric expanding path, was chosen due to its capability of performing precise pixel-level image segmentation with fewer training data requirements. We have experimented with multiple variations of UNet and implemented additional modules like attention and deep supervision in order to increase the overall performance and generalization of the model.

Naval Asset Corrosion and Defect Segmentation Using UNet Architecture 

The goal of this work is to create a robotic subsystem that is capable of effectively fighting fires in a shipboard environment both autonomously and via teleoperation. The system will be designed as a firefighting attachment for mobile robots incorporated into the fleet in the future. SAFERS will use a COTS motorized firefighting monitor, a variable power aimed water cannon, to target and extinguish fires. The targeting control software will allow for both automatic targeting of the fire and manual targeting control by an operator. Both automatic and manual targeting decisions will be informed by the various cameras that make up the perception system. The perception system will also use the video feeds to identify fires and hotspots within the fires to provide ideal targeting suggestions to most effectively fight fires. A variety of different perception sensors will be explored in order to test their effectiveness for firefighting tasks in a complex environment. Additionally, various learning and classical vision algorithms will be trained and tested to evaluate performance in the tasks of fire and hot spot detection across sensor spectrums. Some of the machine learning methods currently being explored are deep convolutional neural networks and visual transformers for object detection, semantic segmentation, and monocular depth estimation. The detected objects and estimated depths of the environments will be used by the control algorithms to make intelligent targeting decisions during firefighting scenarios. Ideally, the developed system will be useful tool for fleet firefighters as both an autonomous and remote system that will lower risk to sailors while simultaneously improving overall firefighting capability.



Semi-Autonomous Fire Extinguishing Robot Subsystem (SAFERS)

Deep learning models such as CNNs have surpassed human
performance in computer vision tasks such as image classi-
fication. However, despite their sophistication, these models
lack interpretability which can lead to biased outcomes re-
flecting existing prejudices in the data. We aim to make pre-
dictions made by a CNN interpretable. Hence, we present a
novel framework called NeSyFOLD to create a neurosym-
bolic (NeSy) model for image classification tasks. The model
is a CNN with all layers following the last convolutional layer
replaced by a stratified answer set program (ASP) derived
from the last layer kernels. The answer set program can be
viewed as a rule-set, wherein the truth value of each pred-
icate depends on the activation of the corresponding kernel
in the CNN. The rule-set serves as a global explanation for
the model and is interpretable. We also use our NeSyFOLD
framework with a CNN that is trained using a sparse kernel
learning technique called Elite BackProp (EBP). This leads to
a significant reduction in rule-set size without compromising
accuracy or fidelity thus improving scalability of the NeSy
model and interpretability of its rule-set. Evaluation is done
on datasets with varied complexity and sizes. We also pro-
pose a novel algorithm for labelling the predicates in the rule-
set with meaningful semantic concept(s) learnt by the CNN.
We evaluate the performance of our “semantic labelling algo-
rithm” to quantify the efficacy of the semantic labelling for
both the NeSy model and the NeSy-EBP model.

NeSyFOLD: A Framework for Interpretable Image Classification

The dot product self-attention (DPSA) is a fundamental component of transformers. However, scaling them to long sequences, like documents or high-resolution images, becomes prohibitively expensive due to the quadratic time and memory complexities arising from the softmax operation. Kernel methods are employed to simplify computations by approximating softmax but often lead to performance drops compared to softmax attention. We propose SeTformer, a novel transformer where DPSA is purely replaced by Self-optimal Transport (SeT) for achieving better performance and computational efficiency. SeT is based on two essential softmax properties: maintaining a non-negative attention matrix and using a nonlinear reweighting mechanism to emphasize important tokens in input sequences. By introducing a kernel cost function for optimal transport, SeTformer effectively satisfies these properties. In particular, with small and base-sized models, SeTformer achieves impressive top-1 accuracies of 84.7% and 86.2% on ImageNet-1K. In object detection, SeTformer-base outperforms the FocalNet counterpart by +2.2 mAP, using 38% fewer parameters and 29% fewer FLOPs. In semantic segmentation, our base-size model surpasses NAT by +3.5 mIoU with 33% fewer parameters. SeTformer also achieves state-of-the-art results in language modeling on the GLUE benchmark. These findings highlight SeTformer applicability for vision and language tasks.

SeTformer Is What You Need for Vision and Language

Real-world data is extremely imbalanced and presents a long-tailed distribution, resulting in models that are biased towards classes with sufficient samples and perform poorly on rare classes. Recent methods propose to rebalance classes but they undertake the seesaw dilemma (what is increasing performance on tail classes may decrease that of head classes, and vice versa). In this paper, we argue that the seesaw dilemma is derived from gradient imbalance of different classes, in which gradients of inappropriate classes are set to important for updating, thus are prone to overcompensation or undercompensation on tail classes. To achieve ideal compensation, we formulate the long-tailed recognition as an multi-objective optimization problem, which fairly respects the contributions of head and tail classes simultaneously. For efficiency, we propose a Gradient-Balancing Grouping (GBG) strategy to gather the classes with similar gradient directions, thus approximately make every update under a Pareto descent direction. Our GBG method drives classes with similar gradient directions to form more representative gradient and provide ideal compensation to the tail classes. Moreover, We conduct extensive experiments on commonly used benchmarks in long-tailed learning and demonstrate the superiority of our method over existing SOTA methods.

Long-Tailed Learning as Multi-Objective Optimization

Single-domain generalization (S-DG) aims to generalize a model to unseen environments with a single-source domain. However, most S-DG approaches have been conducted in the field of classification. When these approaches are applied to object detection, the semantic features of some objects can be damaged, which can lead to imprecise object localization and misclassification.
To address these problems, we propose an object-aware domain generalization (OA-DG) method for single-domain generalization in object detection. Our method consists of data augmentation and training strategy, which are called OA-Mix and OA-Loss, respectively. OA-Mix generates multi-domain data with multi-level transformation and object-aware mixing strategy. OA-Loss enables models to learn domain-invariant representations for objects and backgrounds from the original and OA-Mixed images. Our proposed method outperforms state-of-the-art works on standard benchmarks. Our code will be made available on publication.

Object-Aware Domain Generalization for Object Detection | VIDEO

Recent advancements in implicit neural representations have contributed to high-fidelity surface reconstruction and photorealistic novel view synthesis. However, with the expansion of the scene scale, such as block or city level, existing methods
will encounter challenges because traditional sampling cannot cope with the cubically growing sampling space. To alleviate the dependence on filling the sampling space, we explore using multi-modal priors to assist individual points to
obtain more global semantic information and propose a priorrich multi-modal implicit neural representation network, Pm-INR, for the outdoor unbounded large-scale scene. The core of our method is multi-modal prior extraction and crossmodal prior fusion modules. The former encodes codebooks from different modality inputs and extracts valuable priors, while the latter fuses priors to maintain view consistency and preserve unique features among multi-modal priors. Finally, feature-rich cross-modal priors are injected into the sampling
regions to allow each region to perceive global information without filling the sampling space. Extensive experiments have demonstrated the effectiveness and robustness of our method for outdoor unbounded large-scale scene novel
view synthesis, which outperforms state-of-the-art methods in terms of PSNR, SSIM, and LPIPS.

PM-INR: Prior-Rich Multi-Modal Implicit Large-Scale Scene Neural Representation

Virtual Poster Session 4

poster

### Welcome!
The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. Our Virtual Poster Sessions will take place online Thursday, August 22, 2024.

You are required to register for this event. **Please register [here](https://2024.aclweb.org/registration). **

If you have already registered, please check your inbox for an email from Underline granting you access to ACL 2024 content.

ACL 2024

The 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) will take place in Bangkok, Thailand from August 11th to 16th, 2024. More information will be announced soon.

Virtual Poster Session 3

This poster session includes Main Conference posters, TACL and Demos.

In-Person Poster Session 2

Poster Session 1       

### Welcome to the 8th annual workshop on Naval Applications of Machine Learning!

NAML is the premier annual event showcasing current machine learning research relevant to Naval applications. We welcome participation from the defense community, industry, and academia. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific 

**[More about NAML 2024](https://sites.google.com/go.spawar.navy.mil/naml/)**

**NEW! Join 700+ people - Receive a Special Discount **

Now you can have access to all the lectures & posters and even engage with the speakers (send them emails, ask questions) on the Underline platform. 
Access to NAML 2024 was originally $599 in person. We are offering a one-time only discounted on-demand access to this NAML 2024 content **for only $99** (complimentary to students, government, and military). 
 
**To register, please click** [**here**](https://naml2024.paperform.co/)

NAML 2024

The 8th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Main Track - Computer Vision

technical paper

We are pleased to announce the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24), which will be held in Vancouver, British Columbia at the Vancouver Convention Centre – West Building from 20-27 February, 2024.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-24 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

We expect for AAAI-24 to be an in-person conference – one author of all accepted papers will be expected to present work in person unless there are exceptional circumstances that prevent this.

In order to access the AAAI-24 event page you need to register [here](https://aaai.org/aaai-conference/registration/)

AAAI 2024

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. 

This paper investigates the effects of gender neutralization on the perception of anthropomorphism, gender specificity, and novelty for human voices, comparing spoken and hummed voice modalities. We evaluated gender-neutralized and original voice samples in both spoken and hummed formats using an online survey. Our results confirm that gender-neutralizing filters effectively reduce perceived gender specificity in both modalities, supporting their use in creating gender-neutral voices for humanoid robots. Hummed voices were perceived as more anthropomorphic and less novel than spoken voices, suggesting that non-verbal sound modalities can enhance the human likeness of gender-neutral androids while maintaining gender ambiguity. The study contributes to HRI by highlighting the potential of humming to fulfill users' expectations of interaction with android robots.

Evaluating Gender Ambiguity, Novelty and Anthropomorphism in Humming and Talking Voices for Robots

Social robots are finding their way into our society and must adapt to the social norms of everyday life. It is often unavoidable for robotic assistants to interrupt humans in their activities. These interruptions must therefore be designed in such a way that they have as little negative impact as possible on the ongoing activity and the perception of the robot. In this study, we investigate whether interruptions regarding health-promoting behavior delivered by a social robot in an office situation should be phrased politely rather than directly. A system was implemented to trigger either politely or directly phrased interruptions in a laboratory user study with participants executing an office task. We examined the perception of the robot, the task, and the interruption, as well as whether the health-promoting behavior was performed by the users. The results show that there were no significant differences in the perception of the robot and the interruptions, while the temporal demand of the task was perceived to be higher in the polite condition. In addition, the intention to follow the polite prompts was significantly higher, but there was no difference in the actual compliance with the prompts in the study itself.

Excuse Me, May I Disturb You? the Influence of Politeness of a Social Robot on the Perception of Interruptions

This paper presents the design of a novel social robot, namely Patbot, to engage people in playful interactions to help reduce their anxiety in waiting environments. We introduce the rationale and design decisions made during the development of the robot. We evaluated the robot's usability through an experiment within a simulated waiting environment, followed by interviews with the participants. The study indicated a positive effect of interacting with Patbot regarding STAI-6 anxiety measures. Other quantitative and qualitative measures, including Godspeed questionnaires, observation, and post-interviews, revealed positive impressions made by Patbot regarding its likability and animacy and that it could engage individuals in intuitive and playful activities. Our findings suggest the promise of designing social robots for entertainment and relaxation to enhance people's emotional experiences in waiting environments.

Patbot: Designing a Social Robot to Reduce Anxiety in Waiting Environments

We explore the potential of mobile motion as an expression modality for home social robots with a low expressivity. To gauge the potential of mobile motion, we examine its relative efficacy in terms of emotion perception accuracy, emotion intensity, and impression (anthropomorphism and animacy) compared to screen-based facial expressions. Additionally, we explore how users perceive the emotional intensity of expressions based on different degrees of expression by manipulating emotional features such as motion size and speed. Motion expressions are less accurate than facial expressions, but perform on par with facial expressions in other metrics. Its dynamic and expressive features elicit powerful emotional conveyance, in contrast to the low emotional impact associated with the monotonous nature of screen-based facial expressions. Further research is needed on ME for specific emotions, but in general, the higher the degree of expression, the more intense the emotion conveyed. We show that the degree of expression, i.e., the combination of emotional features and their modulation, can be utilized to express the emotional intensity, situational dependence, and personality of robots. In conclusion, we argue that mobile motion is a promising method to compensate for the weaknesses of screen-based facial expression, which is the dominant expression modality for low expressivity robots.

Exploring the Potential of Wheel-Based Mobile Motion As an Emotional Expression Modality

Integrating intelligent systems, such as robots, into dynamic group settings poses challenges due to the mutual influence of human behaviors and internal states. A robust representation of social interaction dynamics is essential for effective human-robot collaboration. Existing approaches often narrow their focus to facial expressions or speech, overlooking the broader context. We propose employing an adapted Temporal Graph Networks to comprehensively represent social interaction dynamics while enabling its practical implementation. Our method incorporates temporal multi-modal behavioral data including gaze interaction, voice activity and environmental context. This representation of social interaction dynamics is trained as a link prediction problem using annotated gaze interaction data. The F1-score outperformed the baseline model by 37.0%. This improvement is consistent for a secondary task of next speaker prediction which achieves an improvement of 29.0%. Our contributions are two-fold, including a model to representing social interaction dynamics which can be used for many downstream human-robot interaction tasks like human state inference and next speaker prediction. More importantly, this is achieved using a more concise yet efficient message-passing method, significantly reducing the message size from 768 to 14 while outperforming the baseline model.

Modeling Social Interaction Dynamics Using Temporal Graph Networks

A crucial part of traditional reinforcement learning (RL) is the initial exploration phase, in which trying available actions randomly is a critical element. As random behavior might be detrimental to a social interaction, this work proposes a novel paradigm for learning social robot behavior - the use of shielding to ensure socially appropriate behavior during exploration and learning. We explore how a data-driven approach for shielding could be used to generate listening behavior. In a video-based user study (N=110), we compare shielded exploration to two other exploration methods. We show that the shielded exploration is perceived as more comforting and appropriate than a straightforward random approach. Based on our findings, we discuss the potential for future work using shielded and socially guided approaches for learning idiosyncratic social robot behaviors through RL.

Shielding for Socially Appropriate Robot Listening Behaviors

This paper presents a system for diversity-aware autonomous conversation leveraging the capabilities of large language models (LLMs). The system adapts to diverse populations and individuals, considering factors like background, personality, age, gender, and culture. The conversation flow is guided by the structure of the system's pre-established knowledge base, while LLMs are tasked with various functions, including generating diversity-aware sentences. Achieving diversity-awareness involves providing carefully crafted prompts to the models, incorporating comprehensive information about users, conversation history, contextual details, and specific guidelines. To assess the system's performance, we conducted both controlled and real-world experiments, measuring a wide range of performance indicators.

Enhancing LLM-Based Human-Robot Interaction with Nuances for Diversity Awareness

This paper presents a comprehensive framework to enhance Human-Robot Collaboration (HRC) in real-world scenarios. It introduces a formalism to model articulated tasks, requiring cooperation between two agents, through a smaller set of primitives. Our implementation leverages Hierarchical Task Networks (HTN) planning and a modular multisensory perception pipeline, which includes vision, human activity recognition, and tactile sensing. To showcase the system's scalability, we present an experimental scenario where two humans alternate in collaborating with a Baxter robot to assemble four pieces of furniture with variable components. This integration highlights promising advancements in HRC, suggesting a scalable approach for complex, cooperative tasks across diverse applications.

A Modular Framework for Flexible Planning in Human-Robot Collaboration

Vertical take-off and landing (VTOL) aircraft do not require a prolonged runway, thus allowing them to land almost anywhere. In recent years, their flexibility has made them popular in development, research, and operation. When compared to traditional fixed-wing aircraft and rotorcraft, VTOLs bring unique challenges as they combine many maneuvers from both types of aircraft. Pilot workload is a critical factor for safe and efficient operation of VTOLs. In this work, we conduct a user study to collect multimodal data from 28 pilots while they perform a variety of VTOL flight tasks. We analyze and interpolate behavioral patterns related to their performance and perceived workload. Finally, we build machine learning models to estimate their workload from the collected data. Our results are promising, suggesting that quantitative and accurate VTOL pilot workload monitoring is viable. Such assistive tools would help the research field understand VTOL operations and serve as a stepping stone for the industry to ensure VTOL safe operations and further remote operations.

How Is the Pilot Doing: VTOL Pilot Workload Estimation by Multimodal Machine Learning on Psycho-Physiological Signals

Virtual Reality (VR) environments are used as a way to let humans experiment with algorithms, techniques, and situations in various application areas, including emergency management, serious games, smart manufacturing, and precision agriculture. They are especially relevant when experiments in the real world may be harmful to human operators. As VR environments are the closest possible faithful replicas of real environments, many recent works focus on the employment of such tools as a means to generate synthetic datasets that can be used for training machine and deep learning models, especially in situations where obtaining real datasets can be difficult. In this paper, we introduce a strategy to generate a dataset for pose estimation in the challenging scenario of precision agriculture. Finally, the quality of the generated dataset was evaluated.

Generating and Evaluating Synthetic Data in Virtual Reality Simulation Environments for Pose Estimation

A robots ability to effectively recognize human emotions is critical in human-robot collaboration. However, most of the current collaborative robots were designed to improve productivity. Few of these robots consider human emotions. This situation would cause humans to be unwilling to work with robots for a long time. Motivated by this gap, this research developed a human emotion recognition system for enhancing the interaction abilities of collaborative robots. In this project, both speech and facial information were analyzed for robust human emotion recognition in complex working environments like manufacturing assembly environments. In the experiment, the developed system has been tested through three different human-robot co-assembly scenarios: (1) the robot effectively assists humans in finishing the task, (2) the robot is slow in response, leading to the task failing, (3) the robot frequently picks up wrong tools leading to task failure. Experimental results have demonstrated the effectiveness of the developed system in recognizing human co-workers emotions when the human and the robot were working in the above scenarios. It further shows the developed system has the potential to contribute to the development of an empathic collaborative robot companion in the manufacturing area.

A Speech and Facial Information Based Emotion Recognition System of Collaborative Robot for Empathic Human-Robot Collaboration

Human joints are unique in the sense that their stiffness is not fixed and rather adjustable which aids in interacting with the physical world. A robot with servo-adjustable stiffness is intrinsically safe and capable of imitating human-like interactions. Apart from safety, soft joints are efficient in executing cyclic tasks which opens the door to have an agile behaviour. In this paper we extend our prior work on variable stiff joints and use two of them to design a two-link planar robot. Here we modify the dynamic model for this manipulator to include the variable stiffness term in addition to the joint torques as inputs, followed by joint torque calibration method, and then a simplified manipulator control template. This paper serves as an introduction to our ongoing work on vSEA and a foray into true manipulator space application.

R-R Manipulator Using Variable-Stiff Joints for Safe Human-Robot-Interaction

This paper introduces a new companion robot as an open research platform for studying human-robot interaction: The ZENIT Enabling Natural Interaction Technology. Open-source solutions for companion robots are often complicated to build or not thought of holistically. ZENIT is based on a few low-cost components, namely a smartphone and a stationary robot arm in conjunction with an average computer and a separate camera. This concept makes it possible not to be tied to specific hardware. Our platform aims to enable easy use of the multimodality of human communication channels such as body language, spatial behavior, facial expressions and speech so that ZENIT can understand its interlocutor and express itself appropriately. The fusion of various open-source software components in an adaptable, lightweight and resource-efficient distributed system simplifies its replication. The system demonstrated an adequate response time (speech: 2.69s, facial expressions: 0.55s, bodily movements: 0.32s) for perception and expression combined, as well as reliability in public testing. Evaluation of the robot's expressive capabilities in an online survey (N=23) revealed users were able to understand six of eight emotions conveyed by ZENIT using facial expressions and/or body language (except for contempt and fear) and perceived its non-verbal feedback as significantly more expressive when facial expressions were accompanied with bodily movements.

Fusing Components for an Attentive and Emotionally Expressive Companion Robot: Meet ZENIT

Rapid human-robot interactions require fast hardware platforms with minimal latency and high reliability. In response, we present a cost-effective, electrically actuated, tendon-driven robotic hand. This hand features a unique spool-free actuation mechanism that achieves a limit-to-limit flexion movement in less than 60 ms, matching human speed. To our knowledge, it is among the fastest electric motor tendon-driven robotic hands available today. The high speed of the robotic hand was successfully demonstrated in public by playing Rock Paper Scissors at a science fair. This research work outlines the design methodology and introduces a simulation-optimization framework that allows users to preview the motion of the hand, quantify the actuation performance, and customize the design parameters prior to fabrication. The proposed actuation mechanism, along with the simulation and optimization tools, illustrates design principles and computational methods applicable to other dynamic human-robot applications that require fast reaction times. The Dextra hand design is available at https://sensorsini.github.io/dextra-robot-hand.

A Rapid and Robust Tendon-Driven Robotic Hand for Human-Robot Interactions Playing Rock-Paper-Scissors

In recent years, labor accidents and a shortage of skilled workers due to an aging population have become significant issues at construction sites in Japan. To address these challenges, we are developing a Cyber-Physical System (CPS) platform called ROS2-TMS for Construction, which aims to improve both the efficiency and safety of earthwork operations. In this study, we propose a task management system for construction machinery using an open platform named OPERA as an additional function of ROS2-TMS for Construction. This task management system controls construction machinery using environmental information stored in a database, which collects and stores data from sensors deployed throughout the construction site, and an extended Behavior Tree. At the end of this study, the results of the initial validation tests of autonomous earthwork operations using an OPERA-compatible backhoe ZX200 are presented.

Task Management System for Construction Machinery Using the Open Platform OPERA

We propose a novel on-line multi-UAV coverage path planning method for 3D reconstruction. UAVs play ho- mogeneous roles in most existing multi-UAV path planning methods, where their flexibility is not fully exploited. The fact that UAVs can explore with different missions and strategies at different heights simultaneously is not well considered. Therefore, we give UAVs different identities by presenting a global-local pattern, where global UAV locates the interesting regions while local UAV explores these regions in succession. Leveraging the mechanism of sensorimotor coordination, our strategy adjusts the paths of local UAV on-line according to the input from global UAV in order to gather more information and save time. In addition, our method is compatible with real-time SLAM systems. Simulation shows that our strategy takes less time than other strategies based on area-division, and indicates that it gathers more information for better reconstruction.

Sensorimotor Coordinated Multi-UAV Coverage Path Planning

ThDT2 - Sound design for robots

[![](https://assets.underline.io/markdown_image/1/image/f25361a85b7c83a7476c0b3c1c0bb1b2.png)](https://drive.google.com/file/d/18jcAcciCBQHj6R07Dbpm2HCOBEIwAOCU/view?usp=sharing)


Dear Colleagues and Friends,

The 33rd IEEE International Conference on Robot and Human Interactive Communication, IEEE RO-MAN 2024, was held at Pasadena Convention Center in Pasadena, USA.

The theme of the IEEE RO-MAN 2024 was **“Embracing Human-Centered HRI”**.

With the continuous technological effort of understanding human needs, creating human-desired robotic systems, and developing real human user experiences, we are marching together towards building a promising and excellent human quality of life, all for our wellness and our next generation.

The conference provided a forum for researchers and practitioners to discuss a variety of topics related to the broad theme of Robot and Human Interactive Communication involving theories, methodologies, technologies, empirical and experimental studies, and generated more insights for the future of Robot and Human Interactive Communication!

We hope you continue to enjoy the post-conference content on Underline!

A registration is required to access this event page. If you have not already done so, please use the below link to register for the event. 

Please [register here](https://www.ro-man2024.org/participation/registration).



Registration Is Required

IEEE RO-MAN 2024

The 33rd IEEE International Conference on Robot and Human Interactive Communication was held in Pasadena, California, USA. The theme of the IEEE RO-MAN 2024 was “Embracing Human-Centered HRI”.

ThDT3 - Robot Companions and Social Robots II

ThDT4 - Social Intelligence for Robots II

ThDT5 - Multimodal Sensing, Learning, and Control for Robots in Human-Robot Collaborative Contexts

ThDT6 - Innovative Robot Designs

ThCT1 - Degrees of Autonomy and Teleoperation I

NAML 2025 Keynotes

Computer vision

Underline presents the latest conferences in Robotics, showcasing up-to-date research, emerging topics and speakers in the area of Robotics and related disciplines.

Underline digital video library

Browse NAML 2025 Keynotes