Canada

Disagreement in natural language annotation has been predominantly studied from a perspective of biases introduced by the annotators and the annotation frameworks. Here, wepropose to analyze another source of bias: task design bias,which has a particularly strong impact on crowdsourcing linguistic annotations where natural language is used to elicit the interpretation of laymen annotators. For this purpose we look at implicit discourse relation annotation, a task that has repeatedly been shown to be difficult due to the relations&#39; ambiguity. We compare the annotations of 1,200 discourse relations obtained using two distinct discourse relation annotation tasks and quantify the biases of both methods across four different domains. Both methods are natural language annotation tasks designed for crowdsourcing: one reformulating discourse segments as questions and answers and one using discourse connective insertion. We show that the task design can push annotators towards certain relations and that some discourse relations senses can be better elicited with the one or the other NL annotation approach.We also conclude that this type of bias should be taken into account when training and testing models.

ACL 2023

Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases introduced by Task Design

connectives

natural language annotations

discourse relations

annotations

bias

Welcome to the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Continuing its mission of expanding and involving the science community of all European countries, EACL has selected the Malta community for the 18th EACL. Considering the importance of physical interaction among researchers, the conference will be held at the Hotel Radisson Blu, St. Julians, in Malta, from 17 to 22 of March, 2024. The conference will also feature the possibility to participate virtually. As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language. 
#### [Conference Handbook](https://drive.google.com/file/d/13Wn38q0yev6U3RTgnf4Tn5T9GuAuZ-xW/view?usp=sharing) *downloadable*

To access the EACL 2024 event page you need to register [**here**](https://acl.swoogo.com/EACL2024). 

Please register!

EACL 2024

As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

Disagreement in natural language annotation has been predominantly studied from a perspective of biases introduced by the annotators and the annotation frameworks. Here, wepropose to analyze another source of bias: task design bias,which has a particularly strong impact on crowdsourcing linguistic annotations where natural language is used to elicit the interpretation of laymen annotators. For this purpose we look at implicit discourse relation annotation, a task that has repeatedly been shown to be difficult due to the relations' ambiguity. We compare the annotations of 1,200 discourse relations obtained using two distinct discourse relation annotation tasks and quantify the biases of both methods across four different domains. Both methods are natural language annotation tasks designed for crowdsourcing: one reformulating discourse segments as questions and answers and one using discourse connective insertion. We show that the task design can push annotators towards certain relations and that some discourse relations senses can be better elicited with the one or the other NL annotation approach.We also conclude that this type of bias should be taken into account when training and testing models.

poster

### Welcome to ACL 2023, the 61st Annual Meeting of the Association for Computational Linguistics! 
 The conference will be held in Toronto, Canada, July 9-14, 2023. 
Following the succession of the recent conferences in our field, ACL 2023 will adopt a hybrid format.
While the impact of Covid has considerably diminished in terms of traveling, obtaining visas to Canada
entails a very long process. Moreover, the global economic conditions pose challenges for many individuals to travel to conferences. Recognizing these circumstances, we know many participants may not be
able to attend the conference in person. Therefore, we are committed to providing a great virtual platform
so everyone has the opportunity to interact with other participants and enjoy the conference. Based on the
current registered participants, approxiately 30% have chosen to attend the conference virtually. Whether
you join us in person or virtually, we sincerely hope everyone has a remarkable conference experience. 
This General Chair’s message is where I express my gratitude to the many individuals who have made
enormous contributions to the conference over the past year.

Read [**ACL 2023 General Chair's message**](https://docs.google.com/document/d/1WobYM7norbG4dI48s75HfJoD89qgX5a_F-6U8AteLSA/edit?usp=sharing/) in full.

##### **[Conference Handbook](https://2023.aclweb.org/downloads/acl2023-handbook.pdf)**

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing.

This study proposes a new multimodal neural machine translation (MNMT) model using synthetic images transformed by a latent diffusion model. MNMT translates a source language sentence based on its related image, but the image usually contains noisy information that are not relevant to the source language sentence.Our proposed method first generates a synthetic image corresponding to the content of the source language sentence by using a latent diffusion model and then performs translation based on the synthetic image. The experiments on the English-German translation tasks using the Multi30k dataset demonstrate the effectiveness of the proposed method.

Multimodal Neural Machine Translation Using Synthetic Images Transformed by Latent Diffusion Model

Numerous studies found that the linguistic properties of a person's native language affect the cognitive processing of other languages. However, only one study has shown that it was possible to identify the native language based on eye-tracking records of natural L2 reading using machine learning. A new corpus allows us to replicate these results on a more interrelated and larger set of native languages. Our results show that comparable classification performance is maintained despite using less data. However, analysis shows that the correlation between L2 eye movements and native language similarity may be more complex than the original study found.

Native Language Prediction from Gaze: a Reproducibility Study

In this paper, we investigate a type of semantic shift that occurs when a sudden event radically changes public opinion on a topic. Looking at Sweden's decision to apply for NATO membership in 2022, we use word embeddings to study how the associations users on Twitter have regarding NATO evolve. We identify several changes that we successfully validate against real-world events. However, the low engagement of the public with the issue often made it challenging to distinguish true signals from noise. We thus find that domain knowledge and data selection are of prime importance when using word embeddings to study semantic shifts.

Sudden Semantic Shifts in Swedish NATO discourse

Natural Language Inference (NLI) tasks involving temporal inference remain challenging for pre-trained language models (LMs). Although various datasets have been created for this task, they primarily focus on English and do not address the need for resources in other languages. It is unclear whether current LMs realize the generalization capacity for temporal inference across languages. In this paper, we present Jamp, a Japanese NLI benchmark focused on temporal inference. Our dataset includes a range of temporal inference patterns, which enables us to conduct fine-grained analysis. To begin the data annotation process, we create diverse inference templates based on the formal semantics test suites. We then automatically generate diverse NLI examples by using the Japanese case frame dictionary and well-designed templates while controlling the distribution of inference patterns and gold labels. We evaluate the generalization capacities of monolingual/multilingual LMs by splitting our dataset based on tense fragments (i.e., temporal inference patterns). Our findings demonstrate that LMs struggle with specific linguistic phenomena, such as habituality, indicating that there is potential for the development of more effective NLI models across languages.

Jamp: Controlled Japanese Temporal Inference Dataset for Evaluating Generalization Capacity of Language Models

Translation difficulty arises when translators are required to resolve translation ambiguity from multiple possible translations. Translation difficulty can be measured by recording the diversity of responses provided by human translators and the time taken to provide these responses, but these behavioral measures are costly and do not scale. In this work, we use word alignments computed over large scale bilingual corpora to develop predictors of lexical translation difficulty. We evaluate our approach using behavioural data from translations provided both in and out of context, and report results that improve on a previous embedding based approach (Thompson et al., 2020). Our work can therefore contribute to a deeper understanding of cross-lingual differences and of causes of translation difficulty.

Predicting Human Translation Difficulty Using Automatic Word Alignment

The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden representations which makes them unexpectedly close to each other in terms of angular distance (cosine-similarity). Some recent works tend to show that anisotropy is a consequence of optimizing the cross-entropy loss on long-tailed distributions of tokens. We show in this paper that anisotropy can also be observed empirically in language models with specific objectives that should not suffer directly from the same consequences. We also show that the anisotropy problem extends to Transformers trained on other modalities. Our observations tend to demonstrate that anisotropy might actually be inherent to Transformers-based models.

Is Anisotropy Inherent to Transformers?

Transformer Language Models Finally Add Word Frequency Biases

<iframe src="https://app.sli.do/event/uyG7EHDP5trs2LHubv2Gj8/embed/polls/cc529c8d-fcac-4d53-bdef-52e546fd718f" width="300" height="400"></iframe>

Transformer Language Models Handle Word Frequency in Prediction Head

Code search is a task to find programming codes that semantically match the given natural language queries. Even though some of the existing datasets for this task are multilingual on the programming language side, their query data are only in English. In this research, we create a multilingual code search dataset in four natural and four programming languages using a neural machine translation model. Using our dataset, we pre-train and fine-tune the Transformer-based models and then evaluate them on multiple code search test sets. Our results show that the model pre-trained with all natural and programming language data has performed best in most cases. By applying back-translation data filtering to our dataset, we demonstrate that the translation quality affects the model's performance to a certain extent, but the data size matters more.

Constructing Multilingual Code Search Dataset Using Neural Machine Translation

This paper addresses whether masked language models can distinguish (named) entities in internal representations and investigates whether the model's internal representations of the same entity create a sufficiently compact cluster that is distinct from others. The primary contributions of the paper are two-fold. First, we present a novel method for quantitatively assessing the degree to which pre-trained masked language models distinguish between entities in the internal representations. Our approach investigates whether entities belonging to the same object are separated from other embeddings in the embedding space. Second, we conducted experiments, and the findings revealed that internal representations of masked language models were able to distinguish entities from other concepts to a certain degree, even when there was variation in the surrounding context and mentions. (e.g., BERT could distinguish about 70% of the entities from other concepts.) An implication of these results is the possibility that the distinct entities, as reflected in internal representations, are considered evidence of behavior that effectively processes entity strings.

Geometric Locality of Entity Embeddings in Masked Language Models

Pre-trained language models have achieved remarkable results on several NLP tasks. Most of them adopt masked language modeling to learn representations by randomly masking tokens and predicting them based on their context. However, this random selection of tokens to be masked is inefficient to learn some language patterns as it may not consider linguistic information that can be helpful for many NLP tasks, such as multimodal machine translation (MMT). Hence, we propose three novel masking strategies for cross-lingual visual pre-training - more informed visual masking, more informed textual masking, and more informed visual and textual masking - each one focusing on learning different linguistic patterns. We apply them to Vision Translation Language Modelling for video subtitles (Sato et al., 2022) and conduct extensive experiments on the Portuguese-English MMT task. The results show that our masking approaches yield significant improvements over the original random masking strategy for downstream MMT performance. Our models outperform the MMT baseline and we achieve state-of-the-art accuracy (52.70 in terms of BLEU score) on the How2 dataset, indicating that more informed masking helps in acquiring an understanding of specific language structures and has great potential for language understanding.

Downloads

Next from ACL 2023

Multimodal Neural Machine Translation Using Synthetic Images Transformed by Latent Diffusion Model

Similar lecture

Learning Label Embeddings for Mapping Discourse Relations of Different Annotation Frameworks (Extended Abstract)

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2023

Multimodal Neural Machine Translation Using Synthetic Images Transformed by Latent Diffusion Model

Similar lecture

Learning Label Embeddings for Mapping Discourse Relations of Different Annotation Frameworks (Extended Abstract)

Downloads