Canada

We present the system proposed by the MilaNLP team for the Explainable Detection of Online Sexism (EDOS) shared task.
We propose an ensemble modeling approach to combine different classifiers trained with domain adaptation objectives and standard fine-tuning.
Our results show that the ensemble is more robust than individual models and that regularized models generate more ``conservative&#39;&#39; predictions, mitigating the effects of lexical overfitting.
However, our error analysis also finds that many of the misclassified instances are debatable, raising questions about the objective annotatability of hate speech data.

ACL 2023

MilaNLP at SemEval-2023 Task 10: Ensembling Domain-Adapted and Regularized Pretrained Language Models for Robust Sexism Detection

sexism

regularization

hate speech detection

### Welcome to ACL 2023, the 61st Annual Meeting of the Association for Computational Linguistics! 
 The conference will be held in Toronto, Canada, July 9-14, 2023. 
Following the succession of the recent conferences in our field, ACL 2023 will adopt a hybrid format.
While the impact of Covid has considerably diminished in terms of traveling, obtaining visas to Canada
entails a very long process. Moreover, the global economic conditions pose challenges for many individuals to travel to conferences. Recognizing these circumstances, we know many participants may not be
able to attend the conference in person. Therefore, we are committed to providing a great virtual platform
so everyone has the opportunity to interact with other participants and enjoy the conference. Based on the
current registered participants, approxiately 30% have chosen to attend the conference virtually. Whether
you join us in person or virtually, we sincerely hope everyone has a remarkable conference experience. 
This General Chair’s message is where I express my gratitude to the many individuals who have made
enormous contributions to the conference over the past year.

Read [**ACL 2023 General Chair's message**](https://docs.google.com/document/d/1WobYM7norbG4dI48s75HfJoD89qgX5a_F-6U8AteLSA/edit?usp=sharing/) in full.

##### **[Conference Handbook](https://2023.aclweb.org/downloads/acl2023-handbook.pdf)**

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing.

We present the system proposed by the MilaNLP team for the Explainable Detection of Online Sexism (EDOS) shared task.
We propose an ensemble modeling approach to combine different classifiers trained with domain adaptation objectives and standard fine-tuning.
Our results show that the ensemble is more robust than individual models and that regularized models generate more ``conservative'' predictions, mitigating the effects of lexical overfitting.
However, our error analysis also finds that many of the misclassified instances are debatable, raising questions about the objective annotatability of hate speech data.

workshop paper

This paper describes the system we used to participate in the shared task (Kiesel et al., 2023),
as well as additional experiments beyond the
scope of the shared task, but using its data. Our
primary goal is to compare the effectiveness of
transformers model compared to low-resource
dictionaries. Secondly, we compare the difference in performance of a learned dictionary
and of a dictionary designed by experts in the
field of values. Our findings surprisingly show
that transformers perform on par with a dictionary containing less than 1k words, when
evaluated with 19 fine-grained categories, and
only outperform a dictionary-based approach
in a coarse setting with 10 categories. Interestingly, the expert dictionary has a precision
on par with the learned one, while its recall is
clearly lower, potentially an indication of overfitting of topics to values in the shared task’s
dataset. Our findings should be of interest to
both the NLP and Value scientific communities
on the use of automated approaches for value
classification.


TeamEC at SemEval-2023 Task 4: Transformers vs. Low-Resource Dictionaries, Expert Dictionary vs. Learned Dictionary

This paper presents the submissions of the DH- FBK team for the three tasks of Task 10 at SemEval 2023. The Explainable Detection of Online Sexism (EDOS) task aims at detecting sexism in English text in an accurate and ex- plainable way, thanks to a fine-grained annota- tion that follows a three-level schema: sexist or not (Task A), category of sexism (Task B) and vector of sexism (Task C) exhibited. We use a multi-task learning approach in which models share representations from all three tasks, al- lowing for knowledge to be shared across them. Notably, with our approach a single model can solve all three tasks. In addition, motivated by the subjective nature of the task, we incorpo- rate inter-annotator agreement information in our multi-task architecture. Although disaggre- gated annotations are not available, we artifi- cially estimate them using a 5-classifier ensem- ble, and show that ensemble agreement can be a good approximation of crowd agreement. Our approach achieves competitive results, ranking 32nd out of 84, 24th out of 69 and 11th out of 63 for Tasks A, B and C respectively. We finally show that low inter-annotator agreement levels are associated with more challenging examples for models, making agreement information use- ful for this kind of task.

DH-FBK at SemEval-2023 Task 10: Multi-Task Learning with Classifier Ensemble Agreement for Sexism Detection

This presentation describes our approach for SemEval-2023 Task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. For Subtask 1 (News Genre), we propose an ensemble of fully trained and adapter mBERT models which was ranked joint-first for German, and had the highest mean rank of multi-language teams. For Subtask 2 (Framing), we achieved first place in 3 languages, and the best average rank across all the languages, by using two separate ensembles: a monolingual RoBERTa-MUPPETlarge and an ensemble of XLM-RoBERTaLARGE with adapters and task adaptive pretraining. For Subtask 3 (Persuasion Techniques), we trained a monolingual RoBERTa-Base model for English and a multilingual mBERT model for the remaining languages, which achieved top 10 for all languages, including 2nd for English. For each subtask, we compared monolingual and multilingual approaches, and considered class imbalance techniques

SheffieldVeraAI at SemEval-2023 Task 3: Mono and Multilingual Approaches for News Genre, Topic and Persuasion Technique Classification

The MultiCoNER \RNum{2} shared task aims to tackle multilingual named entity recognition (NER) in fine-grained and noisy scenarios, and it inherits the semantic ambiguity and low-context setting of the MultiCoNER \RNum{1} task. To cope with these problems, the previous top systems in the MultiCoNER \RNum{1} either incorporate the knowledge bases or gazetteers. However, they still suffer from insufficient knowledge, limited context length, single retrieval strategy. In this paper, our team \textbf{DAMO-NLP} proposes a unified retrieval-augmented system (U-RaNER) for fine-grained multilingual NER. We perform error analysis on the previous top systems and reveal that their performance bottleneck lies in insufficient knowledge. Also, we discover that the limited context length causes the retrieval knowledge to be invisible to the model. To enhance the retrieval context, we incorporate the entity-centric Wikidata knowledge base, while utilizing the infusion approach to broaden the contextual scope of the model. Also, we explore various search strategies and refine the quality of retrieval knowledge. Our system\footnote{We will release the dataset, code, and scripts of our system at {\small \url{https://github.com/modelscope/AdaSeq/tree/master/examples/U-RaNER}}.} wins 9 out of 13 tracks in the MultiCoNER \RNum{2} shared task. 
Additionally, we compared our system with ChatGPT, one of the large language models which have unlocked strong capabilities on many tasks. The results show that there is still much room for improvement for ChatGPT on the extraction task.

DAMO-NLP at SemEval-2023 Task 2: A Unified Retrieval-augmented System for Multilingual Named Entity Recognition

nlp datasets annotated with human judgments are rife with disagreements between the judges. This is especially true for tasks depending on subjective judgments such as sentiment analysis or offensive language detection. Particularly in these latter cases, the nlp community has come to realize that the approach of ‘reconcil- ing’ these different subjective interpretations is inappropriate. Many nlp researchers have therefore concluded that rather than eliminat- ing disagreements from annotated corpora, we should preserve them–indeed, some argue that corpora should aim to preserve all annotator judgments. But this approach to corpus cre- ation for nlp has not yet been widely accepted. The objective of the LeWiDi series of shared tasks is to promote this approach to developing nlp models by providing a unified framework for training and evaluating with such datasets. We report on the second LeWiDi shared task, which differs from the first edition in three crucial respects: (i) it focuses entirely on nlp, instead of both nlp and computer vision tasks in its first edition; (ii) it focuses on subjective tasks, instead of covering different types of disagreements–as training with aggregated la- bels for subjective nlp tasks is a particularly obvious misrepresentation of the data; and (iii) for the evaluation, we concentrate on soft ap- proaches to evaluation. This second edition of LeWiDi attracted a wide array of participants resulting in 13 shared task submission papers.

SemEval-2023 Task 11: Learning with Disagreements (LeWiDi)

We present the first Afri-centric SemEval Shared task, Sentiment Analysis for African Languages - AfriSenti-SemEval. AfriSenti-SemEval is a sentiment classification challenge in 14 African languages (Muhammad et al., 2023; Amharic, Algerian Arabic, Hausa, Igbo, Kinyarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yoruba), using data labeled with 3 sentiment classes. We present three subtasks: (1) Task A: monolingual classification, which received 44 submissions; (2) Task B: multilingual classification, which received 32 submissions; and (3) Task C: zero-shot classification, which received 34 submissions. The best performance for tasks A and B was achieved by NLNDE team with 71.31 and 75.06 weighted F1, respectively. UCAS-IIE-NLP achieved the best average score for task C with 58.15 weighted F1. We describe the various approaches adopted by the top 10 systems and their approaches. The dataset is available at https://github.com/afrisenti-semeval/afrisent-semeval-2023.

SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)

Subjectivity and difference of opinion are key social phenomena, and it is crucial to take these into account in the annotation and detection process of derogatory textual content. In this paper, we use four datasets provided by SemEval-2023 Task 11 and fine-tune a BERT model to capture the disagreement in the annotation. We find individual annotator modeling and aggregation lowers the Cross-Entropy score by an average of 0.21, compared to the direct training on the soft labels. Our findings further demonstrate that annotator metadata contributes to the average 0.029 reduction in the Cross-Entropy score.

SafeWebUH at SemEval-2023 Task 11: Learning Annotator Disagreement in Derogatory Text: Comparison of Direct Training vs Aggregation

PAI at SemEval-2023 Task 4: A general multi-label classification system
with class-balanced loss function and ensemble module. We employ a multi-label classification model and utilize a class-balanced loss function. Our system wins 5 first places, 2 second places, and 6 third places out of 20 categories of the Human Value
Detection shared task.

PAI at SemEval-2023 Task 4: A General Multi-label Classification System with Class-balanced Loss Function and Ensemble Module

This paper describes our system submission
for SemEval-2023 Task 12 AfriSenti-SemEval:
Sentiment Analysis for African Languages. We
propose an XGBoost-based ensemble model
trained on emoticon frequency-based features
and the predictions of several statistical models
such as SVMs, Logistic Regression, Random
Forests, and BERT-based pre-trained language
models such as AfriBERTa and AfroXLMR.
We also report results from additional experiments not in the system. Our system achieves
a mixed bag of results, achieving a best rank of
7th in three of the languages - Igbo, Twi, and
Yoruba.

Witcherses at SemEval-2023 Task 12: Ensemble Learning for African Sentiment Analysis

We present our approach to the task of identification of persuasion techniques in text, which is a subtask of the SemEval2023 Task 3 on the multilingual detection of genre, framing, and persuasion techniques in online news. Our solution is based on an ensemble of a variety of pre-trained language models (PLMs) fine-tuned on the propaganda dataset. We first describe our system, the different experimental setups we considered, and then provide the results on the dev and test sets released by the organizers. The official evaluation shows our solution ranks 1st in English and attains high scores in all the other languages, i.e. French, German, Italian, Polish, and Russian. We also perform an extensive analysis of the data and the annotations to investigate how they can influence the quality of our systems.

Downloads

Next from ACL 2023

TeamEC at SemEval-2023 Task 4: Transformers vs. Low-Resource Dictionaries, Expert Dictionary vs. Learned Dictionary

Similar lecture

UIRISC at SemEval-2023 Task 10: Explainable Detection of Online Sexism by Ensembling Fine-tuning Language Models

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2023

TeamEC at SemEval-2023 Task 4: Transformers vs. Low-Resource Dictionaries, Expert Dictionary vs. Learned Dictionary

Similar lecture

UIRISC at SemEval-2023 Task 10: Explainable Detection of Online Sexism by Ensembling Fine-tuning Language Models

Downloads