Spain

This paper describes our contribution to SemEval-2020 Task 8: Memotion Analysis. Our system learns multi-modal embeddings from text and images in order to classify Internet memes by sentiment. Our model learns text embeddings using BERT and extracts features from images with DenseNet, subsequently combining both features through concatenation. We also compare our results with those produced by DenseNet, ResNet, BERT, and BERT-ResNet. Our results show that image classification models have the potential to help classifying memes, with DenseNet outperforming ResNet. Adding text features is however not always helpful for Memotion Analysis.

COLING 2020

Utilizing BERT and DenseNet for Internet Meme Emotion Analysis

multi-modal information

internet meme

sentiment analysis

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

poster

In this paper, we present our contribution in SemEval-2020 Task 1: Unsupervised Lexical SemanticChange Detection, where we systematically combine existing models for unsupervised capturingof lexical semantic change across time in text corpora of German, English, Latin and Swedish.In particular, we analyze the score distribution of existing models. Then we define a general classification threshold, adjust it independently to each of the models and measure the models’score certainty. Finally, using both the threshold and score certainty, we aggregate the models forthe two sub-tasks: binary classification and ranking.

Combined Semantic Vector Spaces Models for Unsupervised Lexical Semantic Change Detection

We describe our system for SemEval-2020 Task 11 on Detection of Propaganda Techniques in News Articles. We developed ensemble models using RoBERTa-based neural architectures, additional CRF layers, transfer learning between the two subtasks, and advanced post-processing to handle the multi-label nature of the task, the consistency between nested spans, repetitions, and labels from similar spans in training. We achieved sizable improvements over baseline fine-tuned RoBERTa models, and the official evaluation ranked our system 3rd (almost tied with the 2nd) out of 36 teams on the span identification subtask with an F1 score of 0.491, and 2nd (almost tied with the 1st) out of 31 teams on the technique classification subtask with an F1 score of 0.62.

It Takes Three to Tango: RoBERTa, CRF, and Transfer Learning

We present the MULTISEM systems submitted to SemEval 2020 Task 3: Graded Word Similarity in Context (GWSC). We experiment with injecting semantic knowledge into pre-trained BERT models through fine-tuning on lexical semantic tasks related to GWSC. We use existing semantically annotated datasets and propose to approximate similarity through automatically generated lexical substitutes in context. We participate in both GWSC subtasks and address two languages, English and Finnish. Our best English models occupy the third and fourth positions in the ranking for the two subtasks. Performance is lower for the Finnish models which are mid-ranked in the respective subtasks, highlighting the important role of data availability for fine-tuning.

Fine-tuning BERT for Lexical Meaning

CoSimLex is a dataset that can be used to evaluate the ability of context-dependent word embeddings for modeling subtle, graded changes of meaning, as perceived by humans during reading. At SemEval-2020, task 3, subtask 1 is about ”predicting the (graded) effect of context in word similarity”, using CoSimLex to quantify such a change of similarity for a pair of words, from one context to another. Here, a meaning shift is composed of two aspects, a) discrete changes observed between different word senses, and b) more subtle changes of meaning representation that are not captured in those discrete changes. Therefore, this SemEval task was designed to allow the evaluation of systems that can deal with a mix of both situations of semantic shift, as they occur in the human perception of meaning. The described system was developed to improve the BERT baseline provided with the task, by reducing distortions in the BERT semantic space, compared to the human semantic space. To this end, complementarity between 768- and 1024-dimensional BERT embeddings, and average word sense vectors were used. With this system, after some fine-tuning, the baseline performance of 0.705 (uncentered Pearson correlation with human semantic shift data from 27 annotators) was enhanced by more than 6%, to 0.7645. We hope that this work can make a contribution to further our understanding of the semantic vector space of human perception, as it can be modeled with context-dependent word embeddings in natural language processing systems.

Combining BERT with WordNet sense embeddings to predict graded word similarity changes

Applying pre-trained transformer to select emphasis for written text in visual media [1]. We propose PointWise Regression and PairWise Ranking for the emphasis modeling. We also apply data augmentation and hand-craft features to boost the performance. We have a full investigation across different pre-trained transformers, such as ERNIE 2.0, XLM-ROBERTA, ROBERTA and ALBERT. And we finally take the 1st place in this competition.


Learning Word Emphasis Selection by Pre-trained Language Model

Memes have become an ubiquitous social media entity and the processing and analysis of such multimodal data is currently an active area of research. This poster presents our work on the Memotion Analysis shared task of SemEval 2020, involving the sentiment and humor analysis of memes. We propose a system which uses different bimodal fusion techniques to leverage the inter-modal dependency for sentiment and humor classification tasks. Out of all our experiments, the best system improved the baseline with macro F1 scores of 0.357 on Sentiment Classification (Task A), 0.510 on Humor Classification (Task B) and 0.312 on Scales of Semantic Classes (Task C).

Bi-Fusion Techniques for Deep Meme Emotion Analysis

This paper presents the team BRUMS submission to SemEval-2020 Task 3: Graded Word Similarity in Context. The system utilises state-of-the-art contextualised word embeddings, which have some task-specific adaptations, including stacked embeddings and average embeddings. Overall, the approach achieves good evaluation scores across all the languages, while maintaining simplicity. Following the final rankings, our approach is ranked within the top 5 solutions of each language while preserving the 1st position of Finnish subtask 2.

Contextualised Embeddings for Predicting the (Graded) Effect of Context in Word Similarity

The poster presents our work on Semeval Task 4. Aside from creating competitive English models, we evaluate different approaches to using the system in the Czech language. We also show how models from subtask B can be used to rerank and improve outputs of models in subtask C. Inversely, we demonstrate that accuracy of subtask C models on subtask B correlates better with human judgement than BLEU score, which was used for the official evaluation.

Multilingual commonsense

While Social media platforms can be valuable, they can also be exploited to promote negative views about an individual, race, or group with millions of people. There is thus an urgent need to develop mechanisms by which offensive language can be automatically detected in social media and managed in a timely manner.
We propose an ensemble approach that combines different levels of word embedding models and transfer learning from other sources of emotion-related tasks. The proposed system ranked 9th out of the 52 entries within the Arabic Offensive language identification subtask.


An Ensemble of Different Word Embeddings and Emotion Transfer Learning for Arabic Offensive Language Identification in Social Media

This paper describes our system submitted to task 4 of SemEval 2020: Commonsense Validation and Explanation (ComVE) which consists of three sub-tasks. The challenge is to directly validate whether the system can recognize natural language statements that make sense from those that do not and also require to generate reasonable explanation. Based on BERT architecture with the multi-task setting, we propose an effective and interpretable “Explain, Reason and Predict” (ERP) system to solve the three sub-tasks about commonsense: (a) Validation, and (c) Explanation, (b) Reasoning, following the order of the competition. Inspired by cognitive studies of common sense, our system ﬁrst generates a reason or understanding of the sentences and then choose which one statement makes sense, which is achieved by multi-task learning. The rational experiment validates our assumption and boosts performance. During the post-evaluation, our system has reached 92.9% accuracy in subtask A (rank 11), 89.7% accuracy in subtask B (rank 8), and a BLEU score of 12.9 in subtask C (rank 9).

Downloads

Next from COLING 2020

Combined Semantic Vector Spaces Models for Unsupervised Lexical Semantic Change Detection

Similar lecture

Study of Multimodal Techniques for Memes Analysis

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

Combined Semantic Vector Spaces Models for Unsupervised Lexical Semantic Change Detection

Similar lecture

Study of Multimodal Techniques for Memes Analysis

Downloads