Sentimenti = emotions. We have already written how to analyse emotions correctly whilst having a way to automatically measure or classify them. Today we will tell you how we have collected data to create Sentimenti tools. So, once again, we are writing about how to do something right – to research the emotional meaning of texts.
The text was written on the occasion of a speech at the GHOST Day machine learning conference. Here you can review the presentation shown there, in Polish.
Sentimenti. What emotions?
In order to train machine learning algorithms and automatically indicate the emotions expressed in the text, we first had to ask people how they feel. Such a simple question had to be broken down into several components.
First, what sort of emotions? How do we know how many are there, how they differ, what number of categories will be optimal? This was handled by emotion specialists, people from the LOBI team. In psychology there are several models of emotions, from very simple to complex and multidimensional. We decided on two, which we currently call simply models of sentiment and emotion.
According to the sentiment model proposed in the 1977 article by Russell and Mehrabian, every emotion can be described on two axes: positive-negative and high-low agitation. As far as the model of emotions is concerned, as we have written many times, the Plutchik model won. Apart from scientific reasons, he was supported by the practical ones – part of the Polish Slavic network has already been described emotionally according to him. Thanks to the use of the same model (in the corrected translation, which we also had to work on) we could compare the results of our research with the expert annotations available in this resource. This is one of our tests of the accuracy of results.
What words?
We surely know how to annotate, what though? Our first test was to focus on the emotional meaning of words. We decided to compare our results with the words from the WordNet, and additionally with the NAWL (formerly created by our colleagues from the LOBI). Therefore, some parts of the words must coincide with these resources.
We have strived to create a list of 30,000 words or meanings – after all, there are many ambiguous words, whose emotional overtones also change depending on the context the certain word is used. For example, depression or lowering is something completely different from mood disorders. We have decided we can list a maximum of three meanings of one word, and to show the participants which of these meanings we will show each one in a short phrase: depressed terrain, depression treatment.
Thanks to the WordNet annotation project (our people are also working on it), we knew that about 27% of Polish words have some emotional meaning. From this point of view, they are more interesting than neutral, so they took precedence. Besides, we controlled our list of words in terms of word frequency (to have more of these frequent, but also an appropriate percentage of rare).
Sentimenti project. What about the participants?
We wanted to be able to say something about the overtones of texts written in Polish. For this to happen, we had to find out how an average, typical language user understands them. According to the rules of scientific art, we had to examine a representative group of Poles – one whose structure corresponds to the structure of the population in terms of age, education and other important characteristics.
Such a survey cannot be completed simply through the Internet, in social media or catching people on the street. We needed professionals, therefore we used the services of a nationwide research panel. Our inquiry was won by Ariadna. This company found the participants for us, but we also controlled the course of the survey. We had to provide an appropriate number of grades for each word (minimum 50 people), then give survey participants identifiers to be able to compare their answers with data on age, place of residence and others. 20 thousand people participated in the study.
How to ask about the overtones of words?
Sentiment and emotional arousal are usually described on a scale. Models of emotions are rather categorical – they point out if there is emotion or not. Unless we are talking about Plutchik’s model, which immediately assumes that the intensity of emotions can change: from annoyance through anger to rage, for example. In our model we still wanted to add a zero point, no emotion from the anger spectrum.
After performing many analyses and pilots we created our tool for annotating emotions, thanks to which every meaning of a word written in a phrase could be assessed on scales which are reflecting the operationalisation of sentiment models and basic emotions. The participants of the study pointed out the overtones of the word through the interface shown below. Previously, they had also received an accurate, simple written instruction, which they could return to at any time during the study.
A significant stage in the study’s construction was to determine how many words each participant should see. We couldn’t take too much time for them, as people read the text less carefully and would often give careless answers. The ideal arrangement turned out to be 150 words (phrases) in three rounds. The breaks between the rounds could last even several days – for us, the time spent on each phrase counted (too short and too long had to be rejected from the results).
More than words
Our next study was supposed to be focused on the overtones of texts. For a long time linguists know that from the meaning of a word to the meaning of the whole text leads the winding road. The grammar and layout of the text also express emotions.
The project assumed that we would examine some of the existing opinions – for example mentions about hotels and doctors. It was not difficult to collect opinions, additionally they could be divided into positive and negative, based on the accompanying stars or point scores. The challenge was to select domain-neutral texts. We enriched our corpus with shorter forms, sentences and phrases, coming from Paralingua corpus or our pilot studies, amongst the others. Slightly later, our colleagues working in the WordNet team also created a corpus of opinions (annotated emotionally not by the participants of the research, but by linguists).
To make the results comparable to those derived from the words, we examined the overtones of the texts with the participation of a representative group of people on the same scales as we did with the words. This study involved 2,000 people and 7,000 texts and phrases. Each participant read 50 texts, and each text was rated by at least 25 people.
First people, then AI
Currently, our Sentimenti tools for classifying texts by emotion and sentiment achieve high accuracy for every emotion. The most modern, recognisable and… fancy neural networks or other algorithms cannot perform the same without gathering a solid portion of data. We were able to teach our artificial intelligence of emotions just because we collected a well constructed database of emotional words and texts evaluated by a representative group of Polish language users first.
We have already described the way we constructed and taught algorithms for automatic analysis of emotions. This was revealed both on the blog and in a scientific publication. A part of our word database (about 20%) will be published as a corpus accompanying the publication describing in detail the data collection and analysis. This means that this resource will be available for scientists from all over the world who want to study emotions in Polish. We want the database to be as interactive as NAWL’s list of affective words, with its own dedicated page.