by Agnieszka Czoska | | Sentimenti research
Dr Jan Kocoń is a natural language engineer and the person behind the machine learning process within SentiTool, our solution for analyzing emotions in the text. Dr Kocoń coordinates the work of the linguistics team, integrates individual elements of the tool, and works closely with the IT team.
If you have to describe Sentimenti and the tools to anybody, what would you say first?
Sentimenti is a project meant to analyze emotions hidden in the text. Unlike competitive solutions that recognize the overtones of the text only (positive, neutral or negative), our tools manage to understand the text, assign specific meanings to the words in the text and name the certain emotions people feel about them. These emotions, in turn, provide the knowledge base for a machine learning mechanism that automatically recognizes emotions at the level of sentences and the whole text.
What does it mean that we analyse emotions in the text?
In the research carried out in our project we adapted the Plutchik model. It includes eight basic emotions: joy, sadness, trust, repulsion, expectation, fear, surprise and anger. We are able to estimate to what extent these emotions are expressed in the text.
How do we know what emotions people feel?
The knowledge base that helps our project includes more than 30.000 meanings of words, for which 20.000 unique respondents assign ratings for overtones and emotions. We are talking about “meanings” and not “words” on purpose, because words are ambiguous; for example “dark” means something different in “dark blue” or “dark people” and only in the latter case it carries emotions. Each meaning will ultimately receive 50 marks from different people. This allows us to know what feelings are evoked by certain meanings in the text. However, the emotion of the text is not a simple summation of the emotions assigned to the meanings in the text...
What else makes the emotion analysis tools in the text work?
Two things come to us to help. The first one is our gargantuan database of opinions. It came with associated overtones, derived from different areas: travel, medicine, products, services and more. We have over 10 millions of such texts in our database, which is an excellent source of information about the general feeling of the author. However, in order to find out what emotions a given text evokes in the reader, we also conduct our own research, analogous to research on single meanings.
This time the subject of these studies is the texts. The respondents attribute basic emotions to them, exactly the same way as they do with meanings of the words.
The second pillar of our Sentimenti tool is a combination of various machine learning methods. Experts in natural language processing provide us with tools for text analysis at the syntactic and semantic level, additionally they create rules for the analysis of meanings in context such as: negation, conjecture, weakening or strengthening of the overtones, etc. This is an additional help for automatic methods, such as deep neural networks, which are used to make the right conclusions about the emotions in the measured text.
What do you think automatic emotion analysis can be useful for?
Ultimately, I see many applications for our tools. The very first area that comes to my mind would be the marketing, or, more precisely, display advertising. This certain area covers the market of advertisements displayed in the context of web articles and is matching them with the emotions that the text of the publication evokes in readers. For example, in a sad text there could be an advertisement of an insurance company, and in a merry, joyful text there could be an advertisement for a trip.
Another area that we could cover is brand monitoring, i.e. analyzing how companies’ customers write on the Internet about a given company, its products and what emotions accompany them. Another interesting area could be sorting customers’ email complaints against the emotions contained in them, detecting conflicts arising in employee correspondence, detecting upcoming crises in Social Media, and even the possibility of diagnosing mental illnesses – the potential of Sentimenti tools is really huge!
What else do you plan to do in Sentimenti?
So far, there is a prototype ready with a simple text analysis on the level of meanings with an overtone analysis using our huge opinion resources. Currently in the Sentimenti team in Wroclaw I am managing to build a machine learning mechanism. It will make it possible to aggregate both information from the meaning knowledge base and information from the natural language processing stream. We are constantly receiving new data about the feelings of people reading certain texts, which are our teaching collection. The more data we gather, the better the quality of the tool there is.
by Agnieszka Czoska | | Sentimenti research
You’ve been in Sentimenti from the beginning. What was it like in 2016?
The business idea for the study of emotions in the text came from W3A.PL company from Poznan. After consultations with the environment of Poznań psychologists, cognitive scientists and linguists, a draft of the project for NCBiR (National Centre for Research and Development) was prepared and the search for subcontractors started. After estimating the market, it turned out that two units are capable of undertaking such advanced research work: LOBI IBD PAS and Language Technology Group of Wrocław University of Technology.
Once you got the grant, how did you start working?
As a research manager I was responsible for organizing the work of the team. It was important for me to combine the scientific teams of subcontractors and the business team into one team. The interface between business and science is not easy. In the Sentimenti team everyone – presidents, PhDs and MSc – speaks to each other by name, each person has the right to express their opinion and make decisions.
You are the research manager and scrum master of our team – how much did you have to learn to become one?
I learned the Scrum management methodology for R&D projects in the UK, where I worked in the Argument Analytics project conducted in cooperation with the University of Dundee and financed by Innovate UK, the British equivalent of NCBiR. I understood then that the key issue in the cooperation between science and business is good communication. A common team, preferably working in one place, frequent meetings and evaluation of results to check if this is really what we want – this is the heart of good projects. Many other R&D projects that I have observed did not achieve their goals precisely because of such a lack of communication.
How does the scrum method differ from your previous project experience?
I am a scientist and I have gained most of my experience in academic work and basic research. The transition to applied research was not easy, but I was given a lot by the British culture of openness, communication and respect – the values that are inscribed in Scrum and that we transfer to our team. The three pillars of Scrum are also important: transparency, inspection and adaptation. Transparency means that every person in the team – even new and unfamiliar with the subject – has access to all information (except, of course, confidential information). This helps a lot in overcoming crises, looking for a solution.
And what are inspection and adaptation?
An inspection is a frequent and short “review” meeting, during which we check what has already been completed, whether we do not have any obstacles that the project management should deal with, whether someone has too much or too little work. This helps to master the natural feature of research projects – unpredictability. When the results are different from we expected or when we get information from the business that a solution is not working – we can quickly adapt.
How do you see further development of Sentimenti?
In February, we have already finished our research work and moved on to development work, i.e. we use the collected knowledge and data in the work on Sentitol – our main tool for text analysis. Thanks to the fact that we use an iterative approach, we implement functionalities by adding them in subsequent versions of the product, and simultaneously – according to the Scrum methodology – we finish each Sprint (stage of work in Scrum) with a working product. At the moment, we have working software that recognizes eight emotions in texts in Polish, thanks to research on over 20 thousand people. This is already a solution that exceeds the scope of other solutions present on the market, and we are preparing two more versions.
In the next version of Sentimenti we will include a module using LSS (Lexical Syntactic Structures), i.e. elements of the language that affect the evaluation, e.g. good + no, + very, + a little. Then we will include a module that uses deep neural networks technology, or more precisely – BiLSTM (bidirectional long short-term memory neural networks), so that it can evaluate the emotions throughout the text immediately – and this is a unique solution on a Polish scale, but also worldwide. Our scientific publication about this module will be published soon.
Therefore, in the project we use fast prototyping, and in parallel to the work of the scientific team, the company implements any new solution for customers – because we have a great interest in our solutions. Thanks to this we have already achieved much better results (and faster) than we planned at the beginning.
by Agnieszka Czoska | | Conferences, Sentimenti research
Sentimenti = emotions. We have already written how to analyse emotions correctly whilst having a way to automatically measure or classify them. Today we will tell you how we have collected data to create Sentimenti tools. So, once again, we are writing about how to do something right – to research the emotional meaning of texts.
The text was written on the occasion of a speech at the GHOST Day machine learning conference. Here you can review the presentation shown there, in Polish.
Sentimenti. What emotions?
In order to train machine learning algorithms and automatically indicate the emotions expressed in the text, we first had to ask people how they feel. Such a simple question had to be broken down into several components.
First, what sort of emotions? How do we know how many are there, how they differ, what number of categories will be optimal? This was handled by emotion specialists, people from the LOBI team. In psychology there are several models of emotions, from very simple to complex and multidimensional. We decided on two, which we currently call simply models of sentiment and emotion.
According to the sentiment model proposed in the 1977 article by Russell and Mehrabian, every emotion can be described on two axes: positive-negative and high-low agitation. As far as the model of emotions is concerned, as we have written many times, the Plutchik model won. Apart from scientific reasons, he was supported by the practical ones – part of the Polish Slavic network has already been described emotionally according to him. Thanks to the use of the same model (in the corrected translation, which we also had to work on) we could compare the results of our research with the expert annotations available in this resource. This is one of our tests of the accuracy of results.
What words?
We surely know how to annotate, what though? Our first test was to focus on the emotional meaning of words. We decided to compare our results with the words from the WordNet, and additionally with the NAWL (formerly created by our colleagues from the LOBI). Therefore, some parts of the words must coincide with these resources.
We have strived to create a list of 30,000 words or meanings – after all, there are many ambiguous words, whose emotional overtones also change depending on the context the certain word is used. For example, depression or lowering is something completely different from mood disorders. We have decided we can list a maximum of three meanings of one word, and to show the participants which of these meanings we will show each one in a short phrase: depressed terrain, depression treatment.
Thanks to the WordNet annotation project (our people are also working on it), we knew that about 27% of Polish words have some emotional meaning. From this point of view, they are more interesting than neutral, so they took precedence. Besides, we controlled our list of words in terms of word frequency (to have more of these frequent, but also an appropriate percentage of rare).
Sentimenti project. What about the participants?
We wanted to be able to say something about the overtones of texts written in Polish. For this to happen, we had to find out how an average, typical language user understands them. According to the rules of scientific art, we had to examine a representative group of Poles – one whose structure corresponds to the structure of the population in terms of age, education and other important characteristics.
Such a survey cannot be completed simply through the Internet, in social media or catching people on the street. We needed professionals, therefore we used the services of a nationwide research panel. Our inquiry was won by Ariadna. This company found the participants for us, but we also controlled the course of the survey. We had to provide an appropriate number of grades for each word (minimum 50 people), then give survey participants identifiers to be able to compare their answers with data on age, place of residence and others. 20 thousand people participated in the study.
How to ask about the overtones of words?
Sentiment and emotional arousal are usually described on a scale. Models of emotions are rather categorical – they point out if there is emotion or not. Unless we are talking about Plutchik’s model, which immediately assumes that the intensity of emotions can change: from annoyance through anger to rage, for example. In our model we still wanted to add a zero point, no emotion from the anger spectrum.
After performing many analyses and pilots we created our tool for annotating emotions, thanks to which every meaning of a word written in a phrase could be assessed on scales which are reflecting the operationalisation of sentiment models and basic emotions. The participants of the study pointed out the overtones of the word through the interface shown below. Previously, they had also received an accurate, simple written instruction, which they could return to at any time during the study.
A significant stage in the study’s construction was to determine how many words each participant should see. We couldn’t take too much time for them, as people read the text less carefully and would often give careless answers. The ideal arrangement turned out to be 150 words (phrases) in three rounds. The breaks between the rounds could last even several days – for us, the time spent on each phrase counted (too short and too long had to be rejected from the results).
More than words
Our next study was supposed to be focused on the overtones of texts. For a long time linguists know that from the meaning of a word to the meaning of the whole text leads the winding road. The grammar and layout of the text also express emotions.
The project assumed that we would examine some of the existing opinions – for example mentions about hotels and doctors. It was not difficult to collect opinions, additionally they could be divided into positive and negative, based on the accompanying stars or point scores. The challenge was to select domain-neutral texts. We enriched our corpus with shorter forms, sentences and phrases, coming from Paralingua corpus or our pilot studies, amongst the others. Slightly later, our colleagues working in the WordNet team also created a corpus of opinions (annotated emotionally not by the participants of the research, but by linguists).
To make the results comparable to those derived from the words, we examined the overtones of the texts with the participation of a representative group of people on the same scales as we did with the words. This study involved 2,000 people and 7,000 texts and phrases. Each participant read 50 texts, and each text was rated by at least 25 people.
First people, then AI
Currently, our Sentimenti tools for classifying texts by emotion and sentiment achieve high accuracy for every emotion. The most modern, recognisable and… fancy neural networks or other algorithms cannot perform the same without gathering a solid portion of data. We were able to teach our artificial intelligence of emotions just because we collected a well constructed database of emotional words and texts evaluated by a representative group of Polish language users first.
We have already described the way we constructed and taught algorithms for automatic analysis of emotions. This was revealed both on the blog and in a scientific publication. A part of our word database (about 20%) will be published as a corpus accompanying the publication describing in detail the data collection and analysis. This means that this resource will be available for scientists from all over the world who want to study emotions in Polish. We want the database to be as interactive as NAWL’s list of affective words, with its own dedicated page.
by Agnieszka Czoska | | Sentimenti research
The analysis of emotions in Sentimenti is based on the results of research on what emotions Polish language users associate with specific words, phrases and longer texts. Thanks to the results of subsequent research stages, our tool is still being improved. What research have we conducted so far?
Thousands of words and research participants
The most important part of the project is to collect evaluations of over 30 thousand words, phrases and texts from over 20 thousand Poles. Due to the scale of the project, we decided to carry out the research using two methods: CAPI (Computer Assisted Personal Interview) and CAWI (Computer Assisted Web Interview). Thanks to this, we were able to combine the benefits of both approaches: CAPI research allowed us to maintain strict control and high reliability of the research, while CAWI research allowed us to effectively reach such a large, representative group of Poles. Finally, with CAPI and CAWI data at our disposal, we can directly compare the assessments collected through both methods and check whether the research conducted online is as reliable as that conducted in a laboratory environment.
EmoTool – our research tool
In both studies we used our proprietary EmoTool application, through which the respondents indicated the emotions associated with particular words. The application includes 2 basic evaluation panels: the emotional dimension panel and the emotional category panel. Emotional dimension panel (left side) is used to determine the emotional overtones in the most general sense, i.e. to determine the direction and strength of emotions. The panel of categories of emotions (right side) allows to determine with which basic emotions a given word is associated to the respondents.
Combination of CAPI and CAWI methodology
What data did you collect? A total of 560 came to the laboratory, and more than 20 thousand unique respondents took part in the online research. CAPI participants evaluated nearly 3000 words, while CAWI participants evaluated over 30000 words (including all words from CAPI). Thanks to that, the biggest in Poland and one of the biggest in the world database of emotionally tagged words was created. Each of them has been evaluated at least 50 times, thanks to which we are able to estimate how particular words are perceived in the Polish population. At the same time, we are dealing with a representative group of Poles, which additionally increases the credibility of the results. The collected demographic data will enable the SentiTool tool to be tuned for more specific applications, profiled for selected demographic groups. In other words, we will be able to approximate not only the average reception of a given text, but also how it may vary depending on the age, beliefs or education of the recipient.
One of the most important conclusions from the research to date is that the results of CAWI and CAPI have proved surprisingly consistent. This means that both methods allow us to obtain qualitatively similar assessments. This is important information for us, because it turns out that by conducting research on such a large scale, we do not give up the high quality of the collected data. In other words, we have shown that emotions can be asked about on the Internet as effectively as in the laboratory. What is more, using this method, we can examine emotions related not only to the text but also to other material chosen by the Client, such as emojis, logos, graphics, recordings or video clips.
The most positive and negative words
Among the words considered by the respondents to be the most positive were:
beautiful, loving, sunny, cheerful, tender, gift, delight, joy, warmth and care.
In turn, the worst associations were found:
terrorist, beatings, war, paedophile, genocide, violence, aggression, pervert, murder and cheating.
These results are quite intuitive and thus confirm the reliability of the data collected by Sentimenti. The next comparison leads us to less obvious conclusions. Sometimes it seems to us that negative statements are more emotional, more striking, just stronger. Meanwhile, in the top ten, the words on the scale of strength are almost all positive:
to love pleasure, kiss, orgasm, mother, aggressive, in love, dinner, pass and euphoria..
The results of the research gathered so far through the EmoTool application are directly applicable to SentiTool, our automatic text analysis tool. Although the Sentimenti project can already boast several successes (for example, SentiStock analyses), we are still working on improving our tools. We have recently completed another stage of research, where participants evaluated not only words, but also phrases and whole texts. The data collected in this study will allow us to take a broader context into account in the process of text analysis and thus increase the effectiveness and accuracy of emotion detection.
The text was written together with PhD Monika Riegel and PhD Małgorzata Wierzba from LOBI.
by Agnieszka Czoska | | Sentimenti research
Each data analysis is aimed at understanding what information it contains. Has something changed, or is there a difference between A and B? Do changes in A correlate with those in C or D? Only these steps allow us to draw conclusions about the results.
First stage: measurement. What is the text?
The above statement also applies to the analysis of emotions or sentiment. Its first stage is the MEASUREMENT, checking how many and what kind of emotions we find in a given text or set of them. The result of a simple emotion measurement shows the intensity of each of Plutchik’s 8 basic emotions supplemented by positive and negative sentiment and arousal (overall emotional temperature of the text). Sometimes we can afford to interpret it already at this stage. We did it in one of our first entries, where we analyzed short ads (what is important, we have already managed to improve the way the results are presented). By analyzing the ads we wanted to show something characteristic for the whole type of texts: the most important emotions are joy and trust, only at the very beginning of the story about the product the creators allow themselves to remember the negative ones – to show the hardships of life before the era of the best shampoo or grease in the world.
The correct results of the Emotional Measurement are those that are consistent with people’s feelings, after all, each of us is an expert on feelings. Our tool owes its correctness to the participants of research on emotions in Polish, which we conducted according to the best scientific standards.
Measurement is only the first step towards understanding the message and the emotions it contains. When we deal with many similar texts or collections of texts, we have to do something else. We want to find out which shop has the best opinions? Which version of our marketing content expresses the most enthusiasm or best shows interest in the subject? Which of the texts in the “Beauty” section will delight, move or warn the reader? We are talking about COMPARISON.
The second stage: comparison. Does this text differ from the average?
Comparison is perhaps the most important stage in the analysis of emotions – thanks to it we not only find out what the text is like, but also how it compares with others. We can compare directly – as we did when writing about lipsticks and lipsticks. Then we were interested in which of the topics has an advantage in terms of positive emotions and whether this difference is statistically significant. However, comparing several or a dozen or so different cosmetic brands cannot be done in this way, it would not be the correct approach. That is why in the text about beauty companies we used a comparison to the average – we needed some kind of background measurement, so-called baseline. This approach will be useful, for example, when comparing shops and brands. We then answer the question which brand has better or worse results than most of the industry.
The most general type of baseline would be the sum of emotions that characterize not only the domain, portal or texts of a given author, but simply language. In linguistics, the so-called Polyanna effect is known, which is that there are more positive than negative expressions in every language. Not only in the dictionary, but also in what we say – this effect expresses quite a general tendency of our minds to spend time and energy rather on pleasant things. In our research we very often see this tendency – joys and trust are emotions that appear in the greatest intensity not only in advertising. The fact that language has its emotional mean all the more reason to draw conclusions only based on comparison and not on the measurement itself.
Third stage: trends. What do emotions do?
The analysis of emotions is also about tracking changes in time, i.e. monitoring emotions. We can check whether sadness or revulsion show a growing trend, i.e. there are more and more of them in statements on a given topic or in the opinions of customers. If we notice a trend, which is statistically significant, we can predict what will happen in the future and if by chance it does not mean an impending crisis (depending on the slope of the trend line).
At this stage it is also possible to go beyond the data from the Sentimenti tools. We started with something simple, accessible and yet untouched by others – we compared the emotional temperatures of mentions of listed companies with the prices of their shares, published publicly. Sentistock is great, it allows you to determine what the investor mood really is and how it translates into stock market fluctuations.
This part of the analysis of emotions depends entirely on who and for what purpose wanted to examine the overtones of the text, the notes, the conversation. We have also managed to show which emotions correlate positively with reactions on Facebook and Twitter – that is, how to write, so that the observers would like to like or comment on the post. However, we might as well ask how emotions correlate with remembering information from the text. Studies on the psychology of emotions, including those conducted by our colleagues from LOBI, indicate that the overtones of the text have an impact on what and how well we remember. Correlation between customer feedback and online store sales? Our tools are designed for this type of research.
Why so many stages?
Emotions were not created for themselves. This is our advisory mechanism: they tell us what action to take. Tversky and Kahneman did not receive the Nobel Prize for their research, but for showing that the consumer, including the stock market, is not rational. This statement tells us two things:
- emotions shape the market,
- we need good tools and methods to study this impact.
Trying to understand the emotions “on the eye” we won’t know more than the average customer wanting to buy a new computer, reading all the available reviews and then deciding on the brand for which he or she has (and had) the warmest feelings. Maintaining scientific standards, checking whether differences and trends are statistically significant and even better correlate with other, harder indicators is the best way to find out. After all, we live in the era of big data and data analysis.
