My Issue with BigData Sentiment Bubble: Sorry, Which Is the Variance of the Noise? (NON Verbal Communication)


Why sentiment analysis is so hard? How to interpret the word “Crush” in a tweet? Crush as in “being in love” or Crush as in “I will crush you”? According to Albert Mehrabian communication model and statistics, I would say that on average a tweet for a sentimenter has an accuracy of 7%. No such a big deal, isn’t it?

Let’s think about it by considering, as an example, the case of the sentiment analysis described in My issues with Big Data: Sentiment: crush as in “being in love” (positive) or crush as in “I will crush you” (negative)?

What is a sentimenter? As a process, is a tool that from an input (tweets) produce an outupt like “the sentiment is positive” or “the sentiment is negative“. Many sentimenters are even supposed to estimate how much the mood is positive or negative: cool!

Paraverbal and non-verbal communication

Anyhow, according to Albert Mehrabian the information transmitted in a communication process is 7% verbal, 38% paraverbal (tone of the voice) and the remaining 55% is non-verbal communication (facial expressions, gestures, posture,..).

In a Tweet, as well in a SMS or e-mail, neither paraverbal nor non-verbal communication are transmitted. Therefore, from a single tweet is possible to extract only the 7% of the information available: the text (verbal communication).

So, what about the paraverbal and non verbal communication? During a real life conversation, they play a key role since they count for 93% of all the message. Moreover, since paraverbal and non verbal messages are strictly connected with emotions, they are exactly what we need: sentiments!

Emotions are also transmitted and expressed though words such as “crush” in the example mentioned. However, within a communication process, not always the verbal and non-verbal are consistent. That’s the case when we talk with a friend, he\she saiys that everything is ok while we perceive, more or less consciously, something different from his\her tone or expressions. Thus we might ask: are you really sure that everything is ok? As a golden role, also for every day life, I would recommend to use non-verlbal signals as an opportunity to make questions rather than inferring mislead answers (see also: A good picture for Acceptance: feel the divergences & think how to deal with).

For these reason, the non-verbal messages are a kind of noise that interferes with verbal communication. In a tweet, it is a noise that interferes with the text. Such a noise can be as much disturbing as much the transmitter and the receiver are sensitive to the non-verbal communication. It might be so much disturbing to change completely the meaning of the message received.

Statistic and Information Theory

From a statistic point of view the noise might be significantly reduced by collecting more samples. In Twitter, a tweet is one sample and each tweet have 7% of available information (text) and 93% of noise (non verbal communication) that is the unknown information.

From a prediction\estimation point of view no noise means no errors.

Thus, thanks to BigData, if the sentimenter analyzes all the tweets theoretically it’s possible to reduce the noise to zero and thus having no prediction error about sentiments…...WRONG!!!

Even if the sentimenter is able to provide a result by analyzing all the BigData tweets (see Statistical Truisms in the Age of Big Data Features):

the final error in our predictive models is likely to be irreducible beyond a certain threshold: this is the intrinsic sample variance“.

The variance is an estimation of how much samples are different each others. In the case of a communication process, that means how much emotions are changeable through time. Just for fun, next time, try to talk to a friend by changing randomly your mood happy, sad, angry,..and see what happen with him\her (just in case, before fighting tell him\her that is part of an experiment that you’ve read in this post).

In Twitter, the variance of the samples is an estimation about how much differently emotions are impacting the use of certain words in a tweet, from person to person at a specific time. Or, similarly, by considering one person, how much emotions are impacting the use of words differently through time.

Like in a funnel (see picture), the sentimenter can eliminate the noise and thus reduce the size of the tweet bubbles (the higher the bubble the higher the noise) till a fixed limit that depends on the quality of the sample: its variance.

Sentimenter_Twitter_Funnel

So, I have a question for bigdata sentimenters: which is the sample variance of tweets due to non-verbal communication? Acknowledge the sample variance, the error of prediction of the best sentimenter ever is also given:

error of prediction (size of the bubble sentiment) = sample variance of tweets…

…with the assumption that both samples and algorithm used by the sentimenter are not slanted\biased. If this is not the case, the sentiment bigdata bubble might be even larger and the prediction less reliable. Anyhow, that is another story, another issue for BigData sentimenters (coming soon, here in this blog. Stay tuned!).

Feelink – Feel & Think approach for doing life!

Advertisements

Tripadvisor: a case study to think why bigdata variety matters


tripaadvisor

The recent scandals about fake reviews has put the reliability of TripAdvisor under discussion (see The Guardian).

Such a bad quality of service is not useful for consumers, entrpeneurs as well as in the long run for the reputation of TripAdvisor. So, where is the problem?

Clearly it’s a question of reliability of the sources of information and specifically for TripAdvisor is a question of assessing the reliability of the user that post a new review. Nice and easy…like discovering the hot water. However, thinking also at the practice of the so-called Negative SEO, that is not only an issue of web sites like TripAdvisor but also for all the companies that have to promote theirs brands in the social networks (who think doesn’t need it, raise up the hand).

In order to fix the issue, Tripadvisor developed the service Report Blackmail that tracks and eventually bans the users that are using Negative SEO tactics. For example, 100 user managed by a restaurateur that are reporting cases of colitis and runs in the reviews of the competitor near the corner. Such a solution try to catch fake users when they’ve already done the “attack” as well as, if not properly working, it might ban by mistake honest users. It sound reactive rather proactive, isn’t it?

So, are there other approaches that can fix the problem of malicious reviews proactively? An idea could be use new IT bigdata technologies and re-think the business model. How? (see also MIT Sloan Management Review: technology as a catalyst for change).

An approach could be associating the Tripadvisor user with a unique ID, for example a TripAdvisor idetity card, while to restaurateurs and hotel managers have an ID card reader (RFID, infrared,etc.). Thus, once the consumer eat the meal and goes to pay the restaurateur track the consumer ID that univocally identify the user, plus time and position. Finally, the user have just to fill the form for his\her review that now can be fully validated. Potentially, once the users sign in the TripAdvisor website, a list of pending reviews not already filled might be also provided in order to facilitate the process and thus creating the so-called “customer experience”. Moreover, by tracking precisely the date, it is also possible to provide evaluations that are more meaningful for the customer by giving less importance to aged reviews.

With the technology currently available actually even a smart phone could be a card reader since it might equipped with a RFID or a magnetic stripe reader and, by developing a specific app, the restaurateur could easily and quickly transmit a transaction with the TripAdvisor ID of the customer.

tipadvisor_new_business_model

Apart from the solution proposed, that is an example that stresses the importance, when defining a bigdata strategy, to identify first the information that is really meaningful (user, time, position) as well as having a Variety of sources in order to validate the reliability of the data. In the case of Tripadvisor is crucial to correlate the data coming from the restaurateurs with the reviews of the couple customer\user (together!!!).

Thinking about the definition of BigData by Gartner:

Bigdata is high-volume, -velocity and -variety information assets that demand
cost-effective, innovative forms of information processing for enhanced insight and decision making

So, Variety is one of the “Vs (Volume, Velocity and Variety) and the Volume of data is only what is up to the sea level of the iceberg called BigData.

Do you think that variety matter? I think yes, it matters!

If you think so as well and you have the opportunity to visit Italy I would you recommend (personal advice) to enjoy meals in restaurants where are shown logos such as the following and relying on the word of mouth, an evergreen.

Recensioni

They are not implementing Variety like TripAdvisor as well but reviews are made by professionals and they do not have social media and WEB2.0 visibility risks. Of course, I would recommend to find other sources (use variety!).

Have a nice journey and enjoy the meal!

Feelink – Feel & Think approach for doing life!

Chaos vs. Determinism: why not both? From evolutionary theory to BIG Data challenge


How was the Universe created? It was generated by chance or it was created with a specific purpose?

Chaos vs. Determinism is one of the toughest issue to address for philosophers and it has been debated since the age of the ancient Greece.

Is the world nowadays governed by chaos or determinism? Hard to say, but what I notice is that sometimes Chaos and Determinism together might create an outstanding synergy. When? Here there are at least four examples: the Evolutionary Theory, the New Product Design process, the Lateral & Vertical thinking and the challenge of Big Data with Social Media.

1.  Evolutionary theory

The Darwin’s evolutionary theory is undoubtedly the most meaningful example of how chaos and determinism can work very well together…otherwise we couldn’t be here to discuss how this world works!

Since also Mother Nature cannot foreseen what will happen in the future, how is it possible to survive? By generating continuosly chaotic genetic mutation in the DNA a thus create a large variety of species: simple and brilliant! The generation of new alternatives, through DNA mutations, happens also when the environment is not changing because such variety of species will more likely guarantee the life in our planet Earth if a big change occurs.

Just think what happened 65milion years ago with the extinction of Dinosaurs. The impact of a big asteroid changed radically the climate and the T-Rex, together with his big friends, wasn’t able to adapt to the new environment condition. What happened is that a new family of species more adaptable escape from the extinction: mammals.

Mother Earth is not efficient like human being tends and likes to be. She is effective, likes redundancy and varieties in order to let the life carry on. How many times financial advisors said? “Diversification! That is the way to mitigate the risk of market’s volatility and uncertainty”. Either they consciously know the evolutionary theory or they are survivors from the natural selection.

2.  New Product Design                                                       

Another example is taken from the business world. Words like innovation, internationalization, diversification, mass-customization, not only have inspired the famous “business lingo bingo” game in order to stay awake during a work meeting, but also they have in common the same objective: continuously create new products. A company that doesn’t invest on the development of new product, in order to fill the customer needs that change through time or to reach\establish new market, is doomed to die.

Anyway, a new product is the result of a process: the New Product Design (NPD).Well, such process is divided into many different stages. Briefly, at the beginning there is brainstorming phase in which are collected all the new ideas in terms of needs without thinking if a new idea makes sense or is not feasible. For example, thinking about a new umbrella: “I want to use an umbrella like a parachute!” Why not? …ok, probably using an umbrella as a parachute is not practicable. So, how to organize and select all the ideas that came out from a chaotic brainstorming? A solution is the so called KJ method invented by Kawakita Jiro. It’s a process that organize, prioritize and select all the needs that really matters in a structured way. Probably also a parachute umbrella, is needed who knows!

Once the needs have been classified, the NPD process analyzes systematically all the needs related to the features required by the new product through the QFD (Quality Function Development) and the Pugh matrix. As a result, there is one or a couple of new solutions that are feasible and that fit all the significant needs. Just in case, if doesn’t cost so much effort, also others additional needs like “parachute umbrella” might be added in the new product in order to be “different” in the market.

Now, considering the brainstorming as a genetic mutations and the KJ\Pugh matrix as a natural selection, does the NPD process is like the evolutionary theory applied to products?

3. Lateral vs. Vertical thinking

Considering again the example of the umbrella parachute, it came out during the brainstorming phase without thinking if it would be feasible or not, while during the NPD process it might be more likely eliminated due to many technical as well as reasonable limitations: is there someone that really need a parachute umbrella?

This is the first distinction that Edward de Bono, the inventor of the lateral thinking, suggets between the Lateral and the Vertical thinking. Respectively, one is productive while the other is selective. Not only, Edward de Bono defines many others adjectives that characterized the vertical and lateral thinking as follow:

Lateral thinking: productive, stimulating, discontinuous, incoherent, do not use negations, open to intuitions, unspecific, less probable, open\probabilistic process.

Vertical thinking: selective, analytical, continuous, coherent, use negations, relevance focused, specific, more probable, close\deterministic process.

According to the adjectives mentioned above the aim of the lateral thinking is to find new solutions\ideas in an incoherent and chaotic way in order see the things from different perspective. On the contrary the vertical thinking select the intuitions in a structure way in order to develop a new coherent model. That’s what happened to the father of Quantum Theory Max Planck.

At the beginning ,when he got the intuition to assume that the energy of the particles can change only in discrete amounts, no less that the so called Planck constant, Max Planck was extremely skeptical because such assumption was not coherent with classical physic. Than many others brilliant minds such Bohr, Heisenberg, de Broglie, Einstein, Schrödinger, Pauli and others demonstrated that the assumption of Plank works with physical phenomena at microscopic scales. A new physic model was born thanks to a winning combination between the lateral and vertical thinking: the Quantum Mechanics.

More: see Lateral Thinking by Edward de Bono.

4. Big Data Challenge

Social Media phenomenon is undoubtedly having significant impacts in the way the people communicate and interact as well as the businesses operate. Some decades ago the main trouble was how gathering the needed information while nowadays it’s the opposite: which information is really relevant? The Big Data is going to address this issue, in order to organize, classify and select the relevant information that is generated almost randomly by billions of sources, me included, in the world. Why the information is generate randomly? Well, the Big Data issue is going to be addressed from the technical point of view and many tangible results has been achieved. Think about the mass-customized advertisement and NPD (new product design, see above).

However, Big Data is not only a question of technology. Also the human factor is interested since the information technology and social media might amplify an irrational behavior of groups by creating the so called Social Object’s effect. Retweets call likes, likes call posts and posts calls retweets again into spiral loop. In fact, as Tom Dickson showed: “It blends!

Anyway, why the social object might stimulate an irrational behavior? Prof. Vincent F. Hendricks from the University of Copenhagen underline the fact that the online discussion take place in a kind of echo chambers: “In group polarization, which is well-documented by social psychologists, an entire group may shift to a more radical viewpoint after a discussion even though the individual group members did not subscribe to this view prior to the discussion” (see Information technology amplifies irrational group behavior). That is because the human behaviuor is highly influenced by the group.

The influence of the grpup is one aspect. Than, when I discovered that a social object in Twitter or Facebook reaches its peak of influence only after two hours and then it rapidly declines I realized that also the time factor might force to an irrational behaviour. If you want to follow the peaks you must react quickly, and when a quick reaction is required the human brain rely to the amygdala by asking: flight or fight?

The amygdala is switched on whenever a dangerous or a stressful situation occurs. The amygdala, since activates quick reactions, saved humans (and other species like rabbits!) from extinction when thousands and thousands of years ago the human being were struggling against predators every day. Fortunately a social object doesn’t hurt like a saber-toothed tiger so there is no risk to die physically, possibly only digitally.

Anyway, in order to fully exploit the chaos generated by the social media, dealing amygdala might be useful in order to navigate rather than drifting in the digital see. So feel, think and than just in case post, tweet and like.

Chaos and Determinism: inseparable twin brothers of knowledge!

Feelink – Feel & Think approach for doing life!