Volume vs. Meaning: Effective Customer Listening Requires More Than Keywords
Is it your job to listen to and analyze social media messages and direct customer feedback about your product, brand, or service? Are you leading corporate initiatives that leverage consumer-generated content to uncover meaningful insights about your business? Effective listening and insights analysis allows you to track not just the volume, but also the meaning of online conversations across a complex web of consumer interaction channels.
One of the most common approaches to making sense of customer conversations is to track the presence of keywords within a defined universe of brand conversations (e.g. forums, communities, social networks). While this method highlights text where individual keywords are being used, this approach is unable to automatically evaluate comments for topics, themes, or sentiment. Nevertheless, many companies have implemented keyword systems as a first step to help sift through large volumes of consumer-generated content on the social web. These same companies have then typically hired people to manually read, categorize, summarize, and report on keyword search results at a considerable expense, and with questionable accuracy.
Keyword-Based Online Customer Listening: Fraught with Bias and Subjectivity
The process of selecting and tracking individual keywords in a sea of unstructured content is fraught with bias and subjectivity. Through a combination of personal experience and research, as business user can assemble groups of words that – if contained in a text record – are indicative of a category or sentiment (e.g., “meaning”). For example, a retention analyst attempting to track customer attrition might set up searches for the words “cancel,” “leaving,” and “switching. However, this approach will likely return false positives and miss many records that do not contain the specified combinations.Moreover, this approach does not provide any information why customers are dissatisfied or at risk. To obtain these insights, the analyst must manually read all the messages flagged.
The way people indicate attrition risk may include hundreds if not thousands of different word/phrase combinations. Further, just because a message contains the word “cancel” in the text, that fact does not necessarily indicate an attrition risk. For example, someone could say, “I had to cancel my credit card because I lost it. Could you please call me so I can change my billing information?” Such a message would be included in the example above, even though the customer does not actually want to cancel service.
In terms of sentiment analysis, keyword-based solutions similarly perform tagging functions based upon libraries of positive and negative words. These “one size fits all” libraries offer the lowest level of accuracy as there are – once again – too many variations on how a customer can express positive or negative emotions in a post.
The value of keyword-based monitoring systems is akin to web search or press clippings for the Internet – i.e., the tracking of brand mentions across a generic, predefined universe of media sites to gain a general sense of volume and buzz across the widest possible spectrum. This information can be helpful for high level corporate metrics and general public relations tracking. However, the effort required to read flagged content – and analyze it for topic and sentiment – is significant, and is often accompanied by human subjectivity and fatigue that contributes to the degradation of the resulting insights. Different people rarely read, interpret, and categorize content in the same way; while individuals are notoriously inconsistent and not motivated to deliver insights from this monotonous process. As such, most attempts to leverage keyword monitoring solutions for detailed analysis are flawed.
Natural Language Processing:A More Accurate Approach to Listening
If keyword-based monitoring systems doesn’t sound like a fit for you, it’s time you consider a Natural Language Processing (NLP) -based customer listening solution.NLP is a scientific approach that enables software to discover and match author intent to a virtually limitless set of words and phrases. NLP-based listening systems assign weights to words and phrases, as opposed to simple “exact match” keyword logic. In the attrition risk example, the word “cancel” will most likely produce a strong weight, and be a good indicator of an attrition risk, however, the other words that appear with it (e.g. would, like, to, please, me, subscription, service, etc.) would also be assigned a weight, all of which would be used to score a verbatim record as true or false for a thematic/sentiment category.
NLP-based processing also relies on machine-learning, whereby the system is given examples of text that are “true” for every category if interest. The system then “bubbles up” words that a user would have never thought of to enter in as search words, but are present in the examples with statistical frequency. This approach helps eliminate bias, produces a much smaller percent of false positives, and also increases the recall (number of messages true for a category).
You’re probably wondering how a machine can automatically create the expansive weighted word lists and associations that correspond to individual categories. In a small sample (a few hundred), a human will categorize the messages with a higher degree of accuracy than any machine. However, when you give a human 100,000 to categorize, they will suffer from fatigue, thus greatly reducing the accuracy. Systems such as Overtone’s OpenMic® have been built to leverage a small sample of human categorization, but then take that sample and use machine learning to categorize the rest. This hybrid approach using humans to train a machine produces categories that are much more accurate (minimizing false positives) and find more mentions (maximizing recall) than simple keyword searches.
NLP-based approaches to text classification enable business users to create categories that center on a theme, as opposed to just a set of keywords. Categories like Sentiment, Loyalty, and Attrition Risk are not reduced to a collection of representative words, but rather, an overall set of weightings, and statistically significant word cues that suggest a topic area. Clearly, NLP-based systems perform better than keyword-based systems. The superiority is not slight, it’s considerable.It doesn’t apply to just a few categories; it applies to nearly every category.
Keywords do of course have their place. For example, tracking categories that don’t really contain a theme (e.g. competitor names, product names, etc.) are best handled by simple keywords.But when it comes to accurately categorizing the theme of a message to inform trending and root cause analysis, an NLP-based solution is the way to go.
Overtone, Inc. is a leader in monitoring and analyzing user sentiment from social media and customer feedback sources. We are interested in sharing our expertise in keyword and statistical analysis of of these sources.
Technorati Tags: Customer, Effective, Keywords, Listening, Meaning, More, Requires, Than, Volume
