{"id":1649,"date":"2017-11-30T19:01:19","date_gmt":"2017-12-01T03:01:19","guid":{"rendered":"http:\/\/b-spoke.net\/?p=1649"},"modified":"2026-06-29T22:00:29","modified_gmt":"2026-06-29T22:00:29","slug":"as-you-likert","status":"publish","type":"post","link":"https:\/\/thirdactmedia.com\/b-spoke\/2017\/11\/30\/as-you-likert\/","title":{"rendered":"As You Likert"},"content":{"rendered":"
A Ten Minute Read<\/h6>\n

I recently found myself in a debate about, of all things, data. Specifically, \u201cLikert\u201d-style data, and how frequently it is used (poorly) to draw broad conclusions about The Truth of the Matter\u00a0 – a classic example of using math to make things look scientifically rigorous. Statistical abuses like this are what cause people to quote Disraeli (\u201cThere are three types of lies\u2026<\/a>\u201d)<\/p>\n

We were discussing appropriate measures to drive customer-oriented cultures. The current belle of the ball is \u201cNet Promoter Score,\u201d a metric that “proves” people really like their iPhones and Nordstrom, but not their cable companies (millions of dollars spent reaching that conclusion.)<\/p>\n

My client was attempting to make significant strategic decisions \u2013 impacting marketing, finance, and sales \u2013 based on their analysis of NPS and other surveys.<\/p>\n

So far, so good.<\/p>\n

Trouble was, the client was making mathematically-derived inferences: applying percentages to capital outlays, comparing these \u201cscores\u201d across product lines, and, even worse, benchmarking themselves against their competition.<\/p>\n

On the face of it, this feels like \u201cgood analysis\u201d \u2013 so why isn\u2019t it?<\/p>\n

Likert Data<\/h4>\n

Most everyone has seen a Likert-style survey instrument; these are the questionnaires that ask respondents to answer based on a ranked continuum, such as how strongly they agree or their level of satisfaction.<\/p>\n

\"HowBefore looking at the math involved, there are two things not to like about these surveys:<\/p>\n

No Opinion, No Help<\/strong><\/p>\n

\u201cNeutral\u201d is useless. The point of a survey is usually to discover how people feel about something; gen\"\"<\/a>erally, people aren\u2019t \u201cneutral\u201d about anything<\/em>. Thanks to a central-tendency bias, though, more people will pick \u201cneutral\u201d than actually feel that way. Good surveys force respondents to make a choice by taking away that \u201cmiddle\u201d neutral option. (There are some people who disagree with this. Even some who Strongly Disagree. They\u2019re wrong.)<\/p>\n

Committee-Vetted Verbiage Obfuscates Obvious Connotations<\/strong><\/p>\n

In other words, we can\u2019t be certain that what\u2019s being answered matches what\u2019s being asked. For instance, here is a question from a hotel survey:<\/p>\n

\"\u201cMyThere are many truthful ways to answer that. If it was not everything<\/u> I had hoped it would be, and yet my arrival matched expectations, I could honestly say I \u201cDisagree\u201d \u2013 it was not everything<\/u> I hoped.<\/p>\n

(If there were six things I had hoped would be part of my arrival, but I only experienced five \u2026 Does that mean I Strongly Disagree that it was \u201ceverything<\/u> I had hoped\u201d? Or do I need to miss two of six experiences for it to be \u201cStrong\u201d? And where I might Disagree because it wasn\u2019t \u201ceverything<\/u> I had hoped,\u201d someone else receiving the exact same experience could \u201cStrongly Agree\u201d \u2013 either because they had hoped for less, or because they interpreted the extremes differently.)<\/em><\/p>\n

Surveys often feature absolutes, e.g., \u201cI always stay with Brand X.\u201d I most frequently stay with Brand X, but I don\u2019t always<\/u> stay with them. So do I \u201cDisagree\u201d with the \u201calways\u201d bit, or \u201cAgree\u201d to indicate my preference?<\/p>\n

I recently received a survey that asked: \u201cHow well do you understand SharePoint?\u201d and the answers ranged from \u201cHighly Dissatisfied\u201d to \u201cHighly Satisfied.\u201d I can\u2019t even begin to fathom how I would answer that. (However satisfied I am with SharePoint, my response gives no clarity to how well I understand it! Are they asking how satisfied I am with my own understanding?)<\/p>\n

One particularly egregious example of bafflingly-bad survey questions is in the \u201cStrengthsFinder\u201d instrument that Gallup \u2013 who should know better \u2013 charges a lot of money for. I digress a bit on that pool of dog’s breakfast here<\/a>.<\/p>\n

Perhaps I\u2019m being obtuse: Good survey design is difficult. Now that so many online tools make it easy to create and distribute surveys, we ignore that too often (at our peril). If there\u2019s room to interpret the question, subsequent interpretation of the results increases geometrically; soon, even though we have all this \u201cdata,\u201d we realize it doesn\u2019t actually mean anything.<\/p>\n

But that\u2019s OK \u2013 we still have room to analyze it incorrectly.<\/p>\n

On to the math!<\/p>\n

\u00a0<\/strong>WARNING: Math Ahead<\/h4>\n

\"\"We generally assign numbers to responses on Likert data. Because we learned in second grade that \u201cmath\u201d is what we do to numbers, we try to use \u201cmath\u201d on Likert responses \u2013 which means we average these numbers and hope it tells a story. But they\u2019re made-up numbers, used to keep track of responses: They don\u2019t mean anything. They\u2019re just a ranking. We could use letters (e.g., \u201cA\u201d through \u201cE\u201d) or roman numerals or vegetable emojis.<\/p>\n

The numbers assigned to the answers do not represent real values.<\/p>\n

Likert data are \u201cordinal\u201d data: They put things in ORDER. It\u2019s a simple ranking,\u00a0with no weight or relative strength of choice included.\u00a0First is better than second. \u201c1\u201d is better than \u201c2\u201d (unless \u201c10\u201d is the best). Ordinal data \u2013 these arbitrary numbers – offer no opinion on the gap.<\/p>\n

\u201cStrongly Agreeing\u201d is better than \u201cAgreeing\u201d which is better than \u201cDisagreeing\u201d. But by how much? Is the distance between \u201cStrongly Agree\u201d and \u201cAgree\u201d the same for everyone? (Hint: No.) Is the distance between \u201cStrongly Agree\u201d and \u201cAgree\u201d the same as the distance between \u201cNeutral\u201d and \u201cStrongly Disagree\u201d? Who knows?<\/p>\n

When we treat ordinal data like \u201cinterval\u201d data, things get messy. First, we have to presume the gaps are the same, and quantifiably so (e.g., the distance between \u201cStrongly Agree\u201d and \u201cStrongly Disagree\u201d is exactly four times the distance between \u201cStrongly Agree\u201d and \u201cAgree\u201d).<\/p>\n

Basic mathematics theory tells us that Ordinal data cannot be used as normal math data. Full stop.<\/p>\n

\"One<\/p>\n

Look at what can happen when we try.<\/strong><\/em><\/p>\n

In the most simple scenario, let\u2019s assume we survey ten people. Five of them vehemently love our product, the other five violently loathe it:<\/p>\n

\"\"<\/p>\n

That provides a \u201cMean\u201d score of \u201c3\u201d, or \u201cNeutral\u201d.<\/p>\n

\"\"If we used the Mean, we would interpret this data to represent that no one really cares about our product. This would be very dangerous!<\/p>\n

Either way, the Marketing team has a lot of work to do \u2013 but whether they spend their effort learning what their acolytes appreciate and determining what their apostates detest, or figuring out how to make the masses care, is very different work.<\/p>\n

\"\"<\/p>\n

<\/h4>\n

What\u2019s a Picture Worth?<\/h4>\n

Simply put, do not try to use math to tell the story. If you need detailed statistics to explain your results, you used the wrong tool.<\/p>\n

Use the results in graphical form to highlight difference of opinion.<\/p>\n

Here are three different data sets \u2013 each with the same \u201caverage.\u201d<\/p>\n

\"\"<\/p>\n

These were lifted from leadership alignment reviews with individual clients. Even though each had the same weighted mean score, clearly each has different needs. (Because, after all, we\u2019re averaging fake numbers.)<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

By calling it out graphically, we can easily see where there is variability\/lack of alignment, and how extreme people may or may not feel about the issues.<\/em><\/p>\n

Because of the lack of interval information, you may want to consider combining the \u201cpositives\u201d with each other and the \u201cnegatives\u201d with each other.<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

Here is how we addressed that with \u201cBipolar\u201d data \u2013 which, like, Likert scales, appears to be math-like, but in fact is not:<\/p>\n

\"\"<\/p>\n

Note the lack of a “Neutral” option.<\/em><\/p>\n

Net Promoter Score fans argue that because the instrument aggregates answers \u2013 treating \u201c9\u201d and \u201c10\u201d essentially as one answer (\u201cPromoters\u201d) and \u201c0\u201d through \u201c6\u201d as one (\u201cDetractors\u201d) \u2013 all sins should be forgiven. Strictly following this method is, at best, meh (again, the statistical \"\"\u201cdifference\u201d between an 8 and a 9, or a 6 and a 7, is hard to determine). There is an accepted \u201ccorrect\u201d NPS calculation, yet people still feel the need to play with Excel: if you start seeing \u201cmean\u201d scores, though, you know you\u2019re either being hoodwinked, or listening to someone short on clues.<\/p>\n

As in all things, though, NPS is just a number on a dashboard \u2013 an indicator on how well the pursuit of customer satisfaction is going. Pursuing the metric itself \u2013 any metrics \u2013 is a mistake.<\/p>\n

My college roommate\u2019s high school sweetheart (take a moment, I’ll wait) liked to call out people who were \u201cvictims of their own education\u201d; she would have a field day with those who like to apply statistical techniques that don\u2019t belong.<\/p>\n

Analyzing data is supposed to help us understand our world. Once we start using data in ways it\u2019s not intended, we move into a fictional world. Forget Big Data, it may as well be Fake Data.<\/p>\n

At that point, even though we feel better and\/or smarter about our rigorous (but completely useless) analysis, we make it impossible for a Man of Action to take the information and do anything with it.<\/p>\n

EPILOGUE: The Absolute HorseShit\u2122 That is StrengthsFinder\u2122<\/h4>\n

The Gallup Organization has made bank selling inspirational plaques based on its \u201cStrengthsFinder\u2122\u201d survey. They market this as an accurate assessment of an individual\u2019s, well, strengths, and how the collective fits together on a team.<\/p>\n

First thing to know, it is complete horseshit. It is a fundamentally unsound instrument.<\/p>\n

When people receive assessments \u2013 whether it\u2019s StrengthsFinder, Myers-Briggs (less \"\"controversial, but still only superficially \u201caccurate\u201d), or \u201cAre You a Rory or a Lorelei?\u201d \u2013 they tend to feel they have been accurately described. This is called the \u201cBarnum\u201d effect ( named after the \u201csucker-born-every-minute\u201d guy) or the \u201cacceptance phenomenon\u201d.<\/p>\n

Real research backs this up: in essence, people believe what they want to believe, and are blind to what they don\u2019t want to see. So when a survey comes back and says, \u201cYou are a disciplined \u00a0Man of Action who likes to get things done,\u201d well, who\u2019s going to argue with that?<\/p>\n

The StrengthsFinder survey featured dozens of bipolar questions treated as Likert data. Assessees answer on a scale between he descriptor on the left \u201cStrongly Describes Me\u201d and the descriptor on the right \u201cStrongly Describes Me\u201d, with a Neutral in the middle:<\/p>\n

\"\"<\/p>\n

I have had clients where the \u201ctop achievers\u201d consistently deliver negative<\/u> results, but they were clearly very sick organizations. Conversely, are there people who consistently deliver positive results and are not<\/u> top achievers? That would be a rough place to work.<\/em><\/h3>\n

\"\"Aren\u2019t these almost exactly the same thing? Isn\u2019t making a deadline saying you would do something by a certain time\u2026 and then doing what you said you would do? Is StrengthsFinder\u2122 trying to imply that \u201cmaking deadlines\u201d is somehow cutting corners, a cheat, like rolling through Stop signs in South Philly? <\/em><\/h3>\n

\"\"The implication, I guess, is that \u201cthinkers\u201d do it on their own, and \u201creaders\u201d piggyback off others\u2026 The difference between \u201cexperience\u201d and \u201chigh falutin\u2019 book learnin\u2019.\u201d Apparently, it\u2019s not possible to figure out how something works by reading<\/u> about it. Note the misequivocation \u2013 bookworms love<\/u> what they do; Hard Knox graduates only like<\/u> to figure. <\/em><\/h3>\n

\"\"Just remember this when you\u2019re hiring a babysitter; you can\u2019t have both.<\/em><\/h3>\n

\"\"What about a guy like me, who trusts<\/u> that others will cower as I claw my way to the top?<\/em><\/h3>\n

\"\"Remember this was written before US Senators challenged Betsy DeVos to Muay Thai kickboxing. I can\u2019t even imagine what was in the passed bottle when this question was written: Are there seriously people who are both \u201cpassionate\u201d about enriching the souls of their fellow man, while constantly on the lookout for ways to beat the crap out of same fellow man?<\/em><\/h3>\n

Check your shoes, StrengthsFinder.<\/em><\/h3>\n

\"\"Because NASCAR pit crews, neurosurgery teams, and the 2016 New England Patriots all just wing it as they go. Look, the morons that wrote this personality \u201ctest\u201d clearly believe \u2013 and their output demonstrates \u2013 that working as a team equals sloppy, insipid, and spiritually suffocating work\u2026 But there are some of us who like work done well, and believe that their peers and colleagues do, too.<\/em><\/h3>\n

\"\"If the people who wrote StrengthsFinder were ever invited to a social function \u2013 and, you know, showed up and pulled themselves away from the platter of still-slightly-frozen-in-the-middle Trader Joe\u2019s spanakopita –\u00a0 they may discover that being a \u201cgood listener\u201d is almost exactly half<\/u> of what\u2019s required to be a \u201cgood conversationalist.\u201d<\/em><\/h3>\n

\"\"Apparently, Iron Eyes Cody wrote StrengthsFinder.<\/em><\/h3>\n
\n