Research: Looking For Answers in All the Wrong Places

One of our agency hallmarks is a driving commitment to understand our clients’ customer segments. We begin every client relationship with what we call a “Think” phase in which we typically conduct qualitative research. As simple as some customer interviews or as deep as focus groups, we endeavor to establish baseline perceptions and understand barriers and opportunities that face our client and successful goal attainment. In that context, we recently conducted a client survey that revealed their primary audience sees no difference in the organization’s performance on eight core attributes (considered fundamental, by the client to their overall brand). While performance was positively rated on all of these attributes, the actual perceived values of these attributes clustered together within half a rating point. Nothing stood out as defining the client as ‘best in class.” In other words, they were doing a “fine” job. (At the agency, we have taken to levying a fine on anyone who settles for “fine” in their work.) On the other hand, our client was seen as excelling at nothing.

The search for distinctiveness began as the client looked to the data to reveal an attribute they could “own", something at which they were “best in class” and would continue to drive users to them. Repeated requests were made to look at the data in multiple ways. Rather than presented on the initial 7-point rating scale, we were asked to show only the top three points where all of the mean ratings fell. The numbers did not spread any farther – means of 1.5 – 1.9 were still jammed together. Calculating medians and showing rating distributions made it clear there were some overall variations in performance perceptions among the respondents population but still did not show ownership of any one attribute.

No question: The survey findings were right on. The client is not distinctive on any of these attributes. (Notably, either were their major competitors also queried in this survey.) Yet the search continued to make the data say something that could not be said.

The lesson here: Trying to make data tell you what you want to hear is a futile exercise. We all know that data can be manipulated many ways – often telling different stories. But, in the end, the value lies in respecting the data – whether it’s what we want to hear or not.