Navigant Research Blog

When in Doubt, Take a Survey

Bob Lockhart — May 28, 2013

Cecil Adams of The Straight Dope once supposedly wrote, “Around here, we don’t vote on the facts.”  That was before the age of online surveys.  Once again I have in my inbox a request to participate with other executives in a survey of the current sentiment and outlook of the smart grid industry.

When I see these surveys I wonder, “Who really cares what we think?”  The electrons don’t care.  The untrimmed trees under the high voltage lines don’t care.  The hostile nation-state hackers certainly don’t care.  The ratepayers – sorry, I meant to call them customers – don’t care.  And if someone thinks that I am a “smart grid executive,” then I hate to think who else has been identified as an “executive.”

As a market research professional, I admit that I look down my nose at surveys.  They are not primary research, as many losers in last November’s U.S. elections are now aware.  My research involves a lot of time on the telephone, asking questions of key stakeholders in a given research area, then synthesizing diverse responses into one or two theses.  This is rarely straightforward.  One slide from my conference presentation deck asks, “What is the No. 1 cyber security problem facing utilities?”  During one research project I asked 33 people this question and got 28 distinct answers.  It takes three slides to answer what you’d think is a really simple question.

Not Another Monkey

That is what research looks like.  The notion that you can just run another SurveyMonkey to an anonymous audience, arbitrarily designate your audience as executives, and therefore develop conclusions about the industry… just doesn’t sit well with me.

But surveys produce numbers.  Numbers can be analyzed, operated upon, correlated, summarized.  And no matter the source, numbers somehow convey an air of certainty.  Especially if you have a large enough sample size and can claim a statistical error margin of +/- 3%.  That’s just got to be right, doesn’t it?

Not always.  Surveys of sentiment are qualitative.  This particular survey asks questions such as whether my company’s smart grid investment is going to increase, decrease, or stay the same.  Whether the increase is by $100 or by $1 billion, I tick the same box.  There is nothing quantitative going on here.  Yet we often ascribe to survey results the same strength as weather measurements or time signals.

In the spirit of full disclosure, Navigant Research does publish and sell an annual Smart Grid Consumer Survey.  We are open that we are measuring consumer sentiment, nothing more.  And I have used some of the survey results as supporting data for my research.  But I would never draw conclusions solely from anonymous surveys.

For us, research begins with lots of telephone time discussing issues with key stakeholders.  That research continues with our all-star research associates who spend their days tracking down untold quantities of obscure but useful information.  That is how you begin to understand the direction of a market.

Leave a Reply

Your email address will not be published. Required fields are marked *

Blog Articles

Most Recent

By Date

Tags

Clean Transportation, Digital Utility Strategies, Electric Vehicles, Energy Technologies, Policy & Regulation, Renewable Energy, Smart Energy Practice, Smart Energy Program, Transportation Efficiencies, Utility Transformations

By Author


{"userID":"","pageName":"When in Doubt, Take a Survey","path":"\/blog\/when-in-doubt-take-a-survey","date":"12\/11\/2017"}