Matching Questionnaire Design to Analytical Techniques
In 1947, George Gallup (who gave his name to the Gallup Opinion Poll) observed that “too much attention has been directed toward sample design and too little toward question design.” Thirty years later (1977), Gallup again felt the need to repeat this view when he wrote, “While great strides have been made in improved sample design and technique – there has not been a comparable amount of progress in perfecting question or questionnaire design.”
Almost thirty years later again, it would have to be said that question wording remains an area that still needs improvement. The issues that should be considered when developing a questionnaire are many, with lengthy discussions of these readily available; however, they are not the focus of this discussion. Instead, I would like to concentrate on one, crucial yet recurring error in the writing of many survey questions – namely, a mismatch between the questions asked and the analysis required to answer the stated research objectives.
Much time can be spent on making sure that the questions are not leading, not ambiguous, not asked in the negative in one instance and then the positive in another; making sure that the scale used is appropriate, the anchor descriptions relevant – the list goes on and is unquestionably important. However, given the volume of do’s and don’ts to get through, it is hardly surprising that by the time the question itself is written, too little thought is given to whether the responses allow you to answer the research questions posed.
For example, asking the customers of two competing companies to rate whether or not they are satisfied with the service provided (ie Yes/No), may appear to address the research objective of determining which company has the most satisfied customers – however, all may not be as it seems. If by ‘most’ we mean the greatest number then yes, we can answer the objective by simply counting the number of ‘Yes’ responses for each company and comparing them. But if by ‘most’ we mean which company had the ‘happiest’ customers, then the number of ‘Yes’ responses will not help. Company A may have the same number of satisfied customers as Company B, but the customers of Company A may be highly satisfied compared to only moderately so for Company B.
This is only one of many examples however, the take home message is the same for all – when writing questions it is paramount to consider, in detail, the hypothesised outcomes of the questions and the types of analyses that can be carried out on them.
To help reduce the chance of this critical mismatch occurring, I find the most successful strategy is to invite the researcher to hypothesise what they believe the answer to a particular question will be; or at least, what they are hoping the finding from that question will be. Researchers can usually do this with ease. They might say, I expect that Company A is better than Company B. We should then drill down and ask, in what terms then do you wish to report that? Customers that are more satisfied? – one option could be to use a likert scale to capture average level of satisfaction; greater market share? – use a count of customers.
Whatever the outcome, without this drill down, the question may have been posed as, ‘Who do you think is the better company, A or B?’, with the answer being nothing more than a letter response from which neither of the above results could be reported.
A promising advancement is the current emergence of specialised survey designers in some market research companies. With such a concentration of expertise being directed consistently at the writing of questionnaires, who knows, perhaps it will be only three more years instead of another thirty before this survey question/response mismatch problem is largely resolved.
