Calibration Questions: Interpreting Respondent Ratings
If the debate on scales isn’t tough enough then there is the heroic assumption that we all use the scale the same way. Perhaps for one person 8 is a score that represents a perfectly acceptable performance but for another it means there is some scope for improvement. "Use a semantic scale!" (as opposed to just semantic anchors) you shout, but there are limitations to that when we have an extensive battery of questions.
The 1 – 10 rating scale, as used in many marketing research questionnaires represents, 10 points equally spaced along a line. This conveniently allows the respondent to choose the position that best reflects his or her perceived answer to the question at hand.
In general this works well as it allows easy arithmetic, but such arithmetic ease comes as a double-edged sword. The ease with which these numbers can be processed can lead to calculations being done when one is not even sure what the numbers are measuring. In the respondents mind, is the increment from 5 – 6 the same as from 8 – 9? Alternatively, is the product or service improvement needed to move a respondent’s satisfaction rating from a 5 to a 6 the same as that needed to move them from a 9 to a 10? Would this be the same for all customers? The answer is no; some customers are more easily pleased than others.
In addition, each customer has their own pre-conceived expectations of the quality of that product. These expectations could be based on the reputation of the organisation, previous purchases or background information. The customer’s actual experience with an organisation can result in one of three events: confirm the customers’ expectation where quality is considered equal to expectations, disconfirm the expectation where quality is assessed as differing from that expected (either positively or negatively) or where quality is equal to expectations.
This disconfirmation paradigm means that a rating of say 6 on a 10 point scale cannot be uniformly interpreted. For someone who has a baseline expectation of 5, it would represent positive disconfirmation and be a desirable result, but for someone who has a base level expectation of say 7, it would represent an undesirable, negative disconfirmation.
Thus, there is a need to find some way of interpreting respondents’ ratings so that they are able to be uniformly considered. This can be achieved through the introduction of a calibration question into the questionnaire. The purpose of such a question is to measure the respondents’ base level of expectation. Ratings can then be compared to this calibration rating and the difference calculated.
The exhibit below reports these calibrated differences in terms of whether or not the customer's expectations have been exceeded (positive disconfirmation) or met (confirmation) or the supplier has fallen short (negative disconfirmation). The power of this information is clear. In the case of disclosure responses, the comparison can be made for the individual, further enhancing the explanatory power.
After a number of years of examining how respondents apply the calibration questions, Forethought has a word of advice: there needs to be one question that establishes a baseline performance standard and another that establishes actual expectations. You might be alarmed to find how high the respondents’ expectations are set, particularly if you are reporting a top box score or percentage satisfied as 7-10.
