Multiple Measurements Outperform: A Right of Reply.
The article titled ‘The Juster 11-Point Probability Scale’ featured in the February 2008 edition of Research News. As a result Wollongong academic John Rossiter responded to the article with a letter to the editor, published in the March 2008 edition.
To recap, the original article introduced the Juster scale and Rossiter disagreed with the discussion Forethought put forward.
In essence, what Rossiter proposes as the ‘correct’ approach is to take a single measure (in this case the Juster scale) and re-weight the response in order to better predict business outcomes.
In practice, researchers use multiple measurements to predict business outcomes such as product adoption. For example, at the first level of measurement the sample is selected to comprise respondents who are considered part of the addressable market. The second level of measurement establishes that the respondents do indeed have an unmet need. The third level of measurement entails seeking confirmation that for currently held unsuitable services, there is some probability for defection. At the fourth level, respondents who give a high likelihood of adoption are identified (here is where the Juster scale is applied) and the final measurement involves obtaining an expected timeframe for the adoption. All of this data is then used in the generation of the business case.
Contemporary practice is to count only those respondents who give a high likelihood of adoption. Score-classing the respondents’ ratings is an approach that moves from using a descriptor of the overall population (i.e. mean) to segmenting the sample into purchase likelihood segments. The researcher is then able to draw conclusions about each resulting segment. An example of the application of such score-classing is given by a recent project for the introduction of the Myer credit card. In this project, the respondents indicated their likelihood of adoption using a 0-10 Juster scale. By score-classing the respondents into 3 groups (low likelihood - 0 to 4; moderate likelihood 5 to 7 and high likelihood - 8 to 10), we were able to focus on respondents with a high likelihood of requesting the card. Though no weights were applied to the scale per se, the choice to base uptake estimates on the top end of the scale provides management with a reflection of consumers that have high purchase intention behaviour.
Forethought believes it is wrong for Rossiter to suggest that a researcher should take a respondent who gave a score of one on an 11-point scale of likelihood and re-weight them to have a 0.1 probability of adoption. Our recommendation is to disregard them altogether. Indeed, in these days of stringent requirements for realistic yet conservative business cases, Forethought suggests neither the client nor the professional indemnity insurer would accept including respondents scoring less than eight on a Juster scale in a market estimate.
Rossiter suggests that ‘researchers should calibrate intention scores against subsequent actual purchases’. Was this meant to be a novel idea? Forethought would argue that, generally, the commercial discipline of generating predictive accuracy and maintaining ongoing client trust is far more onerous than the discipline required in academia. Clients and suppliers alike continually monitor predicted verses actual.
Perhaps Rossiter believes that there is little gap between a theoretical construct and marketing research practice, but to use Rossiter’s own words, this is wrong. Rossiter may be interested to know that when calibrating intention scores against subsequent actual purchases, the Juster scale is best used as a dichotomous scale of those scoring 8 or more as having some probability, and those scoring less than eight as having such a remote probability that clients would not accept the commercial risk of building a business case on them.
Ben Ward who last year launched the Myer credit card said recently that the estimates based on this approach had proven to be ‘remarkably accurate’ and Chris Martin, CEO of the OFM Investment Group said that research using the same approach had ‘accurately predicted the volume adoption of reverse mortgages in Australia for the first three years’.
As an organisation that walks both an academic and applied path, Forethought understands that academics find it challenging to acknowledge the wherewithal captured in commercial researchers’ practice since it is not sourced from academic journals.
According to Rossiter, his criticisms are ‘not just academic. They are practically vital in terms of correct managerial interpretation and action taken’. Forethought disagrees. Our practical experience is that multiple measurements are infinitely more accurate than relying on a single re-weighted scale
