We take so much care to produce accurate findings only to be foiled by the sample not being representative of the population it purports to represent. Research vendors have always had a healthy scepticism towards a client-supplied sample; however, this is by no means the sole source of potential sample bias.

There are many issues to contend with when considering the veracity of a sample. There is the sample size formula, which enables you to calculate the smallest sample size required to achieve desired levels of precision and reliability for given levels of variability in the population. There are the various methods of collecting the data depending on the particular scenario i.e. simple random sampling, stratified sampling, quota sampling, and cluster sampling to name a few. There are the different ways of applying weights to the sample data so that it better reflects the population profile of characteristics.

All of these methodologies have at their core the primary objective of ensuring that the sample is representative of the population which it is attempting to estimate, and from which it was drawn. We are only interested in the sample for what it can tell us about the population. Therefore, representativeness is paramount. A biased sample can result in reporting misleading information about the population. Sample bias all too often rears its ugly head, although usually well disguised.

Refreshing the Panel

One example of how sample bias may go undetected is through the natural ageing of panel membership. This can occur when a sample that was originally collected correctly and satisfied all requirements of representativeness, is used repeatedly over time without the demographic profile being checked and refreshed where necessary. This is what was alleged to be the explanation for the problems aired through the audit report comparing two audience measurement services. When a new company was brought in to take over the provision of the TV ratings to the networks, they found significant differences in viewing levels and station shares compared to those previously provided by the incumbents.. One suggested explanation was that the incumbents’ sample panel had relatively too many older grocery buyers and too few younger ones. This is consistent with ageing of the once representative sample. Since one TV channel’s programming tended to target an older demographic, this alleged bias in the sample was said to have resulted in artificially high ratings for that TV station.

List Vendors

Another example of sample bias finding its way into client results is through lists provided by list vendors. This is not to say that their lists are biased however, the end user must be aware of the collection process used to create that list. For example, where people are enticed into completing a survey by free entry into a competition with the chance of winning some prize, the sample is likely to be made up of people who are favourably disposed towards competitions and rewards. If this bias does not impact on the characteristic to be measured from the sample, then perhaps all is well. However, in the real case illustrated below such a sample was analysed via conjoint analysis to estimate the attractiveness of a reward being offered to induce product purchase. Clearly here, the sample bias resulted in artificially high estimates of the importance of the inducement.

Eradicating Sample Bias?

Altogether eradicating sample bias requires a level of rigour which is probably beyond the realistic realm of commercial research. Some lateral thinking coupled with a review of the known characterises of the sample against the known characterises of the population (for example, geographical distribution) is a useful start.

By definition, you will not be able to compare the primary characteristic that you are interested in because it is unlikely to exist in secondary data; – which is why you are conducting the research in the first place. However, comparing the known characteristics enables you to be reasonably comfortable that your sample is representative and perhaps so too therefore, should be your new estimates.