
Precision, Reliability and Sample Size
In market research, our primary goal is to find out what is happening in the market place. In most situations however, a complete census of the relevant market, which would provide that elusive “perfect information”, is prohibited by its large size or lack of documentation (i.e. incomplete population frame). Instead, we seek to examine only part of the population (or market) and still be able to draw valuable conclusions about the market place as a whole. That is, we need to draw a random sample. Given budget pressures, the question becomes, ‘what is the smallest sample we can take whilst still achieving statistically valid results?
In order to calculate an optimal sample size, you need firstly to ascertain what is to be measured from the sample (reporting unit), as this determines which formula is appropriate. Essentially, this will be the mean (i.e. average satisfaction), proportion (i.e. market share), or both
Once this has been ascertained, the following 3 issues need to be addressed:
1. Precision - How precise do the estimates calculated from your sample need to be? (ie. what is the acceptable level of precision/maximum allowable error?)
2. Reliability - How confident do you wish to be that your results correctly represent what’s going on in the population?
3. Variability - How much variability exists in the population? (This is approximated by a preliminary estimate of the population standard deviation) .*
*Budgetary considerations should only be taken into account once the objective, statistically valid sample size has been calculated.
These factors are measured and combined mathematically into a formula.
To avoid introducing statistical notation, this sample size formula can be represented generically as follows:
Optimal Sample Size
|
If we set our ‘benchmark’ measure of required reliability at say 2, required precision at 1 and data variability at 3, then:
|
|
This tells us that, for a given amount of data variability, if we are to achieve the required levels of precision and reliability of the estimates, then the smallest sample size possible is 36. Any more than this would translate to unnecessary costs.
If however, it is decided that the estimates calculated from the sample need to be even more reliable (say, ↑reliability measure to 3), then the smallest sample size required increases to 81. This is assuming that the level of precision required and variability in the data, remains the same.
Alternatively, if reliability and variability are held at their benchmark levels, but increased precision is required (say, allowable error margin reduced to 0.5), the sample size required increases from 36 to 144.
Finally, if both the reliability and precision of the estimates are already sufficient at the benchmark levels, but the data from which the sample is to be drawn is inherently more variable (say variability measure ↑es to 4), then the smallest sample required increases from 36 to 144.
From these examples, we are able to see the role that each of these factors play in determining the smallest possible sample size that will produce both reliable and precise estimates. Both the precision and reliability requirements can of course, be altered to change the sample size required. Where budgetary restrictions are in place, small changes to these factors, if intuitively acceptable, can help to further reduce the size of the sample required and so reduce costs.
