Collecting Less Responses than the Required Sample Size
Statistically valid sample sizes are calculated based on required levels of precision and reliability – but are there situations where it is OK to collect less than this required sample size?
Take product testing for example, and imagine two alternative tastes are being investigated. The required sample size has been calculated to be, say, 200, but after sampling the first 20 respondents, 19 out of 20 prefer Taste A. Do the remaining 180 tests need to be carried out before a conclusion about taste preference can reliably be made?
The answer is held within the components of the sample size formula, but in short is “No”. With this new information, the sample size should be recalculated and the savings realised from the reduced field work required.
In this example of monadic testing, where products are tested one at a time in order to determine what proportion of respondents preferred each product, the sample size calculation would involve using a preliminary estimate of that proportion. This could be set conservatively at ρ=.5 (i.e. indifferent between Product A and B, equal chance of choosing either), or something different if the researcher had reason to believe this was so, such as ρ=.7 (i.e. 70% of the market prefer Product A). Either way, a resulting sample size would be calculated – remembering also that a required level of reliability and precision would need to be stipulated, commonly 95% and 5% respectively.
Using the sample size formula for estimating a population, it can be seen that as estimated precision moves away from the most conservative estimate (ρ=.5), in either direction, the required sample size becomes smaller. So in the case of product testing, it is simply a matter of revising the preliminary estimate of ρ in line with the sample results. For example, for the given levels of reliability and precision, a sample of 385 is required where ρ is estimated at .5. But if after the first 20 tests we find that the sample proportion preferring Product A is .95 (19 out of 20), then recalculating the sample size with this figure results in the required sample size being only 73. This is without any change in the levels of reliability and precision achieved. In this situation, it is OK to collect a sample of 73 instead of the originally stipulated 385. The cost savings from monitoring results whilst still undertaking fieldwork can be considerable.
This same approach applies when sampling to estimate averages as well. To begin, a preliminary estimate of the standard deviation is made and used to calculate the appropriate sample size for given levels of reliability and precision. Ongoing reassessment of this preliminary estimate is needed, in much the same way as described in our earlier product testing example where it was the proportion that had been estimated.
Whilst all endeavours are made to use the best preliminary estimate available, it is none-the-less unknown. So as soon as the sample data starts to come in, a sample standard deviation can be calculated and then compared to the preliminary estimate to see if the planned precision has indeed been achieved. If it has, further sampling is not required. If not, sampling should continue.
Sometimes, then, it is OK to collect less than the originally calculated sample size – the key to knowing when, is to review the accuracy of the pre-fieldwork estimate. The return for this monitoring is not having to waste time and money collecting more sample than you need.
It is rare for marketing research firms – in particular those that derive a significant part of their profit from field work – to re calculate sample size while in field, and yet it is a considerable area for reduction of fieldwork costs.
Downloads
