Modelling Based on Statistical Knowledge and Judgement
Rarely a month goes by that I don’t receive an email for a new or 'improved' piece of statistical software. Many of these software allow for a ‘check box’ approach to analysis whereby the user need only check a box to generate output. If the wider spread use of multivariate techniques by untrained users is not enough, even some CATI software now allows you to check a box to run multiple regression modelling.
To go with the ‘check box’ alternative has been a new breed of heroic researchers all too ready to add the multivariate techniques to their list of capabilities – and why not when they have only to check a box?
‘Education is man going forward from cocksure ignorance to thoughtful uncertainty’ (Kenneth G Johnson). For those trained in statistical analysis, thoughtful uncertainty is the modus operandi in modelling. This is because judgment so often figures predominantly in the solution and checking boxes must be complemented with judgment.
Cause For Thoughtful Uncertainty
I noted in one software vendors sales pitch, that a main selling benefit was that complicated mathematical routines were shielded from the user. The trouble with this is not only are they shielded from such complexity, but also from the requirement of an understanding of the assumptions and judgment that is required in such statistical modelling.
Take multiple regression for example. In the practice in which I work it would be common to run 5 or more models and to critically evaluate each of them before ‘the one’ is chosen. This is because there are many factors that need to be taken into account simultaneously when interpreting the measures reported in the output. They cannot be simply read at face value because they impute no managerial priorities.
In regression output, the indicator of which driver (predictor variable) is significant (p-value) is reported as a percentage. In practice, a cut-off of 5% is usually set such that only variables with a p-value of 0.05 or less are included in the model. In the ‘check box’ approach, only those drivers satisfying this criterion are included in the output – the rest are often excluded from the users view.
Judgment Required
What if the driver that held the most managerial importance and had been tracked for the previous five years had a p-value of 0.056? It would have been removed from the model by such software and reported as not significant. Instead, such a result requires judgment to be applied and should have been investigated by examining other related factors. For example, perhaps a smaller than usual sample size has contributed to this decrease in statistical significance. In this case, the statistical hurdles should have been relaxed using sound, statistically informed judgment that is capable of taking management issues into account.
Alternatively or additionally, there could be another driver in the model which is explaining the same variability in the dependent variable as this one. This is called multicollinearity and can result in one of the involved variables showing as not significant when on its own, it actually is.
What would the implications to management be to exclude this important driver and is it really worth doing based on a third decimal place value?
What I fear is that one day a researcher will find themselves in court answering an accusation of professional negligence. It is highly unlikely that the ‘check-box defence’ will be persuasive in court.
