THE CHOICE OF A FORECASTING ERROR METRIC

(Should we look for a forecasting method with the

Lowest MAD, Lowest MAPE, or lowest Bias, or lowest MSE?)

By Praf Joglekar
La Salle University, Philadelphia, PSA 19141 (joglekar@lasalle.edu)

There are several different ways (or metrics) of measuring the “average error” associated
with a forecasting method.
Often, a particular method may give the smallest value for all the error metrics. Then that
method is superior to all others.
On the other hand, some times, there is a conflict between these error metrics. For
example, Method A may give a smaller MAD but a larger MSE than Method B. Then the
choice between Method A and Method B depends on the costs associated with incorrect
forecasts:
 When the cost of under-prediction and costs of over-prediction are linear and about
the same per unit of error, we should minimize MAD in choosing a forecasting
method. (Minimizing MAPE in these cases makes a little better sense since MAPE is
mean absolute Deviation in percent terms).
 When the costs of large errors are disproportionately larger than the cost of relatively
small errors, we should minimize the MSE.
Minimizing the bias never makes sense, since bias is easy to compensate for. However,
sometimes we prefer forecasts that have a positive bias, and at other times, we prefer
forecasts with a negative bias.
 When the per-unit costs of under-prediction are significantly larger than the costs
of over-prediction, we may prefer to err on the side of over-predicting rather than
under-predicting. That is, in such cases, we prefer a forecasting method with a
positive bias.
 When the per-unit costs of over-prediction are significantly larger than the costs
of under-prediction, we may prefer to err on the side of under-predicting rather
than over-predicting. That is, in such cases we may prefer a forecasting method
with a negative bias.
The diagrams on the next page show the various situations I have discussed above.

1
2