Evaluating Forecasts Through Stochastic Loss Order

We propose to evaluate the performance of  k ≥ 2 forecasting methods by examining the distributions of forecast errors and associated losses. We argue that non-systematic forecast errors are minimized when the distributions are symmetric and unimodal. Also, we suggest that forecast accuracy should be assessed using stochastic loss order instead of the traditional method of expected loss order. Additionally, since the evaluation of forecast performance can be viewed as a one-way analysis of variance, we propose to analyze loss distributions under two scenarios: first, when there exists a strict (but unknown) joint stochastic order among the losses of all forecasting alternatives, and second, when this order applies to subsets of the alternative methods. Under simulation studies, we compared our proposal with existing tests and found that our test is more powerful, particularly when dealing with small sample sizes. Finally, we apply our proposals to a real-world scenario.