Timezone: »

 
Poster
Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts
Étienne Marcotte · Valentina Zantedeschi · Alexandre Drouin · Nicolas Chapados

Thu Jul 27 04:30 PM -- 06:00 PM (PDT) @ Exhibit Hall 1 #322
in Poster Session 6 »
Event URL: https://github.com/ServiceNow/regions-of-reliability »

Multivariate probabilistic time series forecasts are commonly evaluated via proper scoring rules, i.e., functions that are minimal in expectation for the ground-truth distribution. However, this property is not sufficient to guarantee good discrimination in the non-asymptotic regime. In this paper, we provide the first systematic finite-sample study of proper scoring rules for time series forecasting evaluation. Through a power analysis, we identify the ``region of reliability'' of a scoring rule, i.e., the set of practical conditions where it can be relied on to identify forecasting errors. We carry out our analysis on a comprehensive synthetic benchmark, specifically designed to test several key discrepancies between ground-truth and forecast distributions, and we gauge the generalizability of our findings to real-world tasks with an application to an electricity production problem. Our results reveal critical shortcomings in the evaluation of multivariate probabilistic forecasts as commonly performed in the literature.

[ PDF]  [ Poster] 

Author Information

Étienne Marcotte (ServiceNow Research)
Valentina Zantedeschi (INRIA, UCL)
Alexandre Drouin (ServiceNow Research)
Nicolas Chapados (ServiceNow Research)

More from the Same Authors