When deep neural networks are deployed in high-stakes applications, uncertainty estimation is crucial for reliable predictions and decision-making. Despite rich studies in univariate deep regression, multivariate deep regression with accurate uncertainty estimation, especially concerning the covariance matrix, remains largely unexplored. In this paper, we propose a scalable evidential prior to capturing both aleatoric and epistemic uncertainty, including the correlation of the multivariate response vector. Our method formulates a hierarchical probabilistic framework where the evidential prior is fitted using samples generated by a neural network based on moment-matching. Extensive empirical results on real-world multivariate regression tasks demonstrate that our method provides accurate prediction and uncertainty estimation with minimal computational overhead, significantly outperforming existing methods.