Metallic Glasses (MGs) are widely used materials that combine the traits of metals, plastics, and glasses. Their unique properties are attributed to their amorphous atomic structure. While understanding the structure-property relationship of MGs remains a challenge for materials science research, studying their energy barriers as an intermediary step shows promise for addressing this problem. In this work, we utilize Graph Neural Networks (GNNs) to model the atomic structure and study their relationships to the energy barriers. We contribute a new dataset for MG energy barrier prediction as well as a novel Symmetrized GNN (SymGNN) model that exhibits E(3) invariance in expectation. SymGNN handles invariance by aggregating over orthogonal transformations of the graph structure for representation learning. When applied to energy barrier prediction, SymGNN achieves much more accurate prediction performance than molecular dynamics (MD) local-sampling methods and other machine learning models. Compared to precise MD simulations, the inference time on new MGs is reduced from roughly 41 days to less than one second. We also apply explanation algorithms to better reveal the structure-energy barrier relationship of MGs, demonstrating that the important edges we identify possess unique topological properties. Our work enables effective prediction and interpretation of MG energy barriers, bolstering materials science research.