Irregularly, asynchronously and sparsely sampled multivariate time series (IASS-MTS) data occur naturally in practical domains. They are characterized by sparse non-uniform time intervals between successive observations and different sampling rates amongst series. These properties pose substantial challenges to contemporary machine learning models for learning complicated intra-series and inter-series relations within and across IASS-MTS. To address these challenges, we present a time-aware Dual-Attention and Memory-Augmented Networks architecture (DAMA-Net). The proposed model aims at leveraging both time irregularity, multi-sampling rates and global temporal patterns information inherent in time series so as to learn more effective representations and improve prediction performance. We evaluate our model on two real-world datasets for IASS-MTS classification tasks. The results show that our model outperforms state-of-the-art methods in terms of classification performance. Moreover, we conduct the ablation study to demonstrate the contribution made by different mechanisms and modules in our model.