Single-cell multi-omics technology is able to measure multiple data modalities at single cell resolution, such as gene expression level (using single cell RNA-sequencing) and chromatin accessibility (using single cell ATAC-sequencing). Integrating scRNA-seq and scATAC-seq data profiled from different cells is a challenging problem. Existing methods often require that the scRNA-seq and scATAC-seq data cover the same cell types, that is, the same clusters. However, this is often not true for many existing datasets. Here we propose a joint matrix tri-factorization algorithm scJMT that is capable of integrating and clustering cells from both modalities of data in the case where the two data modalities do not share exactly the same cell types. The tri-factorization framework also allows us to obtain clusters of genes and chromatin regions, and the association matrices between cell clusters and gene or region clusters. We show that scJMT is superior to a state-of-the-art method under both scenarios where the two modalities have the same or different cluster compositions.