We propose Epiphany, a light-weight neural network to predict the Hi-C contact map from five commonly generated epigenomic tracks: DNase I hypersensitive sites and CTCF, H3K27ac, H3K27me3, and H3K4me3 ChIP-seq. Epiphany uses 1D convolutional layers to learn local representations from the input tracks as well as bidirectional Long Short Term Memory (Bi-LSTM) layers to capture long term dependencies along the epigenome. To improve the usability of predicted contact matrices, we perform statistically principled preprocessing of Hi-C data using HiC-DC+ \cite{HiCDC+} and train Epiphany using an adversarial loss, enhancing its ability to produce realistic Hi-C contact maps for downstream analysis. We show that Epiphany generalizes to held-out chromosomes within and across cell types, and that Epiphany's predicted contact matrices yield accurate TAD and significant interaction calls.