Recurrent neural networks (RNN) are the primary choice for modelling sequential data, however they are less suitable for modelling irregular time-series data. Continuous time variants of RNN using neural ordinary differential equations (NODE) were shown to perform well on irregular time series data. They learn a better representation of the data using the continuous transformation of hidden states over time, taking into account the time interval between the observations. However, they are still limited in their capability as they use discrete number of layers (depth) over an input in the sequence to produce the output observation. We intend to address this limitation by proposing a RNN model designed based on the principle of heat equation. Our heat diffusion based recurrent neural differential equations(HDR-NDE) model generalizes RNN models by continuously evolving the hidden states in the temporal and depth dimension. HDR-NDE model is based on partial differential equations which treats the computation of hidden states as solving a heat equation over time. We demonstrate the effectiveness of the proposed model by comparing against the state-of-the-art RNN models on real world sequence modeling data sets.