We propose a new method for variable selection using Bayesian neural networks. We focus on quantifying uncertainty in which variables should be selected. Our method provides posterior summaries including posterior inclusion probabilities and credible sets for variable selection. Our framework generalizes the previous Sum of Single Effect model (SuSiE) to deep learning models for incorporating non-linearity. We provide a variational algorithm with several relaxation techniques that enables scalable inference. Our model can be used for both regression and classification tasks. We show that our method has competitive performance in variable selection using simulations. The method is suited for scenarios where input variables are correlated and effect variables are sparse. We illustrate the utility of our method for genetic fine-mapping in statistical genetics with the Stock Mice dataset.