Although Deep Neural Networks (DNNs) are often compared to biological visual systems, they are far less robust to natural and adversarial examples. In contrast, biological visual systems can reliably recognize different objects under a variety of settings. While recent innovations have closed the performance gap between biological and artificial vision systems to some extent, there are still many practical differences between the two. In this Blue Sky Ideas presentation, we will identify some key differences between standard DNNs and biological perceptual systems that may contribute to this lack of robustness. We will then discuss possible avenues for future work by identifying promising DNNs that are constructed with these biological computational motifs but have hardly been examined in terms of robustness.