Recently, large language models (LLMs) have made remarkable progress in natural language processing (NLP). One of the most notable abilities of LLMs is in-context learning (ICL), which enables LLMs to learn patterns from in-context exemplars without training. However, there remains limited intuition for how in-context learning works. In this paper, we show that ICL can be highly related to retrieving from a modern Hopfield Network (MHN), a model of associative memory that is biologically plausible. We establish a theoretical interpretation of ICL based on an extension of the framework of MHNs. Based on our theory, we propose an Active Exemplar Selection approach that is more efficient than commonly used selection methods. Furthermore, we empirically investigate the influence of exemplars on ICL for different tasks. Our study sheds new light on the mechanism of ICL by connecting it to memory retrieval, with potential implications for advancing the understanding of LLM.