In just the past couple of years, we have seen significant advances in the capabilities of (Large) Language Models. One of the most striking capabilities of these systems is knowledge retrieval — Language Models can answer a diverse set of questions, which differ substantially in the domain knowledge needed for their responses, and their input structure. The precise methods for knowledge retrieval vary from the language model directly generating a response (parametric approaches) to a combination of generation and referencing an external knowledge corpus, e.g. retrieval augmented generation, to primarily using an external knowledge corpus with language model embeddings (semi-parametric approaches.) Despite the rapid advances, there remain many pressing open questions on the limits of knowledge retrieval with language models, and connections between these different approaches. How factual are generated responses, and how does this vary with question complexity, model scale, and importantly, different methods of knowledge retrieval? How important is the role of (self-supervised/supervised) pretraining? What are the tradeoffs between few-shot (prompt based) approaches and finetuning when adapting to novel domains? And relatedly, to what extent do different knowledge retrieval approaches generalize to unseen settings? This workshop seeks to bring together a diverse set of researchers across NLP, Machine Learning and Theory to discuss these questions. We hope to share current findings and challenges, identify promising directions for future study, and most importantly, build a community around this topic at this pivotal time.