ICML Workshop on Large Language Models and Cognition

Large Language Models (LLMs) have undoubtedly taken center stage in the AI revolution, showing impressive performance in a wide variety of tasks, including machine translation, standardized tests, and conversational chatbots. It is even more impressive to uncover that these models exhibit unpredictable capabilities in solving unseen tasks. This demonstration of emergent abilities, often credited to the scale of the parameters and data size in the case of LLMs, is being considered as the footprint of intelligence.The goal of this workshop is to assess and understand the position of current LLMs’ abilities in the landscape of intelligent systems, with a strong focus on cognitive abilities. By bringing in experts from different scientific disciplines, such as AI/ML, neuroscience, cognitive science, and psychology, we aim to discuss topics that include but not limited to:• Where do LLMs stand in terms of performance on cognitive tasks, such as reasoning, navigation, planning, and theory of mind?What are the fundamental limits of language models with respect to cognitive abilities?• How do LLMs fine-tuned on specific tasks end-to-end compare to augmented LLMs coupled withexternal modules?• What are the similarities and differences between mechanistic interpretability approaches in AI and inneuroscience? What do they tell us about similarities and differences between LLMs and human brains?• How can we improve existing benchmarks and evaluation methods to rigorously assess cognitiveabilities in LLMs?• Can multimodal and multiagent approaches address some of current limits of LLMs to cognitive tasks?We hope that this workshop will help identify the gaps and opportunities in the current LLM landscape and shape the path for the development of trustworthy and robust systems guided by cognitive science.