Multimodal models that jointly process audio and language hold great promise in many aspects of audio understanding and are increasingly being adopted in the music domain. By allowing users to query models via text and obtain information about a given audio input, these models have demonstrated the potential to enable exploration of music pieces via conversational interfaces. However, it remains unclear how to effectively assess their ability to correctly interpret and analyse music-related inputs. Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music understanding in audio-language models. MuChoMusic comprises 1,187 multiple-choice questions, all validated by human annotators, on 644 music tracks sourced from two publicly available music datasets and covering a wide variety of genres. Questions in the benchmark are crafted to assess knowledge and reasoning abilities, spanning several dimensions that cover fundamental musical concepts and their relation to cultural and functional contexts. Through the holistic analysis afforded by the benchmark, we evaluate five open-source models and identify several pitfalls, including an over-reliance on the language modality, pointing to a need for better multimodal integration. Data and evaluation code are open-sourced.