Automated reasoning with unstructured natural text is an important field of research with many potential applications, and is rapidly growing thanks to recent advancements in Language Models (LMs).Existing benchmarks for automated reasoning assume access to a consistent set of knowledge over which a model reasons, which does not capture common real-world scenarios where information is noisy and sometimes contradictory. In many applications, conflicts can often be resolved by imposing preferences over information sources (e.g., based on the credibility of the source). In this paper, we formulate the problem of reasoning with contradictory information guided by preferences over sources as the classical AI problem of defeasible reasoning, and develop a question-answering benchmark called BoardgameQA for measuring the defeasible reasoning capacity of LMs. Additionally, BoardgameQA also incorporates reasoning with implicit background knowledge, to better reflect reasoning problems in downstream applications. Experiments with three types of models (finetuning with and without proof steps and few-shot prompting) show that LMs perform poorly when reasoning with conflicting information, especially in the few-shot case, and the amount of background knowledge required compounds this difficulty even further.