Preferential Bayesian optimization (PBO) is a framework for optimization of a decision-maker's (DM's) latent preferences. Existing work in PBO assumes these preferences can be encoded by a single latent utility function, which is then estimated from ordinal preference feedback over design variables. In practice, however, it is often challenging for DMs to provide such feedback reliably, leading to poor performance. This is especially true when multiple conflicting latent attributes govern the DM's preferences. For example, in exoskeleton personalization, users' preferences over gait designs are influenced by stability and walking speed, which can conflict with each other. We posit this is a primary reason why inconsistent preferences are often observed in practice. To address this challenge, we propose a framework for preferential multi-attribute Bayesian optimization, where the goal is to help DMs efficiently explore the Pareto front of their preferences over attributes. Within this framework, we propose a Thompson sampling-based strategy to select new queries and show it performs well across three test problems, including a simulated exoskeleton gait personalization task.