In response to everyday queries, humans explicitly signal uncertainty and offer alternative answers when they are unsure. Conformal prediction produces calibrated prediction sets that mimic this human behaviour since larger sets signal greater uncertainty while providing alternatives. In this work, we study the usefulness of conformal prediction sets as an aid for human decision making by conducting a pre-registered randomized controlled trial with conformal prediction sets provided to human subjects. With statistical significance, we find that when humans are given conformal prediction sets their accuracy on tasks improves compared to fixed-size prediction sets with the same coverage guarantee.