Deployed early warning systems in clinical settings often suffer from high false alarm rates that limit trustworthiness and overall utility. Despite the need to control false alarms, the dominant classifier training paradigm remains minimizing cross entropy, a loss function that has no direct relationship to false alarms. While existing efforts often use post-hoc threshold selection to address false alarms, in this paper we build on recent work to suggest a more comprehensive solution. We develop a family of tight bounds using the sigmoid function that let us maximize recall while satisfying a constraint that holds false alarms below a specified tolerance. This new differentiable objective can be easily integrated with generalized linear models, neural networks, and any other classifier trained with minibatch gradient descent. Through experiments on toy data and acute care mortality risk prediction, we demonstrate our method can satisfy a desired constraint on false alarms interpretable to clinical staff while achieving better recall than alternatives.