Robustness to Out-of-Distribution (OOD) samples is essential for successful deployment of machine learning models in the open world. Many existing approaches focus on offline setting and maintaining a true positive rate (TPR) of 95% which is usually achieved by using an uncertainty score with a threshold based on ID data available for training the models. In contrast, practical systems have to deal with OOD samples on the fly (online setting) and many critical applications, e.g., medical diagnosis, demand the system to meet quality constraints in terms of controlling FPR (false positive rate) at most 5%. This is challenging since having adequate access to the variety of OOD data, the system encounters after deployment is hard. To meet this challenge, we propose a human-in-the-loop system for OOD detection that can adapt to variations in the OOD data while adhering to the quality constraints. Our system is based on active learning approaches and is complementary to the current OOD-detection methods. We evaluate our system empirically on a mixture of benchmark OOD datasets in image classification task on CIFAR-10 and show that our method can maintain FPR at most 5% while maximizing TPR and making a limited number of human queries.