We introduce a novel unsupervised method for improving fairness of computer vision algorithms through test-time adaptation on videos. The method is model agnostic and uses the temporal coherence of predictions across sequential frames as a self-supervision signal to update a subset of model parameters. We assess the performance on Casual Conversations Dataset for gender prediction task on face images and show that our approach can significantly improve predictive disparity across different skin tones, ages and lighting conditions.