Automation of machine learning model development is increasingly becoming an established research area. While automated model selection and automated data pre-processing have been studied in depth, there is, however, a gap concerning automated model adaptation strategies for streaming data with non-stationarities. This has previously been addressed by heuristic generic adaptation strategies in the batch streaming setting. While showing promising performance, these strategies contain some limitations. In this work, we propose using multi-armed bandit algorithms for learning adaptive strategies from incrementally streaming data on-the-fly. Empirical results using established bandit algorithms show a comparable performance to two common stream learning algorithms.