Drawing on scholarship across disciplines, we argue that probabilities are constructed rather than discovered and show how this is important for Machine Learning, especially in social settings. We criticise the conventional notion of datapoints as sampled from a true distribution and propose an alternative mathematical framework that allows to analyse learning. We highlight problematic aspects of common reasoning and rhetoric about probabilities in the context of social predictions. We also strengthen the case that (probabilistic) Machine Learning models cannot be separated from the choices that went into their construction and from the task they were meant for.