We discuss the role of humans in algorithmic decision-making (ADM) for socially relevant problems, highlighting tensions arising from the misalignment of the humans with each other and with the algorithms involved. To this end, we assume that a supervisor introduces ADM to achieve strategic goals and that the algorithms’ recommended actions are overseen by agents who makes the final decisions. While the agents should be a corrective, they can counteract the realization of the supervisor’s goals because of misalignment and unmet information needs. This impacts the distribution of power between the stakeholders, and we emphasize the overseeing agents’ implied role as potential political and ethical decision-makers. On a machine learning benchmark dataset we illustrate the significant impact overseeing agents’ decisions can have even if they are constrained to performing only few corrections to the algorithms’ recommendations. Our findings emphasize the need for an in-depth discussion of the role and power of the agents and challenge the often-taken view that just including a human-in-the-loop in ADM ensures its ‘correct’ and ‘ethical’ functioning.