This work considers a dynamic decision making framework for allocating opportunities over time to advantaged and disadvantaged individuals.Here, individuals in the disadvantaged group are assumed to experience a societal bias that limits their success probability.A policy of allocating opportunities stipulates thresholds on the success probability for the advantaged and disadvantaged group. We analyse the interplay between utility and a novel measure of fairness for different dynamics that dictate how the societal bias changes based on the current thresholds while the group sizes are fixed. Our theoretical analysis is supported by experimental results on synthetic data for the use case of college admissions.