Markov Chain from Human Feedback

We propose Markov Chain from Human Feedback (MCHF), an alternative approach for aligning generative models from pairwise human preferences. Unlike RLHF, which reduces comparisons to a scalar reward, and NLHF, which preserves pairwise utilities through a KL-regularized minimax objective, MCHF uses pairwise preferences directly to define a transition mechanism over model outputs. Given a pairwise utility U(x,y) and a reference model muref, we define a Markov kernel P(x,dy) proportional to exp(U(x,y)) muref(dy), and take its stationary distribution as the aligned model. We show that MCHF converges geometrically fast to the stationary distribution, with a convergence rate governed by the additive-defect seminorm inf_{g,f} ||U - g oplus f||_infty, which quantifies the non-transitive structure of the pairwise utility. We further show that a mirror-descent algorithm for NLHF, with an appropriate step size, satisfies an analogous structure-adaptive convergence guarantee. Finally, through a perturbation analysis, we prove that when this additive defect is small, MCHF and NLHF agree up to first order around an RLHF solution, yielding a unified view of reward-based, game-theoretic, and Markovian alignment.