We study how to aggregate the preferences of multiple agents over LLM-generated replies to user queries. The agents are self-interested and may thus misreport their preferences, and new agents may participate for each new query, making fine-tuning LLMs on their preferences impractical. To address these challenges, we propose an auction mechanism that works without fine-tuning or access to model weights. The mechanism is designed to provably converge to the output of the optimally fine-tuned model as computational resources are increased. The mechanism can also incorporate contextual information about the agents when available, significantly accelerating its convergence. Our mechanism ensures that truthful reporting is the optimal strategy for all agents, while also aligning each agent’s utility with her contribution to social welfare - an essential feature for the mechanism's long-term viability. Although our mechanism can be applied in any setting with monetary transfers, our key application is online advertising. In this domain, advertisers try to steer LLM-generated responses based on their interests, while the platform aims to maximize advertiser value and ensure user satisfaction. Experimental results confirm that our mechanism not only converges efficiently to the optimally fine-tuned LLM, but also significantly boosts advertiser value and platform revenue, all with minimal computational overhead.