Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization

Recent advances in tool-integrated language agents have significantly improved their ability to solve complex reasoning tasks. However, existing alignment methods predominantly focus on maximizing task accuracy, while overlooking auxiliary objectives such as tool-use efficiency, which are essential for practical deployment. To address this gap, we introduce \textbf{ParetoPO}, a two-stage multi-objective optimization framework for aligning tool-using large language models (LLMs) under competing objectives. In the first stage, ParetoPO leverages hypervolume-guided dynamic scalarization to adapt reward weights based on global Pareto frontier progress. In the second stage, it replaces scalarized learning signals with Pareto-ranking-based advantage computation, promoting nondominated trajectories through dominance-aware credit assignment. This design enables fine-grained, action-level optimization across multiple conflicting objectives. Experimental results on mathematic reasoning and multi-hop QA tasks show that ParetoPO consistently discovers policies with superior accuracy-efficiency trade-offs compared to static and heuristic baselines. Our code is publicly available at https://github.com/Applied-Machine-Learning-Lab/ICML2026_ParetoPO.

Language models are increasingly used as agents that can call external tools, such as search engines, calculators, or databases, to solve difficult problems. While these tools can make agents more accurate, using them too often can also make the system slower, more expensive, and harder to deploy in real-world applications. Our work studies how to train language agents to balance these competing goals: solving the task correctly while using tools efficiently. We propose ParetoPO, a training method that does not force the model to optimize only one objective. Instead, it helps the agent discover better trade-offs between accuracy and efficiency. ParetoPO first learns how much attention to give to each goal as training progresses, and then encourages solutions that are not clearly worse than others across multiple goals. Experiments on math reasoning and multi-hop question answering show that ParetoPO produces agents that achieve strong performance while using tools more economically.