Curriculum Reinforcement Learning for Black-Box Prompt Tuning via Large Language Models

Black-box prompt tuning (BBPT) aims to optimize input prompts for large models where internal parameters and gradients are inaccessible. However, existing methods fail to simultaneously address the dual challenges of prompt interpretability and query efficiency. To address these challenges, we propose CRL-BPT, a curriculum reinforcement learning framework that utilizes a large language model as an agent to generate human-readable prompts. Specifically, CRL-BPT implements a dynamic curriculum schedule on two auxiliary objectives: an imitation loss and an innovation loss. By dynamically weighting these objectives, CRL-BPT regularizes the RL process, guiding the agent from mimicking reference prompts to discovering novel patterns. Additionally, we introduce tailored stabilization mechanisms comprising historical loss normalization and relative reward calibration to promote more stable training. Extensive experiments demonstrate that CRL-BPT establishes new state-of-the-art performance and generates highly interpretable prompts under a strict budget of API calls. Code is available at https://github.com/GongShuai8210/CRL-BPT.

Current powerful vision-language models are typically provided to users in the form of ''models as a service,'' allowing interaction solely through textual prompts. To accomplish specific tasks, users must supply precise prompts. However, existing automated prompt learning methods are not only computationally expensive and costly but also frequently generate unintelligible, garbled text that is incomprehensible to humans. We propose a novel framework named CRL-BPT, which leverages large language models as agents to automatically generate human-readable prompts. We train this agent using a curriculum learning strategy inspired by human learning: in the initial phase, the agent learns to imitate high-quality prompt templates, and subsequently, it is progressively encouraged to deviate from fixed patterns and explore innovative, novel ways of description. This approach rapidly discovers highly accurate prompts at an extremely low query cost. As a result, users can efficiently apply state-of-the-art AI technologies to their unique tasks, even without access to the model’s internal parameters and under tight budget constraints.