We present an approach for safe trajectory planning, where a strategic task related to autonomous racing is learned sample efficiently within a simulation environment.A high-level policy, represented as a neural network, outputs a reward specification that is used within the objective of a parametric nonlinear model predictive controller.By including constraints and vehicle kinematics in the nonlinear program, we can guarantee safe and feasible trajectories.Compared to classical reinforcement learning, our approach restricts the exploration to safe trajectories, starts with a good prior performance and yields complete trajectories that can be passed to a tracking lowest-level controller.We validate the performance of our algorithm in simulation and show, how it learns to efficiently overtake and block other vehicles.