InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

AI agents are increasingly used to automate research and development, yet existing benchmarks evaluate them on prescribed workflows or narrow action spaces where strong results may reflect memorized recipes rather than genuine optimization. We introduce InferenceBench, where an agent must deploy an OpenAI-compatible inference server and optimize LLM inference speed. Each agent receives a target LLM, one H100 GPU, and a 2-hour budget. Four scenarios isolate prefill latency, decode latency, concurrent request throughput, and a balanced multi-objective setting. Across 15 frontier agent configurations, agents reliably improve over a naive PyTorch baseline (up to 8.08x) and often match or exceed serving engines with default settings (4.05x for vLLM), but still fall below a simple hyperparameter search under the same time budget (up to 11.53x). Although agents enumerate the relevant optimization techniques, they overwhelmingly converge on a single inference framework, test few distinct configurations, and spend the remaining budget re-measuring or repairing hyperparameters rather than exploring substantially different strategies. The bottleneck is not domain knowledge, but the ability to propose diverse configurations, evaluate them systematically, and submit the best identified solution.