AOEB: Benchmarking Agent-Oriented Multimodal Embeddings

LLM agents powered by retrieval and RAG are increasingly prevalent across research and applications. Embedding models play a critical role in these systems, particularly in embedding-based retrieval. However, current benchmarks for embeddings remain focused on general-purpose scenarios, which may fail to align well with the diverse and evolving needs of agentic applications. To close this gap, we introduce Agent-Oriented Embedding Benchmark (AOEB), a comprehensive evaluation suite dedicated to agent-centric retrieval for embedding models. AOEB is characterized by two key features: (1) Multi-Task, covering five essential capabilities for retrieval in LLM agents, including code, tool, reasoning, and memory retrieval; and (2) Multi-Modal, providing evaluation with both textual and visual data for each task category. We evaluate representative embedding models on AOEB and observe that they exhibit distinct strengths across different agent-oriented retrieval tasks. By curating AOEB, we aim to promote a move toward more practically oriented directions within the embedding community and foster further progress.

Large language model (LLM) agents are becoming increasingly common in applications such as coding assistants, digital helpers, and research tools. These systems often rely on “embedding models,” which help computers find the most relevant information from large collections of text, images, tools, or past interactions. However, most existing benchmarks for evaluating embedding models focus on general search tasks and do not reflect the complex needs of modern AI agents. To address this problem, we introduce the Agent-Oriented Embedding Benchmark (AOEB), a new evaluation benchmark designed specifically for AI agents. AOEB tests how well embedding models support important agent abilities, including retrieving code, selecting tools, recalling past memories, and handling reasoning-related tasks. It also evaluates performance across both text and image data, reflecting the multi-modal nature of real-world AI systems.