From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG

With the rapid emergence of personal AI agents based on Large Language Models (LLMs), implementing them on-device has become essential for privacy and responsiveness. To handle the inherently personal and context-dependent nature of real-world requests, such agents must ground their generation in device-resident personal context. However, under tight memory budgets, the core bottleneck is *what to store* so that retrieval remains aligned with the user. We propose EPIC (Efficient Preference-aligned Index Construction), which focuses on user preferences as a compact and stable form of personal context and integrates them throughout the RAG pipeline. EPIC selectively retains preference-relevant information from raw data and aligns retrieval toward preference-aligned contexts. Across four benchmarks covering conversations, debates, explanations, and recommendations, EPIC reduces indexing memory by 2,404$\times$, improves preference-following accuracy by 18.79\%p, and achieves 32.17$\times$ lower retrieval latency over the best-performing baseline. In on-device experiments, EPIC maintains under 1 MB memory and achieves 5.21 to 29.35 ms/query latency across three platforms, while supporting streaming updates under preference drift. Our code and data are available at https://github.com/UbiquitousAILab/EPIC.

Personal AI assistants are becoming more common, but running them directly on phones, laptops, and other small devices is hard. They need personal information to give helpful answers, but repeatedly retraining a large AI model with new personal data is expensive and unrealistic. A more practical option is to store useful information on the device and look it up when answering. The key challenge is deciding what to store, because personal devices can collect far more data than they can safely keep. Our work asks a simple question: what should a personal assistant store in the first place? We propose EPIC, a method that keeps information related to a user’s preferences, such as likes, dislikes, needs, or constraints, while filtering out less useful data before storage. EPIC also looks up stored information in a way that better matches the user’s preferences. Across several benchmarks, EPIC used much less memory, gave answers that better respected user preferences, and responded faster than strong alternatives. Our on-device experiments show that EPIC can run efficiently on resource-limited devices, supporting more private and responsive personal AI assistants.