Interactive Person Retrieval via Multi-Turn Multimodal Conversation

Traditional text-based person retrieval approaches typically rely on single-shot textual queries, which are generally incomplete or vague in real-world scenarios. Recently, chat-based person retrieval methods enable iterative query refinement via question-answering interactions between the system and users. However, these methods fall short of direct user interaction with retrieved candidates during conversation, making it challenging to effectively refine the retrieval results. To address these limitations, we propose multimodal interactive person retrieval (MInterPR), a new retrieval paradigm that allows users to iteratively refine retrieved candidates by providing feedback on visual differences from the target person. To support this task, we establish MInterPEDES, a multimodal conversational dataset constructed by augmenting existing question-answering dialogues with synthesized visual feedbacks. Furthermore, to tackle the challenge of accurate and efficient semantics understanding in multimodal dialogues, we propose a multimodal conversational memory-enhanced framework MNEMO, which incorporates an atomic turn encoding (ATE) module to model each dialogue turn independently, and a dialogue memory aggregation (DMA) module to capture the fine-grained information and cross-turn dependencies. Extensive experiments demonstrate that MNEMO achieves substantial improvements in both retrieval accuracy and generalization ability, highlighting its promising potential in real-world scenarios. The code and dataset are publicly available at https://github.com/Flame-Chasers/MNEMO.

Finding a specific person in a large image collection is difficult when the user only provide a single text description, which is often incomplete or vague. A more natural search process is interactive: the system shows possible candidates, and the user gives feedback such as “the target person has longer hair” or “the clothing color should be lighter.” This paper studies this interactive setting for person retrieval. We introduce a new task that allows users to refine search results over multiple conversation turns by describing visual differences between retrieved candidates and the target person. We build a new dataset for this setting and propose a method that uses both images and conversation history to gradually improve the retrieval results. Experiments show that our method better understands user feedback and progressively retrieves more accurate person images, suggesting its potential for more practical and user-friendly person retrieval systems.