MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models

Modern Vision-Language Models (VLMs) pose significant individual-level privacy risks by linking fragmented multimodal data to identifiable individuals through hierarchical chain-of-thought reasoning. However, existing privacy benchmarks remain structurally insufficient for this threat, as they primarily evaluate privacy perception while failing to address the more critical risk of privacy reasoning: a VLM's ability to infer and link distributed information to construct individual profiles. To address this gap, we propose MultiPriv, the first benchmark designed to systematically evaluate individual-level privacy reasoning in VLMs. We introduce the Privacy Perception and Reasoning (PPR) framework and construct a bilingual multimodal dataset with synthetic individual profiles, where identifiers, such as faces and names, are linked to sensitive attributes. This design enables nine challenging tasks spanning attribute detection, cross-image re-identification, and chained inference. We conduct a large-scale evaluation of over 50 open-source and commercial VLMs. In our controlled benchmark, 60\% of widely used VLMs can perform individual-level privacy reasoning with up to 80\% accuracy, suggesting a significant potential threat to personal privacy. The benchmark is available at https://github.com/CyberChangAn/MultiPriv-PII.

Modern Vision Language Models VLMs can analyze images and text together, making them useful for applications such as assistants, search systems, and document understanding. However, these same abilities also create new privacy risks. A model may combine small pieces of information across images and documents, such as faces, names, locations, or records, to identify a person and infer sensitive details about them. We developed MultiPriv, the first benchmark designed to study this type of individual level privacy reasoning in VLMs. Instead of only testing whether a model can recognize isolated private attributes, our benchmark evaluates whether it can connect distributed clues and reconstruct personal profiles through multi step reasoning. To do this safely, we created a bilingual dataset of synthetic individuals with linked visual and textual information, and designed nine challenging tasks involving identity matching, cross image reasoning, and chained inference. We evaluated more than 50 widely used open source and commercial VLMs. Our results show that many advanced models can successfully perform privacy reasoning at surprisingly high accuracy, revealing significant potential risks for personal privacy. We hope MultiPriv will help researchers build safer and more privacy aware AI systems.