PrivLEX: Detecting legal concepts in images through Vision-Language Models

Privacy laws, such as the General Data Protection Regulation, define personal data as information related to an individual and require this data to be protected. Yet on social media, people often share images that contain personal data, sometimes inadvertently exposing the information they would protect if informed. Therefore, identifying and explaining personal data leaks may help users make informed decisions about their privacy. We present PrivLEX, a novel image privacy classifier that grounds its decisions in legally defined personal data concepts. PrivLEX is the first interpretable privacy classifier aligned with legal concepts that leverages the recognition capabilities of Vision-Language Models (VLMs). Built as a label-free Concept Bottleneck Model, PrivLEX learns the relation between personal data concepts detected by a VLM in an image and the image privacy label. In this way, PrivLEX allows users to be informed about the personal data concepts an image contains before sharing it. It also identifies the privacy of the concepts as perceived by human annotators of image privacy datasets.