From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models

Large language models (LLMs) are increasingly deployed in privacy-sensitive domains, where users must balance the risk of data exposure through external APIs against the high computational cost of local deployment. Split learning has therefore emerged as a promising paradigm for LLM fine-tuning and inference under limited local resources. However, it introduces new privacy risks. Prior work primarily studies leakage of private input prompts, typically via inversion attacks on intermediate representations, while the potential for sensitive information leakage through generative response outputs remains largely unexplored. In this work, we unveil novel vulnerabilities of Split-LLM by presenting Patched Model Inversion with Dual-Sided Initialization(PIDI), a two-stage attack that simultaneously targets both private input prompts and output responses in Split-LLM settings. It combines dual-sided initialization with a patched inversion strategy to tackle long sequences, substantially outperforming prior inversion methods. To counter threats from both sides, we further propose the Adapter-based DualGuard with Mutual Information Defense(ADMI), which integrates an adapter-based local warmup strategy and mutual information regularization to provide a strong empirical privacy protection with minimal impact on task performance. Extensive experiments across diverse tasks and models demonstrate that ADMI effectively defends against PIDI and other state-of-the-art inversion attacks. Our code is publicly available at \url{https://github.com/FLAIR-THU/VFLAIR-LLM}.

Large language models are increasingly used in sensitive areas where sending private data to external AI services may cause privacy risks. A promising solution is to split the model between a user’s device and a cloud server, so that raw data never leaves the user side. However, we found that this setup is still vulnerable: an attacker operating the cloud server may reconstruct not only the user’s private prompts, but also the model’s generated responses. To study this risk, we introduce a new attack called PIDI that can recover both user's query and model's generated response from the intermediate information exchanged during model inference. Our method is especially effective for long conversations and significantly outperforms previous reconstruction attacks. We also propose a defense method called ADMI, which reduces the risk of reconstruction attacks while preserving the model’s performance, which is validated across multiple language models and datasets. Our work highlights previously overlooked privacy risks in split large language models and provides practical tools to make them safer for real-world deployment.