From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction

By processing electronic health records (EHRs) as natural language sequences, large language models (LLMs) have shown potential in clinical prediction tasks such as mortality prediction and phenotyping. However, longitudinal or highly frequent EHRs often yield excessively long token sequences that result in high computational costs and even reduced performance. Existing solutions either add modules for compression or remove less important tokens, which introduce additional inference latency or risk losing clinical information. To achieve lossless compression of token sequences without additional cost or loss of performance, we propose Medical Token-Pair Encoding (MedTPE), a layered method that extends standard tokenisation for EHR sequences. MedTPE merges frequently co-occurring medical token pairs into composite tokens, providing lossless compression while preserving the computational complexity through a dependency-aware replacement strategy. Only the embeddings of the newly introduced tokens of merely 0.5-1.0\% of the LLM’s parameters are fine-tuned via self-supervised learning. Experiments on real-world datasets for two clinical scenarios demonstrate that MedTPE reduces input token length by up to 31\% and inference latency by 34-63\%, while maintaining or even improving both predictive performance and output format compliance across multiple LLMs and four clinical prediction tasks. Furthermore, MedTPE demonstrates robustness across different input context lengths and generalisability to scientific and financial domains and different languages. The code is available in the GitHub repository.

Clinicians and researchers are exploring whether large language models (LLMs) can read electronic health records to help predict outcomes, such as whether an intensive-care patient may survive or be diagnosed with a new disease. A major obstacle is that these records can be very long: before a model reads the text, it breaks it into small text pieces (known as tokens), and medical terms often get split into many tokens, making predictions slow and expensive. To deal with this issue, we proposed MedTPE, a method that lets the model’s text reader combine pairs of medical text tokens into one larger token. This shortens the record without deleting clinical information, and it updates only the meanings of these new tokens rather than retraining the whole model. In tests on two real-world health-record datasets and four prediction tasks, MedTPE shortened the input text by up to 31% and reduced processing time by 34-63%. Importantly, it achieved this efficiency while maintaining or even improving predictive accuracy and producing answers in the required format. This could make LLMs more practical for long medical records, especially when processing time matters. However, since medical language varies across hospitals and regions, these gains should be checked locally before use in patient care.