Invited Talk 4: Kamalika Chaudhuri - Principled Approaches to Measuring Memorization

While memorization of training data has long been a challenge in developing foundation models, how to measure it properly and efficiently in a principled manner is not as clear. In this talk I will discuss two such principled approaches from our recent work. First, I'll present deja vu -- a way to measure memorization in pre-trained representation learning models (without re-training). Then if time permits, I'll present a new way to measure memorization in LLMs motivated by Kolmogorov complexity, and show that our measurement can be used to explain a number of observations about model capacity.