Gravitational wave (GW) parameter estimation has conventionally relied on the assumption of Gaussian and stationary noise. However, noise from real-world detectors, such as LIGO, Virgo and KAGRA, often deviates considerably from these assumptions. In this paper, we use score-based diffusion models to learn an empirical noise distribution directly from detector data, which can then be combined with the forward simulator of the physical model to provide an unbiased model of the likelihood function. We validate the method by performing inference on a simulated gravitational wave event injected in real detector noise from LIGO, demonstrating its potential for providing accurate and scalable GW parameter estimation.