BankCodec: Compact Memory Banks for Multi-Turn Novel View Synthesis

Multi-turn novel view synthesis generates a scene one target view at a time, making it a useful testbed for reliable long-horizon video generation. Current memory-augmented systems maintain persistent state by retrieving past views and feeding their dense tokens to the denoiser, which makes the conditioning interface grow linearly with retrieval size. We ask how small that interface can be when compression is optimized for the behavior that retrieved memory induces in the generator. We show this with BankCodec, which jointly reads retrieved views with learned queries and emits a fixed-rate compact bank trained directly through the generator's flow-matching loss. On the Memory-V2V benchmark, BankCodec exposes only 42 denoiser-facing memory tokens in place of the 16,380 retrieved-memory tokens used by uncompressed Memory-V2V, while preserving camera, subject, image quality, temporal flicker, and motion-smoothness metrics. It remains competitive with a 126-token local codec, while local streams retain better cross-view consistency. BankCodec thus turns a growing memory stream into a compact scene-level state and identifies the regimes where local correspondence cues remain useful. Code will be released upon acceptance.