Learning to Predict Zero: Supervised Subspace Cancellation for Spurious Domain Robustness

We study a seemingly paradoxical objective: training a neural network to map every input to the same scalar target, zero, while still learning non-trivial internal structure. A cancel-only loss enforcing y = w^T z ≈ 0 admits degenerate solutions, such as collapsing the head or compressing the embedding, so we use this principle to motivate zero-projection learning, where selected directions are forced toward zero while task-relevant structure is preserved. We extend scalar cancellation to multi-dimensional subspace cancellation by introducing an orthonormal projection head W in R^{K x d}, maintained via QR re-orthonormalization, and minimizing the projected energy ||Wz||2^2, which suppresses a K-dimensional subspace while preserving the orthogonal complement for task-relevant features. We instantiate this idea for supervised spurious domain cancellation with an asymmetric update schedule: a domain head is trained on a domain-balanced split to capture an injected style, after which the encoder is optimized for the main task while canceling the learned domain subspace. Across MNIST, CIFAR-10, and SVHN under controlled spurious correlations with pcorr = 0.99, our approach reduces domain leakage measured by a linear probe while maintaining competitive accuracy, and can improve generalization under shift, for example on SVHN. Sensitivity analyses over K and beta further reveal a trade-off between invariance strength and main-task performance.