IsoAct: Structure-Preserving Post-hoc Debiasing via Isometric Actions

Learning representations that are robust to nuisance factors is essential for reliable deployment, but remains challenging when the representations lie on non-Euclidean manifolds. Existing debiasing methods typically assume Euclidean representations or apply linear interventions, which can distort the geometry of structured latent spaces. This creates a mismatch between the Euclidean assumptions of the editing operation and the geometric constraints that manifold-valued representations must satisfy. We propose IsoAct, a post-hoc representation-editing framework that reduces nuisance information by composing manifold-native action primitives while maintaining downstream utility and manifold validity. The framework unifies debiasing on hyperspherical, hyperbolic, and SE(3) representations by parameterizing valid transformations within each geometry. We evaluate the approach across synthetic and real-world datasets using debiasing capability, task preservation, and manifold-constraint violation. Across these settings, IsoAct yields a favorable nuisance--utility--validity trade-off relative to Euclidean editing baselines, reducing nuisance recoverability while maintaining competitive task preservation and low manifold-constraint violation.