Weakly Supervised Cross-Modal Learning for 4D Radar Scene Flow Estimation

Due to the difficulty of obtaining ground-truth data for 4D radar scene flow estimation, previous methods typically rely on either self-supervised losses or cross-modal supervision using 3D LiDAR data, 2D images, and odometry. However, self-supervised approaches often yield suboptimal results due to radar's inherently low-fidelity measurements, while existing cross-modal supervised methods introduce complex multi-task architecture and require costly LiDAR sensors to generate pseudo radar scene flow labels from pretrained 3D tracking models. To overcome these limitations, we propose a task-specific iterative framework for weakly supervised radar scene flow learning, using only images and odometry for auxiliary supervision during training. Specially, we establish two novel instance-aware self-supervised losses by exploiting off-the-shelf 2D tracking and segmentation algorithms to obtain tracked instance masks, which are back-projected into 3D space to provide instance-level semantic guidance; for static regions, we integrate vehicle odometry with radar's intrinsic motion cues to construct a rigid static loss. Extensive experiments on the real-world View-of-Delft (VoD) dataset demonstrate that our method not only surpasses state-of-the-art cross-modal supervised approaches that rely on 3D multi-object tracking on dense LiDAR point clouds but also outperforms existing fully supervised scene flow estimation methods. The code is open-sourced at \href{https://github.com/FuJingyun/IterFlow}{https://github.com/FuJingyun/IterFlow}.

The emerging 4D radar is cost-efficient and robust to adverse weather conditions, becoming a promising sensor for scene flow estimation. However, radar point clouds are sparse and noisy, and ground-truth scene flow labels is difficult to obtain, making 4D radar scene flow estimation extremely challenging. Earlier methods either perform poorly under self-supervision or require cross-modal supervision from 2D images, odometry, and expensive 3D LiDAR data. In this work, we introduce a new cross-modal weak-supervision setting for 4D radar scene flow estimation, using only RGB images and odometry during training while requiring radar-only input at inference. We use off-the-shelf 2D tracking and segmentation models to generate 2D instance masks from images and assign instance id to each radar point by 2D-3D projection. With instance-aware guidance, we design two self-supervised losses that alleviate mismatch and ambiguity in sparse radar scene. We also develop a lightweight iterative network in which ball query-based grouping improves robustness over commonly used kNN in sparse radar point clouds. Our proposed weak supervision can largely substitute for expensive flow labels and also highlights the future potential of 2D semantics for 4D radar scene flow estimation.