Abstract: Many problems in machine learning involve data sets in which each data point is a pointcloud in $\mathbb{R}^D$. A growing number of applications require a means of measuringnot only distances between point clouds, but also angles, volumes, derivatives, and othermore advanced concepts. To formulate and quantify these concepts in a coordinate-invariantway, we develop a Riemannian geometric framework for point cloud data. By interpretingeach point in a point cloud as a sample drawn from some given underlying probabilitydensity, the space of point cloud data can be given the structure of a statisticalmanifold -- each point on this manifold represents a point cloud -- with the Fisherinformation metric acting as a natural Riemannian metric. Two autoencoder applicationsof our framework are presented: (i) smoothly deforming one 3D object into another viainterpolation between the two corresponding point clouds; (ii) learning an optimal setof latent space coordinates for point cloud data that best preserves angles anddistances, and thus produces a more discriminative representation space. Experimentswith large-scale standard benchmark point cloud data show greatly improvedclassification accuracy vis-\'{a}-vis existing methods. Code is available at https://github.com/seungyeon-k/SMF-public.