When embedding knowledge graphs, choosing the right geometry can heavily impact the expressivity of the embedding model to accurately predict the relations of the knowledge graph. Importantly, the structure of the chosen space should ideally accommodate the structure in the graph. Existing approaches describe embeddings on non-Euclidean geometries by closed-form analytical equations describing movements on them, limiting the geometry of choice to canonical cases; others forego this by reformulating the problem in a way that removes an interpretation grounded on geometry. In this paper, we generalise the conceptualisation of learning embeddings on manifolds by allowing any choice of metric to be used, whether for which known closed-form equations for movement are known or not. Experimentally, we show that this not only recovers existing approaches, but highlights the practicality of learning embeddings on general geometries.