Learning domain-invariant representations is important to train a model that can generalize well to unseen domains.To this end, we propose a novel approach that leverages the semantic structures inherent in text descriptions as effective pivot embeddings for domain generalization. Specifically, we utilize graph representations of images and their associated textual descriptions to obtain domain-invariant pivot embeddings that capture the underlying semantic relationships between local images and text descriptors.Our approach involves a clustering-based graph-matching algorithm that matches graph-based image node features into textual graphs.Experimental results show the efficacy of our proposed method in enhancing the generalization ability of the model.