Providing Context: Enriching Graph Learning Using Language Model Priors

We introduce GINAT, a graph-language augmentation framework that improves message-passing GNNs by conditioning their edge messages on LM-embedded global graph descriptions. Unlike text-attributed graph methods that rely on node/edge-attached text, GINAT leverages external prompts that capture graph semantics independently of the downstream task. These prompts are injected into message passing through a cross-attention mechanism that preserves permutation equivariance, enabling global contextual guidance within standard MPNN computation. Across a broad suite of node- and graph-level benchmarks, GINAT yields consistent gains over strong GNN baselines, reaching up to 23% relative improvement. Ablations show that the gains arise from the semantic content of prompts rather than random vectors, and an analysis of the learned context scaling reveals where and how strongly different backbones and layers rely on language-conditioned information. Overall, GINAT establishes LM-embedded prompts as a practical, interpretable graph-language signal to strengthen graph representation learning. Our code is available at https://anonymous.4open.science/r/llm4gfm-3FBE/