Parameter-Efficient Transformer Embeddings via Functional Factorization

Embedding tables in transformer sentence encoders are dense matrices whose parameter count scales with vocabulary size. We study a narrow question: can these tables be replaced by a shared functional parameterization in sentence-level representation learning? We propose Parameter-Efficient Transformer Embeddings (PETE), which maps token IDs to normalized scalar coordinates, expands them with fixed Fourier features, and refines them using a lightweight residual MLP. PETE replaces independent token-specific vectors with a shared row-generation rule for the embedding matrix. In controlled NLI-to-STS-B experiments, PETE retains much of the performance of learned lookup embeddings while reducing total parameter count by up to 8x-10x. In ultra-compact regimes, PETE maintains non-trivial semantic alignment with as few as 58K parameters. These results suggest that functional embedding parameterizations are a promising direction for compact sentence encoders, while also motivating future comparisons to explicit low-rank and hash-based embedding compression.