Rethinking Parameter Sharing as Graph Coloring for Structured Compression

Parameter sharing is a key model compression technique, yet existing methods overlook the geometric properties of the loss landscape, often causing severe accuracy degradation under high compression ratios. Inspired by second-order optimization, we propose Curvature-aware Graph Coloring (CGC), a cross-layer parameter sharing framework that treats each network layer as a graph node, with each node assigned to a shared low-rank basis. CGC leverages Hessian eigenspace information to group layers with similar curvature profiles, aligning the perturbations introduced by parameter sharing with the low-curvature (flat) directions of the loss ellipsoid. This effectively mitigates performance loss while enabling flexible, global cross-layer sharing. Experiments on LLaMA-7B and Swin Transformer show that CGC achieves 28\%–50\% parameter compression with Top-1 accuracy loss no more than 0.01\% on Swin—or even accuracy gains on LLaMA—while delivering over 60\% higher inference throughput, significantly outperforming SVD-based and heuristic-based methods. This work demonstrates that curvature-guided, geometry-aware sharing is essential for efficient, stable, and high-ratio model compression.

Large AI models often contain repeated or redundant information, which makes them expensive to store and run. This paper introduces a method for compressing such models by allowing different parts of the model to share compact building blocks, instead of storing every parameter separately. A key challenge is that careless sharing can damage model accuracy. Our method addresses this by estimating which changes to the model are likely to be harmful and which changes are relatively safe. It then chooses sharing patterns that avoid sensitive directions and preserve the model’s behavior as much as possible. We test the method on both language models and vision models. The results show that it can substantially reduce model size and improve inference speed while keeping accuracy nearly unchanged, and in some cases even slightly improving performance. This suggests that carefully designed parameter sharing can make large AI models more efficient and easier to deploy.