LLM-VAE-Early
What it is
A block-wise conditioned Variational Autoencoder (VAE) designed to learn the weight spaces of Large Language Models. It decomposes models into transformer blocks and uses a Dual/Gram-matrix PCA to compress high-dimensional weights into a manageable latent space.
The problem it solves
The high dimensionality of full model weights (millions of parameters) makes direct latent representation and weight-space learning computationally expensive and difficult to generalize across different architectures.
What you could use it for
- Learning compressed representations of LLM weights across different model families.
- Model weight interpolation and manipulation in a shared latent space.
- Efficiently analyzing the structural commonalities between different transformer architectures.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.