Solution Path Pruning (SPP)
What it is
SPP is a pruning framework for pretrained transformers that enables adaptive compression by constructing a 'Transformer Weight Family' with varying sparsity levels in a single search stage. It uses differential inclusions to allow users to deploy models with different compression ratios without re-running the pruning process.
The problem it solves
Traditional pruning methods usually require a separate pruning process for every specific compression ratio, making it inefficient to deploy models with varying levels of sparsity.
What you could use it for
- Compressing Large Language Models (e.g., Llama2, OPT) for deployment
- Reducing parameters in vision transformers (e.g., DeiT, Swin, CLIP) while maintaining accuracy
- Deploying models with multiple different sparsity levels from a single pruning run
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.