Structured pruning offers a practical path to shrinking models for deployment on resource-constrained platforms. At its basic form, the technique removes entire units like channels, filters, or even entire layers. Better yet, unlike unstructured pruning, which requires tracking the locations of nonzero weights, structured pruning produces cleaner, smaller weight matrices that are easier to optimize and deploy.

However, pruning isn’t just about deleting parts of a model. Removing parameters in one layer can impact others, especially in tightly coupled architectures like RNNs. Managing these dependencies is key to keeping pruned models functional and effective.

While PyTorch provides basic pruning utilities via torch.nn.utils.prune, these tools are limited: they only apply masks without reducing actual model size or handling interlayer dependencies.

To solve this, we’ve been working with torch-pruning, an open-source library that physically removes redundant components from a model. Unlike PyTorch’s built-in utilities, torch-pruning builds a dependency graph of the network to determine how layers interact. This enables structured pruning that respects the model’s architecture.

We were recently tasked with work on a speech denoising project and tapped torch-pruning to help us compress a convolutional-GRU model. The Conv2D layers were straightforward to prune, yielding a 25% to 30% reduction in multiply-accumulate operations (MACs) without hurting performance.

But the GRU layers presented a challenge. PyTorch’s GRUs are implemented in C++ and don’t play well with torch-pruning’s dependency graph. Moreover, because GRUs feed their hidden state back into themselves, pruning these inputs wasn’t supported out of the box.

Our solution? Build a prunable GRU from scratch using standard PyTorch layers like nn.Linear and elementwise operations. We also introduced an identity adaptor layer to decouple the hidden state size from the GRU input, enabling safe pruning.

Here’s the workflow:

1. Replace native GRUs with custom prunable GRUs, copying over existing weights.
2. Apply structured pruning using torch-pruning.
3. Convert prunable GRUs back into native GRUs.
4. Fine-tune the model to recover accuracy.
5. Deploy the now-smaller model.

This approach resulted in a 30% reduction in model parameters, in addition to the MAC savings from convolutional pruning.

In our voice activity detection (VAD) project, which uses a mostly convolutional architecture, we pruned an 80k-parameter model down to under 2k parameters while preserving state-of-the-art performance. Key to these changes: carefully pruning later layers while preserving rich early-layer features.