Unlike cloud GPU servers that tap into the power grid and have a broad set of AI tools, edge devices like earbuds rely on ultra-efficient processors designed to run on battery power. Such devices also come with limited memory, requiring models to be small – and that’s just the beginning. A successful edge VAD model must also work within four hard constraints that most research models targeting the same task can quietly ignore.

1. The model needs to use the chip's built-in audio preprocessing. Before any AI runs, raw audio signals must be converted into a usable format. Non-edge models often compute their own custom audio representation instead, adding overhead and breaking compatibility with the rest of the device's audio pipeline. Most edge chips have dedicated hardware to do this - a built-in shortcut, which model designers must adapt to.
2. It can only use math the chip supports. Beyond preprocessing, the model's internal operations must run efficiently on the low-power hardware. Generalist models often rely on specialized calculations that embedded processors handle poorly or can't run at all.
3. It has to decide quickly. Most models need to hear over half a second of audio before deciding whether someone is speaking. That’s a noticeable lag on a device that's supposed to have an instant feel.
4. The model must be accurate in real-world conditions. Real-world deployments rarely make for a flattering dataset when benchmarking models. Due to the low latency required from edge models they cannot use audio from the future (sampled after the initial audio) to help classify the past.

Rather than starting with a large model and shrinking it later, the team built kiloVAD specifically for embedded inference.

The model builds on common microcontroller capabilities: It uses a fully convolutional neural network (CNN) architecture. It avoids recurrent layers, custom digital signal processing (DSP) code, and specialized activation functions. CNNs are often easier to deploy on edge devices than recurrent neural networks (RNNs) because they use fixed computations that can run efficiently in parallel and are well supported by embedded AI runtimes. The model also relies on Mel Spectrograms, a fixed, human hearing-inspired frequency representation that is widely supported in embedded systems.

This keeps the design portable across hardware platforms and compatible with common embedded ML workflows. This simplicity turns out to be a major advantage when aggressive compression is required.

One of the paper's key findings is that VAD performance improves rapidly as audio context grows—but only up to a point.

The researchers evaluated input windows, or audio samples, ranging from 60 ms to 360 ms and found that performance largely plateaued around 200 ms. To illustrate just how short a sample that is, that’s just enough time to utter a single syllable.

At 200 ms, kiloVAD achieved nearly the same accuracy as much longer windows while reducing response time substantially. This is particularly important for always-on systems, where every millisecond delay affects user experience. Compared with several leading compact VAD models that use 630 ms of context, kiloVAD cuts latency by roughly two-thirds.

The team also studied different Mel-spectrogram resolutions, grouping audio frequencies into Mel bins. Each Mel bin is a range that roughly matches how humans perceive differences in pitch. They showed that accuracy degrades gracefully when reducing the number of Mel bins.

This flexibility makes model deployment easier across embedded platforms with varying DSP capabilities.

The most impressive part of the work, however, is how much the model can be compressed.

The full kiloVAD model contains roughly 81,000 parameters. The team sought to prune the model, that is reduce the neural network size by removing underused and duplicate elements with minimal impact on the model’s performance. Using a structured pruning strategy optimized on a layer-by-layer basis, the researchers reduced the original model to just 2,100 parameters—a 97.4% reduction.

Instead of applying the same pruning rate everywhere, the team searched for an optimal pruning ratio for each layer individually. They then recovered lost accuracy using self-distillation, where the original model teaches the compressed version.

The results are striking. Even after massive compression, the 2.1K-parameter model remained within 1.3% on the AVA-Speech benchmark compared to the full model's performance.

Compression is only half the story. Edge devices also benefit from quantization, which replaces floating-point math with low-precision integer arithmetic (INT) that uses less memory space.

For INT8 (8-bit integer) deployment, kiloVAD showed almost no accuracy loss. But the researchers wanted to push further.

Their solution was an angle-aware self-distilling quantization-aware training (QAT) method. Instead of focusing only on prediction accuracy, the technique preserves the geometric relationship between learned features and class representations.

The team argues that parameter count alone does not determine how easy it will be to deploy. A model must also satisfy four practical requirements:

1. Compatibility with standard audio front ends – the audio-processing pipeline that turns microphone signals into signals an AI model can use
2. Portable operations supported by embedded runtimes – relying exclusively on math operations offered by many embedded processors makes the model easier to use more widely
3. Low latency – it’s fast!
4. Strictly causal evaluation – the model works well under real-world conditions using only the audio available at that moment, not the future

kiloVAD is the only model in the authors’ industry comparison that satisfies all four simultaneously while delivering competitive performance. The full model achieves 0.862 AUC (area under curve – a common model precision measure), while the pruned 2.1K-parameter version reaches 0.850 AUC using only 200 ms of context.