Developing a Voice Activity Detection Library

Voice activity detection (VAD) is not merely a simple threshold operation that classifies an audio stream as speech or silence. In a real system, PCM samples must be read correctly, frames must be formed without drift on the time axis, noise must be modeled, short-term decisions must be smoothed, and segment boundaries must be determined consistently. An error in any link of this chain can make an algorithm unreliable in production even when its mathematical model is correct.

When I developed my first VAD library in C, I combined the signal-processing knowledge I had acquired during my undergraduate education with the audio-processing and digital-forensics work on which I focused during graduate study. I later ported the same algorithm to Java and deployed it in an internal audio-processing system. For me, this process was more than an implementation of an algorithm. It became a programming and system-design effort extending from low-level PCM processing and statistical decision models to fixed-point arithmetic and Java memory behavior.

Preparing the PCM stream

The first layer of the algorithm is the reliable acquisition of audio samples. The primary format I used was 16-bit signed little-endian PCM at a sampling rate of 16 kHz. A ten-millisecond frame consists of 160 samples. In a mono stream this frame occupies 320 bytes, while in a stereo stream it occupies 640 bytes.

16000 samples/second x 0.010 seconds = 160 samples

160 samples x 2 bytes = 320 bytes

A single read call from a file or network stream cannot be assumed to return all requested bytes. Partial reads are normal, particularly in real-time streams. If VAD processing begins before a frame is complete, sample boundaries drift. In a stereo stream, the same error also disrupts channel alignment.

The reading layer must therefore accumulate data until it reaches the number of bytes required for one frame. When the stream ends, the treatment of an incomplete frame must be defined explicitly. The incomplete data can be discarded or padded with zeros. Both methods are valid, but the decision must remain fixed so that the same input does not produce different segments across executions.

A little-endian 16-bit sample is reconstructed in Java with the following relation:

sample = lowByte | (highByte << 8)

Because Java byte values are signed, the low byte must not be widened directly. Otherwise, negative values propagate into the upper bits.

final short sample = (short) ((data[offset] & 0xFF) | (data[offset + 1] << 8));

Stereo PCM data is usually interleaved:

L0, R0, L1, R1, L2, R2, ...

The sample counter for each channel must be maintained independently and correctly. Deriving timestamps from the number of processed samples rather than from wall-clock time produces more reliable results:

timeSeconds = processedSamples / sampleRate

This method is unaffected by system load, thread scheduling, or transient delays. The same PCM input produces the same segment boundaries and time values.

Dividing the signal into bands

Raw energy alone is not sufficient to separate speech from noise. A door slam, keyboard sound, or short mechanical impact can produce high energy. A steady fan can continuously carry energy at low frequencies. The distinctive structure of speech lies in the distribution of energy across frequency bands and in how that distribution changes over time.

The first version I developed in C used a processing chain based on fixed-point arithmetic. This choice provided predictable processing cost in embedded or resource-constrained environments. It also required explicit management of overflow, scaling, and sign behavior.

The absolute peak value of the samples is calculated first. If the peak is extremely low, detailed analysis of the frame may be unnecessary. This check does not replace the main VAD decision. It serves only as an early gate for signals close to the numerical floor.

A high-pass filter is then applied. Its purpose is to reduce the direct-current component and very-low-frequency oscillations. Components such as microphone offset or mechanical vibration can increase total energy while contributing little speech information.

The signal is then divided into subbands by a filter bank composed of all-pass filters. An all-pass filter does not change the magnitude response by itself. Low- and high-frequency components can be obtained by taking the sum and difference of paths with different phase responses. Applied in stages, this operation divides the speech spectrum into several subbands.

The structure I used operated on six frequency bands. Energy was calculated for each band and converted into a logarithmic or approximately logarithmic fixed-point representation. This compressed the effect of large amplitude differences and made a wide dynamic range easier to process.

The energy of a band can generally be expressed as:

E_b = sum(x_b[n]^2)

Here, x_b[n] denotes the samples in frequency band b. Direct use of energy can increase overflow risk. In the C implementation, the bit widths of intermediate values and the shift amounts were therefore part of the transformation.

The main benefit of the filter bank is that the speech decision does not depend on a single energy value. Continuous noise concentrated at low frequencies and speech energy distributed across several bands produce different feature vectors.

Statistical speech decision

After the band energies are obtained, speech and noise hypotheses are compared for each frame. The decision layer in the library represented speech and noise distributions with Gaussian mixture models.

For an observed feature value x in each frequency band, the Gaussian probability density has the following form:

p(x | mu, sigma) = 1 / (sigma sqrt(2 pi)) x exp(-(x - mu)^2 / (2 sigma^2))

In a fixed-point implementation, this expression is not evaluated directly with floating-point arithmetic. Mean, variance, inverse variance, and weight values are represented by scaled integers. The exponential function and probability calculation are also converted into the approximate form used by the algorithm.

The speech and noise models contain multiple components rather than a single Gaussian. This structure represents different energy distributions within the same class. A noise model, for example, can track both low-level background noise and higher-level steady noise with different components.

Speech and noise likelihoods are calculated for each band:

L_speech,b = p(x_b | speech model) L_noise,b = p(x_b | noise model)

The local decision is based on the ratio between speech and noise likelihoods in the same band:

R_b = log(L_speech,b / L_noise,b)

The global decision combines the weighted contributions of the bands:

R_global = sum(w_b R_b)

Local and global thresholds are used together. A strong speech indication in a single band can exceed the local threshold. A weaker but consistent distribution across several bands can affect the global decision. This dual structure reduces dependence on a transient rise in a single frequency region.

Model parameters are updated over time. Frames evaluated as noise help the noise means and variances adapt to the environment. The speech model can also be updated in a controlled manner. If the update rate is too high, a short-lived sound distribution rapidly drifts the model. If it is too low, the system adapts slowly to a changing acoustic environment.

Leakage of speech into the noise model is a serious risk during model updates. If speech frames are repeatedly accepted as noise, the decision boundary rises and the algorithm begins to miss later speech. The initial decision, energy floor, model distance, and previous state must therefore be evaluated together.

Temporal decision and state machine

A raw frame-level decision cannot be used directly as a segment boundary. A single negative decision in a ten-millisecond frame may result from a short pause within speech. A single positive decision may be impulse noise.

To address this problem, I used a state machine operating across consecutive frames. Speech onset requires the accumulation of a specified number of positive frames. A segment is not opened after the first positive decision. A candidate state is created and the decision is confirmed over several frames.

With ten-millisecond frames, confirmation over four frames corresponds to approximately 40 ms, while eight frames correspond to 80 ms. Reducing the threshold lowers onset latency but increases the probability of false triggering. Increasing it stabilizes the decision, but the first phonemes of speech may be lost.

A look-back buffer can prevent this loss. When the speech decision is confirmed, several frames preceding the decision are also included in the segment. Confirmation latency therefore does not cause the audio itself to be cut.

A segment is not closed immediately after speech ends. Negative frames are tolerated during a period known as hangover. Short pauses, breathing intervals, and weak phonemes remain within the same segment. The segment ends when consecutive silence exceeds the configured threshold.

A fixed silence threshold is not suitable for speech of every duration. With a short segment, a longer wait prevents a single word from being cut prematurely. With a long segment, a shorter ending threshold reduces latency and unnecessary buffer growth. I therefore used an adaptive gap based on segment duration. A tolerance approaching one second can be applied to short segments, approximately 300 ms to medium-length segments, and approximately 200 ms to segments approaching the target duration.

The segmentation layer must distinguish more than speech and silence. At minimum, the following states are required:

• Silence
• Speech candidate
• Active speech
• End candidate
• Forced termination

The target duration and the hard upper limit must be separate. A segment that reaches the target duration can close at the first suitable pause. Once the hard upper limit is reached, termination is required without waiting for silence. This distinction balances speech integrity against memory and latency bounds.

Stereo speaker ownership

In stereo telephone or conversation recordings, each channel can represent a different speaker. Mixing the channels into a mono signal before processing loses speaker separation and may cause amplitude cancellation during simultaneous speech. I therefore passed the left and right channels through independent VAD chains.

Each frame has four basic states:

00 both channels silent 10 only the left channel is a speech candidate 01 only the right channel is a speech candidate 11 both channels are speech candidates

Energy comparison alone does not provide reliable channel selection. One channel may have higher microphone gain, or its steady noise may carry more energy. Channel ownership should be determined from the VAD decision and decision history.

Under a single-active-segment policy, the current speaker is retained. Ownership does not change unless the other channel becomes a strong candidate for several frames. This delay prevents the selected channel from changing every 10 ms. A transition can occur after the new channel exceeds a defined confirmation period.

Both channels being evaluated as speech at the same time is ambiguous. If the application must produce non-overlapping segments, the current owner can be retained. If genuine overlapping speech must be processed separately, a single-owner state machine is insufficient. Two independent segments or a more advanced source-separation approach is required.

Porting the C implementation to Java

Memory layout, integer overflow, and sign extension were directly visible in the C version. Moving to Java did not eliminate these issues. It only changed their form.

C uint8_t and Java byte do not behave identically. Right shifts, sign extension, and integer-promotion rules must be checked individually during the port. When an intermediate result in fixed-point multiplication can exceed the int range, long must be used. Narrowing the result again requires an explicit shift and bounds control.

Replacing pointer arithmetic from the C code with repeated subarray creation in Java is inappropriate. Allocating a new array, list, or object for every frame in a real-time audio stream creates garbage-collector pressure. In the Java version, I reused fixed-size buffers. Channel separation, filter-bank processing, and feature calculations operated on preallocated arrays.

The algorithm state was held in a single object. The noise model, speech model, filter history, hangover counters, and segment state belonged to the same stream. Reusing the same VAD instance concurrently across multiple audio streams was therefore incorrect. Each independent channel or session required its own state object.

The objective of the port was not to translate the source code syntactically. The decision order, scales, and saturation behavior of the C version had to be preserved. Intermediate values were compared so that filter outputs, band energies, likelihoods, and final VAD decisions could be verified step by step.

Reliability in a live system

A VAD algorithm working on clean audio samples is not necessarily ready for production. Live use requires tests for interrupted streams, zero-length reads, corrupted stereo alignment, sudden amplitude changes, long silence, continuous noise, and uninterrupted data lasting for hours.

Before deploying the library, I evaluated not only decision accuracy but also temporal behavior. Reprocessing the same data had to produce the same segment boundaries. Frame-processing time had to remain below the real-time budget. Buffers could not grow without an upper bound.

False-positive and false-negative rates alone are insufficient metrics. The amount of speech cut from the beginning and end, preservation of short words, and the latency with which long segments close must also be examined. A VAD system can achieve high frame-level accuracy while producing poor segments at application level.

The property that made the library usable in an institutional system was not classification performance alone. The boundaries of the entire chain from PCM reading to segment output were defined. Processing advanced with fixed-size frames, avoided unnecessary object creation during execution, and derived timestamps from the sample counter.

This development process clearly demonstrated the difference between a signal-processing algorithm and production software. The mathematical model formed the core of the decision. Reliable behavior emerged from data acquisition, state management, the time axis, memory use, and error handling. When the work that began in C was moved to Java, the algorithm remained the same but the engineering problem expanded. The final product was not merely a function that detected speech, but a stateful processing system that transformed a continuous audio stream into segments under explicit rules.