Deterministic Speaker Separation in Stereo Audio Streams: VAD, Channel Ownership, and Segment Boundaries

The decisions VL(t) ∈ {0, 1} and VR(t) ∈ {0, 1} separately indicate whether the left and right channels carry valid speech in a stereo recording. When channel mapping is guaranteed by the recording architecture,

left channel  → speaker A
right channel → speaker B

speaker identity is not estimated biometrically; it is derived from the physical input channel. The central problem is then determining which channel carries valid speech over each time interval and in which segment that speech should be processed.

General speaker diarization clusters portions of a single audio stream according to acoustic similarity and assigns speaker labels. Channel-based separation, by contrast, can be deterministic with an appropriate state machine, a reliable voice activity detector, and explicit segmentation rules. The same PCM input produces the same speaker sequence, segment boundaries, and output timestamps.

Channel ownership and VAD decisions

The reliability of channel ownership depends on the following infrastructure assumption:

channelOwner(left)  = A
channelOwner(right) = B

This mapping can be provided in structures such as two-party telephone, radio, or operator systems. However, the presence of energy in a channel does not by itself establish that the channel owner is speaking. Cross-channel leakage, speaker-microphone echo, automatic gain control, environmental noise, tones, short impulses, and codec artifacts can create false activity.

For this reason, the following energy comparison is insufficient:

energy(left) > energy(right) then speaker A

This rule only selects the dominant amplitude; it does not determine speech activity, whether the other channel is truly silent, or whether both channels carry speech simultaneously. A more robust approach produces an independent VAD decision for each channel:

VL = 0, VR = 0 → silence
VL = 1, VR = 0 → left-channel candidate
VL = 0, VR = 1 → right-channel candidate
VL = 1, VR = 1 → simultaneous or ambiguous activity

There is no universally correct policy for simultaneous activity in both channels. Under a single-active-segment and non-overlapping-output constraint, preserving the priority of the current owner is a deterministic choice. When the active segment belongs to A and both channels become active, A continues; B can become the owner only after A activity ends and B activity is confirmed. This approach does not represent every detail of genuine simultaneous speech, but it does not produce overlapping time intervals and is compatible with downstream ASR systems that expect a single audio stream.

Frame, turn, and segment time scales

VAD operates on short PCM frames. In the example configuration, the sampling rate is 16 kHz, the sample format is 16-bit signed little-endian, and the frame duration is 10 ms. The number of samples in a single-channel frame is:

N = 16000 × 0.010 = 160 samples

With two bytes per sample, the mono frame size is 320 bytes and the stereo interleaved frame size is 640 bytes:

160 × 2 = 320 bytes
160 × 2 channels × 2 bytes = 640 bytes

A single 10 ms VAD decision should not be converted directly into a segment-opening or speaker-switching decision. Noise impulses, plosive consonants, echo, and transient leakage can cause short-lived decision changes. Three time scales should therefore be separated:

1. Frame level: the instantaneous VAD observation.
2. Turn level: the short window in which a speaker change is confirmed.
3. Segment level: the longer time interval sent to ASR or another model.

The active owner is not changed until a candidate owner remains dominant for a specified number of consecutive frames:

candidateFrames >= confirmThreshold

With 10 ms frames, confirmThreshold = 4 corresponds to an approximately 40 ms confirmation window. Eight frames provide an approximately 80 ms observation. These thresholds do not measure a linguistic unit; they test the temporal stability of the signal decision.

A state machine with one active segment can be established using the following variables:

owner          ∈ {NONE, A, B}
segmentStart
segmentEnd
candidateOwner ∈ {NONE, A, B}
candidateCount
silenceCount

When there is no active owner and only one channel carries speech for a sufficient duration, a new segment is opened. When A is active and the left channel remains active, the segment is extended. When the left channel becomes silent and the right channel becomes active, a direct A → B transition is not performed; B first becomes a candidate. If the confirmation period is completed, the A segment is closed and a B segment is opened:

A → candidate B → B

Silence, target duration, and mandatory limits

Silence tolerance can vary according to segment length:

segment < 3 seconds       → approximately 1000 ms
3 seconds ≤ segment < 25 → approximately 300 ms
segment ≥ 25 seconds     → approximately 200 ms

The piecewise threshold function is:

          1000 ms, 0 ≤ T < 3
G(T) =     300 ms, 3 ≤ T < 25
           200 ms, T ≥ 25

Here, T is the active segment duration and G(T) is the consecutive silence duration required to close the segment. A longer wait for short segments prevents a natural pause after one or two initial words from closing a segment prematurely. A shorter threshold for long segments limits segment growth, model-call latency, and memory use.

Two distinct duration limits can be used together:

target duration : 25 seconds
hard upper limit: 29 seconds

The application behavior is defined as follows:

T < 25 s
    apply natural silence rules

25 s ≤ T < 29 s
    close at the first suitable silence

T ≥ 29 s
    force closure without waiting for silence

The target duration favors closure at natural pauses. The hard upper limit strictly bounds segment size during uninterrupted speech. These limits help control ASR inference time, temporary tensor sizes, feature-extraction cost, queue latency, and the volume of data to be reprocessed after an error.

The hard limit should be aligned to a frame boundary. With 10 ms frames, 29 seconds equals 2900 frames:

29 × 100 = 2900 frames

To avoid rounding errors, duration can be tracked using processed sample or frame counts rather than floating-point seconds:

frameStart = processedSamples
frameEnd   = frameStart + validSamples

The timestamp is derived from sample position:

timeMs = sampleIndex × 1000 / sampleRate

For 16 kHz:

timeMs = sampleIndex / 16

The integer-division order and rounding policy must remain the same throughout the system.

Minimum speech duration and common routing

Sending every VAD activity directly to the model is inefficient. Short noises, interjections, and partial words can generate unnecessary inference. Two separate lower limits can therefore be used:

minimum actual speech duration : 1 second
minimum model input duration   : 3 seconds

Activity shorter than one second can be rejected as a segment. Valid speech between one and three seconds can be extended to three seconds with silence before it is sent to the model. Validity and model-input size should be evaluated using separate variables:

speechDuration = actual active speech duration
modelDuration  = speech + added silence

The validity decision should be based on speechDuration, whereas the model input size should be based on modelDuration. Padding does not increase content; it only satisfies the model's minimum input requirement.

For mono input, one VAD result can be routed directly to a general speaker label or to A:

V(t) = 1 → A
V(t) = 0 → NONE

For stereo input, channels are separated into distinct buffers and independent VAD is applied. Both input types can pass the same information to a shared route mechanism:

owner
PCM data
valid byte count
frameStart
frameEnd

The segmentation infrastructure therefore remains shared; only the ownership-determination stage differs:

mono   → one VAD result
stereo → two VAD results + ownership policy

The interleaved stereo PCM layout is L0, R0, L1, R1, L2, R2, .... When channels are copied to separate buffers, sample number n in both channels must represent the same time position. The processed sample count should represent the time axis per channel:

processedSamples += validSamplesPerChannel

The total stereo sample count or total byte count should not be added directly to this variable.

Determinism and the limits of channel separation

When fixed frame sizes, preallocated buffers, integer sample counters, and explicit state transitions are used, processing cost per frame remains approximately constant:

O(N)

Here, N is the number of samples in a frame. Total cost grows linearly with audio duration:

O(F × N)

F denotes the total number of frames. Creating new objects, collections, streams, lambdas, or temporary channel arrays for every frame can increase garbage-collector pressure and latency variability. Preallocated buffers and reused VAD states make temporal behavior more predictable. Producing the same segments from the same input allows VAD, channel selection, ASR, and text post-processing to be examined separately during debugging.

Channel-based ownership is not a method that can replace general diarization in every case. Channel identity cannot be used as speaker identity when multiple people use the same channel, microphones are not fixed to speakers, stereo recording is created only for spatial audio, channels are mixed, one speaker is distributed similarly across both channels, or routing changes between recordings. Under these conditions, methods such as embedding-based representations, clustering, speaker verification, or beamforming are required.

Conversely, when channel mapping is reliably supplied by the infrastructure, re-estimating speaker identity can add unnecessary computation and uncertainty. VAD observes only speech activity; channel mapping provides identity; the state machine provides temporal consistency; and segmentation rules preserve the latency and input limits of the downstream model.