• Latency: Delays in response can disrupt the natural flow of interaction and negatively impact user experience.
• Recognition Accuracy: Errors in recognizing speech can lead to frustrating and unreliable performance.
• Interruptibility: AI systems need to be able to reliably detect and handle interruptions from users without causing confusion.
• Noise and Echo Resilience: Voice processing must be able to handle real-world acoustic variability, such as background noise and echoes.

Resource Efficiency: Excessive battery usage, bandwidth consumption, and memory demands can limit the usability of voice-based AI applications.

Device side processing eliminates the need for server round trip delayed decisions about noise, side talk, and real user speech. The result is an experience where the AI can instantly duck its audio and listen to the user when the user speaks or interrupts. Check out our video demos for examples of SMPL vs. Gemini, Meta, Copilot et al..

We have focused our efforts on a device side SDK that brings acoustic excellence and real-time decision making to the user’s device vs. the cloud approaches in production today at the big providers. This means the AI can “hear” and respond to the user’s voice instantly and reliably.

SMPL has developed a codec that presents an ultra-efficient representation of the user’s speech that is more suitable as an input to ML systems. We generate large segments of audio in one go - much faster than realtime. This is opposed to traditional VoIP codecs like OPUS that encode audio in a frame-wise fashion “on the fly”. Because it is ~100X faster than OPUS it is perfect for just-in-time encoding of good voice segments and any re-transmits when network conditions require.

WebRTC provides a good foundation for backend infrastructure and client-side development. However, its default acoustic signal processing (AEC3/AECM) is outdated and falls short of modern standards. This results in subpar audio quality for human-to-human communication and high word error rates in human-to-machine interactions. While the OPUS codec is suitable for human-to-human conversations, it is no longer state-of-the-art. For human-to-machine use cases, OPUS is outdated, computationally intensive, slow, and consumes significant bandwidth.

There are several factors that contribute to making a conversation natural. While some relate to the sound of the agent’s voice, it is also important to feel you are being “heard” by the agent. If it is noisy in the room or the agent is talking and you wish to interrupt, which is a natural part of any verbal exchange, the agent must stop talking immediately when you are talking. If it continues to talk, even for a few hundred milliseconds, then the interaction will be perceived as unnatural and undesirable. This immediate responsiveness is critical to a successful and enjoyable exchange.

Importantly, when you interrupt, it must also understand you. And this comes down to the acoustics processing. Mobile phones have loudspeakers very close to the same microphones that you use to communicate with the agent. The agent’s own voice can be 20 to 40 dB louder on that microphone than your voice. So the acoustic processing that removes the echo of the agent’s voice while it is speaking and preserves your voice while you are trying to interrupt, something called “doubletalk”, is also critical to a natural experience.

Acoustical processing is challenging due to several factors that include the following:

• Noise: Ambient sounds like traffic, wind, or other nearby talkers interfere with the user's voice.
• Echo: The microphone that captures the (desired) user’s voice also records the (undesired) agent’s voice which is an echo to be cancelled. The microphone is typically much closer to the loudspeaker than the user’s mouth, so the echo is significantly louder (and therefore more difficult to cancel) than the user’s voice.
• Distortion: Over-driving the loudspeaker "by turning it to 11" or damaged loudspeakers can render the agent’s voice unrecognizable and difficult to cancel, especially if the user is also talking or trying to interrupt.
• Time-Varying Nature: Constantly changing environments and moving sound sources require real-time adaptation.

These dynamic, interconnected issues demand advanced solutions for clear and consistent audio. Furthermore, because not all users have the latest and most powerful phones, practical solutions that work for the majority of already deployed devices require memory and processor efficient implementations.