OpenAI announced the general availability of its enhanced Realtime API and unveiled gpt-realtime, the company’s most advanced speech-to-speech model, designed for developers and enterprises building production-ready voice agents.
OpenAI’s gpt-realtime marks a significant leap forward in voice AI, delivering substantial advances in audio quality, instruction following, intelligence, and function-calling:
-
Audio Quality
The model generates natural-sounding speech, capable of expressing intonation, emotion, and nuance. It responds accurately to fine-grained voice instructions such as “speak quickly and professionally” or “speak empathetically in a French accent.” OpenAI also introduced two exclusive new voices, Marin and Cedar, while improving its existing eight voices. -
Intelligence and Comprehension
gpt-realtime demonstrates enhanced understanding of native audio, capturing non-verbal cues like laughter, handling mid-sentence language switches, and distinguishing alphanumeric sequences in languages including Spanish, Chinese, Japanese, and French. On the Big Bench Audio reasoning benchmark, the model achieved 82.8 % accuracy compared to 65.6 % by the previous December 2024 model. -
Instruction Following
The model significantly improved at executing developer instructions. On the MultiChallenge audio benchmark, it scored 30.5 %, compared to 20.6 % for the previous model, reflecting stronger adherence to nuanced developer directives. -
Function Calling
With better timing, relevance, and argument accuracy in tool invocation, gpt-realtime scored 66.5 % on the ComplexFuncBench audio evaluation, compared to 49.7 % previously. The model also supports asynchronous function calls, enabling ongoing conversation flow even as long-running tools process in the background.
The upgraded Realtime API includes new capabilities that bolster developer flexibility and agent intelligence:
-
Remote MCP Server Support
Developers can now configure sessions with remote MCP servers, streamlining tool integration without manual wiring. -
Image Input
The Realtime API now supports sending images such as screenshots or photos alongside audio or text allowing agents to interpret visual content and respond contextually. -
Session Initiation Protocol (SIP) Calling
The API now connects directly with public phone networks, PBX systems, and SIP endpoints, enabling voice agents to handle traditional telephony interactions. -
Reusable Prompts
Developers may now save and reuse prompts including developer messages, tools, variables, and sample dialogues across sessions for streamlined workflows.
Also Read: Google Introduces Gemini 2.5 Flash Image: A State-of-the-Art Image Generation and Editing Model
OpenAI emphasized that both gpt-realtime and the Realtime API are built with enterprise-grade safety and privacy measures:
-
Integrated active classifiers monitor for misuse, halting conversations that violate harmful content policies.
-
Developers must clearly disclose AI usage to end users where necessary.
-
EU Data Residency is fully supported, and the API aligns with enterprise privacy commitments.
Since the Realtime API’s initial public beta launch in October 2024, thousands of developers have contributed feedback that shaped today’s production-ready release. The API’s single-model architecture processing audio directly without splitting into speech-to-text and text-to-speech pipelines reduces latency and preserves speech nuance, enabling more authentic conversational experiences.
Experts see gpt-realtime as a major advancement in voice AI. Josh Weisberg, Head of AI at Zillow, noted: “The new speech-to-speech model in OpenAI‘s Realtime API shows stronger reasoning and more natural speech allowing it to handle complex, multi-step requests like narrowing listings by lifestyle needs or guiding affordability discussions with tools like our BuyAbility score. This could make searching for a home on Zillow or exploring financing options feel as natural as a conversation with a friend, helping simplify decisions like buying, selling, and renting a home.”
