-: Hey, in this video, we're gonna have a look
at a implementation of the Realtime API,
which is made by OpenAI.
We're gonna look at how this uses WebSockets
to essentially stream messages from a server
directly to a client,
and a client will then send messages directly to the server.
This was released on October 1st, 2024.
If you actually go and have a look
at the Realtime API documentation,
you'll see this is in beta, so it's still quite early.
It supports things like native speech-to-speech.
It can also do simultaneous multimodal output
and natural steerable voices.
We're gonna have a look at the Realtime console
in this lesson.
And the reason why is there's a demo application
that you can use to have a go
and see how the event stream happens in real time.
I want you to navigate
to openai/openai-realtime-console.
And then what you're going to need to do is click on Code,
click Copy to Clipboard, load up a terminal,
and then we're gonna git clone this repo.
After you've clone that repository,
now you're gonna need to create an environment variable file
at the root level directory with a .env.
And you're gonna need to put both the OPENAI_API_KEY
and also we'll need to add in
the REACT_APP_LOCAL_RELAY_SERVER_URL,
so I'm just gonna copy that as well.
And then make sure to put your API key for OpenAI here.
Okay, so the next thing we wanna do
is just install all of the packages with npm i
and then we wanna run npm run start
and we also want another server,
so go and load up a new tab and do npm run relay.
And then what you should see is a Realtime console.
Go to the bottom and click on Connect.
And there's two different ways that you can do this.
You can see there's a manual way of doing this
and a VAD way.
So the manual way is push talk.
So I can say, "Hey, how's it going? How's everything been?"
And then you can see that the assistant is now replying
like, you know, that everything's been going good.
Or I can do VAD, which will listen in real time
to the audio buffer and will then reply.
So you can now see that just the buffer
is constantly appending,
but then when you hit certain drops in volume,
then the assistant will reply.
So the other things we can do is say,
"I want to get the weather in Chicago."
And if you look on the right-hand side,
the map has now changed to Chicago
and we've got the actual weather output there,
which is great, so you can see that's working,
which is really good.
So you can use tool use within these model.
So the way that this is actually working
is basically we're using WebSockets' technology
to stream events from the server that OpenAI is providing,
which is a stateful backed service.
So it stores all the information for us
and it streams those as WebSocket events
to the client side via a WebSocket connection
inside of React.
I think this is gonna open up a lot of individual use cases
and we'll see the ability to even hook up Twilio
and chat and phone numbers with the OpenAI Realtime.
So we're gonna be able to get
those voice-to-voice interactions.
You're gonna be able to stream
multiple types of modalities, including speech, text,
and also being able to use tools,
giving the agents specific types of tools
that it can use in a real-time setting.
The cost implications for this is slightly more expensive.
As you can see, it's $5 for a million input tokens
and $20 for a million output tokens.
And then you can see,
we've got a certain number of audio tokens.
So these are the ones where it's quite expensive.
So even though these token costs are much larger,
it is actually quite cheap
to generate responses using the realtime API.
Okay, cool, hopefully you found this useful and exciting
and I'll see you in the next one.