Hi, and welcome to this AI and C-Sharp video on the OpenAI Realtime API.
So, the Realtime API is the way we can do speech-to-speech,
meaning we can ask the question in audio
and actually also get audio responses back.
So, let's go into a demo.
So, here I have the code running,
and I have the output over here.
So, let's first see a demo, and then we'll go to the code.
Hi there.
Hello. How's it going? What can I help you with today?
Fine. Can you tell me what the capital of France is?
Of course. The capital of France is Paris.
It's known for its art, culture, and landmarks like the Eiffel Tower.
Cool. Can you tell me how many people live in that city?
Sure. Paris has a population of around 2 million people in the city itself,
and if you include the greater metropolitan area, it's about 12 million.
Okay. Thank you.
You're welcome. If you have any more questions, feel free to ask.
So, that's the demo.
We talked about Paris, but we can talk about anything.
So, let's have a look at what the code actually does
and why it works the way it works.
So, we first get our normal sequence,
and then we make either our OpenAI client or our Azure OpenAI client,
but we're not actually making an Azure OpenAI client,
because if we do that, it will actually come back with an exception
saying the Realtime API is not supported,
and we should use the OpenAI client instead.
I think this is a first step in switching to everything
being in the OpenAI client and then for Azure setting an endpoint,
because this is also the first time we really used
the new slash OpenAI slash v1 endpoint with OpenAI.
So, the way we do it here is then we say Azure OpenAI client
and then set the endpoint,
as we have seen a few times before as well.
And once we have the client, we can get the Realtime client,
because the API is not the chat client or the responses API,
it's what is called Realtime client instead.
We also need models, and this is a model we haven't used before,
so if you're on Azure, you might need to deploy this model.
There's either a GPT Realtime,
there's a... for mini, a non-mini,
and there's a 1.5 version as well.
These are a bit expensive, like $10 per 1 million tokens
and $20 million for output,
and if you go to the non-mini, it's $32 and $64.
In real life, testing it out, I've sat here for a couple of hours
and preparing this demo,
and that has cost me roughly half a US dollar.
So, not too, too expensive, but still.
You can also give one more model for transcriptions,
meaning when we talk,
that they can actually see what we said in text,
as we saw in our output.
This is optional, so if you want to save that,
if you don't need it anyway, you can get rid of it.
I will show a little later how we can do that.
Then, when we ran it, it said we could press Ctrl-C to cancel.
That is just a console.cancel key press here.
And then we take our Realtime Client,
and then we start a conversation,
and that conversation happens with the model.
And then we configure the session,
and there's a bunch of settings we can set on this.
I've taken the most common ones here that I thought we needed.
So, we can give instructions, just like system message,
it should talk in a special way, and so on.
And I'm just giving it that it's a voice assistant,
and it needs to be clear and brief.
Then, we can set some audio options, meaning input audio.
So, for the input, if we want a transcription, we set this.
If we don't, we can just take it away.
You can control some noise reduction, if need be.
And then there's turn detection, which is the main part we do here.
So, it goes in, and you can set various options here,
like whenever I start speaking,
how many milliseconds do it need to go back in the recording
in order to pick up the first words I say,
how long should it wait for a pause until it actually begins answering,
and how long time should it wait
before it tries to do a follow-up for us,
meaning if we are just silent, it says,
hey, I can't hear you, is there something we should talk about?
And finally, we have if we want to be able to interrupt,
meaning in the middle of the AI speaking,
if we want to say something else,
if that should happen or not, or it should finish its sentence.
It doesn't happen instantly,
but at a certain point in the longer conversation it gives back,
it can stop itself and begin to ask in a follow-up question.
Then we have output, where we can control how fast they speak
and which of the different voices we want to have, male or female.
I've used this Marin, it sounds nice.
And for output, we just tell it everything is audio.
Technically, the real-time can also work with images and text,
but that's beyond this video.
Then we do all the in-audio stuff.
In-audio is a NuGet package to work with audio on Windows.
And up here, we will put in an audio player.
So essentially, I don't want to go into this,
but it's essentially just that we can talk with the...
or we can play back streaming audio feedback
that we get from the API.
And in the same manner, we have a microphone streamer
that we can start a recording,
and then we can restart that recording for every time we have something new,
but else the microphone is open all the time,
waiting for us to set.
Again, this is...
This is vibe-coded from using Codecs,
and it's just C-sharp more than anything.
And then we do something a bit special,
because this is an open-ended system.
So instead of doing a wait and async,
we just have these things for tasks.
So we are keeping a continuous task of uploading Microsoft Audio,
and we have a continuous task of receiving updates back.
Upload is fairly simple.
It's just taking the microphone
and read everything that has been said so far into chunks,
and put it into the session.
So session being from the real-time that we created.
And then we receive the updates,
but before we look at that, let's just see that we just start our microphone,
and then we begin to await receiving updates.
And at some point, we will press Ctrl-C or stop the program,
and we will get a cancellation, where we just cancel everything.
We stop the microphone, we clear the audio player, and then we're done.
But the most important thing happens down in receive updates.
So every time, we just have a receive update,
and it can do an await for each update.
And every one of these updates is something different.
The first one we normally get is a session created,
where we can just then write some text out.
We say the session is created, speak naturally,
and wait for the AI to answer back, and press Ctrl-C.
Then there's a buffer speech started,
meaning whenever we begin to speak,
it tells, oh, I'm in listening mode right now.
So we just say, listening back in our console here.
And whenever we stop speaking, we get the speech stopped,
and then it begins to think, because it might take a little while
for our question to be answered based on how difficult the answer is.
Then, if we have to transcribe in place,
we get a transcribe completed object,
where we can just say what we said.
And we can also get the transcribe from the other side
on what the AI said.
We can get it back here in just a done,
but there's also a delta version where we can get it back streaming if need be.
I tend to not to do that.
But what we do need is we need to get a delta of all the audio that's being said.
So every time a small piece of what is being said is put in,
we enqueue that into the audio player,
so it will know when it can begin to give data back.
If we ever clear the buffer, we can just clear the audio player.
And when our response is done, we flush the pending audio,
so we just have it tell out the rest of the data.
And if there's any errors or anything,
we can get, for example, error messages back
if there's something wrong with the response.
And finally, if there's any errors.
So we can try to set some breakpoints here and see it in real life.
This will feel a bit odd because it just begins to...
to say words and pause it again and so on, but let's have a look.
So whenever we, say, start session up here,
start conversation, we get that the session created.
There's also a session updated, which technically happens,
but I haven't had any need for it,
but there's a real-time server update session configuration updated.
But we get this, and we are ready to speak.
And now I spoke something, so it thought that we are listening to that.
And now it hears that we don't speak anymore,
so it's beginning to think about it.
It will become a very odd conversation here.
And now I'd have to transcribe back,
which, yeah, okay.
It couldn't understand what I was saying while we did this,
but now we are getting the audio back in small chunks,
so this one will be hit multiple times.
I'm pressing F5 right now, multiple times.
I'll just get rid of it here.
And now the transcribe is done on what it is it wants to say back.
So, sure, I'm here to help.
That normally comes just before the audio comes back.
So it writes that back, and it now says,
we are done, so now it begins to speak.
So if I press F5 here.
Could you tell me a bit more about what you'd like to talk about
or what kind of support you're looking for?
Of course. I'm here to help.
Take your time and share what's on your mind,
and we can go from there. Thank you.
You're very welcome. Just let me know whenever you're ready,
and we can dive into it together.
So as you can see, it's quite difficult to debug, of course,
hence putting out various messages along the way.
But that's just the nature of trying to speak
while also trying to speak in this video, of course.
So that is actually what's there.
Again, the hard part, for me at least,
was to get someone to make this start and stop recording
of audio streams and playing back the audio streams.
Beyond that, it is fairly simple.
There's tons of settings, and this can do much, much more
than what we are covering in this video.
But beyond that, it's fairly simple to understand
once you go through it line by line.
This is a bit funky, of course, that we are not really making
normal code with the same weight, but it makes sense
when we are just having an open microphone waiting for data.
So I think that's everything. See you in the next one.