Hi, and welcome to this AI and CGI video on an OpenAI and Azure OpenAI specific topic,
which is batch jobs.
So batch jobs is a way for you to get your AI much cheaper, meaning 50% off, if you can
wait for the responses.
So let's see what that is in details.
Because there's a lot to go for batches, because if you check a normal LLM call, it has the
normal model price, depending on the model you choose, of course, while a batch job will
actually only cost you 50% of that, meaning half.
It also gets you much higher rate limits, like 10x rate limits on certain models.
Newer models is slightly less.
So what's not to like, well, there is the difference of that a normal job, you will
be guaranteed instant processing, it begins to work with your data, while in a batch job,
you can technically wait up to 24 hours before you get your job back.
In real life, for low number of requests, it's much faster, which we'll see in one second
in a demo, but they are not guaranteeing it to happen.
So what is actually happening is, when they feel there's no spikes in the number of requests
going to the LLMs, they will process these batch jobs on the fly.
While there's high traffic, you will get less true.
So when there's much traffic in, for example, US waking up, then batching will take longer,
but within 24 hours, you will get your result.
Both of them can do chats, client and responses API, and embeddings.
So you can also take your embeddings and do in batches if need be.
There's some limits in that you can max send out 50,000 requests per file of a batch, because
a batch is happening by sending a file to the LLM, instead of just the individual request.
And it can max be 200 megabytes, that's quite a lot, so we're probably not going to limit you
going to hit, or else you can send multiple batches.
And tool calls are not really supported.
You can technically give a tool, but then the response back from the batch will just be,
hey, I want to run this tool, and then you need to send a new batch with the tool in it.
So it is possible, but in reality, you probably wouldn't want to use that part of it for this.
So let's see this in action.
So I have, under other topics here, I have something called batch jobs,
and right now I'm using OpenAI, but Azure OpenAI is also possible.
We will talk about that at the end, because there's slight differences in what we need to do.
So let's run this.
And the first thing we do is, of course, we make our client like normal,
and then we make a file client.
We have seen file clients before in other topics, but that is just to put up a file into OpenAI.
And, again, I could switch out the OpenAI client with the Azure OpenAI client to be exactly the same.
And then I get something called a batch client as well,
because we need to upload what we want to have processed by the LLM using a file,
and we need to upload one called JSON-L for JSON lists.
And if we look at that file over here, it is just a set of JSON.
So one line equals one JSON.
You can't even put in line breaks.
You need to have them on one line.
And you essentially tell, I want to have a job being done, giving it a unique ID.
It needs to be post.
It needs to go against the completion endpoint.
It could be responses endpoint or embedding endpoint.
It needs to be this model that does it, and then you send your messages.
So in this case, we're just sending a system message with nice AI,
and a role of the user where we can send that along.
So if we do it like this, we can send three requests.
So in this case, we are asking, why is the sky blue?
What is the capital of France?
And what is a JSON-L file?
Which I actually didn't know about these files before that.
So in OpenAI, we just give the model name like normal.
If you're going to do this against Azure OpenAI, it's slightly different,
because there, what you put in model here is the deployments.
And up in Azure, when we do deployments up here with our models,
each model are either set as a global standard or a global batch.
So I made one here called global batch.
And it's only the deployment type global batch that will work together with the system.
But when you deploy a model, it's fairly simple to do.
We take and see if we can choose batch here on this one.
But if you go into custom, you choose here.
In this case, it's codec, so it can't do it.
But you will normally choose the model here.
So let's go to nano and have this one we want to deploy as a batch.
And we can choose global batch or data zone batch if we want to do that as well.
So we can definitely do that up here.
And that's the reason why I have a separate file if I want to do it using Azure.
But back to the code.
So in our case, we're just using the one that used HTTP 4.1 mini.
So we get our file.
And now it's uploaded to the system.
And we need to tell the purpose of that file is batch, else it can't pick it up.
And then we're going to make something called a create batch request.
Because this is a very open-ended system.
So there's no real out-of-the-box objects for these.
So instead, we need to make our own object here,
where we need to have an input file ID, an endpoint, and a completion window.
And we're giving the ID of the file we just uploaded.
We're telling it's completions.
And we're telling it's a window of 24 hours.
It's a little odd that they have this, because it's the only option you can give.
But when we do that, we get something called a binary content.
And we can say we want to create a batch.
We can wait for the batch to complete.
So it will just sit and wait.
But in real life, you will probably not wait for the batch.
I've seen batches come back between 20 seconds and 4 minutes for these three things.
But of course, with much more, it will take longer.
But just to simulate that we're not waiting,
we start a batch.
So you can see it didn't take a long time.
And we just get back that we have a batch ID.
So we're writing out our batch ID here.
And then we're going to go into a while loop and ask, how is our batch doing?
So there's three termination statuses that we need to listen for.
Completed, expired, or canceled.
So we can either complete it, get it completed.
It can expire if it didn't finish within the 24 hours.
Or we have an endpoint that can say, we want to cancel this batch.
We don't want to run it anymore.
Then it's possible to do that.
So we have the options of validating, failed, in progress, and so on.
So if we take these different termination statuses and just start a clock,
we can go in and check if the status of a batch,
which we use the batch client again and say, get batch.
Right now, I have a delay here of 10 seconds we need to wait for first.
Because in real life, it would not complete within those.
We will now get a batch result back.
And this batch result don't really have anything directly for us.
Because again, it's a very open-ended system.
So the real thing we get back is actually this JSON.
And in this case, we can see now it's in progress.
When it started, what is the output file and all kinds of things of the request.
But right now, it's still running.
And if we go to OpenAI, we can go in and check the batches up here.
And it's actually now finished because it ran for 12.
So let's just quickly restart the code with a new batch
so we can see this in more seconds.
So let's get it running.
Let now have a new batch.
We will be able to see there's a new batch up here. It's running.
It's already completed the first of the three.
And if we go to the code, I've just set up that every 10 seconds,
it should give us back how far it's along.
And we can set a breakpoint here.
Because we are always asking for the response back here,
which have a status, some counts, and output files.
So this time, it takes a little longer.
And we are essentially waiting for one of the statuses
to be completed, expired, or canceled.
And once we get that, it will go out of our loop.
Of course, in real life, you wouldn't make a while loop like this.
You would instead have some kind of job that checked every 30 minutes
or however long you feel like your batches will be.
And up here, we can now see it's finalizing.
So we can also see that status here.
So we should be very, very close to getting our result back,
meaning our output file here will end up having a value.
So now our output and status is completed.
We go out of the while loop.
We now have our output file.
And we can download the output.
So in this case, I'm just downloading it to a temp folder.
And then I'm showing it to you in Notepad here.
So if we take this and look at a JSON viewer,
there's no real good JSON viewer for JSONL files.
But if we look in here, we can see we have the first result back,
the second result back, and the third result back.
And the first result was, why is the sky blue?
And you can see we only get the response back.
We don't even get what the questions were.
So we need to match our own ID of the request with our own response here.
So we need to save the request as well and then merge them.
But that's normal, C sharp.
So we won't go into that.
But we can get back why the sky is blue.
We can get back what is the capital of France in the second request.
And the third one was, what is a JSONL file?
So if we have a look at the final part here,
it's just we download the file and we save it.
So we need a little of this extra thing with some objects here.
You can just copy paste mine here.
And there's a bunch more data in them, of course, as we saw.
But this way, we have now asked three questions.
We spent a couple of minutes.
We can see that up here.
I need to refresh here so we can see.
So it took one minute and 26 seconds to get those three messages.
And that would, of course, not be good for a chatbot.
But it would be good if it's just some background job.
And now we have saved 50% of our price just for waiting.
So this is really, really good for some big advanced stuff
where you really, really hit your rate limits.
Or you just want to have it more cheap.
There's more work for you because you need to do all these extra steps in the code.
And the biggest problem, of course, is things like structured output and tools
that is more difficult to work with here.
But we can do it.
And it's really nice that we can.
Azure have exactly the same.
If we go to the models again, go into build.
Go to the models.
We can see our batch jobs.
The only real difference I have seen is it can't really on the fly,
like we saw here, tell how many of them were completed.
It seems like it stays at zero.
And then it suddenly goes to complete it all.
But that's the only difference in what I have seen in the two differences.
But again, we need to deploy a dedicated model for batching.
But else it works exactly the same just with the Azure OpenAI Cloud.
So that's everything there is to know about batching.
Really nice subject if you need to have some really, really big payloads.
So see you in the next one.