Hi, guys, and welcome to this other exciting session in which we are going to look at the Onyx format
of representing machine learning models.
Onyx actually stands for Open Neural Network Exchange.
This open standard for machine learning interoperability was co-developed by Microsoft, Facebook and
AWS.
And in this session, we'll learn how to convert our already trained TensorFlow model into this Onyx
format and then carry out inference on this newly created Onyx model.
So we've gotten to this point where we've fine tuned our hogging phase based vision transformer model.
Now we have this TensorFlow model which we have created and evaluated, and it may happen that another
developer using a different framework, like, for example, PyTorch, wants to make use of this model
which has been trained in TensorFlow.
So thanks to the Onyx format, we can now convert this code which was written in TensorFlow or rather
convert this model which was built in terms of flow into an Onyx model and then later on convert this
model from the Onyx format into PyTorch, such that we now have this PyTorch model which this other
practitioner can use.
Now another possibility is the reverse.
That is, we could go from this PyTorch model to a TensorFlow model.
Thanks to the Onyx format's interoperability.
Another reason why a developer will want to say take this model from PyTorch and make use of say, in
CAFE, for example.
Maybe because that model is more efficiently run in this other framework.
And the reason why we have these kinds of differences for different frameworks is because, for example,
you could have a convolutional neural network or a convolutional layer which is built in PyTorch with
a certain implementation, and the implementation in cafe may be slightly different and maybe more efficient
as compared to that which was done in PyTorch.
Now this is just for demonstrative purposes and we are not saying that the implementations in CAFE are
better than those in PyTorch.
And so in summary, the Onyx format allows models to be represented in a common format that can be executed
across different hardware platforms using the Onix runtime.
So now developers can feel free to build their models with just any framework.
Say, for example, TensorFlow or PyTorch or Padel or whatever framework they actually want to work
with, knowing that they could deploy your model on whatever hardware they want to, since they could
now convert these models to the Onyx format and then run inference on these models via the Onyx runtime,
which is in fact an inference engine which is lightweight and modular and permits us to run our Onyx
models on just any hardware we choose to work with.
So here let's look at the list of supported hardwares.
You see for example, the tensor r rt, which is very popular with the Nvidia GPUs and permit us to
attain very high speeds when carrying out in France on neural network models.
The other great advantage of working with the Onyx models is the ease with which we could convert the
TensorFlow models into this Onyx format.
So right here you see we evaluate in our model this hugging face model, which we fine tuned previously,
and we get in 90% validation accuracy.
Now we could go ahead and save this model.
So let's take this off H.f. model.
Let's call this hugging.
Let's call this white, white or fine tuned OC.
So that's it.
We have this fine tuned white model, which we're going to save.
We check this out here.
There we go.
We see a white fine tuned and you could check.
You see you have the saved model Keras method dealer variables the assets folder.
And then you're if you could see that this is almost one gigabyte model so you could view this here
you see 984 just just check as I hover on this.
You see here 984.39 megabyte model.
And so this means that if you are to deploy this in a real world scenario, then you will need to always
allocate this amount of space in order to run this model.
Now, let's move on to converting this model into the Onyx format.
We shall have this two installs.
So we start by installing the TensorFlow to Onyx Tool and then we'll install the Onyx runtime.
So we run this and then now we'll go ahead to convert our model, which is this one year with fine tuned.
So we just have to specify your white fine tuned into the Onyx format.
So here we just say white fine tuned and then now we have this model or give it the name of our onyx
file.
Who called us white Onyx.
That's it.
Or with Onyx.
TF to Onyx convert.
And that should be it.
So now we'll run this then, while waiting for that run to be complete, you could get to the Onyx GitHub
repo TensorFlow onyx, and then you'll have the documentation for how to convert from TensorFlow to
Onyx.
So let's get back to this still running because see, we have a series of warnings, then specifies
the TensorFlow one, TensorFlow Onyx and TensorFlow two Onyx versions, then the offset, then it's
optimizing.
And now we have this successful conversion.
So on this model is saved, as with Onyx and Onyx.
Now let's open this up.
You see, with Onyx or Onyx, you see you.
We've moved from this.
Let's open this again.
We've moved from this year where we had 984.39 megabytes to this optimized Onyx version, which is at
327 megabytes.
Now, another option will be to convert the model from this class format to the Onyx format.
So this first one, we saved this as a TensorFlow saved model, and then now we could just have this
year.
So let's say we have the model.
We saved this as a class model.
So we have the H five and we run that checking here.
You see we have this still 984.9, eight megabytes, close to one gigabytes.
And then from this Keras format, we are going to now convert this to Onyx.
Now you're we're going to have this specification and then we'll pass in the image size.
So here we have the batch by 256, by two, 56 by three, it's flow 32 and it's our input.
Then we could specify this output path.
So this one was with Onyx.
Let's, let's, let's have this as, let's, let's call this white white cross dot onyx.
Okay.
So this is going to be our output path here.
And then we will have this TensorFlow two Onyx here, which contains this from Keras method, which
takes in our model hog interface model, and then the specifications which we just mentioned right here.
So your specifications right here, we pass this in.
This passes as our input signature.
We have this offset value and then we also have our output path, which we have already specified here.
Then we have our output names, which you shall get automatically.
So that's it.
Let's run this while that's running.
Also, note that you could check out this conversions on the Onyx runtime platform where you can get
the details of all what we were doing.
So let's get back here.
Still still running and that's it complete.
Let's check out our accuracy 30 327.59 megabytes.
Then from here, we move on to the inference where we'll see whether what we get from the Onyx model
coincides with the initial Keras or TensorFlow formats.
So right here we have this provider.
Now, you always specify this provider to be this CPU execution provider as we'll be running this on
the CPU.
And then we have this Onyx runtime as our TX, which we are imported right here and we should we make
enough making use of your.
So when we want to run an Onyx model, you see the first thing you have to notice we do not need TensorFlow
anymore.
So even if we restart this whole process, all we need to do will be just to install the onyx runtime
and then import this this way.
So now we have our Onyx runtime.
We, we, we have this inference session which we create by just specifying this path, our output path
here.
Our output path is the path to this model, the Onyx model, and then the provider is this one year.
So we just need to specify this path and the provider and we're good to go.
So now we have this.
The next thing we want to do is to run the inference.
So the Onyx prediction is this.
MB that run and then now we'll specify the output names.
Now this output names is going from here.
So let's run this, let's print out output names so you see what it contains.
That's it.
You see, we have that dense.
And if we get back to when we're creating this model, let's get back to this.
Okay.
You see, the name here we have is dense.
So if you specify the model name to be different from this, you would have a different output name.
So that's it.
Let's get back to our Onyx inference.
We've converted already and that's it.
So.
So that's it.
We have our output names, which we created.
All right.
I wish we generated automatically from here.
Does it?
And then obviously it's a list because we could have several outputs.
And then from here we pass in our input image.
Now this our input image is simply what we've been having already.
So we just we could copy this, copy this, and then test this out here with our Onyx model.
Let's have this here and this code paces out.
All we need here is just this basically.
So we we could run this.
Let's run this.
That's it.
We have our image, and then let's run this and let's run this.
And then we print out the onyx print so we get in input.
Must be a list of dictionaries or a single non pi array.
So what we're going to do here is instead of TensorFlow use numpy.
So you see already that we do not really need TensorFlow.
Like even with this we could get the the test image from here.
Let's, let's get this test image directly so we can have test image and that's it.
We run that we have our image does not define our test image.
Let's run that again.
That's fine.
We get this other error because this input isn't a float.
So here instead of this we're going to have test image, dot se type and p dot float 32.
Okay, so we have that set.
Now let's run this again.
No, before doing this, let's make sure we pass in this image instead here.
So we have that.
We run that.
That should be fine.
Okay, so let's rerun this.
Again.
And you see, we have now our onyx spread.
So there we go.
You see, it shows us that it's a sad image because this is angry, happy, sad.
So it's sad because the sad.
Has the highest probability.
And you see, the image here is sad.
Now, if we get back to the top here and simply run this the same image, let's run this.
Oh, let's get the probabilities, actually.
So let's let's let's say we want to have let's print out the model image.
Let's get this probabilities.
You see, this is what this model gives us is output.
And here is what we get from the Onyx model right here.
Now, before we continue, another thing we'd like to check out is the speed or the time or the latency
of the model.
So let's get back up here and we could add this code sell.
Let's import time or we could just do it below.
So let's yeah, let's do, let's just do that below.
So let's take this off and then your we could say we want to have less import time, There we go.
And then we have DT one which is time that time.
And then we do have model taking an input image.
Then we record the time, the current time minus the T one, so we could get the time which is elapsed
after running this model.
You see 0.14, that's a 0.14.
And you should know that here we suppose that we run in this on a GPU.
So you check manage sessions, we run this on a GPU.
Now for the Onyx model, you could have your T one time that time and then we could print out the the
difference in time.
So we have that minus RT one.
Let's run this and see what we get.
Input list single non pi.
Oh, let's run this again.
Okay.
So you see with yeah, we're making use of the CPU and we get in 0.27 9 seconds while with the GPU here
we get a 0.14 3 seconds.
Now we can be comparing the Onyx runtime results on the CPU with data of this or hungry phase model
with TensorFlow on a GPU.
You could you could even see from here.
If you do get device, you'll see from here that what the onyx runtime is using is the CPU.
And so if want to make use of the GPU, we would have to install on x runtime GPU version so we could
compare the two models.
So for now, just note that TensorFlow with the GPU is 0.150.1 5 seconds and then we will need to get
TensorFlow with a CPU.
And then we'll also get we got onyx with the CPU, onyx with a CPU give this is 0.3 seconds, give about
0.3 or let's run this again.
We run it again.
We should get from this again.
I'll get in an error.
Now that was because I had moved this file into the drive.
So here we have the crash here and we run this again.
We have about 0.38 C that's it about okay, let's say 0.5.
So let's let's say this is 0.5 here and then we need to get on the GPU and then we also need to get
a tensorflow with the CPU.
So that's it.
What we'll do now is we'll go ahead and install the Onyx GPU.
So yeah, we have PIP install on X runtime CPU and that should be it.
We already installed this OC, so we have this Onyx runtime GPU.
And then if you if you, if you do import on X runtime as our PT and then you as our PT and then you'll
see our PT dot get device, we are getting the CPU.
So what we'll do now is we are going to restart this runtime, so basically restart this runtime so
that this, the GPU version of Onyx will be taken into consideration.
Now this time around we're going to install on X runtime with the GPU.
And as it as you can see now, the device we're using is a GPU.
Okay.
So we have that.
Let's now get back here.
We already have our model in the Onyx format, so we just run this.
And then if you run this, you'll see that since we we specify the provider to the CPU, we shouldn't
have any much difference here.
So let's run this again.
We import a time output names not defined.
Our output names is basically this.
So let's have this list and let's define this here.
So let's have output names.
Output names.
There we go.
Output names dense.
So let's run that again.
That's fine.
And then we get back here.
But note that this output names were generated from yours so you could get back to start generating
this.
Or you could just make yourself the output names.
As you can see, since we started this runtime, we do not have those files anymore apart from the Onyx
which we stored in the drive.
So if we have to do this, we need to retrain our model since we lost them already.
So we have our output names here which was specified, and then we get back here.
We run this again and check out the time it takes to run the model.
You see 0.34.
Not bad.
Uh, 0.34 OC This is 0.34.
So you're normally this should be zero point.
Yeah, that's a 35 OC.
So the TF with GPU 0.15 on X with CPU 0.35.
Now the way we're going to use this is we'll get here.
You see in this documentation you have the provider, you specify the Cuda execution provider, so let's
copy this and then paste it out here.
So instead of instead of F, for example, instead of CPU, now we have the GPU, so we could make use
of the coder execution provider.
So here we'll take this off and then paste this out.
So this all provider now.
Note that since this providers can be a list, you could have several providers like here's when we
do this, when we have CUDA execution provider before the CPU execution provider, it means that the
priority goes to the CUDA execution provider.
And so in the case where we are having a CPU, then we start with this one.
And given that this cannot work in the situation of a CPU, we will then move on to this.
But if we have a GPU then directly we will use this scooter execution provider and that's it.
You see 0.04 8 seconds.
All I say is 0.5 seconds, which is three times less than what we had when running our model or when
running the TensorFlow model.
So you see already that the Onyx framework permits us to optimize our initial TensorFlow model.
So here we have on next GPU.
Now this is 0.05, so it takes us now 50 milliseconds to run this hugging face model.
So that's it.
Now, before we proceed, it's also important to note that generally the way we measure this time it
takes for the model to predict an output.
We usually can test with several input images, or we could repeat this process several times.
So let's say four underscore in range.
Let's say ten, we run this and then we get the time elapsed instead of just testing.
Once, we could test ten times so we could get the average.
So let's run that.
And you see dividing this by ten, we have 0.035.
So that's 35 milliseconds per prediction.
So this means that for us to have 100 predictions, we would take three point 5 milliseconds.
We could test that out.
Let's run this.
And there we go.
We take 2.3 seconds for 100 predictions.
Now, if we divide, let's let's say let's call this number of predictions.
We set this to be 100 and then the time taken.
This is a time time for a single prediction.
There we go.
Time for a single prediction.
All this divided by the number of predictions.
So we get the average.
That's fine.
So let's run that again.
So to get this time.
Okay, so we see 0.02 3 seconds.
That is 23 milliseconds.
Why we repeat the same for the hogging face model.
You see, it takes, uh, 0.1 5 seconds, just as we had already.
Now, you could see that the the difference in speed here.
0.15 divided by .025.
You see, the Onyx model is six times faster than the TensorFlow model.
Now what if we get back and run the TensorFlow model on the CPU?
So change this runtime known and we save that.
So we'll have to rerun all this again.
Now, running our hog interface model, your you see this takes 0.8 seconds, 0.8.
So with the with the, with the CPU, we have 0.8 divided by 0.35.
The Onyx model runs about twice as fast as the TensorFlow model.