Implement our agent.
This is going to be a very long and intensive coding video.
So I'm not going to write each line of code throughout this video.
I'm going to use an existing piece of code that I have, but I will definitely walk you through the
code line by line, and then we'll execute the code on the Google Colab notebook.
Having said that, I have a 400 as the GPU compute with 83 gig memory Ram and 235 disk space.
It supports Python three GPUs.
And now I'll go ahead and first install the haystack AI.
So pip install haystack AI.
And I'll also do pip install fast drag.
It's done.
So now I will go ahead and start the first step which is loading the nutritional data dot JSON.
So here is the nutritional data dot JSON.
We first index our data and for that we need this JSON file I have uploaded the file here in the in
the notebook.
Let's quickly go through the structure of the nutrition data JSON file.
So here is the JSON file.
It has six entries here, with each entry having three fields to it.
Image URL which is where the image with the necessary information is stored.
There is a title and there's a content.
Let me share with you one of the images here with the protein bar nutritional fact and see how this
looks like.
So here is the media Amazon file which has the nutritional facts about the protein bar calories, cholesterol,
carbohydrates and all the details with protein and everything.
So now let's go back to the notebook.
So next we'll have to index the documents to in-memory document store.
So here is our in-memory document store.
And then here we'll use the sentence Transformers all mini six v2 model to create the embedding for
each label description.
And then we'll create a pipeline to index our data to in-memory document store.
So here we have put a pipeline called index pipeline.
And then we add those components the sentence transformer model call it document Embedder.
And then here we have another component documents writer.
We'll name it as Doc Writer.
Then we'll connect the document Embedder documents to document writer documents.
Let's execute the code block.
So next we create document objects with the nutrition label content as the content and store the title
and image URL as metadata before passing them to indexing pipeline for processing.
So here for this indexing pipeline, we have documents with content as the root as a root element.
And then metadata field.
Here is title and image library.
Let's run this.
So we got six documents returned to the in-memory database.
Next, we'll build a retrieval pipeline.
We create a document retrieval pipeline for the documents above.
We'll later use this pipeline in our tool.
This pipeline will consist of sentence transformer text embedder in memory and better retrieval to fetch
top one content, and multimodal prompt builder to construct the prompt that are able to use.
So here I have imported the necessary packages.
Here is the prompt template with image and the content that we get from the document.
Then we have sentence transformer text Embedder component to embed our questions.
Uses all the mini LLM model.
Then we have a in-memory retriever component to fetch the top one document.
And then we have multimodal prompt in prompt Builder to construct the prompt that our agent will use,
and then we'll connect Embedder to retriever.
And the retriever will, once it got all the retriever documents, it will embed it to the prompt builder
with the document here.
And then we'll run this pipeline and print it.
Let's see how this works.
So it ran.
We have prompt builder with the prompt as image.
So in this pipeline the multimodal prompt builder component receives one document object one document
object here from the retriever and renders the prompt.
Notice here we have image placeholder in the prompt template for our model in order to inject the images
into this later on.
Additionally, multimodal prompt builder converts the Keyman image into base 64 screen for image to
be processed by multimodal agent.
So now we have created the retrieval pipeline.
The next phase is creating the multi-modal react agent.
For that, we'll have to first define a tool with our retrieval pipeline ready.
We can create our tool using Doc image haystack query tool component from fast drag.
So here from the fast drag we are importing doc with image Haystack query tool.
Very important to understand that this tool has three important attributes or inputs which is name that
is nutrition tool description and the pipeline or YML file.
So the name and a description of this tool is used by our agent to decide when to use it.
We will provide the agent with our retrieval pipeline when we want to invoke it.
Let's run this with the defining tool.
And now with this tool ready, I'll run this tool with a protein bar as input and print the result.
There you go.
So here we got image tag.
And then this image shows a protein bar with chocolate peanut butter.
Nutrition facts per bag as 50g.
This is what we got from the tool which matches the nutrition data content here.
Protein bar.
Chocolate.
Peanut butter.
Nutrition.
Fats per bar 50g.
So the next step is to initialize the generator.
So now here as you notice we have our multimodal agent.
We have to initialize 535 vision h f generator.
This model processes both text and prompt and base64 encoded images.
This makes it well suited for image text like visual question answering.
The vision HRF generator uses hugging face Image Protect model, which will function as an LLM for our
agent.
So this is the model name model Phi 3.5 vision instruct.
And here's the stop by word text criteria.
And then we have initialized the generator by passing the model name or path along with it.
We also say what task it is for image to text is maximum, token is 100.
Stopping criteria is here the observations stopping word and end.
It's very important to understand and learn this.
And we have defined words as observation and end as stop words.
These stop words are specific to model and react prompting.
So now we pass other more specific parameters.
Storage trust remote code is true.
And then attention implementation is eager and device map as auto.
And then we'll warm up this particular model which will help us download this from the hugging face.
You can go look up this model on the hugging face, which would have all the model details on the model
card.
So it is now downloading this model.
It takes a while.
I'll pause the video.