So far, we have, whenever we talk to
the LLM, just had some hard-coded strings,
like what is the capital in France, and
so on.
So, let's change that now by building a
chat loop.
So, in this sample, the chat loop sample,
we are just making our connection as we've
seen before.
We are making our agent as we've seen
before.
But now we are introducing a while true,
where we just ask different questions.
And we make a little command prompt and
ask for input from the user.
We then use streaming and take that input
instead of a hard-coded value.
We're using the option to get our updates
back from streaming.
So, we can write out the tokens in
and tokens out.
We could also use the run async instead
and not needing to have the streaming part.
But let's see what happens when we run
this.
If we say hello, it will come back
on helping one to assist us.
And it takes eight input tokens and ten
output tokens.
And if we say hello again, exactly the
same happens.
And we can do this over and over
and we'll roughly use the same.
There could be variations over time, but for
now, it's happy.
But what happens if I tell it my
name?
My name is Rasmus.
It will happily greet me, but we would
now think it would know my name.
But the second I ask it, what is
my name?
It doesn't know.
It doesn't have access to my personal information,
it feels like.
Why is that?
Well, this is because this call is just
a fire and forget.
We give a question, get an answer, and
the next time we run it again, it's
a completely new conversation for the LLM.
It has no clue that we asked something
in the past and what we have talked
about so far.
In order to do something about that, we
need something called an agent thread.
So if we go to agent and say
get new thread, we can get one of
these threads.
The thread will internally have a message store
where it saves all the messages.
Let's just call it thread here.
And all we need to do is we
need to tell whenever we run this now
that the thread is involved.
And what will happen is it will add
more and more to the thread so it
knows a conversation.
Just like if we were writing down a
conversation between each other, we could look back
at what was the first thing we talked
about.
So if you run this and now say
hello, it will still come back with how
can I assist you.
But if we say hello again, it will
say hello again.
And that's fine, but also we see that
our input tokens was not just the 8
and 10 like normal.
And if we say it again, this will
now begin to grow.
And this is because all the messages are
in this conversation now because it needs to
know what we talked about before.
So if I say my name is Rasmus
and then ask what is my name, we've
got the conversation now that it knows but
at the cost of spending more tokens.
That is the reason why in ChatGPT and
if we built this out as a chatbot,
we would need to have a new button,
a new chat, because not only are we
spending more tokens again and again as we
ask, but it will also slowly become slower
and slower because it needs to process all
the existing messages before it can answer the
latest one.
So let's try and see behind the scenes
what's actually happening because we can get these
messages out.
So let's do that down here in that
we can put in a thread.getService of
iList of chat messages.
It's a little hidden that we can do
this, but we can get the messages back.
So if we run this, and by the
way, now that if I ask what is
my name, of course it doesn't know because
before it was in memory.
So it's not like forever it knows now,
but what we get back is we can
see we get two messages.
So the first message was from the user,
me, what is my name, and it said,
sorry, I don't know your name.
If we ask again, hello, my name is
Rasmus, we now get four messages.
So whenever we do this right now with
a simple setup like this, we always get
two messages per conversation switch back and forth.
If we began at some point to introduce
tools, there would be more.
There will also be messages from the tools
getting the information from the tools back.
But here we'll just get more and more
of these, and since all of these are
sent to the LLM at every point, so
it can work with the conversation, it will
over time build up.
And the normal simple and often used scenario
is, for example, who is Barack
Obama?
And it'll come back with that and say,
how tall is he?
And it can infer from the first message
that what we're talking about.
But again, at the cost of using extra
tokens.
255 is not a huge amount, so we
can manage.
But, of course, also what we need to
remember is that this is only in memory.
So if we went back and then said,
where does he live?
Because in our mind, the last question was,
who is Barack Obama?
How tall is he?
Then it should also know where he lives.
But, of course, it won't know because now
we are back to zero in the token.
It's like throwing away or forgetting, having short
-term memory loss.
This can be fixed, and we will have
dedicated lectures on that as well, persisting the
memory between sessions.
But for now, what we have is good
enough for us to continue with the chat
in we have now a chat loop to
work with.