WEBVTT 00:00.110 --> 00:00.680 Hey there. 00:00.710 --> 00:01.370 Eden here. 00:01.370 --> 00:08.270 And in this video, I'm going to cover some of the important concerns when dealing with managed large 00:08.270 --> 00:09.320 language models. 00:09.320 --> 00:16.190 And I want to cover here some key points about data retention and privacy. 00:16.190 --> 00:21.410 And I think those are very important issues that we should be aware of at least. 00:21.440 --> 00:28.070 And when taking a generative AI application to production, which uses large language models, then 00:28.070 --> 00:35.510 you have to answer questions like is the data being used for training purposes, or how long the data 00:35.510 --> 00:42.260 is being retained for and for which purposes, copyrights issues, and how can you use the generated 00:42.260 --> 00:48.380 text, and a lot of more things that you need to be aware of and you need to answer. 00:48.410 --> 00:55.190 Now, this is a huge, huge topic and I can go for hours to talk about it. 00:55.190 --> 01:01.670 And there are tons of requirements and rules and regulations, etc. in this video I'm going to go only 01:01.700 --> 01:07.610 on a few of them, so this is definitely not the full list of things you should be aware of, but just 01:07.610 --> 01:13.280 the beginning and something to help you get started and to give you an introduction to it. 01:14.270 --> 01:20.780 And before we begin, just a very important disclaimer, and this disclaimer is super important. 01:20.780 --> 01:21.860 I am not a lawyer. 01:21.890 --> 01:29.780 This is not a legal advice, and you should consult with your legal team and privacy team before integrating 01:29.780 --> 01:33.230 any LLM based solution in your enterprise. 01:33.260 --> 01:40.040 There are a lot of rules and a lot of regulations that I am not aware of, and the topic of data retention 01:40.040 --> 01:45.680 and privacy is very, very sensitive and should have appropriate handling. 01:45.680 --> 01:51.260 I am also not representing any LLM vendor here, and I'm not giving legal advice. 01:51.260 --> 01:58.220 And every LLM vendor is going to have a Eula, an end user license agreement where they have their terms 01:58.220 --> 02:02.240 of services and they specify how they handle your data. 02:02.240 --> 02:05.990 And it's a legal document that you should look into. 02:06.020 --> 02:06.440 Okay. 02:06.470 --> 02:08.510 I'm just giving you my $0.02 here. 02:08.510 --> 02:11.950 And again, I'm not a lawyer and this is not a legal advice. 02:11.950 --> 02:14.530 So this is a very important disclaimer. 02:14.560 --> 02:21.130 You should always talk to your legal team and privacy team and to act according to what they say. 02:21.310 --> 02:26.440 This video is for educational purposes, and I'm going to give you my $0.02 on this topic. 02:26.440 --> 02:30.220 So you should take everything I say in this video with a grain of salt. 02:30.220 --> 02:34.180 And you should really do your own research when it comes to this topic. 02:35.740 --> 02:42.820 Let's start by talking about data retention and what happens to our data that we send to the model to 02:42.850 --> 02:50.140 a managed model like OpenAI, GPT four or Mini, or Google Cloud's Vertex AI Gemini. 02:50.140 --> 02:53.710 And I want to make a very clear distinction over here. 02:53.710 --> 03:02.260 I'm not talking about the B2C direct to consumer products like ChatGPT or Gemini, which was formerly 03:02.260 --> 03:09.670 known as Bard, but I'm talking about the cloud APIs that they expose for enterprises and businesses. 03:09.670 --> 03:12.370 And again, I want to reiterate, I'm not a lawyer. 03:12.400 --> 03:14.590 This is not a legal advice. 03:14.590 --> 03:16.480 And you should do your own research. 03:16.480 --> 03:20.020 And if you're working for an enterprise, you should consult with your legal team. 03:20.020 --> 03:27.460 All managed vendors have a Eula and users license agreement, and you can see all of the details there. 03:27.460 --> 03:31.720 And this is a legal document which you can read and it has all the information. 03:31.720 --> 03:35.080 But I'm going to give you my $0.02 in this topic. 03:35.080 --> 03:39.010 And a lot of people are concerned what happens with their data. 03:39.010 --> 03:46.060 And they're worried that the managed data vendor are going to train their next model on data that they 03:46.060 --> 03:49.510 will be sending to the model or the generated output. 03:49.510 --> 03:56.710 And from what I saw, in most cases, at least in the top tier models, then there is a guarantee that 03:56.710 --> 04:02.980 they're not going to use the data that we send or the generated text to training purposes. 04:02.980 --> 04:08.200 So we get a guarantee that they're not going to use their data for the model training. 04:08.500 --> 04:10.510 At least that's the default behavior. 04:10.510 --> 04:15.760 If we do want to allow them to do this, we can opt in voluntarily for that. 04:15.760 --> 04:22.110 And by the way, this is a very valid concern because if we're working for an enterprise and we want 04:22.110 --> 04:28.650 to integrate an LLM based solution, we need to make sure this is the case, because we might be dealing 04:28.650 --> 04:35.700 with proprietary data of our organization that we do not want to expose, and we do not want to get 04:35.730 --> 04:38.910 leaked or to get trained on, God forbid. 04:38.910 --> 04:45.570 And if we have customers, we of course, do not want their data to be trained on, because probably 04:45.600 --> 04:49.470 we're going to have legal obligations that is going to protect their data. 04:49.470 --> 04:52.290 So this is the first thing I want to address. 04:52.320 --> 04:52.890 All right. 04:52.890 --> 04:56.580 So this was about training on data that we send to the LLM. 04:56.760 --> 05:00.420 Of course you should expect differences between different LLM vendors. 05:00.420 --> 05:02.640 And let's talk about data retention. 05:02.640 --> 05:07.560 And our vendors going to save the data that we send the LLM. 05:07.560 --> 05:09.510 And if so for how long. 05:09.510 --> 05:13.590 What kind of rules do they have for which purposes, etc.. 05:13.980 --> 05:22.230 And in this example of OpenAI, we can see that it clearly says that in order to identify abuse, they 05:22.230 --> 05:27.840 may retain our requests for 30 days and after it it will be deleted. 05:27.840 --> 05:30.720 Or if there are other law requirements. 05:30.720 --> 05:38.880 And they do mention that for some customers they may have a zero retention policy, which is available 05:38.880 --> 05:45.750 where none of the data will be logged or persisted, and it's going to be only used for serving purposes. 05:45.780 --> 05:48.390 And of course, this may also change between vendors. 05:48.390 --> 05:54.900 I did see other vendors where they have a zero retention policy right from the get go, and in order 05:54.900 --> 05:59.610 to log and to retain some of it, you would need to explicitly opt in. 05:59.610 --> 06:04.830 So again, there are differences between the vendors and those may change over time. 06:04.830 --> 06:05.940 All of the rules. 06:06.600 --> 06:08.280 And one last disclaimer. 06:08.280 --> 06:16.410 So even when providers are going to guarantee that they're not going to train their model with our data 06:16.410 --> 06:19.620 and we're going to have zero retention policies. 06:19.650 --> 06:23.070 And for some organizations, this is simply not enough. 06:23.070 --> 06:29.310 Let's take the banking institutions, for example, or insurance companies, they usually have very 06:29.310 --> 06:36.150 strict regulations and very strict policies when it comes to privacy and when it comes to data retention 06:36.150 --> 06:40.680 and sharing customers data, because it is very, very sensitive. 06:40.680 --> 06:48.900 And for those companies, those promises and those guarantees from the LM vendors is not enough. 06:48.900 --> 06:57.030 And usually if they do want to integrate generative AI into their applications, they are going to self 06:57.030 --> 06:58.980 deploy open source models. 06:58.980 --> 07:06.060 And with self deploying open source models in their environment, they will have control over their 07:06.060 --> 07:11.640 data, their retention policies, and they have all the control they need. 07:11.670 --> 07:19.410 However, it will come with a cost because serving LMS is not that simple and they would need to handle 07:19.410 --> 07:27.270 scalability, durability, availability, all those ilities, which is a lot of operations work and 07:27.270 --> 07:30.050 it's not simple to do correctly. 07:30.080 --> 07:37.160 It also costs a lot of money and a lot of effort because they would need to host it on GPUs, and they 07:37.160 --> 07:39.020 would need people to maintain it and to. 07:39.050 --> 07:40.190 Handle the deployment. 07:40.190 --> 07:43.010 And of course they would need to handle the security. 07:43.040 --> 07:46.880 Because even open source models can have vulnerabilities in them, which I. 07:46.910 --> 07:48.290 Didn't discuss at all. 07:48.290 --> 07:50.780 And it comes with a price as well. 07:50.810 --> 07:51.710 And there is also. 07:51.740 --> 07:57.680 This middle ground of hosting the LMS, the open source LMS within their cloud environments. 07:57.680 --> 08:01.520 So they're going to use the managed services of cloud providers to. 08:01.550 --> 08:03.410 Deploy those open source LMS. 08:03.410 --> 08:07.070 So they shift the burden of all those edits to. 08:07.100 --> 08:10.640 The cloud provider, they still have control because it's going to be their. 08:10.670 --> 08:11.570 Cloud environment. 08:11.570 --> 08:14.150 And they can enforce their security controls in. 08:14.180 --> 08:15.470 Their cloud environment. 08:18.380 --> 08:21.470 And just to conclude in this video, I just wanted to. 08:21.500 --> 08:27.140 Introduce you with a couple of things that you need to consider when it comes to privacy and data retention 08:27.140 --> 08:29.780 when using large language models. 08:30.410 --> 08:34.730 This is a very deep topic and we can talk about it for hours. 08:34.730 --> 08:39.980 And my goal here in this video is just to give you an introduction and some things to concern.