WEBVTT

00:04.240 --> 00:08.560
End to end testing tests, in our case, the entire ELT route.

00:08.560 --> 00:14.360
From the moment we extract the data from the API to where we are applying the data quality checks to

00:14.360 --> 00:18.960
do an end to end test for a Dag, you can use the airflow Dag test command.

00:20.320 --> 00:26.960
If we had to use the dash dash option for this command, we would see that this will execute one single

00:26.960 --> 00:27.600
diagram.

00:28.040 --> 00:36.560
All you need to specify is the dag id, so we can run the command using one of the Dag IDs that we have.

00:36.600 --> 00:41.720
The first one being the produce underscore JSON and the python following the execution.

00:41.720 --> 00:44.200
Here we can also see it the airflow UI.

00:45.880 --> 00:47.520
So let's first sign in.

00:51.320 --> 00:53.520
Let's hit the execute button from here.

00:56.880 --> 00:59.720
So you are seeing that this Dag is actually running.

01:00.280 --> 01:02.720
You can also see it from here if we refresh.

01:10.920 --> 01:14.440
So it seems from here the Dag has finished running.

01:14.480 --> 01:15.600
Let's go back to the UI.

01:17.520 --> 01:20.800
And as we expect, the Dag has been successfully run.

01:21.600 --> 01:26.290
What this means is that we will have a new JSON file in the data directory.

01:26.330 --> 01:30.210
You can do the same with the other two DAGs just by replacing the dag id.

01:31.330 --> 01:37.530
So the first one, or should I say the second Dag is the update db.

01:47.450 --> 01:49.610
Here it seems that everything has run successfully.

01:49.770 --> 01:51.930
Let's confirm by going on the UI.

01:55.650 --> 02:00.490
And in fact, it did successfully run for both the staging and the core layer.

02:02.770 --> 02:05.330
And finally we can run the data quality checks.

02:21.050 --> 02:24.850
So I guess we have successfully run all the DAGs that we have created.

02:24.890 --> 02:30.290
A question that you might have is would it be ideal if all this testing we did in this section could

02:30.290 --> 02:32.050
be automated and triggered?

02:32.050 --> 02:38.330
Once we do a change in our code base, this process of automatic testing should be included in a continuous

02:38.330 --> 02:42.890
integration and continuous deployment workflow or for short CI, CD.

02:43.090 --> 02:46.650
And this will be our focus in our last section of this course.