‫Welcome back.
‫In this video and the next one
‫I want to introduce you to the MongoDB
‫aggregation pipeline which is an extremely powerful
‫and extremely useful MongoDB framework
‫for data aggregation.
‫And the idea is that we basically
‫define a pipeline that all documents
‫from a certain collection go through
‫where they are processed step by step
‫in order to transform them into aggregated results.
‫For example, we can use the aggregation pipeline
‫in order to calculate averages
‫or calculating minimum and maximum values
‫or we can calculate distances even,
‫and we can really do all kinds of stuff.
‫It's really amazing how powerful
‫this aggregation pipeline is.
‫All right, but enough talk.
‫Let's now actually start using the aggregation pipeline
‫and I'm gonna create a new header function here
‫because later on we will then define a new route
‫and then use that function for that.
‫But that new route, I'm gonna create it later.
‫So I want to create a function here that's gonna
‫calculate a couple of statistics about our tours.
‫So I'm gonna call this one getTourStats.
‫Okay, so Stats for statistics,
‫and it's gonna be an async function
‫and so a vs code now automatically prefilled off this.
‫All right, let's create or try catch block here.
‫Actually, I'm gonna copy this from right up here,
‫and now in here I will create
‫a variable called stats.
‫Now the aggregation pipeline really is a MongoDB feature.
‫But Mongoose, of course, gives us access to it,
‫so that we can use it in the Mongoose driver, right?
‫So using our tour model in order to access
‫the tour collection, we say
‫tour.aggregate.
‫And so the aggregation pipeline is a bit
‫like a regular query and so using the aggregation pipeline
‫it's a just a bit like doing a regular query.
‫The difference here is that in aggregations,
‫as I already mentioned, we can manipulate the data
‫in a couple of different steps
‫and so let's now actually define these steps.
‫And for that, we pass in an array of so-called stages.
‫So we pass in an array, and then here we will
‫then have a lot of stages.
‫And again the documents then pass through these stages
‫one by one, step by step in the define sequence
‫as we define it here.
‫So each of the elements in this array
‫will be one of the stages.
‫And there are a ton of different stages
‫that we can choose from, but I will just tell you
‫the most common ones in this lecture
‫and also in the next one.
‫But anyway, I now want to take a minute
‫and take a quick look at the MongoDB documentation
‫because I think we didn't do that yet.
‫So MongoDB and we already know this website.
‫So on our MongoDB website we go here to learn,
‫then documentation and then in here we select
‫the MongoDB Manual and this documentation is huge.
‫It really, it has a lot of stuff in here
‫and if you were gonna go through it all,
‫well, then you'd become really a MongoDB master.
‫So it gives you a nice introduction to MongoDB,
‫you have the installation here.
‫You have the CRUD Operations, so a nice overview
‫about a lot of the stuff
‫that we already learned before, right?
‫Then you have stuff about the aggregation
‫and in particular, the aggregation pipeline
‫which we're gonna talk about now,
‫but what I want to really show you here
‫is actually this reference that we have at the bottom
‫because in the reference we have here for example
‫the operators and that's where we have
‫a lot of important stuff that we've been using all the time.
‫So let's take a look here at the query operators,
‫for example and here you see, we actually have
‫these operators that we already used.
‫So for example, greater than or equal,
‫or less than and equal, but if someday
‫in your own applications you need some other operator,
‫well, now you know where you can find them.
‫Now of course, it's also always very helpful
‫to just do a quick Google search, probably find
‫some stack overflow post about it,
‫but believe me, many times it's easier
‫and faster to just quickly jump into the documentation
‫and quickly search for whatever you're needing.
‫You just need to know how the documentation works
‫and once you know that, it's then gonna be quite easy
‫for you to find what you're looking for, all right?
‫Here you also have some logical operators
‫for example, the OR operator we already used.
‫Again, really a lot of stuff, geospatial operators
‫and we're gonna use some actually, later,
‫and yeah, really a lot of stuff.
‫Then what I wanted to show you after this lecture actually,
‫are the aggregation pipeline stages.
‫So here you have a lot of them
‫and what we're gonna use in this lecture here is match,
‫for example and group,
‫where is that?
‫Yeah group, that is a very important one,
‫and so as I mentioned, I'm gonna show you
‫the most important ones in this lecture and the next one.
‫But again, if you in some case need something else,
‫then just go ahead, come here to this documentation
‫and you will find what you're looking for.
‫All right?
‫Anyway, let's now learn how we can actually define
‫one of these stages, all right?
‫And I'm gonna start with match.
‫And match is basically to select
‫or to filter certain documents.
‫And so it's very simple.
‫It's really just like a filter object in MongoDB,
‫such like we've been using so many times.
‫So each of the stages is an object
‫and then here comes the name of the stage.
‫So this one is the match stage, all right?
‫And as I mentioned, it's really just a query.
‫And so let's say that for starters,
‫we only want to select documents
‫which have a ratings average greater or equal than 4.5.
‫So can you do that?
‫I'm just gonna give you a second to do that here.
‫So I hope you did it.
‫So ratings average, should be
‫greater or equal than 4.5.
‫And so that's it, that's the first stage.
‫And usually this match stage is just a preliminary stage
‫to then prepare for the next stages which come ahead.
‫So the next one is now the group stage.
‫So group and then in there
‫we need to always pass just another object.
‫So it looks kinda weird with all these objects,
‫but, yeah, you've seen that before
‫and MongoDB just works this way.
‫It's always objects, inside of objects, inside of objects.
‫And this group here is where the real magic happens
‫because as the name says,
‫it allows us to group documents together,
‫basically using accumulators.
‫And an accumulator is for example,
‫even calculating an average.
‫So if we have five tours, each of them has a rating,
‫we can then calculate the average rating using group.
‫And so let's do exactly that right here.
‫Now the first thing, is we always need to specify
‫is the id because this is where we're gonna specify
‫what we want to group by.
‫For now, we say null here because we want
‫to have everything in one group so that we can calculate
‫the statistics for all of the tours together
‫and not separate it by groups.
‫We will, a bit later then also group by different stuff,
‫for example we can group by the difficulty
‫and we can then, for example calculate the average
‫for the easy tours, the average for the medium tours
‫and the average for the difficult tours.
‫So again, we can group by one of our fields
‫and that field, we are gonna specify in here,
‫but for now, as I said, we want to calculate these averages
‫for all the tours together in one big group.
‫So in that case we say _id and set it to null.
‫Now let's actually calculate the average rating.
‫In order to do that, we simply specify a new field,
‫so let's simply call it the average rating,
‫so like this, and this will be
‫well, the average,
‫which is yet another MongoDB operator, so this one here.
‫You will find it in the reference if you look it up.
‫So this is a mathematical operator calculating
‫the average and now the name of the field.
‫And again, I know that this is gonna look very weird,
‫but in order to specify the field which we want
‫to calculate the average from, we need to use
‫the dollar sign, but in quotes here
‫and then the name of the field.
‫So ratings average in this case.
‫And let's also calculate the average price.
‫So average price, and again, did it wrong here,
‫and so
‫the average and then again,
‫the name of the field and in quotes
‫and with this dollar sign.
‫Right, so it looks kind of weird again,
‫but this is just the way
‫that the aggregation pipeline works.
‫So we have the average price.
‫Let's actually also calculate the minimum price,
‫so the smallest price, and the largest price.
‫So minPrice
‫and from now on, it's gonna all look kind of similar,
‫so now we use the min operator
‫and then again the field name like this.
‫And a comma here and let's now just duplicate this one
‫in order to calculate the maximum price.
‫And now, let's actually take a look
‫at what we already got at this point.
‫So we now need to actually send out this response,
‫and so again, I'm just gonna go head
‫and grab some of these already done responses here.
‫So the data that we want to send out is called stats.
‫And now all we need to do is actually add a new route
‫here in our tour routes and again,
‫I'm gonna edit here, right at the top.
‫So router.route
‫/gettour, or actually just tour-stats, just like this
‫because the get is already defined in the HTTP method.
‫So no need to repeat it in the name.
‫So tourController.getTourStats.
‫Give it a safe and let's actually test it out now.
‫So I'm really exciting to test it out now, actually
‫to see if it works because again it's really an amazing
‫and really powerful tool
‫which allows you to do so much stuff.
‫So tour-stats, send it
‫and that's not really the result
‫that we were waiting for, so that's just the pipeline
‫that we defined, but I already know why that is.
‫So we did not await the result.
‫So this, basically, just like a normal query
‫is gonna return an aggregate object.
‫So .find is gonna return a query, and .aggregate
‫is gonna return an aggregate object.
‫And then only when we await it,
‫it actually comes back with the result.
‫So that's also why we defined this function here
‫as an async function, so that we can then use await
‫in there and so this is the right place.
‫Let's try it again.
‫And here we go!
‫So great!
‫We have the average rating of all our tours,
‫we have the average price which you see is
‫1563 and then the minimum and the maximum, right?
‫And from our data, we can actually confirm
‫that this is true.
‫So great!
‫Really, really great.
‫Now, one more thing that I want to do here
‫is to actually calculate the total number
‫of ratings that we have and also the total number of tours.
‫So we have the average rating here
‫and let's actually do it before,
‫so numRatings,
‫like this
‫and you can probably guess
‫that this one is called sum and then the ratingsAverage,
‫or actually not ratingsAverage, but ratingsQuantity, right?
‫So that's where the number of ratings is stored
‫and so the number of ratings, the total,
‫will be the sum of all of these together.
‫And now, the last one, is the number of tours
‫and that one is a bit trickier,
‫and so that's a nice one to show you here.
‫So we still want to sum, so to add everything together,
‫basically, so we still use sum,
‫but what are we gonna add together?
‫Well, we basically add one for each document,
‫and so we say 1, and that's it.
‫So basically for each of the document
‫that's gonna go through this pipeline,
‫one will be added to this num counter.
‫Let's call it numTours.
‫And so, yeah, let's test it out now, again.
‫And indeed we get our nine tours
‫and we already know that we have nine.
‫So these are all the statistics for all the tours together,
‫but let's now take it to the next level.
‫As I said before, we can now group our results
‫for different fields.
‫And let's actually start out with the difficulty.
‫And so that's quite similar to specifying the fields
‫down there which is simply the dollar sign,
‫and then the name of the field.
‫So $difficulty, give it a safe and that's it.
‫Send it here, and now we have the statistics
‫all defined for each of these three difficulties.
‫So that's really, really amazing.
‫So this is the part that blows my mind.
‫It's absolutely fantastic.
‫I mean, you can start to see all the kinds of stuff,
‫all the kinds of data manipulation
‫that you can do using this pipeline, right?
‫So we really can now start to analyze stuff,
‫for example, we see that the easiest tours
‫are the ones who get the poorest ratings
‫with a 4.6, while the medium tours
‫are the ones who get the highest ratings.
‫We can also see that the difficult tours
‫are the most expensive ones with an average price
‫of 1997, while the easiest ones are also the cheapest ones
‫with a price of 1272, right?
‫We also see that the easiest tours
‫are the ones which we have the most,
‫so four easy ones and only two difficult ones, right?
‫So we can use this, of course,
‫to get all kinds of insights into our data.
‫Let's comment this one out here
‫and let's just try another one,
‫for example, by ratings, why not,
‫let's just try out, see what we get.
‫So for example, we have two tours,
‫so basically with a rating of 4.5,
‫we have two tours with 4.7, one tour with 4.8,
‫and three tours with 4.9.
‫Oh, and also down here, one tour with 4.6,
‫and so for example, one insight that we can get here
‫is that the very expensive tours with this average price
‫of 1997 get extremely good ratings.
‫So all kinds of stuff, really that we can do here.
‫Let's get rid of this one, let's add this one back,
‫and we can now actually even,
‫do some operations with this one.
‫So just for fun, let's put this difficulty to uppercase.
‫So this of course, is gonna be another object here
‫and then here in front is the operator.
‫And so the operator in this case, is called $toUpper.
‫So just, yet another MongoDB operator.
‫So check it out, and now it's spelled uppercase.
‫Great!
‫So we have the group stage here which is quite complete.
‫So let's try another one which is a sort stage.
‫So another object in our array for yet another stage
‫and this one is called sort.
‫So sort and then in there, we need yet another object,
‫and it kind of auto-completed this one, I don't know why,
‫and so here we can now specify which field we want
‫to sort this by and let's actually use the average price.
‫And now here in the sorting we actually need to use
‫the field names that we specified up here in the group.
‫We can no longer use the old names because at this point
‫they are already gone.
‫They no longer exist.
‫So at this point, in the aggregation pipeline,
‫we really already have these results.
‫So these are now our documents basically.
‫So if you want to sort by the average price,
‫then this is the field name we gotta use.
‫So we can say, the average price
‫and then we can say 1 for ascending.
‫So let's try that out.
‫And yeah, indeed.
‫That's the lowest one and that's the highest one.
‫Great!
‫So we did a lot of stuff.
‫Let me just show you that we can also repeat stages.
‫So let's actually do another match here
‫and this one is really just to show you
‫that we actually can repeat stages
‫and I also want to show you another operator.
‫So let's now select by id and remember that id
‫is now the difficulty, right?
‫So we just specified that up here.
‫So we want the id and now a new operator
‫that we didn't use yet.
‫We want it to be not equal to, and let's say easy.
‫And so, just like this we're gonna select
‫all the documents that are not easy.
‫So in this case, the one that says
‫medium and difficult, right?
‫So basically, excluding the one that says easy, right?
‫So again, it's not really useful here
‫because that will take away a lot of our meaningful data,
‫but just to show you that we can of course,
‫match multiple times.
‫So in this case, we matched once before we actually
‫did the group and then we matched once we were ready,
‫doing the grouping.
‫Now, let's actually keep this here,
‫out of our aggregation pipeline,
‫but I'm gonna leave it here,
‫just so that you keep it as a reference.
‫Give it another save, give it another try,
‫and so these are our results
‫for getting our tour statistics.
‫So, really nice, really great.
‫I hope you enjoyed this video
‫and in the next one, we're actually gonna keep using
‫the aggregation pipeline to do some more calculations.
‫And to show you some more different stages.
‫So keep tuned for that one.