If you get stuck in any of the below steps, even after searching online using your personal web browser/ AI chatbot, please raise your issue in the Q&A and we will solve it
Why Docker? Docker is like a magic box for your projects. It bundles everything your project needs, so you don’t have to install various programs on your computer. This keeps your system clean and avoids problems with different software versions.
Docker works on any operating system, so your project runs the same on Windows, Mac, or Linux. Follow the installation link depending on your OS be it either Linux, Mac or Windows - Install Docker
To check that you have docker installed go on your Command Prompt/ Terminal in the search bar and type;
docker --version
Typing in this command will tell us the version of Docker we have installed. If you encounter any errors or messages that this command is not available, Docker might not have been installed correctly, and you may need to redo the installation process.
It is one of, if not the most popular Data Engineering programming language. Simple to read and write compared to other languages like Java or Scala but at the same time still versatile.
Follow the Python installation link and download the latest version of Python for your OS type - Install Python
To check that you have python installed go on your Command Prompt/ Terminal in the search bar and type;
python --version
Same as for docker, any issues might indicate an incorrect installation so would need to re-do the process.
Note: We need to make a distinction between the Python version you download on your local machine and the Python version used inside the airflow-docker environment. In my case I have Python 3.11.4 on my laptop but in the Dockerfile we will set the Python version to 3.10.
An IDE is the editor you use to develop your code. In my case I use Vistual Studio Code (VSCode) but there are other IDEs like PyCharm or Eclipse. Follow this link to install VSCode - Install Visual Studio Code
Once you install VSCode, go to the Marketplace extension, type in python and press install. Make sure to download the official Microsoft extension. The extension provides features like syntax highlighting for readability and debugging, IntelliSense for code completion, linting for identifying and fixing coding errors and much more features
Git is a version control tool which helps you manage and track changes to your code over time. For this course it is needed to be able to version our code and also go through the CI-CD section of this course.
Follow this link to install Git - Install Git
To check that you have git installed go on your Command Prompt/ Terminal in the search bar and type;
git --version
Same as for docker/ python, any issues might indicate an incorrect installation so would need to re-do the process.
Github is a code repository which we will use to store the code that we will build in this course.
If you don't have a Github account, sign up for a free personal account by following these steps;
Navigate to Github
Click Sign up
Follow the prompts to create your personal free account
You need to have a Google Account to get the Youtube API Key. This will be used for the data extraction part of the course. Go to My Google Account and make sure you have an account, if not press on Create Account and follow the steps.